Daniel's Blog

Bare Metal Ingress Controller for Kubernetes

The setup

This client had a Fortinet gateway in front of a Bastion host in front of a Kubernetes cluster. The requests coming in were SSL terminated at the Bastion Host which used LetsEncrypt for it's cert management, then sent to the Kubernetes Cluster where they were to end up at the appropriate service. To do this, an Ingress Controller was needed.

The ingress

For this, the Nginx Ingress was chosen.

The nginx-ingress is maintained by kubernetes.

Web Page Here: https://kubernetes.github.io/ingress-nginx/
Helm Chart Page: https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx

The nginx-ingress class needs to be installed as it routes requests received by the cluster to the appropriate service. That service is defined in an ingress.

Customizing values

The bastion hosts send data to all of the kubernetes nodes in a round robin approach using haproxy as the load balancer. They expected the nodes to be able to receive the requests on port 80 so that needed to be taken into account.

values.yaml

----
controller:
  hostPort:
    enabled: true
  kind: DaemonSet

Enabling the host port makes it listen on port 80 and 443. Changing it from a Deployment to a Daemonset makes it so the bastion host can send data to all nodes.

Installing the IngressClass

k8s$ kubectl create namespace ingress-nginx
k8s$ kubens ingress-nginx
k8s$ helm upgrade --install ingress-nginx ingress-nginx   --repo https://kubernetes.github.io/ingress-nginx  -f values.yaml

Configuration Options

The client needed large-client-header-buffers set as well as gzip enabled, so those were both enabled though the config map.

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  large-client-header-buffers: "16 128k"
  use-gzip: "true"

Patching the config map

Restarting the daemonset is necessary for it to see the new values from the config map.

k8s$ kubectl patch configmap/ingress-nginx-controller --type merge --patch-file cm.yaml
k8s$ kubectl rollout restart daemonset ingress-nginx-controller

Testing

Now it is ready to start taking requests from the bastion host. Any request can be tested on any node using this:

Testing at node

k8s$ curl 127.0.0.1

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Testing at node with active ingress

When an ingress is in place it can be tested using this:

k8s$ curl --header "Host: mydomain.com" 127.0.0.1
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Then you can test at the bastion hosts:

Testing at bastion host

k8s$ curl --header "Host: mydomain.com" XXX.IP.OF.NODE

Checking the generated configuration

k8s$ kubens ingress-nginx
Context "kubernetes-admin@cluster.local" modified.
Active namespace is "ingress-nginx".

k8s$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-8thh5   1/1     Running   0          47h
ingress-nginx-controller-99jjx   1/1     Running   0          47h
ingress-nginx-controller-fs8wp   1/1     Running   0          47h
ingress-nginx-controller-svskp   1/1     Running   0          47h
ingress-nginx-controller-xgqsx   1/1     Running   0          47h

k8s$ kubectl exec ingress-nginx-controller-svskp -- cat /etc/nginx/nginx.conf | head -10

# Configuration checksum: 7696211755756719792

# setup custom paths that do not require root access
pid /tmp/nginx/nginx.pid;

daemon off;

worker_processes 32;

k8s$