My first steps into the world of Kubernetes container orchestration

A few weeks back we started playing around a little bit with Kubernetes. Today we really tried to dive a bit more into it. Kubernetes has a steep learning curve, but the possibilities are many and the sense of ease and control are great.

Today we set up our own Kubernetes cluster. And we started playing around with it, and I can honestly say, (even with some frustration here and there because of missing knowledge and wanting a lot without reading,) I'm a huge fan. We wanted to accomplish a few things:

• Nginx Ingress controller
• Our app deployed in the cluster
• The app exposed through Ingress on a specific host
• Let's encrypt certificates

And I can also happily say we got it all up and running.

Nginx Ingress controller

The thing I struggled with most, was setting up the Nginx Ingress controller. I found a lot of tutorials and examples. But most of them stopped at the point where the controller was up and running but not yet exposed to the outside world on port 80/433 for http/https communication, which is kind of the whole point here... :)

However we did find a good explanation and walkthrough on how to set it up and actually use it. I followed the steps explained here.

What you do very simply put is the following.

• Create a namespace in which the ingress controller objects will live
• Create a nginx configuration file
• Create a file containing the default certificate and key for nginx
• Setup a cluster role
• Deploy the nginx ingress controller
• Expose the nginx ingress controller with either a NodePort or a LoadBalancer

When you follow the steps one by one you'll end up with nginx ingress running on port 80 and 443 to the outside world.

Important to understand here, the nginx ingress controller is not an Ingress object. It merely supplies the feature to handle Ingress objects. This was something that I didn't get at first.

A few thing that are good to know while setting up the controller are:

TLS/SSL

In the first step you'll create a "Secret" object containing a certificate and key. The example they give here will set the default certificate for nginx. So when you go to https://{your-public-ip} this will be the certificate used. These are stored in base64 encoded format.

apiVersion: v1
kind: Secret
metadata:
  name: default-server-secret
  namespace: nginx-ingress
type: Opaque
data:
  tls.crt: {base64 encoded certificate}
  tls.key: {base64 encoded key}

Exposing the nginx controller

In step 4 you'll expose the controller. The example gives you the choice to either use a NodePort object or a LoadBalancer object.
Which one you choose is up to you of course. If you're like me and starting out with Kubernetes i'd suggest the NodePort object, this requires a little less configuration and simply exposes the controller and gives you the ports on which it runs internally.

If you're like me and like some more fancy configuration, you can use the LoadBalancer. This requires a bit more config, but you'll end up with a fancy load balanced setup for your controller.

Once you've setup the initial controller you should be able to access your server on port 80 and 443 and see a 404 page. This means everything is up and running.

Expose an app using Ingress

The same repository you just used to create the ingress controller also has an example on how to deploy and app and expose it on a specific host with Ingress.

You can follow these steps to create a deployment, service and the actual ingress to expose the app.

Here you again create a "Secret" file containing a certificate and key. The example uses the dummy host cafe.example.com. If you have a domain you can add a DNS record to point to your public cluster IP. If that is the case you can use let's encrypt to create certificates for your domain and use those in the secret file.

Auto generating your certificates

You can use certbot on your server to create your certificates, base64 encode them, create a secret file and use it. That'll work just fine. But we're trying to automate things here, so why not automate the certificate creation?

There is a Kubernetes tool called cert-manager. This gives you the possibility to use ClusterIssuer objects and let Ingress objects use it to create certificates for the hosts used.

First install cert-manager. Follow these instructions.

Then create a ClusterIssuer for let's encrypt environment.

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my@email.address
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}

Once this is set you can create a certificate object

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example-app-crt
spec:
  secretName: example-app-crt
  dnsNames:
    - your-domain.com
  acme:
    config:
      - http01:
          ingressClass: nginx
        domains:
          - your-domain.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

This will create a certificate object. You can see it using kubectl get certificates
You can check the status of the issuer using kubectl describe clusterissuer letsencrypt-prod

Then you can create an Ingress object just like you just did in the cafe example. The difference will be the annotations telling the object were using the let's encrypt environment as issuer.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - mydomain.com
    secretName: example-app-crt
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: myAppName
          servicePort: 80

If everything goes as planned you can go to the set host and see https fully operational and the certificate set.

If not use kubectl to see the created objects and see what might be wrong.
Use the kubectl describe to see detailed information on the objects. This helped me figure out I had the wrong url set for Let's Encrypt in the ClusterIssuer object :)

What's next?

Good question, there are a lot of possibilities and I'm very curious and exited to experiment more with the features Kubernetes offers. Some ideas running through my head is using Kubernetes to spin up testing environments for a specific branch of my application on a temporary host, so we can test a specific feature.
Expanding the cluster with more nodes.
Auto scaling pods on heavier load.
Setting up an ELK stack in the cluster for all my applications for searching, logging and monitoring.

I hope to expand my Kubernetes knowledge en maybe write some more extensive tutorials here in the future. But hopefully the information above helped some of you guys out a bit too.