April 19, 2020

Clustering your Raspberry pi's with Kubernetes

Installing a local kubernetes cluster on your raspberry pi's.

Clustering your Raspberry pi's with Kubernetes

The past year we've been starting to get into Kubernetes more and more. And I've gotta say I really like it. It takes some time to get to hang of it and understand how to configure what, where, why and when.

Although it takes time to understand how it works the benefits of using Kubernetes are easily explained. Once it's up and running you no longer need to worry to much about server installations for your customers and configuring them (be it via Ansible or not). Container orchestration can take a lot of chores away from you and allowing you to focus on quick delivery of applications and ensuring high availability and such.

However you might wanna take some time to familiarize yourself with the concepts of Kubernetes, and you probably don't wanna reinstall servers every time you make a mistake and have to start over. (this happened to me a lot :P)
So my idea was to use a few Raspberry pi's and tinker with those. Seeing as wiping an SD card is not as much trouble as reinstalling vm's/servers and you can do it all locally without having to worry to much. Plus it's just fun playing around with raspberry's :)

What was the goal?

  • Create a Kubernetes cluster with 4 raspberry's in my home network
  • Setup as 1 master node
  • And 3 worker nodes
  • Have the cluster accessible from the internet
  • Installing them automated with Ansible

This was all a success :) I came across a few challenges along the way but manged to tackle them all.

How to start

When starting with a raspberry you'll probably get a SD flashed with Noobs. This is nice but for this usecase not necessary. So I created a custom Raspian lite image (debian buster based). I used Raspberry own pi-gen tool. You can find it here.

The reason I created my own image is because I wanted the SSH server to be active by default, I wanted my own public ssh keys already set in the pi user and set a ridiculous long password so the default password would not be easy to guess. You can customize the pi-gen tool to do all of that by simply adding a config file.

Once I had that I flashed it on the 4 SD cards and placed them into my raspberry's. Attached them to the network with cables and looked them up on my machine in the network using nmap.

nmap -sn 192.168.2.0/24

This returns all the devices in your network range. My network range was 192.168.2.x. So the CIDR for my network is 192.168.2.0/24. Yours can be different.
You'll get results like this:

Nmap scan report for kubemaster (192.168.2.40)
Host is up (0.016s latency).
MAC Address: DC:A6:32:71:8A:93 (Raspberry Pi Trading)

Static IP's

I looked up the 4 raspberry pi's. Then created a static DHCP entry in the router for that mac address. Ensuring that the IP address of the device will always be the same. So when you start over you'll at least know what the ip address will be for that device.

Hostname configuration

As I said, I wanted 1 master node and three worker nodes. I will give them representational hostnames to identify them.

  • kubemaster
  • worker-1
  • worker-2
  • worker-3

You can do this using raspi-config on each raspberry. Once the hostname is changed, finish and reboot. Then update your local /etc/hosts file with the names and ip's set. For me that looked like this:

192.168.2.40 kubemaster
192.168.2.41 worker-1
192.168.2.42 worker-2
192.168.2.43 worker-3

After that you can login using pi@kubemaster or pi@worker-1 etc...

First installation

Raspbian lite (debian) already comes with most things in place. But for a first run I at least installed some basic stuff like:

  • curl
  • software-properties-common
  • docker

I used Ansible to create playbooks for these tasks. This makes starting over a lot easier :) I am as any good developer a lazy one. So automating things makes my life a hell of lot easier.

Important!

A challenge I ran into, was that Raspberry processors are slightly different from most VM's/servers. Whereas you commonly run into x86_64, amd type for installation, Raspberry uses armhf. This gave me few hickups with installing Docker. Most documentation you'll find will give you url's for adding GPG keys to your package repositories and such. But these need to be a bit different.
My ansible role for installing docker on raspberry:
Note the apt_key and the apt_repository url's are probably different from those you've found in most documentations online, these are for the raspbian/armhf setup.

Also an important one is the install_recommands: no part in the install docker block. This corresponds with apt-get --no-install-recommends flag. Otherwise it will install a few packages which are not compliant with raspberry / armhf. And will cause problems with updating apt packages afterwards.

- name: "Install docker"
  tags: ["docker"]
  block:
    - name: "Add docker key"
      apt_key:
        url: "https://download.docker.com/linux/raspbian/gpg"
        state: present
      become: yes
    - name: "Add package repository"
      apt_repository:
        repo: "deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable"
        state: present
      become: yes
    - name: "Install docker"
      apt:
        name: docker-ce
        state: latest
        install_recommends: no
        update_cache: yes
        cache_valid_time: 3600
      become: yes
    - name: "Add ansible_user to docker group"
      user:
        name: "{{ ansible_user }}"
        groups: "docker"
        append: yes
      become: yes
      when: '"{{ ansible_user }}" != "root"'
    - name: "Enable docker service"
      service:
        name: "docker"
        state: "started"
        enabled: yes
      become: yes
    - name: "Install python-docker"
      package:
        name: python-docker
        state: present
      become: true

If you're not familiar with Ansible I suggest you give it a look. It takes a little time to get into, but once you've the hang of it you'll be happy you spend the time.

This is what my Ansible inventory looked like:

[cluster]
kubemaster ansible_host=192.168.2.40 ansible_user=pi
worker-1 ansible_host=192.168.2.41 ansible_user=pi
worker-2 ansible_host=192.168.2.42 ansible_user=pi
worker-3 ansible_host=192.168.2.43 ansible_user=pi

[cluster-masters]
kubemaster ansible_host=192.168.2.40 ansible_user=pi

[cluster-workers]
worker-1 ansible_host=192.168.2.41 ansible_user=pi
worker-2 ansible_host=192.168.2.42 ansible_user=pi
worker-3 ansible_host=192.168.2.43 ansible_user=pi

Three groups, 1 containing them all, one group just for the master nodes, and one for the worker nodes.

Rancher K3S

I used K3S for the kubernetes setup because from what I found online in different online discussions was that it was easier to setup and uses less resources, what in this case with Raspberrys is a nice trait.

IP tables

Something that isn't directly pointed out when you take the K3S documentation here, is that you need to enable legacy IP tables on your raspberrys otherwise K3S won't run. It's not a big issue but you need to know. To enable them run these commands on all raspberry's.

sudo iptables -F && sudo reboot
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot

You can find the documentation reference here.
The reason I added the && sudo reboot in the first line, is because when I ran sudo iptables -F I was thrown out of the shell and couldn't get in anymore. A reboot ensures that everything is restarted and then you can login again and run the rest of the commands.

Master node install

My first node had to be the master node and I installed it as such:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --docker --no-deploy traefik" sh

This installs and configures the raspberry as a Kubernetes master node. I added the server --docker --no-deploy-traefik because I want it to run on docker instead of containerd, and it installs an older version of Traefik and I wanna be in control of the Ingress myself. If you're trying this for the first time and have not worked with Traefik or ingress for that matter you can remove that part and it will work out of the box. You can just run the command as seen in the Rancher documentation.

Once completed the master node will be up and running. But this master node needs worker nodes to actually do the work. After the installation you can run find the kube config file at: /etc/rancher/k3s/k3s.yaml
And the token for connection other nodes at: /var/lib/rancher/k3s/server/node-token

Worker node install

On the other raspberry I ran:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --docker" K3S_URL=https://kubemaster:6443 K3S_TOKEN=<<your token here>> sh

Again I added the --docker flag because I want it to use docker.
This command will install K3S as a worker node and connect it to the master node located at the given url.

Keep in mind

The http://kubemaster:6443 only works because I updated all the raspberry hosts files to know each others IP addresses with their corresponding hostnames like this:

192.168.2.40	kubemaster
192.168.2.41	worker-1
192.168.2.42	worker-2
192.168.2.43	worker-3

If you don't do this you'll have to use the IP addresses.

The token you can find at the location I stated here above.

When this is all done you can run k3s kubectl get nodes on the master node and if everything goes as planned you'll get something like this:

NAME         STATUS   ROLES    AGE   VERSION
worker-1     Ready    <none>   17h   v1.17.4+k3s1
kubemaster   Ready    master   18h   v1.17.4+k3s1
worker-2     Ready    <none>   17h   v1.17.4+k3s1
worker-3     Ready    <none>   17h   v1.17.4+k3s1

And then your raspberry pi kubernetes cluster is up and running :).

Connection to the internet

Since I've only got 1 external IP address given by my ISP. I've set the ports 80 and 443 (http/https) to point the kubemaster - 192.168.2.40 in my local network. So external http(s) requests are routed to my Kubernetes master node. No more is needed to enable availability to the outside world to your local kubernetes cluster.

Then you can take your DNS configuration for any domain name and add A records to point to your external ip address.

Non-technical

Though this post is from a very technical point of view I would also like to note that server cluster setups like kubernetes/k3s can have a great benefit from a company wide perspective including sales and marketing. When a production cluster is setup you can go and think of ways that allow sales to spin up a dummy application of your products on the fly while they are sitting next to a potential client, you can state that your company ensures high availability, you can start to think of canary deployment solutions. There is more to say about kubernetes/server clustering then just the technical area.

If you are a developer reading this and want your company to go in this direction you might have to convince non-technical co-workers. So do not only think of your own benefits tech-wise but let them see the (future) benefits it can offer for you, sales, marketing and most importantly your customers.

Conclusion and afterthoughts

It took me some time to understand the issues I ran into like the different processor type in the raspberry, the way the nodes need to work in the network, how to find them etc... But I've gotta say I'm pretty pleased with the result.
What I've got now is a small server cluster at home which serves as a POC to prove that server clustering with kubernetes works, and works very good. Even on smaller machines.

I hope you liked this article and helped or inspire you to go and play around with Kubernetes. Because it offers a lot of possibilities.

Thanks for reading!


References