I wanted to try out Kubernetes for a bunch of reasons, but mostly to get a redundant solution for Frigate and Home Assistant. In the end, I realized Kubernetes is not for me, however here is my journey to get it working. A full K8 Kubernetes cluster takes a lot of doing, so I decided to go with the lighter weight K3.
This video was very helpful in learning more about the basics of Kubernetes – https://www.youtube.com/watch?v=X48VuDVv0do&ab_channel=TechWorldwithNana
I came into possession of three mini PCs, which were to be the basis of my cluster – one master and two workers. To start, I put Proxmox on them, and gave them each static IP. I added my network certificates, and now can reach them by host name. I also added my samba share to all of them so I can more easily add ISOs.
Next, I made an Ubuntu Server VMs on all machines, each with 6gb ram and a 100gb drive. For whatever reason, when I installed Ubuntu, I added the SSH service – but for whatever reason it did not work. I could not SSH into the machine at all. Next time, I will not include this in the installation but instead add it after the machine is built. To fix the problem I had to purge (not simply remove) openssh and then reinstall.
And now I was ready to start building the cluster. I followed this tutorial, but wound up trying a different method in the end:
https://www.techtarget.com/searchitoperations/tutorial/How-to-set-up-a-K3s-cluster
There are a few steps (like disabling swap and enabling bridging) that it didn’t cover, so because of these issues, I tried a different tutorial:
How to Build an Awesome Kubernetes Cluster using Proxmox Virtual Environment
This tutorial was super helpful – but for some reason I couldn’t install the three Kubernetes packages, so i followed Step 1 of this tutorial:
Then I added the overlay network
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/
Documentation/kube-flannel.yml
Now I added worker nodes – this command was given earlier in the process when the master was first created. Will appear under this.
Then you can join any number of worker nodes by running the following on each as root:
Make sure to run with sudo.
That’s it – cluster built
I made a test nginx pod to make sure things are working. I can access this from any browser on my network.
OK – now to start using the cluster.
First, i installed Helm
https://helm.sh/docs/intro/install/
I did it from script, but apt would have worked just fine.
And this is really when I decided to stop. A the start of the process, my (incorrect) understanding of Kubernetes was that it would distribute processes across the nodes – but in reality, it distributes Pods across the nodes. Each pod is like its own container. This is great if you have a webserver or a database that needs redundancy and room to grow – but in my case, I just needed a two servers to run video processing (for Frigate) and Home Assistant. Yes, each could be their own pod, but in reality, Kubernetes is a bit overkill for this. Also, to integrate Frigate with Home Assistant, they need to be on the same machine – in the end, I think I’ll just go with a single VM on one of the mini PCs. And from there I’ll need to get this HA instance to talk to my main HA server over MQTT – basically I just need to send all the alerts to my main HA server. I’m sure that blog post is coming soon.
In the end, this was a very cool exercise, but alas, not what I decided on going with in the end.