Hey! I have been using Ansible to deploy Dockers for a few services on my Raspberry Pi for a while now and it’s working great, but I want to learn MOAR and I need help…
Recently, I’ve been considering migrating to bare metal K3S for a few reasons:
- To learn and actually practice K8S.
- To have redundancy and to try HA.
- My RPi are all already running on MicroOS, so it kind of make sense to me to try other SUSE stuff (?)
- Maybe eventually being able to manage my two separated servers locations with a neat k3s + Tailscale setup!
Here is my problem: I don’t understand how things are supposed to be done. All the examples I find feel wrong. More specifically:
- Am I really supposed to have a collection of small yaml files for everything, that I use with
kubectl apply -f?? It feels wrong and way too “by hand”! Is there a more scripted way to do it? Should I stay with everything in Ansible ?? - I see little to no example on how to deploy the service containers I want (pihole, navidrome, etc.) to a cluster, unlike docker-compose examples that can be found everywhere. Am I looking for the wrong thing?
- Even official doc seems broken. Am I really supposed to run many helm commands (some of them how just fails) and try and get ssl certs just to have Rancher and its dashboard ?!
I feel that having a K3S + Traefik + Longhorn + Rancher on MicroOS should be straightforward, but it’s really not.
It’s very much a noob question, but I really want to understand what I am doing wrong. I’m really looking for advice and especially configuration examples that I could try to copy, use and modify!
Thanks in advance,
Cheers!
You have a lot of responses here, but I’ll tell what k8s actually is, since a lot of people seem to get this wrong.
Just like k8s, docker has many tools. Although docker is packaged in a way, that it looks like it’s just 1 tool. This is docker desktop. Under the hood there is docker engine that is really a runtime and image management service and API. You can look at this more if you wanted. There is containerd, runc, cri-o. These were all created so that different implementations can all talk to this API in a standard way and work.
Moving on to k8s. K8s is a way to scale these containers to run in different ways and scale horizontally. There are ways to even scale nodes vertically and horizontally to allow for more or less resources to place these containers on. This means k8s is very event driven and utilizes a lot of APIs to communicate and take action.
You said that you are doing kubectl apply constantly and you say feels wrong. In reality, this is correct. Under the hood you are talking with the k8s control plane and it’s taking that manifest and storing it. Other services are communicating with the control plane to understand what they have to do. In fact you can apply a directory of manifests, so you don’t have to specify each file individually.
Again there are many tools you can use to manage k8s. It is an orchestration system to manage pods and run them. You get to pick what tool you want to use. If you want something you can do from a git repo, you can use something like argocd or flux. This is considered to be gitops and more declarative. If you need a templating implementation, there are many, like helm, json net, and kustomize (although not a full templating language). These can help you define your manifests in a more repeatable and meaningful way, but you can always apply these using the same tools (kubectl, argocd, flux, etc…)
There are many services that can run in k8s that will solve one problem or another and these tools scale themselves, since they mostly all use the same designs that keep scalability in mind. I kept things very simple, but try out vanilla k8s first to understand what is going on. It’s great that you are questioning these things as it shows you understand there is probably something better that you can do. Now you just need to find the tools that are right for you. Ask what you hate or dislike about what you are doing and find a way to solve that and if there are any tools that can help. https://landscape.cncf.io/ is a good place to start to see what tools exist.
Anyway, good luck on your adventure. K8s is an enterprise tool after all and it’s not really meant for something like a home lab. It’s an orchestration system and NOT a platform that you can just start running stuff on without some effort. Getting it up and running is day 1 operations. Managing it and keeping it running is day 2 operations.
I see, that makes sense actually! Thanks for the message!
I saw the landscape website before, that’s a LOT of projects! =O
I would add that you can run kubectl apply on directories and/or have multiple yaml structure in the same taml file (separated with —, it’s a yaml standard).