I’m redoing everything I have from scratch. This week I have FreeIPA set up from OpenTofu + Ansible configs, and enrolls most of my other servers against FreeIPA. I am still migrating TrueNAS to use FreeIPA’s Kerberos Realm for auth, and I need to chown a lot of files for the new UIDs and GIDs homed in FreeIPA. After that, I’m setting up FreeRadius for auth to switches, APs, and Wifi. And then after that, I’m back to overhauling my k8s stack. I have Talos VMs running but didn’t finish patching in Cilium. And after the real fun begins.

Decided to buy a raspberry pi, it arrived, I installed pihole on it and put it into my dad’s house, all in a few days. Biggest win: I just took action and did it, instead of researching, brainstorming and writing down stuff for weeks and then never execute.

Finally took the time to setup Woodpecker CI to replace Drone. Also finally linked it not only to my self hosted gitea, but also to github, so I can automate a few builds there as well.

In the process I also learned, that I can set up a whole bunch of pods in a single kube definition for podman/quadlets, which allows me to have a much cleaner setup. Previously I was only aware that you can define a single pod with multiple containers. It makes sense, but it never occurred to me before.

I plugged in an NVIDIA gpu in my server and enabled ollama to use it, diligently updated my public wiki about it and now enjoying real time gpt: OSS model responses!

I was amazed, time cut from 3-8 minutes down to seconds. I have a Intel Core7 with 48gb ram, but even an oldish gpu beats the crap out of it.

This week I saw my 3 machine cluster flailing trying to stay online, digging around identified it as an issue with communication with my NAS. It was running NFS3 and so I swapped that to NFS4.1 and did some tuning and now my services have never been faster!

The nextcloud AIO instance that hadn’t been working since September suddenly started working after I updated it. This was all after their forums did fuck all to help except tell me to get gud. I knew the problem wasn’t on me or my config and I feel so vindicated

I’ve been running all my apps on my NAS as docker containers, but some get ‘stuck’ occasionally, requiring a reboot of the whole machine. Using the NAS was mostly out of convenience.

I also had an old laptop running k3s, hosting a few stateless services.

This week I picked up three Wyse 5070 devices and started setting up a more permanent Kubernetes cluster. I decided to use Talos Linux, which is a steep learning curve, but should hopefully reduce the amount of ongoing work for upgrades. I’ll be deploying everything with FluxCD this time around too.

I’ve stumbled a bit with the synology-csi-driver. It didn’t work with Talos out of the box, but turns out the latest commits have a fix. The only thing remaining before I can start porting the apps over is figuring out how to spin up a new CA and generate client certificates for mTLS. I currently do that in Vault but it seems like something cert-manager could handle going forward.

Hum. I’ve been smooth sailing for a while now. I’ve tried installing OwnTracks again and made some progress by figuring out cloud flare tunnels are a problem (at least the way I configured them). New to MQTT. So the app still doesn’t work properly but now I have an idea why and I’m not just banging my head on the wall anymore.

I have tried out Openclaw in a container, and it wasn’t hard at all.

All the warnings of danger are right, though. But if anything goes wild, I still know how to delete a container :-)

I got Terminus for the TRMNL set up using Podman on my server running NixOS.

Although I’m actually planning on replacing Terminus with my own simple server app that way it can be even more declarative (no Postgres database of devices/users/screens) and easier for me to customize. The API I’ll have to implement is extremely straightforward, so I don’t anticipate it taking too long.

The table (dm) might finally make the switch from roll20 to foundry for a campaign!

I had enough time to install sort of pihole.

I got gitea running on my VPs cluster that I use to host keyboard vagabond services. I moved my repository from my home PC into it, and set up an action runner to automate a build and deploy of piefed, so it runs my build script, pushes to harbor registry (internal), and then deletes and recreates a job to run db migrations and restarts the web and worker pods.

I’m going to migrate the other build services to it as well, and after that I should be able to finally get all of my services behind cloud flare tunnels and tail scale, and finally remove the last bits of ingress-nginx. The registry was the only thing still on ingress-nginx because I needed to push larger image files than are permitted by cloud flare. since all of that is internal now, I get to finally seal those bits off.

The build is also faster since I don’t have to rely on wifi

Still waiting for my success. Pihole randomly doesn’t answer DNS requests in time, causing a lot of trouble between my services. It’s happening since I switched to dnsmasq in opnsense (which is upstream for my local domain for Pihole), but also for external domains. Can’t nail it down and am this short of reconsidering my whole network setup. It used to work fine for over a year though…

Opnsense dnsmasq is DHCP for my servers and also resolves them as local hosts. (e.g. server1.local.domain) and Pihole conditionally forwards there. Since the issue is also when resolving external domains, it shouldn’t be related, but the timing is suspicious. I also switched the general upstream DNS.

Pihole does have some logs indicating too many concurrent requests, but those are not always correlating with the timeouts.

I know it’s DNS, I just don’t know where yet.