I have something similar to host paths with node selectors: an NFS provisioner for PVs. The provisioner is tied to the node with the large disk. It’s not resilient to node outages, but allows me to spread pods across the nodes. For my deployments, I’m preferring to use S3 storage wherever possible.

Excellent write-up. I had Nextcloud running on K3s with its files on a NAS which were shared with Minio and it worked well. I’m looking into Longhorn, but only have 2 nodes and it wants at least 3. I’m reevaluating my resiliency needs in favour of simplification.

I’ve been looking to do this, but haven’t found a good, easy to use pull thru proxy for docker, ghcr.io and some other registries. Most support docker only.

This one looks promising but overly complicated to set up.

A few times now, I’ve gone to restart a container and the repo’s been moved, archived or paywalled. Other times, I’m running a few versions behind and the maintainer decided to not support it, but upgrading would mean a complete overhaul of my Helm values file. Ugh!

I was considering a docker registry on separate ports for each upstream registry I’d like to proxy/cache.

I’m self hosting to learn. I’ve been hacked before and I lost stuff and then I refined my technique and started over again. Nothing I do is “mission critical”, so I now have the mindset that it will fail, I will lose data and time and I will get hacked. Honestly, it’s helped me to be better at home and at my workplace to have this mindset. Always plan for failure (and keep backups).

I came here to say the same thing except that I have a pi locally and one at a relative’s house. I back up to the local pi and a nightly cron starts rsync to pull my local copy.

I chose this so that i could control the rstnc start time, bandwidth and stop time but also so I could leave the remote network vanilla with no open ports, etc. With bandwidth limiting, it may take a few days to catch up from full backups, but a differential is same day.

Be sure to use a RO filesystem or overlay FS on the Pi card. I’ve had them go corrupt.

My mantra is “plan to be hacked”. Whether this is a good backup strategy, a read-only VM, good monitoring or serious firewall rules.

It’s a VGA connection and, yes, my primary concern is resource usage. I’m running 2-3 VMs on it so that I can easily migrate the VM around.

I had plain old top and it was boring. I did not know how many alternatives there were.

I’ll also have to check out cmatrix.

This is an excellent idea!

Generally, no. On some cases where I’m extending the code or compiling it for some special case that I have, I will read the code. For example, I modified a web project to use LDAP instead of a local user file. In that case, I had to read the code to understand it. In cases where I’m recompiling the code, my pipeline will run some basic vulnerability scans automatically.

I would not consider either of these a comprehensive audit, but it’s something.

Additionally, on any of my server deployments, I have firewall rules which would catch “calls to home”. I’ve seen a few apps calling home, getting blocked but no adverse effects. The only one I can remember is Traefik, which I flipped a config value to not do that.

I was hacked years ago. I was hosting a test instance of a phpbb for a local club. Work blocked SSH, so I opened up telnet. They either got in from telnet or a php flaw and installed password sniffers and replaced some tools (ps, top) with tools that would hide the sniffer service they installed.

After that, I changed my model. My time lab is for learning and having fun. I’m going to make mistakes and leave something exposed or vulnerable and hackers are going to get in. Under this new model, I need to be able to restore the system easily after a breach. I have a local backup and a remote backup and I have build scripts (ansible) so that I can restore the system if I need to. I’ve had to do this twice. Once from my own mistake and one from hardware failure.

Same. I have spent way more time troubleshooting a pipeline than it saves. I like the idea of automation but laziness prevails.

For my own curiosity, how do you perform a build? Is it all done in pipelines, kicked off on change? Do you execute the whole infra build each time you release an update?

As others have said, a traditional off site backup will work. How do you plan to perform a restore, though? If you need the self hosted source repo, it won’t be available until the infrastructure is stood to creating another circular dependency.

I’m still in the early stages of exploring this, too. My solution is to run a local filesystem git clone of the “main” repo and execute it with a Taskfile that builds a docker image from which it can execute the ansible infrastructure build. It is somewhat manual but I have performed a full rebuild a few times after some Big Mistakes.

After breaking “prod” many times, I have a Dev (local machine), Test (small VM) and Prod (big VM). My test is just less RAM and space and I need to spin down certain K8s things to spin up others, but it’s a close mirror of Prod, just less.

Thanks for the feedback. I plan to do some reading on NFSv4 domain mapping this weekend.

I think this is exactly what I’m looking to do. Thanks for such a detailed writeup!

I did some reading last night and think it lines up with what you’re saying. I found docker-mailserver with some configuration. The only thing I need to add is mail filtering to folders and I think that’s included.

I’d like to hide behind the service that I’m paying for without incurring extra fees for retaining it all. I can figure out the pull side by using fetchmail or something to a server that hosts dovecot, but the sending side is confusing since I’d need something that can receive my email and send it via the service. It’s only 1 email address, so I’m not looking for a mail relay, but something like a full caching mail proxy.

I started watching the video. I was not aware that LetsEncrypt supported wildcard certificates. Does this mean that your internal network uses the same domain name as your externally-hosted services?

I tried step-ca to start with, but my primary use case was for certs in the cluster, which cert-manager is more suited for natively. Maybe step-ca has improved, I was using it in the early days. My goal isn’t a short lived cert as much as it is to have an easy configuration and to learn.