I’m self hosting to learn. I’ve been hacked before and I lost stuff and then I refined my technique and started over again. Nothing I do is “mission critical”, so I now have the mindset that it will fail, I will lose data and time and I will get hacked. Honestly, it’s helped me to be better at home and at my workplace to have this mindset. Always plan for failure (and keep backups).

I came here to say the same thing except that I have a pi locally and one at a relative’s house. I back up to the local pi and a nightly cron starts rsync to pull my local copy.

I chose this so that i could control the rstnc start time, bandwidth and stop time but also so I could leave the remote network vanilla with no open ports, etc. With bandwidth limiting, it may take a few days to catch up from full backups, but a differential is same day.

Be sure to use a RO filesystem or overlay FS on the Pi card. I’ve had them go corrupt.

My mantra is “plan to be hacked”. Whether this is a good backup strategy, a read-only VM, good monitoring or serious firewall rules.

It’s a VGA connection and, yes, my primary concern is resource usage. I’m running 2-3 VMs on it so that I can easily migrate the VM around.

I had plain old top and it was boring. I did not know how many alternatives there were.

I’ll also have to check out cmatrix.

This is an excellent idea!

Generally, no. On some cases where I’m extending the code or compiling it for some special case that I have, I will read the code. For example, I modified a web project to use LDAP instead of a local user file. In that case, I had to read the code to understand it. In cases where I’m recompiling the code, my pipeline will run some basic vulnerability scans automatically.

I would not consider either of these a comprehensive audit, but it’s something.

Additionally, on any of my server deployments, I have firewall rules which would catch “calls to home”. I’ve seen a few apps calling home, getting blocked but no adverse effects. The only one I can remember is Traefik, which I flipped a config value to not do that.

I was hacked years ago. I was hosting a test instance of a phpbb for a local club. Work blocked SSH, so I opened up telnet. They either got in from telnet or a php flaw and installed password sniffers and replaced some tools (ps, top) with tools that would hide the sniffer service they installed.

After that, I changed my model. My time lab is for learning and having fun. I’m going to make mistakes and leave something exposed or vulnerable and hackers are going to get in. Under this new model, I need to be able to restore the system easily after a breach. I have a local backup and a remote backup and I have build scripts (ansible) so that I can restore the system if I need to. I’ve had to do this twice. Once from my own mistake and one from hardware failure.

Same. I have spent way more time troubleshooting a pipeline than it saves. I like the idea of automation but laziness prevails.

For my own curiosity, how do you perform a build? Is it all done in pipelines, kicked off on change? Do you execute the whole infra build each time you release an update?

As others have said, a traditional off site backup will work. How do you plan to perform a restore, though? If you need the self hosted source repo, it won’t be available until the infrastructure is stood to creating another circular dependency.

I’m still in the early stages of exploring this, too. My solution is to run a local filesystem git clone of the “main” repo and execute it with a Taskfile that builds a docker image from which it can execute the ansible infrastructure build. It is somewhat manual but I have performed a full rebuild a few times after some Big Mistakes.

After breaking “prod” many times, I have a Dev (local machine), Test (small VM) and Prod (big VM). My test is just less RAM and space and I need to spin down certain K8s things to spin up others, but it’s a close mirror of Prod, just less.

Thanks for the feedback. I plan to do some reading on NFSv4 domain mapping this weekend.

I think this is exactly what I’m looking to do. Thanks for such a detailed writeup!

I did some reading last night and think it lines up with what you’re saying. I found docker-mailserver with some configuration. The only thing I need to add is mail filtering to folders and I think that’s included.

I’d like to hide behind the service that I’m paying for without incurring extra fees for retaining it all. I can figure out the pull side by using fetchmail or something to a server that hosts dovecot, but the sending side is confusing since I’d need something that can receive my email and send it via the service. It’s only 1 email address, so I’m not looking for a mail relay, but something like a full caching mail proxy.

I started watching the video. I was not aware that LetsEncrypt supported wildcard certificates. Does this mean that your internal network uses the same domain name as your externally-hosted services?

I tried step-ca to start with, but my primary use case was for certs in the cluster, which cert-manager is more suited for natively. Maybe step-ca has improved, I was using it in the early days. My goal isn’t a short lived cert as much as it is to have an easy configuration and to learn.

I think it may support it, but it’s not well documented. I’ll need to read up a bit. I started with helm charts but like how operators, um operate. They upgrade on their own and are very stable. Honestly, though, it was mostly because I wanted to learn how they work.

I think this is what I’m going to do.

Yes, monthly is too fast. I’m using a K8s operator for cert-manager which defaults to a month. I think I can patch the CSV with an annotation that will bump that out, but when the operator updates the CSV then I need to repatch it.

I was polling the community to see if there’s something that is easy to use but I was not able to find in my searches. It seems like a common problem.

Part of my problem is that I chose to use a K8s operator for cert-manager which isn’t easy to configure. Had I used a helm chart, i’d have bumped the root cert to 10 years and forgotten about it.