• 0 Posts
  • 20 Comments
Joined 2 years ago
cake
Cake day: June 20th, 2023

help-circle


  • I hear you. I worked for an msp where some customers would refuse to invest in backup solutions and we either declined to renew their contract or they suffered an event and we were then setting up backups.

    I was in the middle of a migration from OVH to Hetzner. I knew I had good backups at home so the plan was to blow away OVH and restore from backup to Hetzner. This was the mistake.

    Mid migration I get an alert from the raid system that a drive has failed and had been marked as offline. I had a spare disk ready, as I planned for this type of event. So I swapped the disk. Mistake number 2.

    I pulled the wrong disk. The Adaptec card shit a brick, kicked the whole array out. Couldn’t bring it back together. I was too poor to afford recovery. This was my lesson.

    Now I only use ZFS or MDRAID, and have multiple copies of data at all times.


  • I’m lucky enough to run a business that needs a datacenter presence. So most my home-lab (including Lemmy) is actually hosted on a Dell PowerEdge R740xd in the DC. I can then use the small rack I have at home as off-site backups and some local services.

    I treat the entirety of /var/lib/docker as expendable. When creating containers, I make sure any persistent data is mounted from a directory made just to host the persistent data. It means docker compose down --rmi all --volumes isn’t destructive.

    When a container needs a database, I make sure to add an extra read-only user. And all databases have their container and persistent volume directory named so scripts can identify them.

    The backup strategy is then to backup all non-database persistent directories and dump all SQL databases, including permissions and user accounts. This gets run 4 times a day and the backup target is an NFS share elsewhere.

    This is on top of daily backuppc backups of critical folders, automated Proxmox snapshots for docker hosts every 20 minutes, daily VM backups via Proxmox Backup Server and replication to another PBS at home.

    I also try and use S3 where possible (seafile and lemmy are the 2 main uses) which is hosted in a container on a Synology RS2423RP+. Synology HyperBackup then performs a backup overnight to the Synology RS822+ I have at home.

    Years ago I fucked up, didn’t have backups, and lost all the photos of my sons early years. Backups are super important.












  • I recently decommissioned my old poweredge T620. Beast of a thing, 5U heavy af. It had 8x10T drives and was the primary media server.

    Now that it is replaced I bought 2x Synology RS822+ and filled them with the old disks. Using SHR2. They are mixed brands bought at different times so I’ve made sure each NAS has a mix of disks.

    Lowest is 33k hours, highest is 83k.




  • In 2010 I self hosted a Xen hypervisor and used it for everything I could. It was fun!

    I had a drive failure in my main raid 5 array so bought a new disk. When it came to swap it out, I pulled the wrong disk.

    It was hardware raid using an Adaptec card and the card threw the raid out and refused to bring it together again. I couldn’t afford recovery. I remember I just sat there, dumb founded, in disbelief. I went through all the stages of grief.

    I was in the middle of a migration from OVH to Hetzner and it occurred at a time where I had yet to reconfigure remote backups.

    I lost all my photos of our first child. Luckily some of them were digitised from developed physical media which we still had. But a lot was lost.

    This was my lesson.

    I now have veeam, proxmox backup server, backuppc and Borg. Backups are hosted in 2 online locations and a separate physical server elsewhere in the house.

    Backups are super, SUPER important. I’ve lost count how many times I’ve logged into backuppc to restore a file or folder because I did a silly. And it’s always reassuring how easy it is to restore.