I think most options have been covered here, but I’d like to consider some other bits…

User accounts & file permissions:- if you have >1 account, note that the UserID is internally numbered (starting from 1000, so Bob=1000, Sue=1001) and your file system is probably setup using the numerical UserID… so re-creating the users in a different order would give Sue access to Bob’s files and vice versa.

Similarly, backing up /etc /var etc… you should check if any applications (ie databases) need specific chmod and chown settings

Rsync, tar, etc can cover some of this, you just need to check you rebuild users in the correct order.

Maybe Ansible is another approach? So your disaster recovery would be:

1. Install plain OS on new drive
2. Get Ansible access to it (ie basic netwroking)
3. Rebuild OS and instsll applicstions automatically with Ansible
4. Restore application & home folders (again with Ansible)

When you get this working, it’s amazing to watch an entire system being rebuilt

It is a question I’ve spent a lot of time trying to work out. Can’t speak to docker.

Some of the specifics of Keeps and Dontkeeps depend on details of your system. You have to find out where the distro, DM and other apps keep the following:

Dontkeeps:

• trashes
• temp files
• file indexes … IMHO these dont backup properly if you leave them in and will prevent you from completing the task
• device files

Keeps:

• list of installed packages — explicit and deps separate if possible
• config files: /etc, ~/.config, ~/.* on a case by case basis… I say remove the obvious large temp dirs and keep the rest by default for simplicity
  • for the system configs I’ve had a tool called etckeeper running for a while because it was highly recommended but I’ve never actually restored from it…
• personal documents and other files such as typically kept in the home directory
• /root occasionally has something you need

Ways to investigate:

• use a disk usage utility to find out where your storage is being used up … It’ll help you find large Dontkeeps
• watch for recently modified files
• dirs and files that are modified all the time are usually temp dirs. But sometimes they have something useful like your firefox profile.

Most backup solutions are ONE of the following:

1. User files
2. System files

Don’t spend too much time crying about needing two solutions. Just make your backup today and reach perfection later.

Remember: sync isn’t backup. Test your backup if you can (but its not as easy as it sounds). Off site your most precious files.

I usually just use SyncThing and sync the Home directory to another device.

You need to make your own plan about what’s important to you. Don’t waste time backing up stuff you can quickly download again - like docker images. Just know where the stuff is that you care about (such as any mounted volumes in docker with your own data on) and back that up, and make everything automated. If you do it manually, chances are you’ll gradually stop doing them.

I wrote some things about properly backing up data that may be of use.

I have used a combination of rsync and borg to backup to an external drive. Then I was feeling adventurous and tried borgbackup2. It was a mistake. Then I learned of rdiff-backup. Easy to setup, but even incremental backups of my home dir took many hours to complete. This is no solution to have regular backups for me.

I decided to go with btrfs snapshots instead. I recently reinstalled everything and I have finally btrfs. I bought a new external NVME drive for my backups where all the snapshots will go. Btrfs has even a parent-option to copy incremental snapshots to another filesystem (the parent snapshots being on both filesystems).

I did not finish my setup, so I cannot share any scripts yet.

Why? I have a hard time imagine a use case where restoring the OS itself would be appropriate.

I can imagine restoring data, obviously, and services running with their personalization … but the OS is something generic that should be discarded at whim IMHO. You probably chance few basic configuration of some services and most likely that’s stored in /etc but even then 99% is default.

You can identify what you modified via shell history, e.g. history | grep /etc and potentially save them or you can also use find /etc -type f -newerXY with a date later than the OS installation and you should find what you modified. That’s probably just a few files.

If you do back up anything beyond /home (which should be on another partition or even disk than the OS anyway) you’ll most likely save garbage like /dev that might actually hinder your ability to restore.

So… sure, image the OS if you actually have a good reason for it but unless you work on archiving and restoring legacy hardware for a museum then I doubt you do need that.

I usually only keep documents and media. Programs can be redownloaded and reinstalled (and it might be better to reinstall them in case you move to a new OS anyway to ensure compatibility).

For docker specifically, only keep stuff that’s specific for your instance; which you normally setup as an external volume anyway. Docker is designed such that you should be able to nuke the container, and all persistent data is restored via an external volume on the host. If you’re not doing that, you should immediately go and set that up now (to get the data out safely, setup a volume connection such that the container path is new - that way you don’t accidentally destroy what’s there, copy the stuff you need out, then readjust the path so it’s correct)

I’ve been working with the rsync command for a while now and I have created a pretty solid backup system on my own with that command. I’ve had some terrible luck using back/restore programs like Timeshift on Linux Mint and so I went out of my way to learn how to make a “simpler” backup/restore process for myself.

I am using Alpine Linux so the following commands are based around that file system. It’s possible to modify the commands to fit your Operating System’s file system by using the lsblk command to see all the mountpoints for your OS. Each mountpoint requires a separate rsync command.

For myself, I have four mount points. / for the root directory. For the boot directory, I will need a command for /boot/ and /boot/efi/ directories. Since my home directory is in a separate partition, I also need a command for /home/.

The following are all the expanded commands I use to backup my entire OS which is backed up to the /backup/ directory:

# BACKUP  
# /root/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=/backup/*' '--exclude=/boot/*' '--exclude=home/*' '--exclude=proc/*' '--exclude=sys/*' '--exclude=dev/*' '--exclude=tmp/*' '--exclude=run/*' '--exclude=mnt/*' '--exclude=media/*' / /backup/`  

# /boot/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /boot/ /backup/boot/`  

# /boot/efi/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /boot/efi/ /backup/boot/efi/`  

# /home/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' '--exclude=.cache/*' /home/ /backup/home/`  

The root partition is the most involved command so I’ll break it down slightly. You can check out the rsync man page for more detailed information about each command option.

• --dry-run- is used to perform a test run. If you have any verbose options, it can be used to see what files will be moved without performing any actions. This is important for testing but I also use it to view or review any changes before I commit anything. This option must be removed if you want to commit any changes. I’m leaving it here incase anyone decides to blindly run this command. I don’t want to be responsible for any lost data.
• --archive --acls --one-file-system --xattrs --hard-links --sparse are all used to preserve any file/folder attributes and permissions. --archive is commonly used, even for single files while the rest are more important for the operating system in the event of a restore. You can check the man page for more information as I’ve kind of forgotten their purposes over time.
• --verbose --human-readable --partial --progress is both for verbose outputs and for continuing in case a previous transfer was incomplete.
• --delete rsync will compare the source directory and the destination directory for any changes and will delete any files or directories in destination directory to match the source directory.
• --numeric-ids is used to preserve ownership in number form. I find it useful since I keep multiple device backups on an external SSD.
• --exclude=... This excludes files or directories from being synced. For the / directory it’s important to exclude /backup/. If you don’t, you will end up recursively saving to the /backup/* directory until your storage is full. /boot/* home/* are excluded because they will be handled by the following rsync commands. proc/* sys/* dev/* tmp/* run/* mnt/* media/* are directories that are used to run the operating system or devices and will not be required after a system reboot.
• / /backup/ is two parts. / is the source directory, what you want backed up. /backup/ is the destination directory. It’s where you want your data to end up.
• --exclude=lost+found is not important and can be excluded
• --exclude=.cache/* can also be excluded from the home directory if you want to save space.

You’ll most likely need to create the /backup/ directory in order to perform the backup commands.
Running the commands in order as listed, it will first perform a backup on the root directories while creating empty directories for the following commands. After that, each command will be able to complete it’s transfer.

Restoring is as simple as using the same command but reversing the Source and Destination directories. If backing up root was / /backup/, then to restore you now use /backup/ /. After performing a restore, it’s wise to reboot your system.

# RESTORE  
# /root/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=/backup/*' '--exclude=/boot/*' '--exclude=home/*' '--exclude=proc/*' '--exclude=sys/*' '--exclude=dev/*' '--exclude=tmp/*' '--exclude=run/*' '--exclude=mnt/*' '--exclude=media/*' /backup/ /`  

# /boot/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /backup/boot/ /boot/`  

# /boot/efi/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /backup/boot/efi/ /boot/efi/`  

# /home/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' '--exclude=.cache/*' /backup/home/ /home/`  

For docker containers, I usually shut them all down before a backup, perform the backup and start the containers again with the following commands:

# CONTAINERS DOWN  
docker stop --timeout 30 $(docker ps -a -q)  

# PERFORM BACKUP  

# CONTAINERS UP  
docker start $(docker ps -a -q)  

For a restore with docker containers, I usually shut down the containers, perform the restore and reboot my system. If you have any containers that require building, you may need to spend time rebuilding after a restore. I haven’t figured out how to address that issue yet. It may be helpful to run the command docker system prune -a && docker volume prune -f to remove all docker containers before a restore. Sometimes docker containers remain after performing a restore and must be deleted by hand. You can see which directories can be removed by performing a dry run and looking for the delete errors.

You don’t have to use any of this but for me I at least got some insight into how backing up and restoring a file system works with the help of rsync. Hopefully if can give you some insight as well for your needs.

I use these commands near daily since I like to keep my systems as neat and organized as possible. Often I’ll create a backup, do my experimenting, perform a restore, make any changes I want and perform another backup with the latest changes. I’ve had more consistent results going this route compared to using other backup programs.

I know there is a way to create smaller backups using links but I don’t think I do that. I prefer a full backup copy anyways since I run my systems in a more minimal way. Space isn’t an issue for me to worry about.

everything. download clonezilla, perform a bare-metal backup and if the worst happens, you can reflash your new disk to exactly how the old one was without skipping a beat.

I back up all user data, for me that’s some folders in /var, all of /etc (don’t need everything, but it’s small enough), mysqldumps and pgsqldumps of my databases and the output of dpkg --get-selections. And the backup script itself.

Everything is rsynced to a server in another location.

I want to switch to Borg backup but haven’t found the time yet.

If you need to restore it, back it up.

How do you plan to restore if the whole drive dies?

Don’t forget to grab any /etc configurations you’ve made to adapt the system to your hardware or services. You don’t need all of /etc, just whatever you’ve changed to not be default.

you could “dd” your root partition to a partition on that HDD every now and then. and if you don’t have your home on a different partition do that and it’s easy to back this up with a syncing tool if u decided to do that or at least the .config .local and .cache directories because many programs put their configuration files in there.

BE VERY CAREFUL WHEN USING “dd” BECAUSE IT CAN MAKE YOU LOSE DATA FOR A WHOLE PARTITION OR DEVICE

docker keeps some data in ~/.local/share/docker btw

Backup everything as a bootable system image, that’s the best method for recovery IMO.