Backup – houghi.org

Backing up of one or more computers from one or more backup servers. An explanation on how I did it. There are many other ways to do it It is decided in several parts

This is all based on rsync -a --stats --delete --link-dest /previous /backup /data

It is an incremental backup and could be used as a basis for something much more complex. There are several parts to it

Things to know and have
Installing of the needed software
Configuration of the client
The rsync command we are using
The script
What else
Conclusion.

Things to know and have

To do this you need to have a basic knowledge of Bash. i.e. know how to make a script executable, move files around and install software. I assume you have two machines on the same network that are able to reach each other. The client is the one with the data and the server is the one that gets the data and in this case does the backup. You must have root rights.

I use Debian. It should be usable on any distro with slight modifications.

Installing of the needed software

Install xinetd and rsync on both machine. Here what to do

apt update && apt -y upgrade
apt install rsync xinetd

This will install the needed software, rsync is what the actual backup does and xinetd sees that the connection can be made to the correct port. It is possible that one or both are already installed.

Configuration of the client

On the client with root rights (use sudo or become root with sudo -i or su -)

echo "
# default: off
# description: The rsync server is a good addition to an ftp server,
# as it allows crc checksumming etc.
service rsync
{
disable = no
flags = IPv6
socket_type = stream
port = 873
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon
log_on_failure += USERID
}

.
" > /etc/xinetd.d/rsyncd
/etc/init.d/xinetd restart

This will write a file to /etc/xinetd.d/rsyncd and then restart the service. This will open port 873. That is the default port, so be sure your server can reach it. Do not open ports to the outside world!

Also do the following with root rights:

cat << _EOF_ >> /etc/rsyncd.conf
pid file = /run/rsynd.pid
lock file = /run/rsyncd.lock
log file = /var/log/rsync.log
uid = root
gid = root
read only = true
timeout = 300

[MyBackup]
path = /
comment = backup of the client
exclude from = /etc/rsyncd.exclude

_EOF_

as well as

cat << _EOF_ >> /etc/rsyncd.exclude
- /dev
- /run
- /sys
- /lost+found
- /media
- /mnt
- /proc
- /tmp
- /backup
_EOF_

See man rsyncd.conf for more details. What this does is make the service on the client read only, so no accidental overwriting, a complete backup of EVERYTHING starting at / and have a file with what you do NOT want to backup, starting at /. And the name of the module is MyBackup.

If you want to only backup e.g. /home/user/Music, edit the path. Y ou can have more than one module.

The second is the file that tells you what to exclude from the backup, starting at path. Adapt both files to your needs. You could start with just a small directory, like /etc and when all works, you can add the rest. Just add all directories from / to /etc/rsyncd.exclude with a minus in front of it, except /etc

That is the setup done. The client is ready. The rest will be done on the server.

The `rsync` command we are using

We are now going to test this on the server. Just do rsync –list-only rsync://client where client is either a name or the IP address. You will see the module name and the information you added.

user@server : rsync --list-only rsync://client
MyBackup        backup of the client

Next we are going tyo see what will be backed up

user@server : rsync --list-only rsync://client/MyBackup
drwxr-xr-x          4,096 2019/12/09 17:27:06 .
lrwxrwxrwx              7 2019/08/06 21:04:11 bin
lrwxrwxrwx             30 2019/09/24 22:44:03 initrd.img
lrwxrwxrwx             30 2019/08/06 21:05:11 initrd.img.old
lrwxrwxrwx              7 2019/08/06 21:04:11 lib
lrwxrwxrwx              9 2019/08/06 21:04:11 lib32
-SNIP-

See if that is what you are willing to backup. Understand that a lot of data takes up a lot of space AND time. So again: best see that you only do e.g. /etc for now and exclude everything else. Make a directory /backup and do your first backup with the following:

rsync -avzh –delete rsync://client/MyBackup /backup/client/MyBackup/2020-02-04_21:05

Now add a file and run it again (with the identical command, including the same date/time. Now delete that file and run it again. Your backup now works. Edit the files so you will be able to do a full backup of everything you need. If all you want to do is copy files as a backup, you are done. Again, this can take a LONG time.

The script

To make an incremental backup, rsync uses hardlinks, so it does not take up extra space on your drive. You will need to know the old directory name and the new directory name. A script will take care of that.

#!/bin/bash
# Backup script
set -x #This is for debugging. Remove when the script works
PS4='# ${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]}() - [${SHLVL},${BASH_SUBSHELL},$?] ' #Also for debugging

PARAMETERS="-avzh --delete"
BACKUPDIR="/backup:
CLIENT="client" #Can be an IP address as well
MODULE="MyBackup"
OLDDIR=$(ls -1t $BACKUPDIR/$CLIENT/$MODULE|head -n 1)
NEWDIR=$(date +%F_%R)

rsync $PARAMETERS --link-dest $BACKUPDIR/$CLIENT$MODULE/$OLDDIR rsync://$CLIENT/$MODULE $BACKUPDIR/$CLIENT/$MODULE/$NEWDIR
# Remove old files
find $BACKUPDIR/$CLIENT/$MODULE --max-dept 1 -type d -name '????-??-01_*' -prune -o -mtime +15 -exec rm {} +

Make it executable and run it as root. It is best places in /usr/local/bin as e.g. backup.sh To run it daily, put it in your cron. Even though it still needs way less space, an incremental backup will grow in size. The directory structure takes some space and file that are changed, e.g. logfiles, will take up new space. So we must delete older files, while still keeping other things.

The last line will delete all directories older that 15 days, except the first of the month. This will work if your date format is done with date +%F_%R

What else?

Turn around the role of client and server, so the client take the backup of the server data. You can also add more clients and/or modules. You need to add some loops. First you have a loop that goes through a list of client. Each client will show the modules with the --list-only option.

#!/bin/bash
# Backup script
set -x #This is for debugging. Remove when the script works
PS4='# ${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]}() - [${SHLVL},${BASH_SUBSHELL},$?] ' #Also for debugging

PARAMETERS="-avzh --delete"
BACKUPDIR="/backup:

for CLIENT in "client new_client 192.168.1.10 localhost"
do
    for MODULE $(rsync --list-only rsync://$CLIENT|awk '{print $1}')
    do
        OLDDIR=$(ls -1t $BACKUPDIR/$CLIENT/$MODULE|head -n 1)
        NEWDIR=$(date +%F_%R)
        mkdir -p $BACKUPDIR/$CLIENT/$MODULE/$NEWDIR
        rsync $PARAMETERS --link-dest $BACKUPDIR/$CLIENT/$MODULE/$OLDDIR rsync://$CLIENT/$MODULE $BACKUPDIR/$CLIENT/$MODULE/$NEWDIR
        # Remove old files
        find $BACKUPDIR/$CLIENT/$MODULE --max-dept 1 -type d -name '????-??-01_*' -prune -o -mtime +15 -exec rm {} +
    done
done

Conclusion

Incremental backups are a must. Making your own will make it a lot easier to adapt to your own needs. e.g. you can edit the output, so it looks nice.

Things to know and have

Installing of the needed software

Configuration of the client

The rsync command we are using

The script

What else?

Conclusion

The `rsync` command we are using