Unix Tar front-end
Don't waste your time writing buggy script ! Use always the same with different configuration.
Last release supports incremental and differential backups.
Every Un*x administrator has already used the tar command, and sometimes written scripts to backup data. With MKSBackup he can focus on what and where to backup ! And icing on the cake, he can even receive an email report.
- How it works
- Differential and incremental GNU tar backups
- Getting started
- Job parameters
- Mail report
How it works
The tar back-end use the well known Un*x find utility to do the file selection. This allows to include and exclude files and directories on their pattern and location, but also on their size, owner, date and type. Here is a typical job configuration section.
[TAR] program=tar include=/home /data exclude_dir=/home/nobody tar_option=-z destination=<normal=mon-fri>/backup_place/backup-${nweekdayname}.tgz <none=sat-sun>
Differential and incremental tar backups
GNU tar handle incremental backups by storing additional metadata in a standalone file called snapshot. At every backup, tar update this snapshot file. Differential backups can be done by always reusing the same snapshot copy.
MKSBackup handles only three levels of backups. The full backups. The differential backups are always done against the last full backup. And incremental backups are always done against the last full or differential backup.
The location of file containing metadata if defined by parameter snapshot.
snapshot=/var/lib/mksbackup/snapshot.snar
Every full backup creates or overwrites the snapshot file. Any differential backup creates a copy into a file with identical name ending by -1. In this case this is /var/lib/mksbackup/snapshot.snar-1 Any incremental backup uses the snapshot.snar-1 file if it exists or create it from original snapshot file. Of course, every full backup also removes the snapshot.snar-1 file.
The snapshot file must be stored on a static location and be always available for all backups. You should not store it on a removable disk !
When doing incremental backup from NFS disk, you must know about tar option --no-check-device and add it to tar_option parameter if required.
To restore from incremental backup read the GNU tar documentation.
For example to do a full backup every first Sunday of the month, a differential other Sundays and an incremental every other day of the month. And if you want to keep every monthly backup, up to 5 last differential backup and last 6 daily backups. And split backup between two destination every month:
destination=<full=1-7,sun,jan,mar,may,jul,sep,nov>/place/backup-full-${nmonthname}.tgz <diff=sun,jan,mar,may,jul,sep,nov>/place/backup-diff-${nmonthweek}.tgz <inc=mon-sat,jan,mar,may,jul,sep,nov>/place/backup-inc-${nweekdayname}.tgz <full=1-7,sun,feb,apr,jun,aug,oct,dec>/other/backup-full-${nmonthname}.tgz <diff=sun,feb,apr,jun,aug,oct,dec>/other/backup-diff-${nmonthweek}.tgz <inc=mon-sat,feb,apr,jun,aug,oct,dec>/other/backup-inc-${nweekdayname}.tgz
Getting started
We are hosting websites on a host and need to backup some of them. We will make a full backup every days and keep the last 7 days. We will name them with the name of the day : mon, ....sun. We want also keep the last 4 sunday's backups.
Create a file /etc/mksbackup/mksbackup.ini like this one. Don't forget to configure your mail settings !
[DEFAULT]
smtp_host=smtp.myprovider.com
smtp_port=25
sender=backup.operator@server01.example.com
recipients=me@example.com
[WEB_SITES]
program=tar
include=/sites/www.magikmon.com
/sites/www.magiksys.net
exclude_dir=/sites/www.magikmon.com/cache
/sites/www.magiksys.net/cache
tar_option=-z
destination=<normal=sun,wsun/4>/backup_place/backup-week${nweek}.tgz (a)
<normal=mon-sat>/backup_place/backup-${nweekdayname}.tgz
(a) means, backup on Sunday and the cycle restart every the 4 weeks on Sunday. The line after, means from Monday to Saturday, backup to an archive having its name varying with the day. More on the destination syntax.
You can manually test your job before to schedule it in your cron. Depending the way you have installed, the Python package, your command line can vary. Look here to learn how to start it.
# mksbackup -d -c /etc/mksbackup/mksbackup.ini check WEB_SITES
The check checks the parameters and displays a planing for next backups. This is helpful to tune the destination option.
16:39:53 INF start version=0.9.20 cmd=['./mksbackup', '-q', '-c', 'mk.ini', 'check', 'WEB_SITES'] 16:39:53 INF find_cmdline=find /sites/www.magikmon.com /sites/www.magiksys.net -path /sites/www.magikmon.com/cache -prune -o -path /sites/www.magiksys.net/cache -prune -o -not -type s -print 16:39:53 INF find_bin=/bin/find 16:39:53 INF tar_cmdline=tar cvTf - /backup_place/backup-Mon.tgz --no-recursion 16:39:53 INF tar_bin=/bin/tar 16:39:54 WAR Warnings in section: WEB_SITES 16:39:54 WAR exclude_dir: directory /sites/www.magiksys.net/cache not found 16:39:54 INF Destinations day by day: 16:39:54 INF Mon 21 Feb 2011 normal /backup_place/backup-Mon.tgz 16:39:54 INF Tue 22 Feb 2011 normal /backup_place/backup-Tue.tgz 16:39:54 INF Wed 23 Feb 2011 normal /backup_place/backup-Wed.tgz 16:39:54 INF Thu 24 Feb 2011 normal /backup_place/backup-Thu.tgz 16:39:54 INF Fri 25 Feb 2011 normal /backup_place/backup-Fri.tgz 16:39:54 INF Sat 26 Feb 2011 normal /backup_place/backup-Sat.tgz 16:39:54 INF Sun 27 Feb 2011 normal /backup_place/backup-week3.tgz 16:39:54 INF Mon 28 Feb 2011 normal /backup_place/backup-Mon.tgz 16:39:54 INF Tue 01 Mar 2011 normal /backup_place/backup-Tue.tgz 16:39:54 INF Wed 02 Mar 2011 normal /backup_place/backup-Wed.tgz 16:39:54 INF Thu 03 Mar 2011 normal /backup_place/backup-Thu.tgz 16:39:54 INF Fri 04 Mar 2011 normal /backup_place/backup-Fri.tgz 16:39:54 INF Sat 05 Mar 2011 normal /backup_place/backup-Sat.tgz 16:39:54 INF Sun 06 Mar 2011 normal /backup_place/backup-week0.tgz 16:39:54 INF Mon 07 Mar 2011 normal /backup_place/backup-Mon.tgz 16:39:54 INF Tue 08 Mar 2011 normal /backup_place/backup-Tue.tgz 16:39:54 INF Wed 09 Mar 2011 normal /backup_place/backup-Wed.tgz 16:39:54 INF Thu 10 Mar 2011 normal /backup_place/backup-Thu.tgz 16:39:54 INF Fri 11 Mar 2011 normal /backup_place/backup-Fri.tgz 16:39:54 INF Sat 12 Mar 2011 normal /backup_place/backup-Sat.tgz 16:39:54 INF Sun 13 Mar 2011 normal /backup_place/backup-week1.tgz 16:39:54 INF Mon 14 Mar 2011 normal /backup_place/backup-Mon.tgz 16:39:54 INF Tue 15 Mar 2011 normal /backup_place/backup-Tue.tgz 16:39:54 INF Wed 16 Mar 2011 normal /backup_place/backup-Wed.tgz 16:39:54 INF Thu 17 Mar 2011 normal /backup_place/backup-Thu.tgz 16:39:54 INF Fri 18 Mar 2011 normal /backup_place/backup-Fri.tgz 16:39:54 INF Sat 19 Mar 2011 normal /backup_place/backup-Sat.tgz 16:39:54 INF Sun 20 Mar 2011 normal /backup_place/backup-week2.tgz
Verify you have no errors or warnings, then replace check by checkmail to test your mail configuration. Finally do it for real.
# /usr/bin/mksbackup -d -v -l /tmp/mksbackup.log -c /etc/mksbackup/mksbackup.ini backup WEB_SITES
16:53:54 INF start version=0.9.20 cmd=['./mksbackup', '-q', '-c', 'mk.ini', 'backup', 'WEB_SITES'] 16:53:54 INF find_cmdline=find /sites/www.magikmon.com /sites/www.magiksys.net -path /sites/www.magikmon.com/cache -prune -o -path /sites/www.magiksys.net/cache -prune -o -not -type s -print 16:53:54 INF find_bin=/bin/find 16:53:54 INF tar_cmdline=tar cvTf - /backup_place/backup-Mon.tgz --no-recursion 16:53:54 INF tar_bin=/bin/tar 16:53:54 WAR Warnings in section: WEB_SITES 16:53:54 WAR exclude_dir: directory not found 16:53:54 INF start command=backup job=WEB_SITES archiver=tar 16:53:54 INF end command=backup job=WEB_SITES archiver=tar 16:53:54 INF name=WEB_SITES 16:53:54 INF program=tar 16:53:54 INF version=0.9.20 16:53:54 INF hostname=f11asx.asxnet.loc 16:53:54 INF status=OK 16:53:54 INF tar_exit_code=0 16:53:54 INF find_exit_code=0 16:53:54 INF start=Mon Feb 21 16:53:54 2011 16:53:54 INF end=Mon Feb 21 16:53:54 2011 16:53:54 INF file_count=4 16:53:54 INF file_excluded=0 16:53:54 INF target=/backup_place/backup-Mon.tgz 16:53:54 INF target_size=10240 16:53:54 INF target_mtime_epoch=1298303634 16:53:54 INF target_mtime=Mon Feb 21 16:53:54 2011 16:53:54 INF target_free_space=1040936960 16:53:54 INF start_epoch=1298303634 16:53:54 INF end_epoch=1298303634 16:53:54 INF cmd_tar=tar cvTf - /backup_place/backup-Mon.tgz --no-recursion 16:53:54 INF cmd_find=find /sites/www.magikmon.com /sites/www.magiksys.net -path /sites/www.magikmon.com/cache -prune -o -path /sites/www.magiksys.net/cache -prune -o -not -type s -print 16:53:54 INF mail sent to: root@max.asxnet.loc 16:53:54 INF final exit code=0
The log contains a lot of useful information, including command line used to start the back-end. After the backup, look inside the target directory :
$ ll /backup_place/ total 20 -rwxrwxr-x 1 asx asx 44 2011-02-21 16:51 backup-Mon.err -rwxrwxr-x 1 asx asx 0 2011-02-21 16:51 backup-Mon_find.err -rwxrwxr-x 1 asx asx 120 2011-02-21 16:51 backup-Mon.idx -rw-rw-r-- 1 asx asx 10240 2011-02-21 16:51 backup-Mon.tgz $ cat /backup_place/backup-Mon.err tar: Removing leading `/' from member names
Also verify the log file is there !
$ ll /tmp/mksbackup.log -rw-rw-r-- 1 asx asx 2102 2011-02-21 16:59 /tmp/mksbackup.log
When you are ready, schedule the job in your cron.
Job parameters
program (mandatory)
To use the tar backend, program must be set to tar.
program=tar
include (mandatory)
This is the main files and directories selector. Put one file or directory per line, additional lines must start with at least one space character.
include=/home
/data
exclude_dir (optional, default is empty)
You can exclude directories by specifying them one by one, one per line. Additional lines must start with at least one space character. This parameter is reserved for directory only.
exclude_dir=/home/nobody /proc /sys /tmp
exclude (optional, default is empty)
You can exclude files using pattern including Unix shell-style wildcards. The match is done by Python function fnmatch.fnmatchcase.
exclude=/var/log/*.log.[0-9] /home/*/Documents/*.bak
Be careful using wildcards ! For example, patterns above also exclude file like /home/foo/Documents/sub/mydoc.bak !
You can use -regex in find_options if you need powerful regular expressions.
destination
This specifies where the tar archive will be created. This parameter follows the destination syntax described here.
destination=<normal=mon-fri>/backup/backup-${nweekdayname}.tgz
<normal=sat,wsat/4>/backup/backup-week${nweek}.tgz
<none=sun>
The type can be normal, full, diff, inc. normal and full are identical, diff, inc are for differential an incremental backup.
tar_options (optional, default is empty)
These options are added to the tar command line. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .
tar_options=-z
find_options (optional, default is empty)
These options are added to the find command line. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .
find_options=-size -100M
This option allows you to fine tune your file selection. You can find all the powerful options of find command in its manual.
gzip_options (optional, default is empty)
Set GZIP environment variable of the tar process to modifie the default compression parameters used by gzip. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .
If you use this option, you should have added -z to tar_options .
gzip_options=-9
bzip2_options (optional, default is empty)
Set BZIP2 environment variable of the tar process to modifie the default compression parameters used by bzip2. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .
If you use this option, you should have added -J to tar_options .
bzip2_options=-9
index (optional, default is auto)
This is the filename where MKSBackup save the output of tar command when -v is set. This file will contain a list of all archived files.
Special value can be used:
- auto stores the file near the backup archive with the .idx extension.
- no send STDOUT output to /dev/null.
index=/tmp/tar.idx
error (optional, default is auto)
This is the filename where MKSBackup will save errors (aka stderr) from the tar command.
Special value can be used:
- auto will store the file near the backup archive with the .err extension.
- no send STDERR output to /dev/null.
error=/tmp/tar_error.output
error_find (optional, default is auto)
This is the filename where MKSBackup will save errors (aka stderr) of the find command.
Special value can be used:
- auto will store the file near the backup archive with the _find.err extension.
- no will redirect the output to /dev/null
error_find=/tmp/find_error.output
Mail report
The mail report reproduce the usual structure but can also include the following attachments.
- err.txt This is the STDERR of the tar command.
- dir.txt This is the listing of the target directory.
- err_find.txt This is the STDERR of the find command.