Emailgency logo
Your email
100%
available.
Tar 1.23-7 available on Fedora 14 and RHEL 6 is buggy and segfault when doing incremental backup. tar-1.23-8 don't segfault.

Unix Tar front-end

Don't waste your time writing buggy script ! Use always the same with different configuration.

Last release supports incremental and differential backups.

Every Un*x administrator has already used the tar command, and sometimes written scripts to backup data. With MKSBackup he can focus on what and where to backup ! And icing on the cake, he can even receive an email report.

How it works

The tar back-end use the well known Un*x find utility to do the file selection. This allows to include and exclude files and directories on their pattern and location, but also on their size, owner, date and type. Here is a typical job configuration section.

[TAR]
program=tar
include=/home
        /data
exclude_dir=/home/nobody
tar_option=-z
destination=<normal=mon-fri>/backup_place/backup-${nweekdayname}.tgz
            <none=sat-sun>
    

Differential and incremental tar backups

GNU tar handle incremental backups by storing additional metadata in a standalone file called snapshot. At every backup, tar update this snapshot file. Differential backups can be done by always reusing the same snapshot copy.

MKSBackup handles only three levels of backups. The full backups. The differential backups are always done against the last full backup. And incremental backups are always done against the last full or differential backup.

The location of file containing metadata if defined by parameter snapshot.

snapshot=/var/lib/mksbackup/snapshot.snar
    

Every full backup creates or overwrites the snapshot file. Any differential backup creates a copy into a file with identical name ending by -1. In this case this is /var/lib/mksbackup/snapshot.snar-1 Any incremental backup uses the snapshot.snar-1 file if it exists or create it from original snapshot file. Of course, every full backup also removes the snapshot.snar-1 file.

The snapshot file must be stored on a static location and be always available for all backups. You should not store it on a removable disk !

When doing incremental backup from NFS disk, you must know about tar option --no-check-device and add it to tar_option parameter if required.

To restore from incremental backup read the GNU tar documentation.

For example to do a full backup every first Sunday of the month, a differential other Sundays and an incremental every other day of the month. And if you want to keep every monthly backup, up to 5 last differential backup and last 6 daily backups. And split backup between two destination every month:

destination=<full=1-7,sun,jan,mar,may,jul,sep,nov>/place/backup-full-${nmonthname}.tgz
            <diff=sun,jan,mar,may,jul,sep,nov>/place/backup-diff-${nmonthweek}.tgz
            <inc=mon-sat,jan,mar,may,jul,sep,nov>/place/backup-inc-${nweekdayname}.tgz
            <full=1-7,sun,feb,apr,jun,aug,oct,dec>/other/backup-full-${nmonthname}.tgz
            <diff=sun,feb,apr,jun,aug,oct,dec>/other/backup-diff-${nmonthweek}.tgz
            <inc=mon-sat,feb,apr,jun,aug,oct,dec>/other/backup-inc-${nweekdayname}.tgz

Getting started

We are hosting websites on a host and need to backup some of them. We will make a full backup every days and keep the last 7 days. We will name them with the name of the day : mon, ....sun. We want also keep the last 4 sunday's backups.

Create a file /etc/mksbackup/mksbackup.ini like this one. Don't forget to configure your mail settings !

[DEFAULT]
smtp_host=smtp.myprovider.com
smtp_port=25
sender=backup.operator@server01.example.com
recipients=me@example.com

[WEB_SITES]
program=tar
include=/sites/www.magikmon.com
        /sites/www.magiksys.net
exclude_dir=/sites/www.magikmon.com/cache
            /sites/www.magiksys.net/cache
tar_option=-z
destination=<normal=sun,wsun/4>/backup_place/backup-week${nweek}.tgz  (a)
            <normal=mon-sat>/backup_place/backup-${nweekdayname}.tgz

(a) means, backup on Sunday and the cycle restart every the 4 weeks on Sunday. The line after, means from Monday to Saturday, backup to an archive having its name varying with the day. More on the destination syntax.

You can manually test your job before to schedule it in your cron. Depending the way you have installed, the Python package, your command line can vary. Look here to learn how to start it.

 # mksbackup -d -c /etc/mksbackup/mksbackup.ini check WEB_SITES

The check checks the parameters and displays a planing for next backups. This is helpful to tune the destination option.

16:39:53 INF start version=0.9.20 cmd=['./mksbackup', '-q', '-c', 'mk.ini', 'check', 'WEB_SITES']
16:39:53 INF     find_cmdline=find  /sites/www.magikmon.com  /sites/www.magiksys.net  -path  /sites/www.magikmon.com/cache  -prune  -o  -path  /sites/www.magiksys.net/cache  -prune  -o  -not  -type  s  -print
16:39:53 INF     find_bin=/bin/find
16:39:53 INF     tar_cmdline=tar  cvTf  -  /backup_place/backup-Mon.tgz  --no-recursion
16:39:53 INF     tar_bin=/bin/tar
16:39:54 WAR Warnings in section: WEB_SITES
16:39:54 WAR exclude_dir: directory /sites/www.magiksys.net/cache not found
16:39:54 INF Destinations day by day:
16:39:54 INF     Mon 21 Feb 2011 normal       /backup_place/backup-Mon.tgz
16:39:54 INF     Tue 22 Feb 2011 normal       /backup_place/backup-Tue.tgz
16:39:54 INF     Wed 23 Feb 2011 normal       /backup_place/backup-Wed.tgz
16:39:54 INF     Thu 24 Feb 2011 normal       /backup_place/backup-Thu.tgz
16:39:54 INF     Fri 25 Feb 2011 normal       /backup_place/backup-Fri.tgz
16:39:54 INF     Sat 26 Feb 2011 normal       /backup_place/backup-Sat.tgz
16:39:54 INF     Sun 27 Feb 2011 normal       /backup_place/backup-week3.tgz
16:39:54 INF     Mon 28 Feb 2011 normal       /backup_place/backup-Mon.tgz
16:39:54 INF     Tue 01 Mar 2011 normal       /backup_place/backup-Tue.tgz
16:39:54 INF     Wed 02 Mar 2011 normal       /backup_place/backup-Wed.tgz
16:39:54 INF     Thu 03 Mar 2011 normal       /backup_place/backup-Thu.tgz
16:39:54 INF     Fri 04 Mar 2011 normal       /backup_place/backup-Fri.tgz
16:39:54 INF     Sat 05 Mar 2011 normal       /backup_place/backup-Sat.tgz
16:39:54 INF     Sun 06 Mar 2011 normal       /backup_place/backup-week0.tgz
16:39:54 INF     Mon 07 Mar 2011 normal       /backup_place/backup-Mon.tgz
16:39:54 INF     Tue 08 Mar 2011 normal       /backup_place/backup-Tue.tgz
16:39:54 INF     Wed 09 Mar 2011 normal       /backup_place/backup-Wed.tgz
16:39:54 INF     Thu 10 Mar 2011 normal       /backup_place/backup-Thu.tgz
16:39:54 INF     Fri 11 Mar 2011 normal       /backup_place/backup-Fri.tgz
16:39:54 INF     Sat 12 Mar 2011 normal       /backup_place/backup-Sat.tgz
16:39:54 INF     Sun 13 Mar 2011 normal       /backup_place/backup-week1.tgz
16:39:54 INF     Mon 14 Mar 2011 normal       /backup_place/backup-Mon.tgz
16:39:54 INF     Tue 15 Mar 2011 normal       /backup_place/backup-Tue.tgz
16:39:54 INF     Wed 16 Mar 2011 normal       /backup_place/backup-Wed.tgz
16:39:54 INF     Thu 17 Mar 2011 normal       /backup_place/backup-Thu.tgz
16:39:54 INF     Fri 18 Mar 2011 normal       /backup_place/backup-Fri.tgz
16:39:54 INF     Sat 19 Mar 2011 normal       /backup_place/backup-Sat.tgz
16:39:54 INF     Sun 20 Mar 2011 normal       /backup_place/backup-week2.tgz

Verify you have no errors or warnings, then replace check by checkmail to test your mail configuration. Finally do it for real.

 # /usr/bin/mksbackup -d -v -l /tmp/mksbackup.log -c /etc/mksbackup/mksbackup.ini backup WEB_SITES
16:53:54 INF start version=0.9.20 cmd=['./mksbackup', '-q', '-c', 'mk.ini', 'backup', 'WEB_SITES']
16:53:54 INF     find_cmdline=find  /sites/www.magikmon.com  /sites/www.magiksys.net  -path  /sites/www.magikmon.com/cache
 -prune  -o  -path  /sites/www.magiksys.net/cache  -prune  -o  -not  -type  s  -print
16:53:54 INF     find_bin=/bin/find
16:53:54 INF     tar_cmdline=tar  cvTf  -  /backup_place/backup-Mon.tgz  --no-recursion
16:53:54 INF     tar_bin=/bin/tar
16:53:54 WAR Warnings in section: WEB_SITES
16:53:54 WAR exclude_dir: directory not found
16:53:54 INF start command=backup job=WEB_SITES archiver=tar
16:53:54 INF end command=backup job=WEB_SITES archiver=tar
16:53:54 INF     name=WEB_SITES
16:53:54 INF     program=tar
16:53:54 INF     version=0.9.20
16:53:54 INF     hostname=f11asx.asxnet.loc
16:53:54 INF     status=OK
16:53:54 INF     tar_exit_code=0
16:53:54 INF     find_exit_code=0
16:53:54 INF     start=Mon Feb 21 16:53:54 2011
16:53:54 INF     end=Mon Feb 21 16:53:54 2011
16:53:54 INF     file_count=4
16:53:54 INF     file_excluded=0
16:53:54 INF     target=/backup_place/backup-Mon.tgz
16:53:54 INF     target_size=10240
16:53:54 INF     target_mtime_epoch=1298303634
16:53:54 INF     target_mtime=Mon Feb 21 16:53:54 2011
16:53:54 INF     target_free_space=1040936960
16:53:54 INF     start_epoch=1298303634
16:53:54 INF     end_epoch=1298303634
16:53:54 INF     cmd_tar=tar cvTf - /backup_place/backup-Mon.tgz --no-recursion
16:53:54 INF     cmd_find=find /sites/www.magikmon.com /sites/www.magiksys.net -path /sites/www.magikmon.com/cache -prune -o -path /sites/www.magiksys.net/cache -prune -o -not -type s -print
16:53:54 INF mail sent to: root@max.asxnet.loc
16:53:54 INF final exit code=0

The log contains a lot of useful information, including command line used to start the back-end. After the backup, look inside the target directory :

 
$ ll /backup_place/
total 20
-rwxrwxr-x 1 asx asx    44 2011-02-21 16:51 backup-Mon.err
-rwxrwxr-x 1 asx asx     0 2011-02-21 16:51 backup-Mon_find.err
-rwxrwxr-x 1 asx asx   120 2011-02-21 16:51 backup-Mon.idx
-rw-rw-r-- 1 asx asx 10240 2011-02-21 16:51 backup-Mon.tgz
$ cat /backup_place/backup-Mon.err
tar: Removing leading `/' from member names

Also verify the log file is there !

 
$ ll /tmp/mksbackup.log
-rw-rw-r-- 1 asx asx 2102 2011-02-21 16:59 /tmp/mksbackup.log

When you are ready, schedule the job in your cron.

Job parameters

program (mandatory)

To use the tar backend, program must be set to tar.

program=tar

include (mandatory)

This is the main files and directories selector. Put one file or directory per line, additional lines must start with at least one space character.

include=/home
        /data

exclude_dir (optional, default is empty)

You can exclude directories by specifying them one by one, one per line. Additional lines must start with at least one space character. This parameter is reserved for directory only.

exclude_dir=/home/nobody
            /proc
            /sys
            /tmp

exclude (optional, default is empty)

You can exclude files using pattern including Unix shell-style wildcards. The match is done by Python function fnmatch.fnmatchcase.

exclude=/var/log/*.log.[0-9]
        /home/*/Documents/*.bak

Be careful using wildcards ! For example, patterns above also exclude file like /home/foo/Documents/sub/mydoc.bak !

You can use -regex in find_options if you need powerful regular expressions.

destination

This specifies where the tar archive will be created. This parameter follows the destination syntax described here.

destination=<normal=mon-fri>/backup/backup-${nweekdayname}.tgz
            <normal=sat,wsat/4>/backup/backup-week${nweek}.tgz
            <none=sun>

The type can be normal, full, diff, inc. normal and full are identical, diff, inc are for differential an incremental backup.

tar_options (optional, default is empty)

These options are added to the tar command line. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .

tar_options=-z

find_options (optional, default is empty)

These options are added to the find command line. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .

find_options=-size -100M

This option allows you to fine tune your file selection. You can find all the powerful options of find command in its manual.

gzip_options (optional, default is empty)

Set GZIP environment variable of the tar process to modifie the default compression parameters used by gzip. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .

If you use this option, you should have added -z to tar_options .

gzip_options=-9

bzip2_options (optional, default is empty)

Set BZIP2 environment variable of the tar process to modifie the default compression parameters used by bzip2. Options are separated by spaces. If one option contains space, it must be quoted using double quote character " .

If you use this option, you should have added -J to tar_options .

bzip2_options=-9

index (optional, default is auto)

This is the filename where MKSBackup save the output of tar command when -v is set. This file will contain a list of all archived files.

Special value can be used:

index=/tmp/tar.idx

error (optional, default is auto)

This is the filename where MKSBackup will save errors (aka stderr) from the tar command.

Special value can be used:

error=/tmp/tar_error.output

error_find (optional, default is auto)

This is the filename where MKSBackup will save errors (aka stderr) of the find command.

Special value can be used:

error_find=/tmp/find_error.output

Mail report

The mail report reproduce the usual structure but can also include the following attachments.