$> /usr/bin/rsync -a --exclude-from=excludes.cfg --delete / /mirror/
Rsync is a command line utility traditionally used in synchronizing files between two computers over the network, but rsync can also be used as an effective backup tool.
This article explains how to use rsync to backup your whole Linux system setup to a second drive attached to your system. You can use a removable drive, such as an external USB hard drive, so that you can store the backups in a safe place away from your working environment.
I am running Debian 3.1 (Sarge) on a fanless LEX NEO configuration. As I have installed the whole system on a 1GB CF-card, I would like to have a mirror of it on another partition on my harddisk (Let's say I'm going on vacation and desperately need another CF-card and the stores are already closed…). I'd like to be able to boot from that second source.
I have mounted /mirror to /dev/hdd1, my backup partition of the same size like my root partition (in my case the CF-card).
Now, let's copy all the data over to our backup partition:
$> /usr/bin/rsync -a --exclude-from=excludes.cfg --delete / /mirror/
Make sure you exclude all of the following directories/files. I have specified the exclude list in a separate file excludes.cfg:
/proc/ /tmp/ /mnt/ /mirror/ /etc/fstab
The rsync-command above we could run e.g. daily by cronjob. On a regular system this should not take more than 2 minutes as rsync only transfers the data that has been changed in a incremental manner. (besides, Rsync's protocol is pretty smart, see Tom's article on nope.ch).
Copy the original /etc/fstab over to /mirror/etc/fstab and edit it that the new partition get's correctly mounted as /.
In my case this results in the following line:
/dev/hdd1 / ext2 defaults,errors=remount-ro 0 1
Also, make sure, the directories we are never going to synchronize exist on your mirror partition as linux needs them as a mounting point:
$> cd /mirror $> mkdir -m 1777 tmp $> mkdir -m 555 proc $> mkdir -m 755 mnt
Finally, configure grub to correctly boot from the mirrored partition:
title Debian GNU/Linux, kernel 2.6.8-2-386, MIRROR ON /dev/hdd1 root (hd1,0) kernel /boot/vmlinuz-2.6.8-2-386 root=/dev/hdd1 ro init=/sbin/preinit ide=nodma initrd /boot/initrd.img-2.6.8-2-386 savedefault boot
now, run:
$> grub-install /dev/hdd
If the command fails and grub answeres with something like "/dev/hdd does not have any corresponding BIOS drive", make sure, the device drive has been entered into /boot/grub/device.map, e.g.:
(hd0) /dev/hdc (hd1) /dev/hdd (hd2) /dev/sda
That's it! We're done. reboot and run the mirrored system from the grub menu.
documented on: 8/21/2005 by Pipo
This option might be a lot faster than using tar when the target machine already has some version of Linux installed that you want to override.
Mount the partition where you want to leave the root of your filesystem, cd into it and run the following command:
When using rsync to maintain incremental backups significant space can be saved on the backup server by excluding directories that have non-critical or transient data such as /tmp, /proc, /mnt, /cdrom, object and backup files, and caches.
### transient directories - contents # ref http://aplawrence.com/Bofcusm/2409.html[] ### transient ### + /proc/ - /proc/** + /sys/ - /sys/** + /dev/ - /dev/** ### tmp ### + tmp/ - **/tmp/** ### cache ### # netscape, mozilla, .opera/cache4/, samba etc + Cache/ - **/Cache/** + cache/ - **/cache/** - **/cache[0-9]*/** + ccache/ - **/ccache/** + .ccache/ - **/.ccache/** + xover-cache/ - **/xover-cache/** ### backup ### - *~ - *.old - *.org - *.bak - *.sav - *.orig - *.dpkg-old
Newsgroups: gmane.linux.debian.user Date: 2007-04-05
Is there any tools that can backup disk partitions logically, for both Linux & Windoze partitions?
By logical backup I meant, at least the backup tools don't do blind sector to sector backup. Ideally it would be something like Norton Ghost. I hope I can use it to backup Windoze partitions under Linux, and restore to VmWare virtual drives.
Last time I checked, about a year or two ago, such tools didn't exist under Linux…
> Today I was also looking for a disk image tool, and came across this > list of tools: > http://www.thefreecountry.com/utilities/backupandimage.shtml[] > > I ended up using SystemRescueCD, after trying PiNG (Ping is not Ghost). > Both were based on partimage, and if I remember correctly claimed to > support NTFS via captive driver.
I tried Partimage before, when it was v0.6.2. But by then it was not totally a logical backup tool. E.g., it insisted that partition size has to be exactly the same, during back/restore. This is insane. Hope this has been fixed. Anybody has such positive experience with Partimage?
FYI, here is what I have noted when I tried it:
partition size has to be exactly the same
The partition is too small to be restored: Original partition size:........2097414144 bytes Destination partition size:.....1052803584 bytes
I only have 427M used on it, which should well fit into the 1G destination drive. With this limitation, I think by now partimage is practically useless.
Newsgroups: comp.os.linux.misc Date: 31 Dec 2006 04:37:07 GMT
I'm trying to compile a comprehensive document on cloning root partitions. My immediate goal is to clone my current working Linux to external USB HD, so that I can use it wherever I go.
By comprehensive I mean it should not be as simple minded as
dd if=/dev/hda2 of=/dev/sda2
or
cp -a / /mnt/point
or
tar -p -m cf - / | (cd /mnt/point; tar xf - )
I know they work, but there are so many things have been left out. By comprehensive, I mean I want to know all relevant things that need to be considered.
For example for dd, let alone its rigid limitation, if you use it, at least the 'conv=sync,noerror bs=4k' options should be used: sync,noerror just means continue and zero fill any error blocks, bs=4k just writes 4k at a time which will speed things up a lot. For cp, at least 'cp -ax' should be used.
But there are still much more to it.
First, directories that don't need to copy over, like /tmp, /proc. With modern Linux that uses udev, the /dev and /sys don't need to be copied either. Anything else (besides distro specifics like /var/cache/apt/archives)?
2nd, the clone partition should be made bootable, by grub or lilo.
Anything else? Like the concerns of /etc/fstab…
Last, with all the above concerns, how to achieve them with various tools?
*Tags*: tar rsync find cpio dd
On Sat, 30 Dec 2006 16:25:25 -0600, Dances With Crows wrote:
>>> the clone partition should be made bootable, by grub or lilo. > > No, really? Are you having trouble with this step? Explain the trouble > you're having.
First of all, thank you all who replied. No I don't have problems with that. Actually, as said in OP, I was looking for *concerns* over cloning root partitions.
> I'd say it's a better idea to use GRUB here, since > booting from things that aren't IDE can cause weird rearrangements in > BIOS boot order depending on the x86 you're using. (Boot from IDE > CD-R*, SATA disk is (hd2). Boot from SATA disk, SATA disk is (hd0). > BTDT.)
So I read. GRUB is not very reliable for booting usb. I read that the most reliable way is to use SYSLINUX, but having played with SYSLINUX for the whole day, I think I'll rule out SYSLINUX and use back to GRUB.
> GRUB gives you a shell you can use to try to recover. LILO > doesn't.
yep, that'd be the last resort to refer to. On the contrary, using SYSLINUX would *just work*.
>>> For example for dd > > dd is not a backup method. Forget it.
I think so too.
> cp -a all directories that aren't /proc, /sys, or /dev (if using > udev/devfs), edit /etc/fstab to reflect new position of drive, reinstall > bootloader. 3 steps--which one are you having trouble with?
Actually I want to rule out 'cp' as well, since "cp -a all directories that aren't /proc, /sys, or /dev" is easy to say than done in script. rsync should be a much better option.
T
Newsgroups: gmane.linux.debian.user Date: Sat, 30 Dec 2006
I did this only yesterday - but in my case I wanted a mirror image of the entire system, not just the root partition.
The simplest most bullet proof procedure I could come up with was:
dd if=/dev/hda of=/dev/sda
vi /etc/fstab in the copy and 1,$s/hda/sda/
either
edit /boot/grub/menu.lst on the internal drive to add a boot obtion passing the USB root to the kernel, or
if your host supports USB booting, update the boot sector on sda to look for the stage 2 boot in the USB partition.
I used for first option, so am not sure if I have covered everything required for a direct USB boot…
I booted using the USB copy and everything looked to be working fine.
Didn't use 'sync,noerror' in the dd operation because I count on having perfect media, and if I don't I want to know about it!
Digby Tarvin
> For cp, at least 'cp -ax' should be used.
I've never used option flags for this purpose, and I'm not sure what the -ax options could mean in this context, since copying from device to device ignores not only the filesystem but partition data as well. Used in this way, cp streams data in byte order from device to device, copying partition tables, boot loaders and filesystem indexing tables as well as the filesystems, regardless of the target device geometry. Since I am unsure of the implications of this, I use this method only between identical devices, and only when there are no bad blocks on either device. In practice this means that I only rarely use this approach, but I have been considering it for cloning backup pc archives.
> But there are still much more to it. > > First, directories that don't need to copy over, like /tmp, /proc. With > modern Linux that uses udev, the /dev and /sys don't need to be copied > either. Anything else (besides distro specifics like /var/cache/apt/archives)? > > 2nd, the clone partition should be made bootable, by grub or lilo. > > Anything else? Like the concerns of /etc/fstab... > > Last, with all the above concerns, how to achieve them with various tools? > > Keywords: tar rsync find cpio dd
You don't mention rsync, which is currently my preferred method. It's simpler to use than tar, provides the option of incremental updates, and unlike cp -a it works around linux' lack of an "lutimes" system call, taking care of time stamps of symbolic links. For cloning root filesystem drives I use a small script that performs an rsync backup followed up by fixups to the /dev directory, /etc/fstab, and /etc/lilo.conf and then runs lilo -r to make the backup bootable. (A similar approach could be used for grub.) The script runs in a few minutes if I disable rsync checksumming.
Here is the entire script, which uses /dev/hda1 as my backup root drive:
mount /dev/hda1 /mnt/hda1 time rsync -vxaHD --delete / /mnt/hda1/ rsync -xaHD --delete /dev/ /mnt/hda1/dev/ cp -a /etc/fstab.hda1 /mnt/hda1/etc/fstab cp -a /etc/lilo.conf.hda1 /mnt/hda1/etc/lilo.conf rm -rf /mnt/hda1/dev/.udevdb lilo -r /mnt/hda1 umount /dev/hda1
This script presumes that udev is installed. For it to work, I have to manually maintain /etc/fstab.hda1 and /etc/lilo.conf.hda1.
Disclaimer for newbies: please do not use this script unless you completely understand how it works, and are not using /dev/hda1 as your root filesytem!
The backup drive can be selected through the BIOS boot menu if the main root drive craps out. The only complication is that one my applications (mythtv) uses mysql to keep track of files on an non-root filesystem, which creates the possibility that the database could be out of sync with the mythtv data files in the event of a main filesystem crash.
Marty
> For cloning root filesystem drives I use a small script that > performs an rsync backup followed up by fixups to the /dev directory, > /etc/fstab, and /etc/lilo.conf and then runs lilo -r to make the backup > bootable. (A similar approach could be used for grub.) The script runs in a > few minutes if I disable rsync checksumming. > > Here is the entire script, which uses /dev/hda1 as my backup root drive: > > mount /dev/hda1 /mnt/hda1 > time rsync -vxaHD --delete / /mnt/hda1/ > rsync -xaHD --delete /dev/ /mnt/hda1/dev/ > cp -a /etc/fstab.hda1 /mnt/hda1/etc/fstab > cp -a /etc/lilo.conf.hda1 /mnt/hda1/etc/lilo.conf > rm -rf /mnt/hda1/dev/.udevdb > lilo -r /mnt/hda1 > umount /dev/hda1 > > This script presumes that udev is installed. For it to work, I have to manually > maintain /etc/fstab.hda1 and /etc/lilo.conf.hda1.
Thanks Marty, your rsync switch is the most comprehensive one I've seen.
Just FYI, when cloning the root partition, there are directories that don't need to copy over, like /tmp, /proc. So you may want to add —exclude switches to your script next time.
Further, with modern Linux that uses udev, like what you are using, the /dev and /sys don't need to be copied either. You can —exclude them as well.
Moreover, you may want to reconsider whether you want to clone /var/cache/apt/archives next time.
The point is that I don't want to clone more than necessary.
T
> Further, with modern Linux that uses udev, like what you are using, the > /dev and /sys don't need to be copied either. You can --exclude them as > well.
Found that my Debian (grml to be exactly) can't boot up with an empty /dev directory, kernel panic about no console device found.
T
documented on: 2007-09-16
I wrote a bash script which will clone your gentoo partition on another partition, so you are safe to do anything advetureous…
Why is the script? Isn't "cp -a / /mnt/clone" would be enough? Unfortunately, this would require to unmount separate partitions, like /home, and could cause problems with /dev partition which gentoo uses. Certain special files would also be left out. So, after a reading a long discussion on gentoo user list, I chosen the possibly safest solution (at least what I tried) and wrote a scrip around it to make it easy to use.
the whole script could be done in two actual lines in it.
find -mount -print | cpio -pdm $CLONE_DIR
99% of it is checking the parameters, printing out docs making it as automatic as possible.
lakicsv Dec 31, 2002
An alternative way to do backup your partition into a compressed file is this:
tar -X tar.exclude -cvjpf backup.tar.bz2 /*
With this being an example tar.exclude file:
/mnt /tmp /usr/src /usr/portage/distfiles /var/tmp /proc
Be sure to put the path for the backup tar.bz2 archive somewhere on a different partition then your / partition.
Freak_NL Jan 23, 2003
much much simpler way would be.
mount drive to wich would hold the backup
mount /dev/hdc3 /backup
then copy files over
rsync -va --deleate --exclude=/proc/* --exclude=/dev/* --exclude=/backuo/* / /baclup
then have some code to change the fstab, and add the entry in to your grub config. all done.
then later you can add the -u option to rsync and just update what has changed.
kappax Feb 02, 2003
Newsgroups: comp.os.linux.networking, comp.os.linux.misc Date: 2001-08-26 03:09:36 PST
> > [. . .] How can I copy the disc 1:1 to the ohter computer. they
>if all the harddisks are the same kind (i.e. 6.5gb Seagate), attach >three disks beside the original (let's say that original is /dev/hda >and others are hdb, hdc and hdd. Now copy with device dump (dd). >dd if=/dev/hda of=/dev/hdb bs=10000 & >dd if=/dev/hda of=/dev/hdc bs=10000 & >dd if=/dev/hda of=/dev/hdd bs=10000 &
This sounds nice, but will not avoid any bad sectors that might be on the new drive. (If you are sure that the new drive has no bad sectors, you might be okay.) Also, for efficiency, the "bs=" parameter should probably be set to a power of 2, or at least to a multiple of 512. Note that this also requires you to move all of the hard disks into one computer (might be a mess) and then move them back, instead of doing it over the network like you probably want.
First, read the hard disk upgrade instructions at http://www.linuxdoc.org/HOWTO/mini/Hard-Disk-Upgrade/index.html
However, don't start quite yet. You have a few things to keep in mind:
That document recommends using "cp -x" as one of the ways to back up a single file system. However, in some versions of GNU fileutils, the "-x" option to stay within one file system is broken — it just keeps rolling along. If you are not sure that you have a properly working version of the "cp" command, do not depend upon this option. My workaround for this was to use "cp -a" on each top-level directory and "cp -dp" on the top-level files that I needed to copy (basically the third method in the above-referenced document), back when I was doing a disk- to-disk copy (different than the tarball copy method that I have been slogging through recently and finally figured out today). At the time I was doing this (last fall), the latest alpha version of fileutils had this fixed; this has presumably been released now, but many Linux installations are likely to have the broken version. (For instance, I was using MontaVista Hard Hat Linux 1.1, which is a close derivative of Red Hat 6.1 — not the latest, but not exactly ancient either.)
Caution: when readjusting Lilo for the new hard disk, proofread /etc/lilo.conf CAREFULLY. Referencing a non-existent partition or misspelling something in this file can cause failures that are quite an unwanted challenge to diagnose.
If you deviate from that document and instead use a tarball on intermediate media (for instance, a CD-R or a network file server) as I did (and it sounds like you probably want), it wouldn't hurt to do a compare between the backup tarball and your original disks after creating the tarball, just to make sure that it didn't get corrupted. This also applies to doing a disk-to-disk copy (especially if you are not yet sure of the quality of your IDE connection — I have found that this can cause corruption before it causes total failure). My backup command (the version that worked) was (executed in the root directory of the original disk):
tar -zcvpf /mnt/backup_volume/backup_file.tgz -T backuplist.txt
where backuplist.txt was a list of top-level files and directories to back up (easiest way to get this is "ls -1 > backuplist.txt" and then edit the file to get rid of /mnt and /proc.
The compare command would be:
tar -zdvpf /mnt/backup_volume/backup_file.tgz
I didn't do this on the original drive, because I was trying to save time in my experimentation, but I did do it on the destination drive after unpacking the tarball (see below), because I suspected hard disk corruption on the destination.
The restore command was (executed in the root directory of the destination disk):
tar -zxvpf /mnt/backup_volume/backup_file.tgz
On the destination disk, you need to create the /mnt directory tree and /proc manually after doing either a disk-to-disk copy or a restore from a tarball. (The version of the "tar" command from RedHat 6.1 or later has no problem with the /dev nodes.) After I did all this, everything seems to work properly (including having the right permissions on the restored /tmp directory, which I noticed were wrong if the "-p" option was not in the "tar" commands).
One pain in the behind I have had is having to use 2+1/2 different boot disks for this. Tom's Root Boot Disk is able to mount the remote NFS volume containing the tarball, but has a cheap plastic imitation of the "tar" command that can't handle a tarball, so I had to compile the "tar" and "gzip" commands statically linked (or they wouldn't work with the libraries on Tom's Root Boot Disk) and put them on the remote NFS volume. Tom's Root Boot disk also does something that makes Lilo abort with an error message like "System map too big", even if I try to run a statically linked recent version of Lilo from the remote NFS volume. Therefore, I have to boot off a Red Hat 6.2 or 7.0 installation CD in rescue mode, which is really brain-dead (the 7.0 version being even worse than 6.2) and doesn't create most of the hard disk nodes in /dev, so I have to do it manually; in addition, the Red Hat rescue mode seems to be unable to get proper network access (which is why I have to use Tom's Root Boot Disk for the first part of the restore). I haven't yet had a chance to mess with other distributions' rescue modes (where I work, we are stuck with Red Hat by management decision — be glad Linux is even allowed into product at all) or DemoLinux, but these might alleviate this problem.
Lucius Chiaraviglio
Newsgroups: gmane.linux.debian.user Date: Sun, 30 Apr 2006 11:08:12 -0400
> This medium makes it practical. > > Which tools are best and simplest for this?
Personally, I prefer systemimager. There is a great howto on snapshot-style backups with rsync:
All I did was to substitute systemimager for rsync. Now, systemimager uses rsync under the hood, but lots of the things that howto has you do manually are easily configurable in systemimager. So, you can take your periodic snapshots and then burn them to DVD.
Roberto C. Sanchez
> Which tools are best and simplest for this? > > (I do not like partimage because it insists on identical sizes for restoring > and this might not be the case as I have learned in moving my partitions > around.)
Not so AFAIK: just the partition to restore from cannot be bigger that the one restored to.
It's main *con* seems to me that you cannot use partimage on a running system.
I use mondo and how to use it to burn DVD's was posted in February of 2006: http://lists.debian.org/debian-user/2006/02/msg00203.html
Hugo Vanwoerkom
> Which tools are best and simplest for this?
I use DAR and a script called DARomizer which is designed to backup a hard drive to multi DVD-RW disks.
Bill Thompson
The following urls can be savely ignored: