rsync: --sparse cannot be used with --inplace
virtsync is a $49 commercial Linux command-line tool for synchronizing the contents of huge files (such as virtual machine disk images and databases).
chris@server:~$ time virtsync -v /var/lib/libvirt/images/vsws.img backup.barricane.com:/home/chris/ syncing /var/lib/libvirt/images/vsws.img to backup.barricane.com:/home/chris/vsws.img (dot = 1 GiB) [........>.........................................] done - 53687091200 bytes compared, 4096 bytes transferred. real 2m47.201s user 0m48.821s sys 0m43.915s
It can update a remote copy of a 50GB file, across broadband internet links, in just a few minutes.
"Just tried it with one file - it not only works, but it's way faster than rsync, excellent!"
- Taneli Leppä (crasman.fi)
VirtSync can update an older copy of a file to the latest content. Ideal for backups - either to a single backup drive, or to a set of rotated backup media.
VirtSync's efficiency allows multi-gigabyte files to be replicated to remote backup and hot standby locations. It is currently synchronizing 50GB+ disk images across the internet in under 3 minutes each
Remote synchronisation requires that the machine's public certificate has been installed in the target machine's authorized_keys file. If login to the remote machine is not possible without a password prompt, virtsync will fail. See "man ssh-copy-id".
chris@server:~$ time virtsync -v /var/lib/libvirt/images/vsws.img backup.barricane.com:/home/chris/
syncing /var/lib/libvirt/images/vsws.img to backup.barricane.com:/home/chris/vsws.img (dot = 1 GiB)
[..................................................]
done - 53687091200 bytes compared, 0 bytes transferred.
real 2m44.662s
user 0m48.730s
sys 0m44.000s
(or under 20 minutes when there are 800MB of changes to synchronize).
chris@server:~$ time virtsync -v /var/lib/libvirt/images/vsws.img backup.barricane.com:/home/chris/
syncing /var/lib/libvirt/images/vsws.img to backup.barricane.com:/home/chris/vsws.img (dot = 1 GiB)
[>.......>.>.>.>.>.>.>.>.>.....>.......>.......>...]
done - 53687091200 bytes compared, 810745856 bytes transferred.
real 17m42.061s
user 0m47.590s
sys 0m46.870s
(The above tests were done using 5 year old servers, and FTTC broadband.)
VIRTSYNC(1) VIRTSYNC(1)
NAME
virtsync - synchronise huge sparse files efficiently
SYNOPSIS
local:
virtsync [OPTIONS] source_file dest_file
virtsync [OPTIONS] source_files dest_dir
virtsync [OPTIONS] source_dir dest_dir
remote: (via ssh)
virtsync [OPTIONS] source_file [user@]host:dest_file
virtsync [OPTIONS] source_files [user@]host:dest_dir
virtsync [OPTIONS] source_dir [user@]host:dest_dir
DESCRIPTION
Synchronise source_file to dest_file or dest_dir, multiple source_files
to dest_dir, or source_dir to dest_dir.
Remote synchronisation requires that the machine's public certificate has
been installed in the target machine's authorized_keys file. If login
to the remote machine is not possible without a password prompt,
virtsync will fail. See "man ssh-copy-id".
Destination files are updated in-place, with only changed blocks being
written.
Destination files and directories are created, if they do not already
exist.
OPTIONS
-V, --version
Output program version.
-h, --help
Output help information.
-v, --verbose
Enable verbose output.
-j, --json-stats
Enable JSON stats.
REPORTING BUGS
Report virtsync bugs to virtsync@barricane.com
VirtSync home page: http://www.virtsync.com
AUTHOR
virtsync was written by <chris.dew@barricane.com>.
COPYRIGHT
Copyright (C) 2011-2013, Barricane Technology Ltd. All Rights Reserved.
September 18, 2011 VIRTSYNC(1)
loading libcrypto.so... loading libcrypto.so.0.9.8... loading libcrypto.so.0.9.8e... loading libcrypto.so.6... libcrypto.so.6: cannot open shared object file: No such file or directory
This is a bug in v0.9.5 - it will be fixed in the next release. You can work around this issue by installing the openssl-devel package (for .rpm users) or libssl-dev (for .deb users).
Yes, if the VM is not writing to it's disk for the duration of the synchronization. Otherwise, look at LVM snapshots.
When there is no old version of the file as a target, the entire contents of the file must be sent. For a remote sync, this will be limited by the speed of your network connection. For a local sync, this will be limited by the sequential IO rate of your drives.
Evaluation copies are licensed for 60 days from the date of download.
The majority of modern servers run 64 bit operating systems. If you need
a 32 bit package then
please
If you would like to use virtsync beyond the expiry of your evaluation copy, you must buy licenses to use the software on one or more machines.
Licensed copies of virtsync will cost $49 per machine, or $79 for machines with Xeon or Opteron processors. Performing remote synchronization requires at licenses for at least two machines.
We will have payment processing in place shortly, well in advance of the first evaluation copies expiring in July 2013.
You can contact virtsync's author by email at chris.dew@virtsync.com. virtsync is a product of Barricane Technology Ltd., a micro-ISV based in England.