Fast VMware VCB Backups
Update 11th Mar 2010 : Less noisy output and retries backup request if it failed
Update 29th Nov 2009 : Major improvement to VM selection rules
Update 18th Nov 2009 : Bug fix release.
Update 15th Nov 2009 : Now handles zombie processes correctly so will terminate correctly when called from .cmd batch scripts.
Update 10th Nov 2009 : Now deletes old backup snapshots left behind by failed previous backups.
This is a new program which uses the VMware "Virtual Consolidated Backup" (or VCB) system to work, and will run on any Windows system. It is highly parallel, very fast and will easily saturate a 1Gbps network link. I have had performance of over 100Mbytes/second across our network, with a sustained rate of over 70Mbytes/second for several hours.
It is far faster than my previous JKFBackup.sh script mentioned in a previous article. It also does not require any ssh or other "unsupported" access to the VMware host server.
It takes a command-line parameter which is the full pathname of the configuration file that it uses. This can be surrounded in double-quotes (") if there are spaces in the path or filename.
The configuration file is a simple text file:
The INCLUDE and EXCLUDE lines work like rules in a firewall. A rule can match several VMs by using “*” wildcard characters in them. As a simple example, the line “INCLUDE kanga-*” would tell the program to backup all VMs whose names start with “kanga-”. The first rule that matches the name of the VM says whether it will be backed up or not. I strongly advise you put either “INCLUDE *” or “EXCLUDE *” as the last rule in the file, to specify the default action for any VMs that do not match any other rules, such as VMs that have been added since you updated the configuration file.
Currently the script uses a different directory for each day of the week, so you have the last week's worth of backups on the disk. If you want to change this, learn a little Perl and you will find the code is very simple and straightforward.
As an example, the backups for Thursday for the VM named "Dummy" will be stored in DIRECTORY\Thu\Dummy.
If you want to backup several VMware host servers, then have a separate configuration file for each server. This script is currently aimed at relatively small vSpheres where you only have a few VMware host servers. If you have hundreds of VMware host servers, then please feel free to rip the script apart and use any bits of it you want. If you are prepared to let me include your improvements in a future release, then please contact me!
To backup VMs as fast as possible, the script splits the list of VMs into 3 and does 3 backups simultaneously in parallel.
Using VCB, data recovery is very simple. Just look in the directory named after the VM you want to restore, and use the VMware Standalone Converter to turn the files in there back into a VM on your VMware server. Easy as that!
You can download version 1.10 of VCBBackup.pl here.
I suggest you right-click on that link and choose "Save as..." to download the file.
It is written in Perl so you will need to install the free Perl distribution from www.activestate.com. You will also need to download and install the "VMware Consolidated Backup" distribution from VMware, which you should be able to access if you have purchased VMware products.
I do my backups onto a little Dell 860 1U rackmount server running Windows Server 2008R2 x64. It backs up to cheap external disk connected via a Firewire-800 interface. This is quite sufficient and can maintain over 700 Mbits/second backup speed.
Every couple of nights I swap out the hard disk and take it off-site. That provides us with backups of the last fortnight with some off-site storage.
I always welcome constructive feedback, and suggestions and requests for new features. If you need any help, just contact me!
Update 29th Nov 2009 : Major improvement to VM selection rules
Update 18th Nov 2009 : Bug fix release.
Update 15th Nov 2009 : Now handles zombie processes correctly so will terminate correctly when called from .cmd batch scripts.
Update 10th Nov 2009 : Now deletes old backup snapshots left behind by failed previous backups.
This is a new program which uses the VMware "Virtual Consolidated Backup" (or VCB) system to work, and will run on any Windows system. It is highly parallel, very fast and will easily saturate a 1Gbps network link. I have had performance of over 100Mbytes/second across our network, with a sustained rate of over 70Mbytes/second for several hours.
It is far faster than my previous JKFBackup.sh script mentioned in a previous article. It also does not require any ssh or other "unsupported" access to the VMware host server.
It takes a command-line parameter which is the full pathname of the configuration file that it uses. This can be surrounded in double-quotes (") if there are spaces in the path or filename.
Configuration
The configuration file is a simple text file:
- All leading and trailing spaces are ignored
- All blank lines are ignored
- Comments start with a hash character (pound sign in the USA) like this "#" and continue to the end of the line
- Each line must start with one of the following keywords, followed by its single parameter:
- DIRECTORY : the full path, including drive letter, under which all the backups will be stored
- HOST : the name of the VMware host server running the VMs to be backed up
- USER : the username to access the VMware host server (usually root)
- PASSWORD : the password for the stated USER
- INCLUDE : backup this named VM. Use the “*” character as a wildcard to mean “0 or more characters” to specify several VMs
- EXCLUDE : do not backup this named VM. Use the “*” character as a wildcard to mean “0 or more characters” to specify several VMs
The INCLUDE and EXCLUDE lines work like rules in a firewall. A rule can match several VMs by using “*” wildcard characters in them. As a simple example, the line “INCLUDE kanga-*” would tell the program to backup all VMs whose names start with “kanga-”. The first rule that matches the name of the VM says whether it will be backed up or not. I strongly advise you put either “INCLUDE *” or “EXCLUDE *” as the last rule in the file, to specify the default action for any VMs that do not match any other rules, such as VMs that have been added since you updated the configuration file.
Currently the script uses a different directory for each day of the week, so you have the last week's worth of backups on the disk. If you want to change this, learn a little Perl and you will find the code is very simple and straightforward.
As an example, the backups for Thursday for the VM named "Dummy" will be stored in DIRECTORY\Thu\Dummy.
If you want to backup several VMware host servers, then have a separate configuration file for each server. This script is currently aimed at relatively small vSpheres where you only have a few VMware host servers. If you have hundreds of VMware host servers, then please feel free to rip the script apart and use any bits of it you want. If you are prepared to let me include your improvements in a future release, then please contact me!
To backup VMs as fast as possible, the script splits the list of VMs into 3 and does 3 backups simultaneously in parallel.
Data Recovery
Using VCB, data recovery is very simple. Just look in the directory named after the VM you want to restore, and use the VMware Standalone Converter to turn the files in there back into a VM on your VMware server. Easy as that!
Download
You can download version 1.10 of VCBBackup.pl here.
I suggest you right-click on that link and choose "Save as..." to download the file.
It is written in Perl so you will need to install the free Perl distribution from www.activestate.com. You will also need to download and install the "VMware Consolidated Backup" distribution from VMware, which you should be able to access if you have purchased VMware products.
My Setup
I do my backups onto a little Dell 860 1U rackmount server running Windows Server 2008R2 x64. It backs up to cheap external disk connected via a Firewire-800 interface. This is quite sufficient and can maintain over 700 Mbits/second backup speed.
Every couple of nights I swap out the hard disk and take it off-site. That provides us with backups of the last fortnight with some off-site storage.
Feedback
I always welcome constructive feedback, and suggestions and requests for new features. If you need any help, just contact me!