Configuring rsync

From Metro Studios Knowledgebase

Jump to: navigation, search

Contents

[edit] References

The following references were used to create this guide:

http://www.askapache.com/security/mirror-using-rsync-ssh.html
http://troy.jdmz.net/rsync/index.html

[edit] Source Slice

[edit] Determine Private Network IP

rsync needs to be run over the private network only. To identify the private network use the ifconfig command:

ifconfig

This will give you output that looks like this:

 
    eth0      Link encap:Ethernet  HWaddr 40:40:eb:8a:23:e6  
              inet addr:173.203.209.224  Bcast:173.203.209.255  Mask:255.255.255.0
              inet6 addr: fe80::4240:ebff:fe8a:23e6/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:112815106 errors:0 dropped:0 overruns:0 frame:0
              TX packets:82057975 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:50843824588 (50.8 GB)  TX bytes:115244445160 (115.2 GB)
              Interrupt:24 
 
    eth0:1    Link encap:Ethernet  HWaddr 40:40:eb:8a:23:e6  
              inet addr:173.203.241.221  Bcast:173.203.241.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              Interrupt:24 
 
    eth1      Link encap:Ethernet  HWaddr 40:40:2d:d2:80:db  
              inet addr:10.177.144.244  Bcast:10.177.159.255  Mask:255.255.224.0
              inet6 addr: fe80::4240:2dff:fed2:80db/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:212 errors:0 dropped:0 overruns:0 frame:0
              TX packets:244 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:34652 (34.6 KB)  TX bytes:35734 (35.7 KB)
              Interrupt:25 
 
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:66162451 errors:0 dropped:0 overruns:0 frame:0
              TX packets:66162451 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:20856531860 (20.8 GB)  TX bytes:20856531860 (20.8 GB)

Look for the inet addr value that starts with 10.
In this case that is inet addr:10.177.144.244

[edit] Install rsync

If the server does not yet have rsync installed, use apt-get to install it:

sudo apt-get install rsync

[edit] Create RSA keys

So a password does not need to be stored in plain text on the server, public/private key authentication will be used. First, prepare the directory which will store the keys:

sudo mkdir -p /root/.ssh
sudo chmod 700 /root/.ssh

Then create the keys (be sure that you create a key with no passphrase):

sudo ssh-keygen -t rsa -b 2048 -f /root/.ssh/rsync.key

[edit] Transfer public key to destination slice

The public key file that was generated in the previous step now needs to be transferred to the destination slice. The scp command can be used to transfer this file. Be sure that you use the private network to transfer this file by specifying the private IP address of both the source and destination servers:

sudo scp -o BindAddress=10.177.144.244 -P 11200 /root/.ssh/rsync.key.pub USERNAME@10.177.138.143:/home/USERNAME/

In this case 10.177.144.244 is the source server and 10.177.138.143 is the destination server. Also be sure to replace USERNAME with your account username on the destination slice.

[edit] Setup SSH configuration

To make the connection string simpler when doing the rsync, setup the SSH configuration file:

sudo nano /root/.ssh/config

Add the following to that file:

Host dev1
   IdentityFile /root/.ssh/rsync.key
   Port 11200
   Protocol 2
   User rsync
   BindAddress 10.177.144.244
   HostName 10.177.138.143
   PasswordAuthentication no

[edit] Test rsync and run initial backup

Before you can test rsync, setup the destination slice so it is ready to accept the transferred files.

Once that has been setup you can run a test to check that everything is working:

sudo rsync -e 'ssh' --delete --dry-run --verbose --progress -url /var/www slice1:~/backup

This will show some output showing the files that would be transferred. If you get any errors, check the configuration on both the source and destination slices.

Once the test is working well, you can run the initial rsync. Use this command:

sudo rsync -e 'ssh' --delete --verbose --progress --chmod=Du+x -url /var/www slice1:~/backup

This will run the first backup, which may take several minutes (maybe hours) to complete. Once you have verified that the backup worked properly, it's time to setup a cron job to automate the backup.


[edit] Setting up a cron job

All you will have to do to set the rsync scheduled job up is add a templated command to the root user's crontab:

crontab -e

The templated command will look like:

rsync -e 'ssh' --delete --verbose --progress --chmod=Du+x -url /var/www {SSH host alias}:~/backup 2>&1 >> /root/backups/rsync.log

Replace {SSH host alias} with the SSH host alias you set up in the "/root/.ssh/config" configuration file.

Scheduling of the job should happen in a staggered manner that does not affect any production processes of the server. Typically this would be run at night or early morning. Refer to the Slice Backup Schedule in order to figure out a time for the process to fire.

After the cron job is setup you will need to edit/add the logrotate.d/rsync file in order to keep the generated rsync logs under control.

[edit] Destination Slice

Determine the private network IP address and install rsync as you did on the source slice.

[edit] Add rsync user

An unprivileged user account will be used for the rsync transfers between slices. To create that user, execute this command:

sudo adduser rsync

Do not set a password, just keep hitting enter when requested!

[edit] Setup public key

You now need to setup the public key that you transferred from the source slice. That needs to go into the rsync user's home directory:

sudo mkdir -p /home/rsync/.ssh
sudo chmod 700 /home/rsync
sudo touch /home/rsync/.ssh/authorized_keys
sudo chmod 700 /home/rsync/.ssh
sudo chmod 600 /home/rsync/.ssh/authorized_keys
sudo chown -R rsync:rsync /home/rsync/.ssh
sudo bash -c "cat /home/USERNAME/rsync.key.pub >> /home/rsync/.ssh/authorized_keys"
sudo rm /home/USERNAME/rsync.key.pub

[edit] Allow SSH connections for the rsync group

The SSHd configuration needs to be updated to allow connections from users in the rsync group. Open the SSHd configuration file for editing:

sudo nano /etc/ssh/sshd_config

Near the bottom of that file you should see the AllowGroups setting:

AllowGroups server-admins

Add rsync to that list:

AllowGroups server-admins rsync

Save the file and then reload it in SSHd:

sudo service ssh reload

[edit] Restrict connections by IP address

The public key file can be modified to allow connections only from a specific IP address. Edit that file to make this change:

sudo nano /home/rsync/.ssh/authorized_keys

The contents of that file will look like this:

ssh-rsa AAAABBLAHBLAHBLAHBLAH

Add the from option like so:

from="10.177.144.244" ssh-rsa AAAABBLAHBLAHBLAHBLAH

Where 10.177.144.244 is the private IP address of the source slice.

[edit] Logging

In order to log the output of our rsync processes we tack a "2>&1 >> /root/backups/rsync.log" onto the end of our cron command. the "2>&1" redirects STDERR to STDOUT and then we use the ">>" targeting "/root/backups/rsync.log" to append the log.

This should record any output that normally would be reported to the screen for said command.

In addition to setting up logging we will need to rotate the generated logs using logrotate: rsync Log Rotate

[edit] Notes/Extras

sudo apt-get install mailutils
Personal tools
Namespaces
Variants
Actions
Wiki Navigation
Knowledgebase
Toolbox