Automated Remote Backups with Rdiff-backup

One subtle feature of rdiff-backup is that it allows users to make remote backups over Internet using SSH, which makes remote backups very secure since data transferred is encrypted.

One problem is that SSH requires a password for logging, which is not convenient if we want to run rdiff-backup as a cron job. Here we show how to initiate rdiff-backups from a central backup server, and pull data from a farm of hosts to be backed up. For security reasons, the central server uses a non-root user account (rdiffbk) to perform backups, whereas root account is used on each host being backed up. Though root accounts are used on hosts being backed up, they are protected by SSH public-key authentication mechanism with forced-command-only option.

For convenience, I'll call the central backup server canine and three hosts to be backed up beagle, shepherd and terrier. For short, only works on canine and beagle will be shown.

Here is the procedure for backup server canine:

1. generate one passphrase-free SSH key pair for each host being backed up,
2. move corresponding ssh key to each host,
3. create SSH configuration file, and
4. create a cron job file

Step 1:  generate one passphrase-free SSH key pair for each host being backed up

To generate RSA type pair for host beagle, we issue

ssh-keygen -t rsa -f id_beagle-backup


where private key will be saved in file id_beagle-backup and public key id_beagle-backup.pub.

Step 2: move corresponding ssh key to each host

To move id_beagle-backup.pub to host beagle, we may choose to use any preferred method (for example, ftp, sftp, or ssh-copy-id), since public key is not sensitive. Other hosts can be done similarly.

Step 3: create SSH configuration file

To define how to connect to host beagle with backup key, we place the following lines into file ~rdiffbk/.ssh/config. Other hosts need to be configured similarly.

host beagle-backup
    hostname beagle
    user root
    identifyfile ~rdiffbk/.ssh/id_beagle-backup
    protocol 2

Step 4: create a cron job file

The following cron job file automates the remote backups daily at 200am, 210am, and 220am, respectively.

0 2 * * * rdiff-backup beagle-backup::/remote_dir beagle/remote_dir
10 2 * * * rdiff-backup shepherd-backup::/remote_dir shepherd/remote_dir
20 2 * * * rdiff-backup terrier-backup::/remote_dir terrier/remote_dir



By default setting, rdiff-backup uses SSH to pipe remote data. Therefore, both SSH server and rdiff-backup are required in hosts to be backed up. 
What left on host beagle and others (shepherd, terrier) is simply to give permission to canine to access it (through SSH) and run rdiff-backup. This can be done in the following two steps:


Step I: create an authorized-keys file for root account

To enable SSH public key authentication for root account, we need to create the file /root/.ssh/authorized_keys, which consists public key for user rdiffbk@canine, forced command and other options. The public key (id_beagle-backup.pub) should be available for beagle once we have done Step 2. A sample authorized_keys file is as follows:

command="rdiff-backup --server --restrict-read-only /",from="canine",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB3.... rdiffbk@canine

Here, for security reason, rdiff-backup server is restricted to real only, and
we disable port-forward, X11-forward and pty options. See here for more details.

Step II: configure SSH server for root access

As we saw here, this can be done by put the following line in the SSH server configuration file (sshd_config):

PermitRootLogin forced-commands-only

Tips on Forced Command for SSH

In general, an SSH connection invokes a remote command chosen by the client. There are times that server should decide which command the client will run. The forced command enables us to achieve this goal.

There are two ways to forced command. One is through public-key authentication configuration in the file authorized_keys as we saw here. The other is thought the usage of the keyword ForceCommand in sshd_config. To restrict users run nothing but the alpine command, we put the following line in sshd_config:

ForceCommand /usr/bin/alpine

The major difference between these two are: configuration though public-key authentication applies to one user, and each user may have her/his own option; configuration through ForceCommand keyword may be system-wide, keyword Match should be used combinedly if ForceCommand should apply to certain user(s).

What if we want the user to not only execute a single command, but few fixed commands at user's choice, such as:

• show process list (ps aux),
• print system information (uname -a),
• show who is logged on (who), or
• start rdiff-backup server (rdiff-backup --server --restrict-read-only /)

A simple wrapper script combined with the ForceCommand will suffice. Here is an example that allows user backup to invoke four different commands at user's choice.


With environment variable SSH_ORIGINAL_COMMAND,
the following script (wrapper.sh) wraps all permitted commands:


#!/bin/sh
# Script: /usr/local/bin/wrapper.sh

case "$SSH_ORIGINAL_COMMAND" in
    "ps")
        ps aux
        ;;
    "uname")
        uname -a
        ;;
    "who")
        who
        ;;
    "rdiff")
        rdiff-backup --server --restrict-read-only /
        ;;
    *)
        echo "Only the following commands are available to you:"
        echo "ps, uname, who and rdiff"
        exit 1
        ;;
esac

The configuration (sshd_config) of ForceCommand with Match (user backup) is as follows:

Match User backup
    ForceCommand /usr/local/bin/wrapper.sh


To show process list on ssh server, one issues:

ssh backup@server ps

where original command "ps" was passed to the wrapper script by environment variable SSH_ORIGINAL_COMMAND.

To backup directory tree /path_to_src on server to local directory /path_to_dst, one issues:

rdiff-backup --remote-schema "ssh -C %s rdiff" backup@server::/path_to_src /path_to_dst

Root Access Control for SSH

Sshd has a separate access control mechanism for the root (superuser). The keyword PermitRootLogin specifies its usage.

The argument (option) for PermitRootLogin must be "no", "yes'', "without-password'', or ``forced-commands-only''. If this option is set to "no'', root is not allowed to log in.

If this option is set to "without-password'', password authentication is disabled for root. However, root may login in with GSSAPIAuthentication, HostbasedAuthentication or PubkeyAuthentication, if they are set properly.



If this option is set to "forced-commands-only'', root login with public key authentication is allowed, but only if the command option is specified (which may be useful for remote backup as we saw in the example of public-key-based configuration).  All other authentication methods are disabled in this setting.

Public-key-based Configuration for SSH server

Public key is one of the frequently used authentication methods in SSH. To set up public-key authentication for one's account on an SSH server, one creates an authentication file named authorized_keys (for OpenSSH), and lists key and options that provide access to one's account.

Each line (SSH protocol 2) in authorized_keys may contain:

1. An (optional) set of authorization options for the key.
2. A (required) key type string: ssh-dss for a DSA key, or ssh-rsa for an RSA key.
3. The (required) base64-encoded public key.
4. An (optional) descriptive comment.

The optional options consist of comma-separated option specifications, where no space is allowed, except within double quotes.  Some common option specifications are:

command="command": Specifies that the command to be executed
from="pattern-list": Specifies the permitted client name or IP address
no-port-forwarding: Forbids TCP forwarding
no-X11-forwarding: Forbids X11 forwarding
no-pty: Prevents tty allocation

The following example file specifies that:
the command "rdiff-backup --server --restrict-read-only /" to be executed if client is from the machine named "beagle" where no port, X11 forwarding is allowed. Notice that all settings are in one line.

command="rdiff-backup --server --restrict-read-only /",from="beagle",no-port-forwarding,no-X11-forwarding ssh-rsa AAAAB3.... root@beagle

Tips on SSH Client Configuration

OpenSSH client ssh obtains configuration data from sources in the following order:

1.  command-line options,
2.  user's configuration file (~/.ssh/config),
3.  system-wide configuration file (/etc/ssh/ssh_config)

Since each parameter may be defined in different sources, order of parameter definition is important. The ssh manual page (ssh_config) says that:

For each parameter, the first obtained value will be used.

Here are some frequently used parameters:
Each configuration file contains sections separated by host specifications that applies to all matching hosts specified by the Host parameter. The host is the hostname argument given on the command line.

Hostname: Specifies the real host name to log into.

IdentityFile: Specifies a file from which the user's public key authentication identity is read.

Port: Specifies the port number to connect on the remote host (if it is not 22).


User:  Specifies the user to log in as.

Here is an example configuration file (~/.ssh/config) for remote machine robert.some.net:

host bob
        hostname robert.some.net
        identityfile /somepath/.ssh/id_rsa_bob
        port 2222
        user root

With the above configuration file, once we issue:

ssh bob

which is equivalent to

ssh -i /somepath/.ssh/id_rsa_bob  -p 2222 root@robert.some.net

Using public keys for SSH authentication

SSH supports many methods for user authentication. Public key, password, and host-based are three main methods specified in RFC 4252 (The Secure Shell (SSH) Authentication Protocol). With the public key method, the possession of a user's private key serves as authentication. Host-based authentication works similarly: the possession of host's (client's) private key enables the authentication based on  the user names on the server and the client.

SSH authentication using public keys can be achieved in two steps:

1. create a public and private key pair on client side, and
2. copy your public key to server.

Once these are done, we should be able to login remotely  without being prompted for a password.

The command ssh-keygen allows us to generate and manage authentication keys. To generate a pair of RSA keys, we use:
  
ssh-keygen -t rsa

It generates a pair of RSA keys and saves them in directory $HOME/.ssh; the default name for public (private) key is id_rsa.pub (id_rsa). Always use passphrases to protect your private keys.

Public keys are not sensitive data in general, we may choose any method to copy them. However, command ssh-copy-id provides an easy way to accomplish this. You use:

ssh-copy-id user@server

to copy your public key to server. After that, your public key file will be copied/appended to file $HOME/.ssh/authorized_keys in server. Don't forget to specify (option -i) the path of your public key file if it is not in $HOME/.ssh/id_rsa.pub on client side.

An ssh-agent is very helpful in using public keys for SSH authentication; it is strongly recommended.

How to test Django development server remotely?

Django development server is intended only for use while developing. There are times that we need to test Django applications remotely through development server. Thought we may bind the development server to some public IPs to achieve this, it is not a good practice in general for various reasons. Ssh port forwarding provides a much secure way for remote testing of Django development server. Let serv be the server where Django development server resides, test the remote host where you want to conduct the tests. Here is a how-to.

1. On host serv, start your development server:
python manage.py runserver 8080
where 8080 is any port number of your choice.

2. On host test, start ssh port forwarding:
ssh -L 9090:localhost:8080 user@serv
where 9090 is the port number providing remote service.

3. On host test, point your browser to localhost:9090 to perform tests.


If you want to test the development server through other hosts, say  third, you should enable gateway forwarding on host test by issuing:

ssh -g -L 9090:localhost:8080 user@serv

Then you may perform tests through host third by pointing your browser to test:9090.

an advanced usage of screen

The following script is an advance usage of command screen, which allows us to connect to a list of machines with few keystrokes (if key-based ssh logins is enabled). This is quite handy if we need to manage a long list of servers from our desktop.

#!/bin/bash
screen -d -m -S srvA   ssh serverA.domain.name
screen -d -m -S srvB   ssh serverB.domain.name
screen -d -m -S srvZ   ssh serverZ.domain.name

where options
-d -m        creates a new screen session in  detached mode
-S srvA  names identify of the session with the name srvA


To work on srvB, you just attach to its screen by typing:

screen -r srvB

Key-based ssh logins

Key-based authentication is a much more secure mode of authentication usable with (Open)SSH. It may provide a way for ssh automatic login. Here is the procedure to set up a client:

1. generate a key pair
The following command will generate RSA keys with a bit length of 4096 for use by SSH protocol version 2:

ssh-keygen -t rsa -b 4096


2. transfer client (public) key to server
The public key is normally stored in the file $HOME/.ssh/authorized_keys. This can be done by issuing:

ssh-copy-id -i server

where -i specifies that the key to be copied is stored in its default location ($HOME/.ssh/id_rsa.pub, in our example). In fact, it can be done by any your preferred method to move file from client to server.


For ssh automatic login, we need the help from ssh-agent, which allows us to store (using ssh-add, for example) our private key identities to the authentication agent.

ssh port forwarding

There are two pairs of client-server involved in ssh port forwarding: a pair of application (AP) client-server and a pair of ssh client-server. The ssh client-server provides a secure channel for communication between AP client and AP server, and there are many other usages of ssh port forwarding.

Ssh port forwarding can be categorized into local and remote forwarding. In a local forwarding, the AP client sits beside the ssh client (and the AP server sits beside the ssh server). In a remote forwarding, the situation is reversed: the AP client sits beside the ssh server (and the AP server sits beside the ssh client).

Let matt be the ssh client, mark the ssh server, clnt the AP client, and serv the AP server providing its service on port 143.

A local port forwarding can be set up by issuing the following command on host matt:

ssh -L 8080:serv:143 mark

Once the ssh channel is established, clnt is able to access serv by connecting to port number 8080 on host matt as shown in the following figure:

    clnt                          serv
                                      143
      |                                 |
      |                                 |
    8080
    matt <=======>  mark


The ssh server (mark) should be configured to enable port forwarding, which can be done by setting the option AllowTcpForwarding to yes in its configuration file. Similarly, if clnt is a different host from matt, we need to enable the option GatewayPorts on host matt by issuing the command:

ssh -g -L 8080:serv:143 mark



A remote port forwarding can be set up by issuing the following command on host matt:

ssh -R 8080:serv:143 mark

Once the ssh channel is established, clnt is able to access serv by connecting to port number 8080 on host mark as shown in the following figure:

    clnt                          serv
                                      143
      |                                 |
      |                                 |
    8080
    mark <=======>  matt

The AllowTcpForwarding and GatewayPorts options should be set properly on host mark.



One question remains: which forwarding to choose?
A quick rule is to look for the client application. If the client is running locally on (or, close to) ssh client machine, use local forwarding. Otherwise, one should choose remote forwarding.