Rdiff-backup by Examples

Rdiff-backup is a python script that backs up one directory  to  another. Some features of rdiff-backup as claimed in its official site are: easy to use, creating mirror, keeping increments, and preserving all information. Here are some examples of its usage.

1. simple backing up (backup local directory foo to local directory bar):

rdiff-backup foo bar

2. simple remote backing up (backup local directory /some/local_dir to directory /whatever/remote_dir on machine hostname.net):

rdiff-backup /some/local_dir  hostname.net:://whatever/remote_dir

Ssh will be used to open the necessary pipe for remote backing up.

3. simple restoring from previous backup (restore from bar/dir to foo/dir):

cp -a bar/dir foo/dir

4. simple restoring from the latest remote backup (restore from hostname.net:://whatever/remote_dir to local directory /some/local_dir):

rdiff-backup -r now hostname.net:://whatever/remote_dir /some/local_dir

5. restoring from a certain version of a remote backup (restore from backup done 15 days ago):

rdiff-backup -r 15D hostname.net:://whatever/remote_dir /some/local_dir

6. restoring from an increment file (restore file pg.py to its version dated 2011-11-30T00:28:38+08:00)

rdiff-backup hostname.net:://remote-dir/rdiff-backup-data/increments/pg.py.2011-11-30T00:28:38+08:00.diff.gz  /local_dir/pg.py

Tips on Some Stat Commands for FreeBSD

FreeBSD provides some handy commands to get various information:  Command netstat shows network status, sockstat lists open sockets, fstat identifies active files, and procstat gets detailed process information. Here are some tips on their usages.

1. Show Internet routing table:

netstat -rn

2. Show all active Internet (IPv4) connections (including servers):
netstat -f inet -a -n
or,
sockstat -4
which gives us more information on command name and process identifier (PID) for each connection.

3. Show all Active TCP connections (including servers):
netstat -f inet -p tcp -a -n
or,
sockstat -4 -P tcp

4. Identify processes using a file or directory:
fstat -v /path/to/the/fileORdirectory

5. Identify all files opened by the specified process:
fstat -v -p PID
or,
procstat -f PID

6. Identify all files opened by the specified user:
fstat -v -u user

7. Get detailed process information on all processes:
procstat -a

8. Get environment variables on the specified process:
procstat -e PID

Some Usages of fuser on Linux Systems

Command fuser allows us to  identify processes using files or sockets. Here are some example usages of fuser.

1. Identify processes using a file or directory:

fuser -v /path/to/the/fileORdirectory

2. Identify processes using a particular TCP/UDP port#:

fuser -v -n tcp port#

where option -n enables us to specify an object in a different name space, tcp port# here.

3. Kill processes that are executing a particular command:

fuser -v -k -i /path/to/the/command

where option -i specifies asking the user for confirmation before killing a process.

4. Kill processes using a particular TCP/UDP port#:

fuser -v -k -i -n tcp port#

How to Mount LVM partitions/disks

Logical volume manager (LVM) is suitable for many occasions, e.g., managing large disk farms, easily re-sizing disk partitions on small systems, and etc. The following quote from wiki LVM page best describes its common uses:

One can think of LVM as a thin software layer on top of the hard disks and partitions, which creates an illusion of continuity and ease-of-use for managing hard-drive replacement, repartitioning, and backup.

Here are steps on how to mount LVM partitions/disks:

1. scan all disks for volume groups:

vgscan


2. scan all disks for logical volumes:

lvscan

The output consists of one line for each logical volume indicating if it is active and its size.

3. change the availability of the logical volume (if it is inactive):


lvchange -a y /dev/vg_name/lv_name

where vg_name is the name of the volume group found by vgscan and lv_name name of the logical volume found by lvscan. You may use vgchange to change the availability of all logical volumes in a specified volume group.

4. mount the logic volume:

mount /dev/vg_name/lv_name /mount/point

where /mount/point is the mount point for the logic volume.

Load Python Module Dynamically

There are times that we need to import some python module and, for whatever reasons, we will not know the name of the module until run-time. This can be achieved by built-in function __import__().

Dynamic loading python module is useful, for example, if we plan to read configuration from a file. Here is an example that allows us to load the module specified in sys.argv[1]:

config_file = sys.argv[1]
config = __import__(config_file)

For newer versions (2.7 and up), there are convenience wrappers for __import__(). We may use module importlib:

import importlib
config_file = sys.argv[1]
config = importlib.import_module(config_file)

Both __import__() and importlib.import_module() search modules in certain locations (e.g., sys.modules, sys.meta_path, sys.path). There are times that modules being loaded are not in those default locations. This can be achieved by using imp.load_source(), which allows us to load and initialize  modules implemented as  Python source files:

import imp
pathname = sys.argv[1]
config = imp.load_source(name, pathname)

where pathname is the path name of the configuration file.

A quick and simple way to wipe a hard drive

For various reasons, it is necessary to wipe hard drives before disposing them. Writing disk with all zeros should suffice most occasions. This can be done as:

dd if=/dev/zero of=/dev/sdXY

where /dev/sdXY is the device name of the hard drive to work on. To get a better performance, we may need to set bs option, e.g.,

dd if=/dev/zero of=/dev/sdb1 bs=10M

to make dd read and write in 10M bytes at a time.

Some Usages of cpio

The command cpio allows us to copy files between archives and directories. There are three operation modes: output (-o), input (-i), and pass-through (-p).

To create an archive for directory tree src, we issue:


find src -print0 | cpio -ov0 > src.cpio


To extract files from its archive just created, we issue:

cpio -ivd < src.cpio


To copy files from directory tree src to directory des, we issue:

find src -print0 | cpio -pmd0 dest


To copy files from src to dest that are less than 2 days old and whose names contain 'txt', we issue:

find src -mtime -2 -print0 | grep txt | cpio -pmd0 dest


To copy files from src to dest that are less than 2 days old and which  contain the word 'txt', we issue:

find src -mtime -2 | xargs grep -l txt | cpio -pmd0 dest




Combining with ssh (or, natcat if you are not concerned with security),  cpio allows us to access remote hosts. Here are some example:

To backup the directory tree src on a remote host, we may issue:

find src -print0 | cpio -oaV0 -H tar -O user@remotehost:src.tar

which uses ssh to copy archive file from localhost to remote host.


To copy files from src to a remote host, we issue:

find src -print0 | cpio -oaV0 -H tar | ssh user@remotehost "cpio -imd"

Access UFS File System under Linux

Unix file system (UFS) is widely used in many Unix systems, for example, FreeBSD, OpenBSD, and HP-UX. There are times that we need to access UFS under Linux systems. The following command allows us to mount UFS2 for read-only (ro) under Linux systems:

mount -t ufs -o ufstype=ufs2,ro /dev/sdXY /mnt/path

Write support for UFS is not compiled into Linux kernels by default. One needs to properly configure and compile kernels for write support.