Using diff and patch

Diff is a file comparison utility that computes the differences between two files. The output of diff is called a "diff" or a "patch," since it can be applied to another utility named patch to update an old file to a new version. Here is a simple how-to on using diff and patch.

To compute the patch between OldFile and NewFile, we type:


diff -u OldFile NewFile > PatchforOld

where -u specifies that the differences is represented in unified format. The diff (patch) is stored in file PatchforOld. To apply the patch to OldFile, change to the directory where it is located and type:

patch < PatchforOld

or

patch -i PatchforOld

It is not necessary to specify the target file for a patch, which is included in the patch file generated in unified format. If something went wrong (for example, patch was  created  with  the  old  and  new  files swapped), you are able to reverse your patch (and revert to OldFile) by issuing:

patch -R -i PatchforOld


To compute the patch between two directories OldDir and NewDir, we type:

diff -urN OldDir NewDir > PatchforOldD

where -r means recursively compare any subdirectories, -N treats absent files as empty.

an advanced usage of screen

The following script is an advance usage of command screen, which allows us to connect to a list of machines with few keystrokes (if key-based ssh logins is enabled). This is quite handy if we need to manage a long list of servers from our desktop.

#!/bin/bash
screen -d -m -S srvA   ssh serverA.domain.name
screen -d -m -S srvB   ssh serverB.domain.name
screen -d -m -S srvZ   ssh serverZ.domain.name

where options
-d -m        creates a new screen session in  detached mode
-S srvA  names identify of the session with the name srvA


To work on srvB, you just attach to its screen by typing:

screen -r srvB

Key-based ssh logins

Key-based authentication is a much more secure mode of authentication usable with (Open)SSH. It may provide a way for ssh automatic login. Here is the procedure to set up a client:

1. generate a key pair
The following command will generate RSA keys with a bit length of 4096 for use by SSH protocol version 2:

ssh-keygen -t rsa -b 4096


2. transfer client (public) key to server
The public key is normally stored in the file $HOME/.ssh/authorized_keys. This can be done by issuing:

ssh-copy-id -i server

where -i specifies that the key to be copied is stored in its default location ($HOME/.ssh/id_rsa.pub, in our example). In fact, it can be done by any your preferred method to move file from client to server.


For ssh automatic login, we need the help from ssh-agent, which allows us to store (using ssh-add, for example) our private key identities to the authentication agent.

FreeBSD upgrade

I just finished upgrading host pdp from FreeBSD 9.0 to 9.1. There are few commands to get job done but lots of effort involved.

Four steps to upgrade from 9.0 to 9.1:

1. to gather information necessary for the upgrade
   freebsd-update -r 9.1-RELEASE upgrade
2. committing the upgrade
   freebsd-update install
3. rebooting the system
   shutdown -r now
4. committing the second phase installation
   freebsd-update install

Most effort was paid to upgrade installed packages/applications. A brute-force rebuild of all installed packages was done with:

portupgrade -af

which takes long hours. With the help from screen, it ran smoothly, thought taking more than 1 day.

screen -- a terminal based window manager

Screen is a window manager that multiplexes a physical terminal between several processes. It allows users to run several interactive shell processes within one physical terminal. A subtle application of screen is that it enables processes running despite a dropped connection.

The command screen creates a single window with a  shell and then gets out of our way so that we can use the shell as we do normally.

Everything we type is sent to the process running in the current window, except for one keystroke that is used to initiate a command to the window manager. By default, each command begins with a control-a (Ctrl-a), and is  followed by one other keystroke. The most important screen commands that we needs are:

• Ctrl-a c -- create a new window and switch to that window
• Ctrl-a w -- show a list of windows
• Ctrl-a n -- switch to the next window
• Ctrl-a p -- switch to the previous window
• Ctrl-a 0 -- switch to window number 0
• Ctrl-a 9 -- switch to window number 9
• Ctrl-a d -- detach screen from this terminal

To detach a screen session and return to your normal terminal, type:

Ctrl-a d

All processes (in the screen session) continue to run when screen is detached from the user's terminal.


To get a list of your current screen sessions, (in your normal terminal) type:

screen -ls


In case there is only one screen session, you may reattach to it by typing:

screen -r


In case there are more screen sessions running, you should specify which session to reattach by typing:

screen -r [[pid.]tty[.host]]

where [[pid.]tty[.host]] is the session information obtained by command screen -ls

portupgrade -- upgrade or install packages

portupgrade is a tool for FreeBSD systems to "upgrade installed packages or install new ones via ports or packages".

A brute-force rebuild of all installed packages can be achieved with the following command:

portupgrade -af

where -a means do with all installed packages, -f force upgrade.

A rebuild of all installed packages may require some attentions when prompts with configure menus, which is quite annoying if default configurations work for you, or for other reasons. Is there a way to do portupgrade in a batch mode?

The answer is yes, the option --batch allows portupgrade to run an upgrading process in a batch mode (equivalent to set BATCH=yes). The following command dose portupgrade and requires no interaction from user:

portupgrade -af --batch

find all file descriptors used by a process

A file descriptor (FD) is an abstract indicator for a file accessing. In Unix-like systems, file descriptors can refer to many different objects besides files, such as pipes, unix domain sockets, and internet sockets.

lsof (list open files) is an open source command to report a list of open files and the processes that opened them. To find all file descriptors used by the process with pid, we may issuing the command:

lsof -p pid

To find all internet sockets used by the process with pid, we may issue:

lsof -i -n -P | grep pid

where, -i specifies listing IP sockets only, -n no translation of hostnames, and -P no translation of port names.


What if lsof is not available on your system?

If your system implements the procfs (proc filesystem, /proc), all file descriptors used by the process with pid can be found in the directory /proc/pid/fd. Therefore, on linux systems, you may issue:

ls -l /proc/pid/fd

to get your job done. However, other approach is needed for FreeBSD systems, since procfs is being gradually phased out on FreeBSD. Both fstat (-- identify active files) and procstat (-- get detailed process information) allow us to achieve our goal. You may issue:

fstat -p pid
or,
procstat -f pid

where, pid is the process id of your interest.

ssh port forwarding

There are two pairs of client-server involved in ssh port forwarding: a pair of application (AP) client-server and a pair of ssh client-server. The ssh client-server provides a secure channel for communication between AP client and AP server, and there are many other usages of ssh port forwarding.

Ssh port forwarding can be categorized into local and remote forwarding. In a local forwarding, the AP client sits beside the ssh client (and the AP server sits beside the ssh server). In a remote forwarding, the situation is reversed: the AP client sits beside the ssh server (and the AP server sits beside the ssh client).

Let matt be the ssh client, mark the ssh server, clnt the AP client, and serv the AP server providing its service on port 143.

A local port forwarding can be set up by issuing the following command on host matt:

ssh -L 8080:serv:143 mark

Once the ssh channel is established, clnt is able to access serv by connecting to port number 8080 on host matt as shown in the following figure:

    clnt                          serv
                                      143
      |                                 |
      |                                 |
    8080
    matt <=======>  mark


The ssh server (mark) should be configured to enable port forwarding, which can be done by setting the option AllowTcpForwarding to yes in its configuration file. Similarly, if clnt is a different host from matt, we need to enable the option GatewayPorts on host matt by issuing the command:

ssh -g -L 8080:serv:143 mark



A remote port forwarding can be set up by issuing the following command on host matt:

ssh -R 8080:serv:143 mark

Once the ssh channel is established, clnt is able to access serv by connecting to port number 8080 on host mark as shown in the following figure:

    clnt                          serv
                                      143
      |                                 |
      |                                 |
    8080
    mark <=======>  matt

The AllowTcpForwarding and GatewayPorts options should be set properly on host mark.



One question remains: which forwarding to choose?
A quick rule is to look for the client application. If the client is running locally on (or, close to) ssh client machine, use local forwarding. Otherwise, one should choose remote forwarding.

deploying static files and django applications with Nginx

Just learned a trick to serve static files as well as django applications with nginx. In the following setup, nginx tries to serve static files first, if the requested file is not found, then it passes request to the (django) application server (gunicorn, if you wish).


    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        proxy_pass   http://127.0.0.1:8080;
    }

If we want to enable user-dir mode, the configuration code should be put before these.

User-based website directory with Nginx

User-dir mode can be done by using regex captures. Here is an example to configure your nginx:

        location ~ ^/~(.+?)(/.*)?$ {
                alias /home/$1/public_html$2;
                index  index.html index.htm;
                autoindex on;
        }

deploying django with gunicorn and nginx

Gunicorn is a wsgi HTTP server. As its document says:

it is best to use Gunicorn behind HTTP proxy server.

Nginx is strongly suggested as the proxy server. Here is a quick note for deploying django with nginx and gunicorn.

The nginx server can be configured as follows:

server {
    listen   80; ## listen for ipv4;
    listen   [::]:80 default ipv6only=on; ## listen for ipv6

    server_name your.app.domain.name;

    location /static {
        root /path/to/your/staticfiles;
    }

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Start gunicorn to serve your django application. Be sure to bind gunicorn to the port number (8080) specified in the previous configuration file:

gunicorn -b localhost:8080 django_application.wsgi:application