Netcat for File Transfer

File transfer is one of the most practical usages of netcat.

To transfer a file named filename from client to server, we first issue the following command on server:

nc -l -p 1234 > filename

where 1234 is the port number used by the server. On the client side, we issue:

nc -q 10 server 1234 < filename

where '-q 10' specifies to wait 10 seconds and then quit after EOF on stdin. This would cause the server to quit.

To transfer a director tree named /path from client to server, we issue the following command on server:

nc -l -p 1234 | tar xvzf -

On the client side, we issue:

tar cvzf - /path | nc -q 10 server 1234



If we want to reverse the direction of file transfer, i.e., client pulls file from server, we use:

nc -q 10 -l -p 1234 < filename

on server, and

nc server 1234 > filename

on client. Similarly, to reverse the direction of directory tree transfer, we use:

tar cvzf - /path | nc -q 10 -l -p 1234

on server, and

nc server 1234 | tar xvzf -

on client.

Using findutils to delete files

Here is a summary from the deleting files page.

The most efficient and secure method to delete any file with name ending in '~' in the directory /path is:

find /path -name \*~ -delete

Using command xargs may allow us to achieve same efficiency but is not as secure:

find /path -name \*~ -print0 | xargs -0 /bin/rm

where '-print0' specifies using ASCII NUL to separate the entries in the file list, and similarly '-0' for command xargs.

If the '-delete' action is not available, we may use action '-execdir' or '-exec':

find /path -name \*~ -execdir /bin/rm {} \+

find /path -name \*~ -exec /bin/rm {} \+

Action '-execdir' is secure but less portable. On the other hand, action '-exec' is most efficient portable but insecure. These two actions can be used for doing things other than deleting files.

Use command find to clean up your file system

Some editors create back-up files with the the same file name and a '~' suffix. Once you are done with editing, there is no need to keep these back-up files. The command find allows us to find and remove all such back-up files in working directory:

find . -name \*~ -exec rm {} \;

If you want to remove files not been accessed for more than one year (365 days), you may issue:

find . -atime +365  -exec rm {} \;

Command find comes with many other options, for example, you may use: 'mtime' for last modified, 'ctime' for last change, or even 'inum' for inode number.

Using public keys for SSH authentication

SSH supports many methods for user authentication. Public key, password, and host-based are three main methods specified in RFC 4252 (The Secure Shell (SSH) Authentication Protocol). With the public key method, the possession of a user's private key serves as authentication. Host-based authentication works similarly: the possession of host's (client's) private key enables the authentication based on  the user names on the server and the client.

SSH authentication using public keys can be achieved in two steps:

1. create a public and private key pair on client side, and
2. copy your public key to server.

Once these are done, we should be able to login remotely  without being prompted for a password.

The command ssh-keygen allows us to generate and manage authentication keys. To generate a pair of RSA keys, we use:
  
ssh-keygen -t rsa

It generates a pair of RSA keys and saves them in directory $HOME/.ssh; the default name for public (private) key is id_rsa.pub (id_rsa). Always use passphrases to protect your private keys.

Public keys are not sensitive data in general, we may choose any method to copy them. However, command ssh-copy-id provides an easy way to accomplish this. You use:

ssh-copy-id user@server

to copy your public key to server. After that, your public key file will be copied/appended to file $HOME/.ssh/authorized_keys in server. Don't forget to specify (option -i) the path of your public key file if it is not in $HOME/.ssh/id_rsa.pub on client side.

An ssh-agent is very helpful in using public keys for SSH authentication; it is strongly recommended.

Hard and Symbolic links

A hard link is an entry in a directory file that associates a name with an (existing) file on a file system, which allows a file to appear in multiple paths.

Unix/Linux systems do not allow hard links on directories, since it may create endless cycles. Hard links are limited to files on the same volume, because name and file association in each hard link is through inode. Most file systems that support hard links use link count to keep track on the total number of links created to point to the inode (file). To find all the files which refer to the same file as NAME, we may use command find with the option '-samefile NAME' or '-inum INODE', where INODE is the inode number of NAME. The command ls with option '-il' gives you information on link count and inode for files.


A symbolic link is a special type of file that contains a text string which is interpreted by the operating system as a path to another file/directory. The other file/directory is usually called the "target". A symbolic link is another file that exists independently of its target, i.e., they are two files/directories indexed by two different inodes, as opposed to hard links. Symbolic links are different from hard links in that:

• a symbolic link may point to a directory, and
• a symbolic link may point to a directory/file in different volume

There is one issue with symbolic links. If a symbolic link is removed, its target remains unaffected. However, there is no automatic update for a symbolic link if its target is moved, renamed, or deleted. The symbolic link continues to exist and point to the original target, which no longer exists. This is called a broken link.

Inode

An inode is a data structure in Unix/Linux file systems that stores information (metadata) about a file system object, except file content and file name. The command stat allows us to retrieve most information stored in an inode. Here is an example of system file bash on a Linux box:

  File: `/usr/bin/bash'
  Size: 902036        Blocks: 1768       IO Block: 4096   regular file
Device: 808h/2056d    Inode: 279026      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:shell_exec_t:s0
Access: 2013-05-01 13:59:45.680608666 +0800
Modify: 2013-01-31 22:47:10.000000000 +0800
Change: 2013-03-21 10:29:27.135450340 +0800
 Birth: -

We can see that regular files have the following attributes:

• Size in bytes
• Device ID where the file is stored
• Inode number
• Link count
• User ID of the file
• Group ID of the file
• Permissions (access rights) of the file
• Timestamps on last access (atime), modify (mtime), and change (ctime)

To get the file content, we need to consult to the inode pointer structure that is also part of information stored in inode. The inode pointer structrure consists of four different pointers:

• direct pointer,
• singly indirect pointer,
• doubly indirect pointer, and
• triply indirect pointer

These pointers point directly or indirectly to block locations where pieces of file content are stored.


One question remains: where is the file name stored? The answer is that it is stored in the content of the directory that contains the file. Unix/Linux directories are lists of association structures, each of which consists of one file name and one inode number for that file. That is why we need to specify (implicitly or explicitly) the path whenever we want to access a file in the file systems.

Create Multiple User Accounts on Linux Systems

adduser is a system command on FreeBSD that allows root to create new user accounts in interactive or batch modes.

useradd is a system command on Linux systems that enable root to create a new user account; to create multiple user accounts on Linux systems in batch mode, we may use the newusers system command.

Each input line for newusers is in the same format as the standard password file with some exceptions. To use command newusers, we need to specify the password for user or leave it blank as for null password. There are times that we need to create multiple user accounts with random passwords. The following bash script allows us to accomplish this with few more options:

#!/bin/bash
function mkpw()
{
    head /dev/urandom | uuencode -m - | sed -n 2p | cut -c1-10;
}

SAVEFILE=newAccounts
chmod 0400 $SAVEFILE
IFS=':'
while  read USER PASSWORD uUID GID FULLNAME HOMEDIR SHELL
do
    PASSWORD=${PASSWORD:-`mkpw`}
    GID=${GID:-"users"}
    SHELL=${SHELL:-"/bin/bash"}
    echo "Adding user '$USER, $FULLNAME'"
    useradd -m $USER -p `mkpasswd -p $PASSWORD` \
        -c "$FULLNAME" -s $SHELL -g $GID
    echo "$USER:$PASSWORD" >> $SAVEFILE
done

where mkpasswd is the command that encrypts the given password. The input file for this script is in the same format as for command newusers. The output file, named newAccounts, contains all newly created account ids and passwords in plain text; you need to take proper action on it.