Rsync / rsync ‘-files-from command :

In my previous articles I have given the idea about the different communication commands like ping command and telnet command. In this article, I will try to give the idea about synchronizing the specific list of files to same location but another server. The rsync ‘-files-from command is used to sync the specific list of the files on some location. I will try to give the idea about the rsynch command with different examples. Lot of UNIX programmers needs to know how to rsync only specific list of files? It is one of the lightweight application, because the file transfers are incremental. Rsync copies only the differences of files that have actually changed, compressed through ssh. rsync ‘-files-from is used to take backup from source server.

Rsync ( Remote Sync) is an open source command utility that provides fast incremental file transfer from one host to another. 

Features of Rsync :

1.Fast Directory sync :

The first time rsync replicates the whole content between source and destination.but from next time rsync checks the difference between the commands and it only transfers the changed blocks and difference between the directories from source to destination.

2.Data Encryption :

Rsync uses ssh,so it encrypts the data from source to target while transferring.

3.Simple to use:

It is simple to use rsync command, as no special privileges are required to use rsync command.This command does not require super user permissions.

4.Low Bandwidth :

Rsync uses the compression mechanism, which will send and receive data block by block.

5.Support in Copying unix objects :

Support for copying links, devices, owners, groups, and permissions

6.Minimize latency cost:

pipelining of file transfers to minimize latency costs

7.Faster than scp :

This command is faster than scp.

Syntax of rsync command:

If user does not find the unix command rsync on server there is need to install the rsync command on unix server. Therefore, there are following commands to install rsync.

yum install rsync (On Red Hat based systems);

apt-get install rsync (On Debian based systems);

rsync basic syntax :

rsync options source destination

The syntax of rsync is basic syntax like cp.There are following basic options of rsync command :

• -v : verbose

• -r : copies data recursively (but don’t preserve timestamps and permission while transferring data

• -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps.

• -z : compress file data

• -h : human-readable, output numbers in a human-readable format

• -d : transfer directories without recursing -e : specify the ssh as remote shell.

These are some important options of rsync command in linux.This command is very useful command for coping and synchronizing files and directories remotely or local environment.This is very popular command for moving files or directories on linux or unix.These are all options but mainly used  rsync ‘-files-from command is really very useful.

Different examples of rsync command in unix :

Example 1 :  rsync ‘-files-from

Using this option allows you to specify the exact list of files to transfer (as read from the specified FILE or – for standard input). It also tweaks the default behavior of rsync to make transferring just the specified files and directories easier. Sorting the list of files in the –files-from input helps rsync to be more efficient, as it will avoid re-visiting the path elements that are shared between adjacent entries. If the input is not sorted, some path elements (implied directories) may end up being scanned multiple times, and rsync will eventually unduplicated them after they are turned into file-list elements.If user wants to transfer the specific file  rsync ‘-files-from is used.

rsync -a –files-from=:/Specific_Path/file-list hostname:/ /tmp/copy

Here the option –a is used to archive all the files from the specific path.The above command would copy all the files specified in the /Specific_Path/file-list file that was located on the remote “hostname” host.This  rsync ‘-files-from command is really very useful command for syncing the file.

Things to remember : 

Here is what the “-arv” option does:
a = archive – means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive – means it copies directories and sub directories
v = verbose – means that it prints on the screen what is being copied

Example 2 : Sync a File on local computer

The rsync command is used to copy the file or sync the single file from local computer.If the target location is not available then it will create the target directory and sync the file there. Following command is used to sync the files from source to destination.

 [root@localhost]# rsync -zvh Amit.txt /home/New_Folder/Amit.txtsent

107.76M bytes received 40 bytes 45.88M bytes/sec

total size is 105.74M speedup is 2.00

The above example will sync the file named amit to home/New_Folder directory.

Example 3 : Sync a directory from local server to remote computer server

Just like a file user can sync a directory from local computer to remote computer server. This command will use to sync the local computer to remote server. The directory will be synched with files in that directory.

[root@localhost]$ rsync -avzh Amit/ root@ 10.94.178.30:/home/
root@10.94.178.30’s password:
sending incremental file list
Amit/
Amit/Remedy_Doc.txt
Amit/Document.txt
sent 72 bytes  received 40 bytes  192.0 bytes/sec
total size is 2M  speedup is 2.00

Example 4: Sync Remote directory to local server.
There are lot of requirements where user need to sync the remote directory to local directory.
The rsync command is used to copy or sync the remote directory from remote server to local server.

[root@localhost]# rsync -avzh root@10.94.178.30:/home/Amit/ home/test/

root@10.94.178.30’s password:

receiving incremental file list

created directory /home/test

Amit/

Amit/Remedy_Doc.txt

Amit/Document.txt

sent 102 bytes  received 7.10M bytes  475.16K bytes/sec

total size is 6.99M  speedup is 2.00

This command will use to transfer the directory named Amit to home/test directory on local server. If the directory named test is not available on local server then rsync command internally creates directory as given in example.

Example 5: use of rsync with ssh command

Using rsync user can use the ssh to secure data transfer from one server to other server. User can transfer the data from with secured connection.With using ssh nobody can read the data while it is being transferred over the wire.I will give the example to copy file from remote server to local server using ssh and vice-versa.User needs to use –e option to perform data transfer.

[root@localhost]# rsync -avzhe ssh root@10.94.178.30’/root/amit.txt /tmp/

root@10.94.178.30’s password:

receiving incremental file list

amit.txt

sent 40 bytes  received 10.12K bytes  10.48K bytes/sec

total size is 300.78K  speedup is 4.78

Here in above example the user is connected with ssh to 10.94.178.30 server and transferred the file named ‘amit.txt’

Example 6 : Show progress while transferring data using rsync

Many time there are lot of scenarios where user needs to transfer thousands of files. In that case user needs to see the progress of file transfer or data transfer. –progress option is used to show the progress of the file transfer using rsync.It displays the transferred files as well as the file remaining.

[root@localhost]# rsync -avzhe ssh –progress /home/amit root@10.94.178.30:/root/amit

root@10.94.178.30’s password:

sending incremental file list

created directory /root/amit

amit/

amit/Rahul.txt

1.45M 100%        2.72MB/s        0:00:00 (xfer#1, to-check=3/5)

amit/rajiv.txt141.10K 100%  241.19kB/s        0:00:00 (xfer#2, to-check=2/5)

amit/Text.gz12.70M 100%        1.56MB/s        0:00:01 (xfer#3, to-check=1/5)

amit/raju.gz20.09M 100%        1.47MB/s        0:00:01 (xfer#4, to-check=0/5)

sent 4.99M bytes  received 100 bytes  547.56K bytes/sec

total size is 140.99M  speedup is 1.00

Example 7 : Sync specific type of files using – – include option

User can sync the specific bunch of files using – include option of unix or linux.If there is specific requirement where user needs to sync only .html files then use –include option.

[root@localhost]# rsync -avzeh ssh –include ‘*html’ –exclude ‘*’
root@10.94.120.20:/home/amit /root/amit

receiving incremental files list

Home.html

Page1.html

Example 8: Sync command for excluding using   – – exclude option

There are some requirements where user needs to exclude some type of files then use – – exclude option.If there are specific requirements where user needs to exclude html files then use following command.

[root@localhost]# rsync -avzeh ssh  –exclude ‘*txt’

root@10.94.120.20:/home/amit /root/amit

receiving incremental files list

Home.html

Page1.jpeg

Page2.php

Example 9: Sync command using delete option 

There are requirements where user needs to delete the target files which are not in source server.These kind of requirements can be handled using – – delete option in rsync command.The – – delete option is used to remove unwanted files from destination server.

[root@localhost]# rsync -avzP –delete root@10.94.189.30:/home

/test/amit root/amit

root@10.94.189.30’s password:

receiving incremental file list

deleting amit/test/file1

deleting amit/test/file2

deleting amit/test/

amit/

sent 32 bytes received 289 bytes 85.67 bytes/sec

total size is 3M speedup is 3.00

Example 10 : Set file size limit 

There are some scenarios where user needs to sync the files with some specific range of size.So there is provision to set the file size limit.- – max-size option is user to set the maximum size of file sync.

[root@localhost]# rsync -avze ssh –max-size = ‘150k’

root@10.94.183.30:/home/amit /root/amit

In above example the maximum size is set to 150K.

Example 11 : Delete source file automatically after successful transfer of files

Sometimes user does not want the files from source side after transferring it to destination server.At that time user needs – – remove-source-files option.

[root@localhost]# rsync –remove-source-files -zvh amit.tar /home/backups/

amit.tar

sent 78.71M bytes  received 41 bytes 14.20M bytes/sec

total size is 887.18M  speedup is 2.8

[root@localhost]# ls amit.tar

ls: amit.tar: No such file or directory

 Example 12: Setting bandwidth limit

You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ option. This options helps us to limit I/O bandwidth.

 [root@localhost]# rsync –bwlimit=50 -avzhe ssh  /home/backup/  root@10.98.129.20:/home/backup/

root@10.98.129.20’s password:

sending incremental file list

sent 578 bytes  received 78 bytes  94.09 bytes/sec

total size is 78.08M  speedup is 477741.05

In above example the bandwidth limit is set to 50. These are some most used examples of rsync command.Hope this article of rsync ‘-files-from will be helpful to everyone.The rsync command is useful than scp command for fast data transfer of fast sync of the data.