rsync-dist

NAME

rsync-dist.pl - A Perl script for managing distributions of binary files

SYNOPSIS

rsync-dist.pl -c <config-file> -m -t dist

rsync-dist.pl -c <config-file> -m -L change-links <filemask>...

DESCRIPTION

Features

This script can be used to copy files (usually binary distributions of programs) to a remote server while preserving all the old versions of these distributions. It can also create and change symbolic links that point to certain distributions. Among the programs features are:

Every distribution is placed in a directory named after the current time and date complying with ISO 8601. By this, each directory is unique.
Files that didn't change with respect to the last version are saved as a hard-link, occupying almost no additional disk space.
Files are transferred via rsync and ssh. Rsync only copies files that have changed with respect to the last version. Changes are detected either by comparing file-creation times or a MD5 sum.
During distribution, files can be transferred to several servers in one single command.
A lock file is created before critical actions on the server(s), so several instances of this script can be started at the same same by different people without interfering with each other
Log files on the server are created that show when and by whom things were changed
A local log file in the users home tracks all actions performed that changed something on the server.
A version can be tagged with a single line string. That tag is placed with other information in the log file. Later on, the logfile can be searched for that tag or by a regular expression
Each version has an optional log message that is placed into the log file.
Versions can be moved to an "attic" directory. By this, the main distribution directory can be cleared of obsolete or broken versions.
Symbolic links can be created that point to the version directories. Manipulation of these links is also logged. With this mechanism you can select for an arbitrary number of systems which version they actually use.
Remote host specifications can contain an ssh port number as in HOST:PORT.

Prerequisites

The rsync program has to be installed on the server and the client machine.
The user of the client machine must have an account on the server machine where he can log on via ssh without the need to enter a password or a passphrase (requires proper setup of the ssh-agent).

The program also needs to run rsync on the remote server to get files from your local host. This works directly when you can log from the remote machine back to your local machine with ssh. If your local machine is unknown on the remote server but you can connect from local to remote with ssh, you need an ssh reverse tunnel. rsync-dist.pl use or create reverse tunnels, see further below.

You can test if this works with the following commands, replace REMOTEUSER and REMOTEHOST with the proper values for your setup:
```
ssh REMOTEUSER@REMOTEHOST
```
This must always work without the need to enter a password.

Then, on the remote host enter the following command, replace LOCALUSER and LOCALHOST with the the user name and hostname where you started:
```
ssh LOCALUSER@LOCALHOST
```
If this works, again without the need to enter a password, you have everything you need. If not, you need to use an ssh reverse tunnel. Use option --ssh-reverse-tunnel-mode or set RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE in the configuration file.

A note on passwords: Even if your ssh private key is not encrypted with a passphrase, you need to have an ssh agent set up and running in order to avoid the question for a passphrase or password. See ssh documentation for further details.
perl (of course...) must be installed on the client machine
The server must run a unix/linux operating system since this program needs at least a standard bourne-shell
There must be a directory on the server where the remote user has write access.

Using a configuration file

Certain values like the name of the remote host or the directory on the remote-host can be stored in a configuration file. The program can then read that configuration file in order to obtain these values.

Variables for the configuration file are shown when you run

rsync-dist.pl -h

and look for option "--config".

A configuration file is easy to create. Example:

rsync-dist.pl -H MYHOST -u MYUSER \
    -p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
    --prefix-distdir 1 --checksum show-config

This command prints the contents of such a file to the screen where all settings match the ones given as command line parameters. Here is how you create that file directly:

rsync-dist.pl -H MYHOST -u MYUSER \
    -p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
    --prefix-distdir 1 --checksum write-config CONFIGFILE

You can easily extend this file. Here is for example, how you add the "world-readable" option:

rsync-dist.pl --config CONFIGFILE --world-readable write-config CONFIGFILE

Later on, you can start a distribution command like this:

rsync-dist.pl --config CONFIGFILE dist

or like this

rsync-dist.pl -c CONFIGFILE dist

The program takes all parameter from "CONFIGFILE". Parameters it needs but that are still not specified (like the local path) are requested from the user interactively by starting an editor (default: vi).

Distributing files

In order to distribute files the user must at least specify the remote-server-name, the remote-dist-path and the local-path. The local and remote-dist-path are relative to your user-home unless they start with a '/'.

For day to day work, the server-name and paths will be specified in a configuration file that you created as it is described in the previous chapter. Here is how you distribution command then looks:

rsync-dist.pl -c CONFIGFILE -t TAG -m LOGMESSAGE dist

Additionally you can specify your account-name on the remote machine (especially when this is different from your account on the local machine). This can be done with the "-u" option or by the config-file.

Example:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -t TAG dist

If you prefer not to have a tag or log-message the command looks like this:

rsync-dist.pl -c CONFIGFILE -m -t dist

Inspecting a remote directory with distributed files

The remote directly can be listed with this command:

rsync-dist.pl -c <config-file> ls dist

Note that since the command "ls" accepts an argument, "dist" is an argument here, not a command.

The word "dist" after "ls" means that the distribution directory is listed. In order to see the link directory, you have to specify "links" after "ls".

On the remote directory you see the following files and directories:

version directories

These directories are named like: "2019-10-04T13:57:35" or "2019-10-04T14:03:12". This is an ISO 8601 format for timestamps. The numbers within the string are (in this order) the year, month, day, hour, minute, seconds. The letter "T" separates the date from the time. Your distributed files are placed within these directories. Files that are equal among several directories are stored with hard-links in order to preserve disk space.

the file LAST

This file contains the name of the most recently created version directory. This file is always present after you call this script for the first time. It is modified each time you distribute files to the server. When a new version is distributed, the directory mentioned in "LAST" is copied as a hard-link copy, then rsync is called to transmit all changed/added or removed files.

branches

When your development takes place on several branches at the same time, the hard-link copy of the last distributed version may be inefficient with respect to the space it occupies on the hard disk. For these cases, branches can be used. A branch simply means that another "LAST" file is used in order to create the hard-link copy. A branch can be created from trunk or another branch with the create-branch command. It can be used together with the dist command with the "--branch" option. See also the online help when you execute "rsync-dist.pl -h".

the file CHANGES-DIST

This file contains the changes of the most recent version compared to the previous version. All files that were changed or added are listed. Note that files that were deleted ARE NOT listed. This is an example of the contents of this file:

VERSION: 2019-10-05T11:39:06
CHANGED/ADDED FILES relative to 2019-10-05T10:10:06:
2019-10-05T11:39:06/local/idcp/C

VERSION: 2019-10-05T12:22:16
CHANGED/ADDED FILES relative to 2019-10-05T11:39:06:
2019-10-05T11:39:06/local/idcp/B
2019-10-05T11:39:06/local/idcp/A

The first line of each block is the name of the directory which is also the date and time when it was created

the file LOG-DIST

This is the log-file that is appended each time new files are distributed. It contains the information when and by whom the distribution was done. It also contains a log-message and, optionally, a tag for each version. This is an example of the log-file:

VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
%%
VERSION: 2019-10-05T11:39:06
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test

The first line in each block is the name of the directory which also shows the date and time when the action took place. After "ACTION:" you find the action that took place, "added" or "moved to attic" or "moved from attic". The part after "FROM:" is the person who started the command together with the name of the client-host. After "TAG:" you find the optional tag, after "LOG:" the optional log message with may span several lines. A "%%" seperates entries in the log-file.

Listing the CHANGES-DIST file

The file CHANGES-DIST, as it is described above, can be listed with this command:

rsync-dist.pl -c <config-file> cat-changes

Listing the LOG-DIST file

The file LOG-DIST, as it is described above, can be listed with this command:

rsync-dist.pl -c <config-file> cat-log dist

The word "dist" after "cat-log" means that logfile of the distribution directory is listed.

Searching for tags

The log-file can be searched for a given tag or a (grep compatible) regular expression that matches one or more tags. Example:

rsync-dist.pl -c <config-file> ls-tag <tag or regexp>

A real-world example:

rsync-dist.pl -c <config-file> ls_tag TAG

The output of this command looks like this:

VERSION: 2019-10-05T11:27:51
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
--
VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test

In this case, the tag (seen as a regular expression) matched two existing tags "TAG1" and "TAG2". Both entries from the logfile are listed separated by "--". Note that there is only the first line shown of each log-message.

Managing symbolic links

Symbolic links can be used to connect a specific version with a fixed path on the server.

This feature can for example be used to select startup-scrips of EPICS IOCs. The startup-script determines which version of the EPICS application the IOC boots.

The path to the startup-script configured on the IOC can stay the same and rsync-dist is used to select the actual version of the EPICS application that is loaded.

The log-file in the source directory

Note that there is a file created in the DISTDIR directory that contains a list of all links that point to a certain version. The filename of that file contains the name of the LINKDIR with slashes replaced by dashes like in this example:

LINKS-opt-IOC-Releases-idcp-links

This file contains information on symbolic links in the directory "opt/IOC/Releases/idcp/links". Here is an example of the contents of this file:

2019-10-09T10:28:13 idcp12
2019-10-09T10:28:13 idcp15
2019-10-09T10:28:13 idcp3
2019-10-09T10:28:13 idcp7
2019-10-09T10:28:13 idcp9
2019-10-09T10:33:09 idcp13

The first line means, that in directory "opt/IOC/Releases/idcp/links" there is a symbolic link named "idcp12" that references the distribution-directory "2019-10-09T10:28:13".

Adding symbolic links

Any new symbolic link you intend to use has to be created first. You do this with the "add-links" command like this:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
              add-links VERSION,LINK1,LINK2...

Note that if at least one the given links already exists, the command fails. Also note , that the directory VERSION must exist. The log-file is updated when this command is run.

Changing symbolic links

Symbolic links are changed like this:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE 
              change-links VERSION,LINK1,LINK2...

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE 
              change-links VERSION,FILEMASK...

Note that (especially in case one with all links explicitly listed) all of the given link-names must exist and must be symbolic links. The log-file is updated when this command is run.

If you just distributed a version you can use the "-L" option to specify the last distributed version. In this case, you can omit the VERSION parameter. Example:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -L
              change-links FILEMASK...

Inspecting the remote link-directory

The remote directly can be listed with this command:

rsync-dist.pl -c CONFIGFILE ls links

Note that since the command "ls" accepts an argument, "links" is an argument here, not a command.

The "links" after "ls" means that the link directory is listed.

the file LOG-LINKS

This is the log file where lines are appended each time new links are added or existing links are changed. It contains the information when and by whom the change was done. It also contains an optional log-message for each change and a directory listing of the links before and after the change.

This is an example of the log-file:

DATE: 2019-10-05T14:09:07
FROM: pfeiffer@tarn.acc.bessy.de
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc1 -> ../2019-10-05T13:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc2 -> ../2019-10-05T13:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44

DATE: 2019-10-05T14:11:58
FROM: pfeiffer@tarn.acc.bessy.de
LOG: changed nothing
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc2 -> ../2019-10-05T14:02:44

The first line in each block is the date and time when the action took place. The part after "FROM:" is the person who started the command together with the name of the client-host. After "LOG:" is the optional log message with may span several lines.

Listing the LOG-LINKS file

The file LOG-LINKS, as it is described above, can be listed with this command:

rsync-dist.pl -c CONFIGFILE cat-log links

The word "links" after "cat-log" means that the logfile of the link directory is listed.

QUICK REFERENCE

distribute a version

without any log messages

rsync-dist.pl -c CONFIGFILE -m -t dist

with log message and tag

rsync-dist.pl -c CONFIGFILE -m "my message" -t "my tag" dist

enter log-message and tag interactively

rsync-dist.pl -c CONFIGFILE dist

change a link

Note that FILEMASK may be a name of a single symbolic link like "idcp11" or a file glob pattern like "idcp*".

link to the last dist-version without log message

rsync-dist.pl -c CONFIGFILE -m -L change-links FILEMASK...

link to the last dist-version with log message

rsync-dist.pl -c CONFIGFILE -m "my message" -L change-links FILEMASK...

link to the last dist-version and enter log-message interactively

rsync-dist.pl -c CONFIGFILE -L change-links FILEMASK...

link to a specific version without log message

rsync-dist.pl -c CONFIGFILE -m change-links VERSION,FILEMASK...

VERSION is usually a date-string like "2019-04-02T11:23:14"

inspect remote distribution directory

list directory

rsync-dist.pl -c CONFIGFILE ls dist

show last entries in log-file

rsync-dist.pl -c CONFIGFILE tail-log dist

show last entries in CHANGES file

rsync-dist.pl -c CONFIGFILE tail-changes dist

inspect remote link directory

list directory

rsync-dist.pl -c CONFIGFILE ls links

show last entries in log-file

rsync-dist.pl -c CONFIGFILE tail-log links

show last entries in CHANGES file

rsync-dist.pl -c CONFIGFILE tail-changes dist

Tips and tricks for the server administration

Logging on the server is required for these examples.

show all distribution directories

for i in 2[0-9][0-9][0-9]*; do echo $i; done

list all existing distribution directories mentioned in LOG-DIST

for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
      sed -e 's/VERSION: //'

list directories mentioned in LOG-DIST that do no longer exist

for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
      sed -e 's/VERSION: //' > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
      diff - TMP

compare existing directories with the ones mentioned in LOG-DIST

for i in 2[0-9][0-9][0-9]*; do echo $i; done > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
      diff - TMP

list directories used in LOG-LINKS

grep -- '->' LOG-LINKS | \
      sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|sort|uniq

compare directories used in LOG-LINKS with existing ones

(cd ../dist && for i in 2[0-9][0-9][0-9]*; \
      do echo $i; done) > TMP
grep -- '->' LOG-LINKS | \
      sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|\
      sort|uniq|diff - ../dist/TMP  | grep '<'

AUTHOR

Goetz Pfeiffer <goetzpf@googlemail.com>

LICENSE

GNU GENERAL PUBLIC LICENSE Version 3