rsync-dist.pl - A Perl script for managing distributions of binary files
rsync-dist.pl -c <config-file> -m -t dist
rsync-dist.pl -c <config-file> -m -L change-links IOC*
This script can be used to copy files (usually binary distributions of programs) to a new created VERSION directory on a remote server. Old VERSION directories are never deleted. The active VERSION is selected by a symbolic LINK that is created and managed by the program. Programs features are:
Every distribution is placed in a VERSION directory named after the current time and date. The default for this name is of the form YYYY-MM-DDThh:mm:ss, complying with ISO 8601. With option "--short-date" or RSYNC_DIST_SHORT_DATE from the configuration file, the form YYYY-MM-DDThhmmss is used, which is not strictly ISO 8601 compliant. In any case, the name of the created directory is unique.
Files that didn't change with respect to the last version are saved as a filesystem hard-link, occupying almost no additional disk space.
Files are transferred via rsync and ssh. rsync only copies files that have changed with respect to the last version. Changes are detected by comparing the file's checksum (via -c/--checksum option to rsync).
With the distribute command, files can be transferred to several servers with one single command.
A lock file is created before critical actions on the server(s) are run. So several instances of this script can be started at the same time without interfering with each other.
Log files on the server are created that show when and by whom things were changed.
A local log file in the user's home directory tracks all actions performed that changed something on the server.
A version can be tagged with a single line string, a TAG. That TAG is placed with other information in the log file. Later on, the logfile can be searched for that TAG by a fixed string or by a regular expression
Each version has an optional log message that is placed into the log file at the server.
VERSIONS can be moved to an "attic" directory. By this, the main distribution directory can be cleared of obsolete or broken VERSIONS.
Symbolic links can be created that point to the VERSION directories. Manipulation of these links is documented in a log file. With this mechanism you can select for an arbitrary number of systems which VERSION actually use.
Remote host specifications can contain an ssh port number as in HOST:PORT.
The rsync program has to be installed on the server and the client machine.
The user of the client machine must have an account on the server machine where he can log on via ssh without the need to enter a password or a passphrase (requires proper setup of the ssh-agent).
The program also needs to run rsync on the remote server to get files from your local host. This works directly when you can log from the remote machine back to your local machine with ssh. If your local machine is unknown on the remote server but you can connect from local to remote with ssh, you need an ssh reverse tunnel. rsync-dist.pl can use or create ssh reverse tunnels, for details see further below.
Here is a short test that shows if your ssh setup can be used. Replace REMOTEUSER and REMOTEHOST with the proper values for your setup:
ssh REMOTEUSER@REMOTEHOST
This must work without the need to interactively enter a password.
Then, on the remote host enter the following command, replace LOCALUSER and LOCALHOST with the the user name and hostname where you started:
ssh LOCALUSER@LOCALHOST
If this works, again without the need to enter a password, you have everything you need. If not, you need to use an ssh reverse tunnel. Use option --ssh-reverse-tunnel-mode or set RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE in the configuration file.
A note on passwords: Even if your ssh private key is not encrypted with a passphrase, you need to have an ssh agent set up and running in order to avoid the question for a passphrase again and again. If you don't the the concept of ssh agents, look this up in ssh documentation.
Perl (of course...) must be installed on the client machine.
The server must run a unix/linux operating system since this program needs at least a standard bourne-shell
There must be a directory on the server where the remote user has write access.
Certain values like the name of the remote host or the directory on the remote-host can be stored in a configuration file. The program can then read that configuration file in order to obtain these values.
Variables for the configuration file are shown when you run:
rsync-dist.pl -h
and look for option "--config".
A configuration file is easy to create. Example:
rsync-dist.pl -H MYHOST -u MYUSER \
-p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
--prefix-distdir 1 show-config
This command prints the contents of the config file to the screen where all settings match the ones given as command line parameters. Here is how you create that file directly:
rsync-dist.pl -H MYHOST -u MYUSER \
-p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
--prefix-distdir 1 write-config CONFIGFILE
You can easily extend this file. Here is for example, how you add the "world-readable" option:
rsync-dist.pl --config CONFIGFILE --world-readable write-config CONFIGFILE
Later on, you can start a distribution command like this:
rsync-dist.pl --config CONFIGFILE dist
or like this
rsync-dist.pl -c CONFIGFILE dist
The program takes all parameters from "CONFIGFILE". Parameters it needs but are not specified (like the local path) are requested from the user interactively by starting an editor (default: vi).
The following variables can be defined in a configuration file
Name of the distribution host or hosts. More than one host can be specified as a comma-separated list. Associated command line option: --hosts.
The username(s) rsync-dist.pl uses to log on to the remote hosts. See also --user. When several hosts (see above) are defined, you can define several (possibly different) users for the hosts. Associated command line option: --users.
Specify the local ssh command that is used. Associated command line option: --ssh-local.
Specify the remote ssh command that is used. Associated command line option: --ssh-remote.
If not empty, use a ssh reverse tunnel to connect from the server back to the local host. Two values are allowed for this variable are: 'exists' : use an *existing* reverse tunnel. 'create' : create a reverse tunnel for the operation and remove it when the operation is finished Associated command line option: --ssh-reverse-tunnel-mode
When reverse tunnel mode is set (see RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE or --ssh-reverse-tunnel-mode) use this port for the reverse tunnel on the remote server. The must be *no other* application using this port on the remote server. The default for this port is 53591 (0xd157). Associated command line option: --ssh-reverse-tunnel-port.
The remote distribution directory. Associated command line option: --distpath.
The remote link directory. Associated command line option: --linkpath.
If empty or '1', prepend the distribution directory to the link-source path. If this is a non-empty string prepend this to the link source path. This effects only the connamds 'add-links' and 'change-links'. Associated command line option: --prefix-distdir.
The local directory or directories. More than one directory can be specified in a comma-separated list. Associated command line option: --localpath.
The prefix that is prepended to local paths (see above). Associated command line option: --localprefix
When set to 1, make all distributed files world-readable. Associated command line option: --world-readable.
When set to 1, do not cross filesystem boundaries on the local host. Associated command line option: --one-filesystem
When set to 1, symbolic links are never dereferenced and always copied as links. Associated command line option: --preserve-links.
When set to 1, symbolic links always dereferenced. Associated command line option: --dereference-links.
Specify a file on the local host that contains files that are to be excluded. Associated command line option: --exclude-list.
Defunct option, retained for compatibility.
Use shortened date for dist directories. Associated command line option: --short-date.
When this variable is defined, use this branch for distributing files. Associated command line option: --branch.
When set to 1, make a complete copy of the last version on the server but distribute only some files from the client to the server. --localpath may contain file-glob patterns in this case. Associated command line option: --partial.
The default editor to enter options that the used didn't specify on the command lines. The default is "vi". Associated command line option: --editor.
When set to 1, the editor does not show defaults, when called. Associated command line option: --no-editor-defaults.
When set to 1, show progress of rsync on console. Associated command line option: --progress.
Specify a default-tag, may also be empty. In this case the program does not request a tag if no one is given on the command line. Associated command line option: --tag.
Specify a default log-message, may also be empty. In this case the program does not request a message if no one is given on the command line. Associated command line option: --message.
When set to 1, output from remote ssh commands is filtered to make it more human readable. This affects the commands "change-links", "add-links", and "ls links". Associated command line option: --filter-output.
An optional command to be run after dist on the target host. Associated command line option: --postproc .
An optional command to be run after change-links on the target host. Associated command line option: --link-postproc.
In order to distribute files the user must at least specify the remote-server-name, the remote-dist-path and the local-path. The local and remote-dist-path are relative to your user-home unless they start with a '/'.
For day to day work, the server-name and paths will be specified in a configuration file that you created as it is described in the previous chapter. Here is how you distribution command then looks:
rsync-dist.pl -c CONFIGFILE -t TAG -m LOGMESSAGE dist
Additionally you can specify your account-name on the remote machine (especially when this is different from your account on the local machine). This can be done with the "-u" option or by the config-file.
Example:
rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -t TAG dist
If you prefer not to have a tag or log-message the command looks like this:
rsync-dist.pl -c CONFIGFILE -m -t dist
The remote directly can be listed with this command:
rsync-dist.pl -c <config-file> ls dist
Note that since the command "ls" accepts an argument, "dist" is an argument here, not a command.
The word "dist" after "ls" means that the distribution directory is listed. In order to see the link directory, you have to specify "links" after "ls".
On the remote directory you see the following files and directories:
These directories are named like: "2019-10-04T13:57:35" or "2019-10-04T140312". The first one is an ISO 8601 format for timestamps, the second one is used when option --short-date or variable RSYNC_DIST_SHORT_DATE are given. The numbers within the string are (in this order) the year, month, day, hour, minute, seconds. The letter "T" separates the date from the time. Your distributed files are placed within these directories. Files that are equal among several directories are stored with hard-links in order to preserve disk space.
This file contains the name of the most recently created VERSION directory. This file is always present after you call this script for the first time. It is modified each time you distribute files to the server. When a new version is distributed, the directory mentioned in "LAST" is copied as a hard-link copy, then rsync is called to transmit all changed/added or removed files.
When your development takes place on several branches at the same time, the hard-link copy of the last distributed version may be inefficient with respect to the space it occupies on the hard disk. For these cases, branches can be used. A branch simply means that another "LAST" file is used in order to create the hard-link copy. A branch can be created from trunk or another branch with the create-branch command. It can be used together with the dist command with the "--branch" option. See also the online help when you execute "rsync-dist.pl -h".
This file contains the changes of the most recent version compared to the previous version. All files that were changed or added are listed. Note that files that were deleted ARE NOT listed. This is an example of the contents of this file:
VERSION: 2019-10-05T11:39:06
CHANGED/ADDED FILES relative to 2019-10-05T10:10:06:
2019-10-05T11:39:06/local/idcp/C
VERSION: 2019-10-05T12:22:16
CHANGED/ADDED FILES relative to 2019-10-05T11:39:06:
2019-10-05T11:39:06/local/idcp/B
2019-10-05T11:39:06/local/idcp/A
The first line of each block is the name of the directory which is also the date and time when it was created.
This is the log-file that is appended each time new files are distributed. It contains the information when and by whom the distribution was done. It also contains a log-message and, optionally, a tag for each version. This is an example of the log-file:
VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
%%
VERSION: 2019-10-05T11:39:06
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test
The first line in each block is the name of the directory which also shows the date and time when the action took place. After "ACTION:" you find the action that took place, "added" or "moved to attic" or "moved from attic". The part after "FROM:" is the person who started the command together with the name of the client-host. After "TAG:" you find the optional tag, after "LOG:" the optional log message with may span several lines. A "%%" seperates entries in the log-file.
The file CHANGES-DIST, as it is described above, can be listed with this command:
rsync-dist.pl -c <config-file> cat-changes
The file LOG-DIST, as it is described above, can be listed with this command:
rsync-dist.pl -c <config-file> cat-log dist
The word "dist" after "cat-log" means that logfile of the distribution directory is listed.
The log-file can be searched for a given tag or a (grep compatible) regular expression that matches one or more tags. Example:
rsync-dist.pl -c <config-file> ls-tag <tag or regexp>
A real-world example:
rsync-dist.pl -c <config-file> ls_tag TAG
The output of this command looks like this:
VERSION: 2019-10-05T11:27:51
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
--
VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test
In this case, the tag (seen as a regular expression) matched two existing tags "TAG1" and "TAG2". Both entries from the logfile are listed separated by "--". Note that there is only the first line shown of each log-message.
Symbolic LINKS can be used to connect a specific version with a fixed path on the server.
This feature can for example be used to select startup-scrips of EPICS IOCs. The startup-script determines which version of the EPICS application the IOC boots.
The path to the startup-script configured on the IOC can stay the same and rsync-dist is used to select the actual version of the EPICS application that is loaded.
Note that there is a file created in the DISTDIR directory that contains a list of all links that point to a certain version. The filename of that file contains the name of the LINKDIR with slashes replaced by dashes like in this example:
LINKS-opt-IOC-Releases-idcp-links
This file contains information on symbolic LINKS in the directory "opt/IOC/Releases/idcp/links". Here is an example of the contents of this file:
2019-10-09T10:28:13 idcp12
2019-10-09T10:28:13 idcp15
2019-10-09T10:28:13 idcp3
2019-10-09T10:28:13 idcp7
2019-10-09T10:28:13 idcp9
2019-10-09T10:33:09 idcp13
The first line means, that in directory "opt/IOC/Releases/idcp/links" there is a symbolic link named "idcp12" that references the distribution-directory "2019-10-09T10:28:13".
Any new symbolic LINK you intend to use has to be created first. You do this with the "add-links" command like this:
rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
add-links VERSION,LINK1,LINK2...
Note that if at least one the given LINKS already exists, the command fails. Also note , that the directory VERSION must exist. The log-file is updated when this command is run.
Symbolic LINKS are changed like this:
rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
change-links VERSION,LINK1,LINK2...
or
rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
change-links VERSION,FILEMASK...
Note that (especially in case one with all links explicitly listed) all of the given link-names must exist and must be symbolic links. The log-file is updated when this command is run.
If you just distributed a version you can use the "-L" option to specify the last distributed version. In this case, you can omit the VERSION parameter. Example:
rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -L
change-links FILEMASK...
The remote directly can be listed with this command:
rsync-dist.pl -c CONFIGFILE ls LINKS
Note that since the command "ls" accepts an argument, "links" is an argument here, not a command.
The "links" after "ls" means that the link directory is listed.
This is the log file where lines are appended each time new LINKS are added or existing links are changed. It contains the information when and by whom the change was done. It also contains an optional log-message for each change and a directory listing of the LINKS before and after the change.
This is an example of the log-file:
DATE: 2019-10-05T14:09:07
FROM: pfeiffer@tarn.acc.bessy.de
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc1 -> ../2019-10-05T13:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc2 -> ../2019-10-05T13:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44
DATE: 2019-10-05T14:11:58
FROM: pfeiffer@tarn.acc.bessy.de
LOG: changed nothing
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc2 -> ../2019-10-05T14:02:44
The first line in each block is the date and time when the action took place. The part after "FROM:" is the person who started the command together with the name of the client-host. After "LOG:" is the optional log message with may span several lines.
The file LOG-LINKS, as it is described above, can be listed with this command:
rsync-dist.pl -c CONFIGFILE cat-log links
The word "links" after "cat-log" means that the logfile of the link directory is listed.
rsync-dist.pl -c CONFIGFILE -m -t dist
rsync-dist.pl -c CONFIGFILE -m "my message" -t "my tag" dist
rsync-dist.pl -c CONFIGFILE dist
Note that FILEMASK may be a name of a single symbolic link like "idcp11" or a file glob pattern like "idcp*".
rsync-dist.pl -c CONFIGFILE -m -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -m "my message" -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -m change-links VERSION,LINK...
VERSION is usually a date-string like "2019-04-02T11:23:14"
rsync-dist.pl -c CONFIGFILE ls dist
rsync-dist.pl -c CONFIGFILE tail-log dist
rsync-dist.pl -c CONFIGFILE tail-changes dist
rsync-dist.pl -c CONFIGFILE ls links
rsync-dist.pl -c CONFIGFILE tail-log links
rsync-dist.pl -c CONFIGFILE tail-changes dist
Logging on the server is required for these examples.
for i in 2[0-9][0-9][0-9]*; do echo $i; done
for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
sed -e 's/VERSION: //'
for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
sed -e 's/VERSION: //' > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
diff - TMP
for i in 2[0-9][0-9][0-9]*; do echo $i; done > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
diff - TMP
grep -- '->' LOG-LINKS | \
sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|sort|uniq
(cd ../dist && for i in 2[0-9][0-9][0-9]*; \
do echo $i; done) > TMP
grep -- '->' LOG-LINKS | \
sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|\
sort|uniq|diff - ../dist/TMP | grep '<'
You can download the source code or distribution packages at:
sourceforgehttps://sourceforge.net/projects/rsync-dist
This help describes all parameters of the program, some which are not mentioned in this document.
You get this help on these by the command:
rsync-dist.pl -h | less
or by this link: commandline arguments
Goetz Pfeiffer <goetzpf@googlemail.com>
GNU GENERAL PUBLIC LICENSE Version 3
rsync-documentation