rsync-dist

NAME

rsync-dist.pl - A Perl script for managing distributions of binary files

SYNOPSIS

rsync-dist.pl -c <config-file> -m -t dist

rsync-dist.pl -c <config-file> -m -L change-links IOC*

DESCRIPTION

Features

This script can be used to copy files (usually binary distributions of programs) to a new created VERSION directory on a remote server. Old VERSION directories are never deleted. The active VERSION is selected by a symbolic LINK that is created and managed by the program. Programs features are:

Prerequisites

Using a configuration file

Certain values like the name of the remote host or the directory on the remote-host can be stored in a configuration file. The program can then read that configuration file in order to obtain these values.

Variables for the configuration file are shown when you run:

rsync-dist.pl -h

and look for option "--config".

Creating a a configuration file

A configuration file is easy to create. Example:

rsync-dist.pl -H MYHOST -u MYUSER \
    -p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
    --prefix-distdir 1 show-config

This command prints the contents of the config file to the screen where all settings match the ones given as command line parameters. Here is how you create that file directly:

rsync-dist.pl -H MYHOST -u MYUSER \
    -p DISTDIR -P LINKDIR --localpath DISTRIBUTIONDIR \
    --prefix-distdir 1 write-config CONFIGFILE

You can easily extend this file. Here is for example, how you add the "world-readable" option:

rsync-dist.pl --config CONFIGFILE --world-readable write-config CONFIGFILE

Using a a configuration file

Later on, you can start a distribution command like this:

rsync-dist.pl --config CONFIGFILE dist

or like this

rsync-dist.pl -c CONFIGFILE dist

The program takes all parameters from "CONFIGFILE". Parameters it needs but are not specified (like the local path) are requested from the user interactively by starting an editor (default: vi).

Configuration file variables

The following variables can be defined in a configuration file

RSYNC_DIST_HOST

Name of the distribution host or hosts. More than one host can be specified as a comma-separated list. Associated command line option: --hosts.

RSYNC_DIST_USER

The username(s) rsync-dist.pl uses to log on to the remote hosts. See also --user. When several hosts (see above) are defined, you can define several (possibly different) users for the hosts. Associated command line option: --users.

RSYNC_DIST_SSH_LOCAL

Specify the local ssh command that is used. Associated command line option: --ssh-local.

RSYNC_DIST_SSH_REMOTE

Specify the remote ssh command that is used. Associated command line option: --ssh-remote.

RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE

If not empty, use a ssh reverse tunnel to connect from the server back to the local host. Two values are allowed for this variable are: 'exists' : use an *existing* reverse tunnel. 'create' : create a reverse tunnel for the operation and remove it when the operation is finished Associated command line option: --ssh-reverse-tunnel-mode

RSYNC_DIST_SSH_REVERSE_TUNNEL_PORT

When reverse tunnel mode is set (see RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE or --ssh-reverse-tunnel-mode) use this port for the reverse tunnel on the remote server. The must be *no other* application using this port on the remote server. The default for this port is 53591 (0xd157). Associated command line option: --ssh-reverse-tunnel-port.

RSYNC_DIST_PATH

The remote distribution directory. Associated command line option: --distpath.

RSYNC_DIST_LINKPATH

The remote link directory. Associated command line option: --linkpath.

RSYNC_DIST_PREFIX_DISTDIR

If empty or '1', prepend the distribution directory to the link-source path. If this is a non-empty string prepend this to the link source path. This effects only the connamds 'add-links' and 'change-links'. Associated command line option: --prefix-distdir.

RSYNC_DIST_LOCALPATH

The local directory or directories. More than one directory can be specified in a comma-separated list. Associated command line option: --localpath.

RSYNC_DIST_LOCALPREFIX

The prefix that is prepended to local paths (see above). Associated command line option: --localprefix

RSYNC_DIST_WORLDREADABLE

When set to 1, make all distributed files world-readable. Associated command line option: --world-readable.

RSYNC_DIST_ONE_FILESYSTEM

When set to 1, do not cross filesystem boundaries on the local host. Associated command line option: --one-filesystem

When set to 1, symbolic links are never dereferenced and always copied as links. Associated command line option: --preserve-links.

When set to 1, symbolic links always dereferenced. Associated command line option: --dereference-links.

RSYNC_DIST_EXCLUDE_LIST

Specify a file on the local host that contains files that are to be excluded. Associated command line option: --exclude-list.

RSYNC_DIST_CHECKSUM

Defunct option, retained for compatibility.

RSYNC_DIST_SHORT_DATE

Use shortened date for dist directories. Associated command line option: --short-date.

RSYNC_DIST_BRANCH

When this variable is defined, use this branch for distributing files. Associated command line option: --branch.

RSYNC_DIST_PARTIAL

When set to 1, make a complete copy of the last version on the server but distribute only some files from the client to the server. --localpath may contain file-glob patterns in this case. Associated command line option: --partial.

RSYNC_DIST_EDITOR

The default editor to enter options that the used didn't specify on the command lines. The default is "vi". Associated command line option: --editor.

RSYNC_DIST_EDITOR_NO_DEFAULTS

When set to 1, the editor does not show defaults, when called. Associated command line option: --no-editor-defaults.

RSYNC_DIST_SHOW_PROGRESS

When set to 1, show progress of rsync on console. Associated command line option: --progress.

RSYNC_DIST_TAG

Specify a default-tag, may also be empty. In this case the program does not request a tag if no one is given on the command line. Associated command line option: --tag.

RSYNC_DIST_MESSAGE

Specify a default log-message, may also be empty. In this case the program does not request a message if no one is given on the command line. Associated command line option: --message.

RSYNC_DIST_FILTER_OUTPUT

When set to 1, output from remote ssh commands is filtered to make it more human readable. This affects the commands "change-links", "add-links", and "ls links". Associated command line option: --filter-output.

RSYNC_DIST_POSTPROC

An optional command to be run after dist on the target host. Associated command line option: --postproc .

RSYNC_DIST_SSH_REVERSE_TUNNEL_MODE

An optional command to be run after change-links on the target host. Associated command line option: --link-postproc.

Distributing files

In order to distribute files the user must at least specify the remote-server-name, the remote-dist-path and the local-path. The local and remote-dist-path are relative to your user-home unless they start with a '/'.

For day to day work, the server-name and paths will be specified in a configuration file that you created as it is described in the previous chapter. Here is how you distribution command then looks:

rsync-dist.pl -c CONFIGFILE -t TAG -m LOGMESSAGE dist

Additionally you can specify your account-name on the remote machine (especially when this is different from your account on the local machine). This can be done with the "-u" option or by the config-file.

Example:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -t TAG dist

If you prefer not to have a tag or log-message the command looks like this:

rsync-dist.pl -c CONFIGFILE -m -t dist

Inspecting a remote directory with distributed files

The remote directly can be listed with this command:

rsync-dist.pl -c <config-file> ls dist

Note that since the command "ls" accepts an argument, "dist" is an argument here, not a command.

The word "dist" after "ls" means that the distribution directory is listed. In order to see the link directory, you have to specify "links" after "ls".

On the remote directory you see the following files and directories:

version directories

These directories are named like: "2019-10-04T13:57:35" or "2019-10-04T140312". The first one is an ISO 8601 format for timestamps, the second one is used when option --short-date or variable RSYNC_DIST_SHORT_DATE are given. The numbers within the string are (in this order) the year, month, day, hour, minute, seconds. The letter "T" separates the date from the time. Your distributed files are placed within these directories. Files that are equal among several directories are stored with hard-links in order to preserve disk space.

the file LAST

This file contains the name of the most recently created VERSION directory. This file is always present after you call this script for the first time. It is modified each time you distribute files to the server. When a new version is distributed, the directory mentioned in "LAST" is copied as a hard-link copy, then rsync is called to transmit all changed/added or removed files.

branches

When your development takes place on several branches at the same time, the hard-link copy of the last distributed version may be inefficient with respect to the space it occupies on the hard disk. For these cases, branches can be used. A branch simply means that another "LAST" file is used in order to create the hard-link copy. A branch can be created from trunk or another branch with the create-branch command. It can be used together with the dist command with the "--branch" option. See also the online help when you execute "rsync-dist.pl -h".

the file CHANGES-DIST

This file contains the changes of the most recent version compared to the previous version. All files that were changed or added are listed. Note that files that were deleted ARE NOT listed. This is an example of the contents of this file:

VERSION: 2019-10-05T11:39:06
CHANGED/ADDED FILES relative to 2019-10-05T10:10:06:
2019-10-05T11:39:06/local/idcp/C

VERSION: 2019-10-05T12:22:16
CHANGED/ADDED FILES relative to 2019-10-05T11:39:06:
2019-10-05T11:39:06/local/idcp/B
2019-10-05T11:39:06/local/idcp/A

The first line of each block is the name of the directory which is also the date and time when it was created.

the file LOG-DIST

This is the log-file that is appended each time new files are distributed. It contains the information when and by whom the distribution was done. It also contains a log-message and, optionally, a tag for each version. This is an example of the log-file:

VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
%%
VERSION: 2019-10-05T11:39:06
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test

The first line in each block is the name of the directory which also shows the date and time when the action took place. After "ACTION:" you find the action that took place, "added" or "moved to attic" or "moved from attic". The part after "FROM:" is the person who started the command together with the name of the client-host. After "TAG:" you find the optional tag, after "LOG:" the optional log message with may span several lines. A "%%" seperates entries in the log-file.

Listing the CHANGES-DIST file

The file CHANGES-DIST, as it is described above, can be listed with this command:

rsync-dist.pl -c <config-file> cat-changes

Listing the LOG-DIST file

The file LOG-DIST, as it is described above, can be listed with this command:

rsync-dist.pl -c <config-file> cat-log dist

The word "dist" after "cat-log" means that logfile of the distribution directory is listed.

Searching for tags

The log-file can be searched for a given tag or a (grep compatible) regular expression that matches one or more tags. Example:

rsync-dist.pl -c <config-file> ls-tag <tag or regexp>

A real-world example:

rsync-dist.pl -c <config-file> ls_tag TAG

The output of this command looks like this:

VERSION: 2019-10-05T11:27:51
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG1
LOG: test
--
VERSION: 2019-10-05T11:28:56
ACTION: added
FROM: pfeiffer@tarn.acc.bessy.de
TAG: TAG2
LOG: test

In this case, the tag (seen as a regular expression) matched two existing tags "TAG1" and "TAG2". Both entries from the logfile are listed separated by "--". Note that there is only the first line shown of each log-message.

Symbolic LINKS can be used to connect a specific version with a fixed path on the server.

This feature can for example be used to select startup-scrips of EPICS IOCs. The startup-script determines which version of the EPICS application the IOC boots.

The path to the startup-script configured on the IOC can stay the same and rsync-dist is used to select the actual version of the EPICS application that is loaded.

The log-file in the source directory

Note that there is a file created in the DISTDIR directory that contains a list of all links that point to a certain version. The filename of that file contains the name of the LINKDIR with slashes replaced by dashes like in this example:

LINKS-opt-IOC-Releases-idcp-links

This file contains information on symbolic LINKS in the directory "opt/IOC/Releases/idcp/links". Here is an example of the contents of this file:

2019-10-09T10:28:13 idcp12
2019-10-09T10:28:13 idcp15
2019-10-09T10:28:13 idcp3
2019-10-09T10:28:13 idcp7
2019-10-09T10:28:13 idcp9
2019-10-09T10:33:09 idcp13

The first line means, that in directory "opt/IOC/Releases/idcp/links" there is a symbolic link named "idcp12" that references the distribution-directory "2019-10-09T10:28:13".

Any new symbolic LINK you intend to use has to be created first. You do this with the "add-links" command like this:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
              add-links VERSION,LINK1,LINK2...

Note that if at least one the given LINKS already exists, the command fails. Also note , that the directory VERSION must exist. The log-file is updated when this command is run.

Symbolic LINKS are changed like this:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
              change-links VERSION,LINK1,LINK2...

or

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE
              change-links VERSION,FILEMASK...

Note that (especially in case one with all links explicitly listed) all of the given link-names must exist and must be symbolic links. The log-file is updated when this command is run.

If you just distributed a version you can use the "-L" option to specify the last distributed version. In this case, you can omit the VERSION parameter. Example:

rsync-dist.pl -c CONFIGFILE -m LOGMESSAGE -L
              change-links FILEMASK...

The remote directly can be listed with this command:

rsync-dist.pl -c CONFIGFILE ls LINKS

Note that since the command "ls" accepts an argument, "links" is an argument here, not a command.

The "links" after "ls" means that the link directory is listed.

This is the log file where lines are appended each time new LINKS are added or existing links are changed. It contains the information when and by whom the change was done. It also contains an optional log-message for each change and a directory listing of the LINKS before and after the change.

This is an example of the log-file:

DATE: 2019-10-05T14:09:07
FROM: pfeiffer@tarn.acc.bessy.de
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc1 -> ../2019-10-05T13:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:07 ioc2 -> ../2019-10-05T13:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44

DATE: 2019-10-05T14:11:58
FROM: pfeiffer@tarn.acc.bessy.de
LOG: changed nothing
OLD:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:09 ioc2 -> ../2019-10-05T14:02:44
NEW:
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc1 -> ../2019-10-05T14:02:44
lrwxr-xr-x 1 usr exp 22 Oct 5 14:12 ioc2 -> ../2019-10-05T14:02:44

The first line in each block is the date and time when the action took place. The part after "FROM:" is the person who started the command together with the name of the client-host. After "LOG:" is the optional log message with may span several lines.

The file LOG-LINKS, as it is described above, can be listed with this command:

rsync-dist.pl -c CONFIGFILE cat-log links

The word "links" after "cat-log" means that the logfile of the link directory is listed.

QUICK REFERENCE

distribute a version

without any log messages
rsync-dist.pl -c CONFIGFILE -m -t dist
with log message and tag
rsync-dist.pl -c CONFIGFILE -m "my message" -t "my tag" dist
enter log-message and tag interactively
rsync-dist.pl -c CONFIGFILE dist

Note that FILEMASK may be a name of a single symbolic link like "idcp11" or a file glob pattern like "idcp*".

rsync-dist.pl -c CONFIGFILE -m -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -m "my message" -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -L change-links LINK...
rsync-dist.pl -c CONFIGFILE -m change-links VERSION,LINK...

VERSION is usually a date-string like "2019-04-02T11:23:14"

inspect remote distribution directory

list directory
rsync-dist.pl -c CONFIGFILE ls dist
show last entries in log-file
rsync-dist.pl -c CONFIGFILE tail-log dist
show last entries in CHANGES file
rsync-dist.pl -c CONFIGFILE tail-changes dist
list directory
rsync-dist.pl -c CONFIGFILE ls links
show last entries in log-file
rsync-dist.pl -c CONFIGFILE tail-log links
show last entries in CHANGES file
rsync-dist.pl -c CONFIGFILE tail-changes dist

Tips and tricks for the server administration

Logging on the server is required for these examples.

show all distribution directories
for i in 2[0-9][0-9][0-9]*; do echo $i; done
list all existing distribution directories mentioned in LOG-DIST
for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
      sed -e 's/VERSION: //'
list directories mentioned in LOG-DIST that do no longer exist
for i in 2[0-9][0-9][0-9]*; do grep $i LOG-DIST; done | \
      sed -e 's/VERSION: //' > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
      diff - TMP
compare existing directories with the ones mentioned in LOG-DIST
for i in 2[0-9][0-9][0-9]*; do echo $i; done > TMP
grep VERSION LOG-DIST |sort|uniq| sed -e 's/VERSION: //' | \
      diff - TMP
grep -- '->' LOG-LINKS | \
      sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|sort|uniq
(cd ../dist && for i in 2[0-9][0-9][0-9]*; \
      do echo $i; done) > TMP
grep -- '->' LOG-LINKS | \
      sed -e 's/^.*->.*\(2[0-9][0-9][0-9]\)/\1/'|\
      sort|uniq|diff - ../dist/TMP  | grep '<'

Further reading

Source code

You can download the source code or distribution packages at:

sourceforge
https://sourceforge.net/projects/rsync-dist

Command line parameters

This help describes all parameters of the program, some which are not mentioned in this document.

You get this help on these by the command:

rsync-dist.pl -h | less

or by this link: commandline arguments

AUTHOR

Goetz Pfeiffer <goetzpf@googlemail.com>

LICENSE

GNU GENERAL PUBLIC LICENSE Version 3

SEE ALSO

rsync-documentation