Mirror Master

Lee McLoughlin

and

Zoë Leech

22 May 1998
lmjm@icparc.ic.ac.uk
zl@icparc.ic.ac.uk

Warning

mm is not for the faint-hearted

Introduction

mm [flags] [configuration file]

Description

mm is a Perl program designed to manage the running of multiple mirrors. It will keep as many mirrors running in parallel as it can, up to a user defined maximum, automatically starting another when one finishes. It will also handle locking so that only one mirror of a package can occur at a time and allows configuration of the minimum time between attempts to mirror a package.

mm presumes that all the package details are stored in a directory called packages. All the packages for a given site must be in a file whose name is the same as that of the site. So in packages/sunsite.org.uk will be all the package details for the host sunsite.org.uk.

Flags

-t  Ignore the minimal retry timers. 
-o site:package  Only mirror the given site and package. 
-debug  Enable debugging. If this argument is given more than once the level of debugging will increase. Currently the maximum possible level is four. 
-s  Enable status debugging. This will print out the status of any subprocesses as they are spawned or as they exit. 
Any unrecognized arguments are passed as arguments to mirror for example:
              mm -d mmin
will pass mirror the -d flag (so mirror will run in debug mode).
 

Configuration File

The configuration file is parsed as a series of statements. Blank lines and lines beginning with a hash are ignored. There are are two kinds of statements: control statements and site:package details.

Control statements are of the form:

Whitespace around the equals sign is ignored.

Statements are obeyed in sequence as they are reached.

Here is a list of the keywords and their values and any defaults:
home Directory that mm chdirs to before doing any work. The default is the current directory (i.e. .) 
max Maximum number of mirrors that can be running in parallel at any one time. The default is 6. 
mirror This is the way to call mirror. 
The default is: 
exec ./mirror \$args -p'\$package' packages/\$site > logs/\$ site:\$pkg 2>&1 

$args, $package and $site are replaced with the appropriate values from the package file. Any $pkg is replaced with a version of $package with all characters that cause problems in filenames converted to underscore. 
cmd Run the given command. This can occur multiple times. It can be used to delete temporary files or send mail.
cmdin Run the given command and use its output as mm input. This can occur multiple times. For example 
cmdin=./pkgs_to_mmin packages/*
skip Skip the given site:package when it occurs. 
In addition to control statements there are also package details. Each package details statement is of the form:

site:package  min-restart-last-ok  min-restart-last-notok  mirror-args
The site and package fields are matched against those in the files in the packages directory. A mirror for this site:package will only be run if the last mirror run was successful and was more than min-restart-last-ok hours ago. If the last mirror of this was unsuccessful then min-restart-last-notok will be used to decide if the mirror will be run. The information about the last mirror run is stored in a file called mm.status in the directory defined by home
 
 

Examples

Here is a  mm configuration file to manage the mirroring of two packages.
   

However maintaining this file where there are a large number of mirrors can be a problem. To overcome this the program pkgs_to_mmin, was
designed to take a list of package files as input and turn these into the necessary input for mm

In this  mm configuration file pkgs_to_mmin is used to manage all the packages in the /public/mirror/packages directory.
 

See Also

perl(l), ftp(1), mirror

Author

Written by Lee McLoughlin <lmjm@icparc.ic.ac.uk>.