This section describes a set of procedures for managing ioc software.
EVERYTHING is for the sole purpose of supporting iocs, thus it does not
discuss management of high level applications. The procedures are
intended for an operational facility. During commissioning activities
some more relaxed procedures may be appropriate.
The Configuration Management process described here relies on a
certain level of understanding and responsibility by the engineers who
implement it, and may not always rigorously maintain a previous "known
working" set of software online for an IOC to be rolled back to in the
event of problems while making changes. It is always possible to
retrieve the last version from the system backup tapes, but this may
not be acceptable for some high-availability applications. Sites
desiring more beaurocratic but "safer" procedures may wish to contact
JLAB or SNS to find out about their approaches.
The key features of the APS Configuration Management System are:
Before EPICS base release 3.13, APS/ASD used an Application Source
Release control system called appSR. appSR uses SCCS for source file
control. Since appSR was written several things that impact S/R control
All human created/edited files are put in the cvs repository
dedicated to ioc applications
The complete set of ioc software is divided into <top>
areas. Each <top> area is managed independently. Each follows
its own development and release schedule.
This is a <top> dedicated to a set of iocs that are booted
A <top> area that contains products shared across multiple
Each <top> can have multiple releases.
- Operations and Development areas
Releases of <top> appear in an operations tree.
Operational iocs are booted from this tree. Developers can checkout any
<top> in a private tree.
- <supporttop> releases are not modified in the operations
Because a <supporttop> is used by multiple <ioctops>
it should never be modified in the operations tree. Any modifications
require a new release. It may be appropriate to relax this rule during
a period of commissioning.
- <ioctop> releases may be modified in the operations tree.
This makes it easy to make small ongoing changes to operational
For the above reasons we decided to redo the APS/ASD IOC Applications
S/R control system. Our goals were:
- With release 3.13 the method of configuring record/device/driver
- Both EPICS base and extensions have switched from sccs to cvs
- base/config, i.e. definitions and rules for GNU make, has
evolved so that is now possible to create makefiles that are much
simpler, more powerful, and extendable.
A major decision was What directory structure should we use.
Before the original appSR was written a lot of discussion went
into this topic. There were several meetings between the Application
Developers and Bob Zieman, who implemented appSR. This directory
structure was used to implement the entire APS/ASD contol system
software. Thus a lot of correct decisions were made. The only major
problem is that it was not easy to share code, mainly
record/device/driver support, across <top> areas. This
resulted in sharing by copying source modules from one <top>
area to another. When one developer would make changes to a source
module, the other developers would often not even be aware of the
changes. Thus over time the source modules evolved in different
- SCCS will no longer be used.
- All functionality performed by appSR will be done with a
combination of CVS and GNU make.
Since the overall directory layout appears to be correct, it is kept
in the new system. A brief description of the APS/ASD environment may
help explain why the directory layout works.
One other topic to discuss before describing the overall directory
structure is the idealized Application Development Cycle. It
consists of the following steps:
- APS accelerator
- The APS accelerator complex is composed of four major subsystems:
Linac, PAR (Positron Accumulator Ring), Booster Syncrotron, and Storage
Ring. Each subsystem is controlled by a separate set of IOCs. Thus it
is entirely appropriate to have separate <top> areas for each
- Each subsystem can be viewed as a separate set of applications,
e.g. RF, Magnet Power Supplies, Diagnostics, etc.
- IOC responsibilities.
- A particular application may be spread over multiple IOCs. For
example the control for storage ring applications is normally spread
over many IOCs just because of the physical size of the storage ring.
- A particular IOC may contain parts of multiple applications. For
example vacuum and power supplies normally share IOCs.
- Application Developers.
- For the Linac, a single Application Developer has final
responsibility for all controls applications, i.e. RF, vacuum,
- For the par, booster, and sr, Application Developers are assigned
application areas. For example the same person is responsible for
almost all diagnostic controls for the par, booster, and storage ring.
In reality there is overlap between these steps. In addition as new
needs arise the three steps again have to be performed. However, an
application developer tends to spend a large part of his/her attention
on each step before moving on to the next step.
The cvs tree is arranged in two parts:
- Define I/O requirements.
This involves meeting with the user, in this case the engineers who
are responsible for the application in order to decide the types and
number of I/O modules needed.
- Assemble control hardware and software.
This includes IOCs, I/O modules, software device/driver support,
etc. If EPICS software support is not already available, it has to be
written and tested.
- Build databases, sequence programs, etc.
So at APS/ASD the repository tree is:
This section provides guidelines for creating a new <top> application
that will be maintained in the local cvs repository, i.e. it does NOT
apply to <top> applications maintained somewhere else.
Assume you have a directory tree starting at <working>/<newtop>
that you want to put in the cvs repository at location
$CVSROOT/<system>/<newtop>Just issue the commands:
where <system> is either support or
gnumake clean uninstall
cvs import -m "Creating" <system>/<newtop> <newtop> start
A .cvsignore file should appear in all <top>
areas and also in all <top>/iocBoot/ioxxxx directories.
<top>/.cvsignore should contain:
<top>/iocBoot/iocxxx/.cvsignore should contain:
.cvsignore files are added to the repository just
like any other file.
Each <top> should provide some documentation describing the
application and also release notes describing each release. The release
notes should mention any dependencies on other <top> applications it
uses. The recommended documentation format, at least for release notes,
is HTML. These documents can be put in any directory containing a
Makefile.Host. Adding the lines:
to this Makefile will result in xxx.html being installed
into the directory $(INSTALL_LOCATION)/html/., where
$(INSTALL_LOCATION) is normally <top> The HTMLS_DIR value
can be used to target a subdirectory under $(INSTALL_LOCATION)/html
, useful for longer documents with many sections and graphics files.
An operations tree has the structure:
HTMLS += xxx.html
HTMLS_DIR = .
If only bug fixes are being applied to EPICS base then a new
RX.XX.X directory is generally not needed. Instead a new
support/base/x-y-site release can be created.
- A new RY.YY.Y directory will be needed whenever a
significantly changed version of base is needed which requires that all
ioc and support applications be rebuilt to use the new version. Some
examples of such major changes are:
- A new version of vxWorks is being used.
- The underlying epics structures are modified.
- The set of fields in dbCommon are changed.
- <supporttop>/x-y releases are created via cvs export
commands because it is not permissible to modify an operational
- <ioctop>/x releases are created with a cvs checkout
command because it is permissible to modify an operational ioc
release. See below for reasons.
A portion of the actual APS/ASD operations area is:
A <supporttop> must follow a few simple rules in order to be
used by an <ioctop>.
3-13-1-asd1/ This is actually 3.13.1 with bug fixes
1-3/ What is left after unbundling
others as they get converted to 3.13.1
Thus it must have the following directory structure:
- If it uses EPICS_BASE or another <supporttop>, it must
have a config/RELEASE file.
- It must install everything intended for use by an <ioctop>
into a directory immediately under its <top>.
NOTE: Most <supporttop>s will need only a subset
of bin, lib, dbd, db and include
At least the following two types of support areas may be used by an
In addition an application might use commercial products. Since these
products are not under our control, each <ioctop> has to
decide how to use the product. Wherever possible, however, the product
location should be defined by a line in the <ioctop>/config/RELEASE
- A <supporttop> that is developed in the same cvs
repository as the <ioctops>
- A <supporttop> that follows the above directory structure
but is maintained in a different cvs repository. Examples are epics
base and epics unbundled products such as the allenBradley support. To
allow local bug fixes to be made to any of these remotely obtained
products, the following management procedure is used:
NOTE: EPICS base will follow this procedure. This makes it easy to
make local bug fixes rather than having to create and use a completely
new EPICS release for even minor changes.
- The product is exported from the cvs repository where it is
maintained, or otherwise copied into a local directory area.
- This area is then imported into the local cvs repository as a
- Developers develop in a private area checked out from the main cvs
- When a new version is ready for operations, all changes are
committed and a release tag is created.
- Operations creates an exported version of the new version in the
- If it is necessary to patch an older release, a branch is created
based on that release and the patch applied to the new branch.
A private working version of <supporttop> is checked out via the
- Have a standard place to do development. At APS/ASD it is
recommended that each developer use the following paths, which should
be created with mkdir commands:
- Checkout ONLY the <supporttop> applications for which you are
- If cvs watch on has been set then all files will be
checked out read only. If you want to edit a file, first issue the
DO NOT just issue a chmod to change the file permissions.
cvs edit <filename>
- Try to do all development on the main cvs branch. It should seldom
be necessary to create branches for <supporttops>. If it is get help
unless you understand the procedures described below. This section only
discusses changes on the main branch.
- If a new application is being created get help unless you
understand the procedures described previously.
When changes have been made and it is time for operational use, all
changes should be committed and a new release tag created:
cvs checkout -d <supporttop> support/<supporttop>
all development done here
Tags should take the form Rx-y, where x
is a major release number for the module and y is a minor
release number. For example the first release is R1-1, the
next minor release R1-2, etc.
make sure all changes have been committed
cvs tag Rx-y
You can see all previous tags by issuing the commands:
After tagging the new release you should notify
operations to install it in the operations tree.
If you also want to create a tar file of containing the new release
for use elsewhere, you should not just tar up your working
directory, but instead use the following commands:
cvs status -v Makefile
Note that the naming conventions used for the export
directory and tar file above (a hyphen `-' separates the name
and release number, and the release number has a period `.'
between major and minor parts) are those used for most open source
software. The -kv option to cvs export instructs it
to expand RCS keywords so that they will be preserved without further
change if the resulting files are ever imported into a different CVS
repository, thus preserving their history.
NOTE: This should seldom be necessary.
cvs export -kv -d <supporttop>-x.y -r Rx-y support/<supporttop>
tar cf <supporttop>-x.y.tar <supporttop>-x.y
If it becomes unavoidable to apply patches to an old release
instead of upgrading to a newer version, a branch must be created in
the repository to hold these changes. The following procedure
The cvs rtag -b can be issued from anywhere as
it creates the branch tag Bx-y by working directly on the
files in the repository. The checkout command specifying -r Bx-y
then retrieves that new branch. All subsequent cvs commands issued in
the new area will apply to the branch rather than the versions on the
main trunk. The command cvs tag Rx-y-1 makes a tag for
modified version on this branch. A good convention to follow is that
branch releases should have the same tag as the release with -i
appended, where i represents the patch release number for
this branch. Thus the first patch is Rx-y-1, the second
cvs rtag -b -r Rx-y Bx-y support/<supporttop>
cvs checkout -d <supporttop>/Bx-y -r Bx-y support/<supporttop>
make, test and commit changes
cvs tag Rx-y-1
Installing a new release for use in the Operations Area
Operations installs a new <supporttop> by exporting the new release
into the operational directory tree:
Note that the operations version is created using cvs
export rather than cvs checkout. This means that no CVS
directories appear in the operations version of a <supporttop>
. This directory area should never be modified once operational
<ioctops> start using it. If a modification is required a new
(possibly patch) release must be made using the methods described in
cvs -r export -d x-y -r Rx-y support/<supporttop>
Installing a patch release to the operations area
Patch releases are installed in the operations tree exactly like any
other release, using the commands:
This applies to products like EPICS base, allenBradley, mpf, etc. The
basic rules are:
cvs -r export -d x-y-i -r Rx-y-i support/<supporttop>
If possible the local release tags assigned to products maintained
elsewhere should identify the imported release. For example epics base
would be imported with a command like:
- The first time a product is put in the local repository:
- It is imported into support/<product> giving a vendor tag
reflecting the product and a release tag reflecting the release version.
- Any local modifications that are needed (e.g. changes to config,
NOT bug fixes) are committed on the main branch.
- Any local bug fixes are made on a new branch, NOT the main branch.
- When a new release of the product from the maintainer is to be
- It is imported again, using the same vendor tag but with a release
tag reflecting the new maintainer's release version number.
- Any local changes that occurred on the main branch between the two
import operations will automatically be merged into the new
maintainer's release and the result placed at the head of the main
- Any conflicts due to overlapping changes between the newly
imported release and local modifications will be highlighted to be
After making and committing any local configuration
changes this would be given a tag of R3-13-1. If it is
necessary to patch this release of base, the first patch tag would be
cvs import -m "import base 3.13.1" support/base base release3-13-1
The following examples are for mpf. Assume that the first version of
mpf is version 1-1 and the second version is 2-1.
Importing for the first time
Assume that a tar file containing version 1-1 of mpf has been
extracted into the directory <working>/mpf
Now checkout the new product from the main branch, make
local changes, tag it, and delete the working copy.
cvs import -m "Import mpf version 1-1" support/mpf mpf release1-1
/bin/rm -rf mpf
Only changes that are needed for the local environment
should be made here, for example the config/RELEASE file
should be modified. Bug fixes should NOT be applied here - see below
for how to make bug fixes. The local release tag R1-1 matches
the mpf version number, indicating that the software is identical to
that of the official mpf release but with local configuration steps
cvs checkout -d mpf support/mpf
make and test local configuration changes
cvs commit -m "Configuration changes for release 1-1"
cvs tag R1-1
/bin/rm -rf mpf
Importing a new version
Assume that version 2-1 of mpf has been extracted from a tar file into
The cvs import command will report any conflicts
that have to be merged by hand, and provides instructions on how to do
this if it is necessary. In our example this was the case, and the
subsequent cvs checkout command merged the changes between the
two vendor releases onto the main branch.
cvs import -m "Import mpf version 2-1" support/mpf mpf release2-1
The import will indicate any conflicts that need resolving.
/bin/rm -rf mpf
cvs checkout -d mpf -j release1-1 -j release2-1 support/mpf
Make changes to resolve conflicts and add any new local configuration
changes needed. The following commit is needed to remove files that have
not been locally modified and which are not present in the new release.
cvs commit -m "Merged local changes with mpf 2-1"
cvs tag R2-1
/bin/rm -rf mpf
Applying Local Patches
If it is necessary to make local patches to software that is
maintained elsewhere, the changes must be made on a side branch. If
the changes were made on the main branch then importing later versions
of the product that had these changes incorporated would flag these up
as conflicts that had to be resolved manually. The disadvantage is
that local bugfixes which have not been included in an official release
will have to be re-applied on another new branch, but this can be
regarded as an incentive to ensuring that the official maintainer be
sent copies of all bug fixes found.
Assume that it is necessary to apply a patch to version 1-1 of mpf:
The new release tag that operations should install is
cvs rtag -b -r R1-1 B1-1 support/mpf
cvs checkout -d mpf -r B1-1 support/mpf
make any necessary changes and commit them
cvs tag R1-1-asd1
Exporting a new release to the Operations Area
Releases are prepared for operations just like for a <supporttop>
being maintained in the local repository.
The procedures for <ioctop>s are designed so that simple changes to
the operational tree are easy. It is expected that there will be
frequent (weekly or even daily) small changes such as adding database
records, changing HOPR, LOPR values, etc. All these minor changes are
made on a side branch rather than the main branch of the cvs tree.
Developers check out a private copy of this branch. The operational
area, from which the iocs are booted, has the same branch checked out.
Changes appear in the operational area by a developer committing
changes and operations performing cvs updates.
The following diagram shows how an <ioctop> evolves over
A new branch is needed when:
Assume the current operational system is
- It is not possible to boot all iocs from the same directory
structure, for example while converting to a new release of EPICS base.
- Major changes are being made to <ioctop> and it must be
possible to quickly revert to the old working system.
Each <ioctop> application developer has a
Both of these areas have been checked out via the
All cvs commands issued in an <ioctop>/1 area
will be made to the B1 cvs branch. Until it is time for a new release
no changes are made to the main cvs branch.
cvs checkout -d 1 -r B1 ioc/<ioctop>
Assume that a new release of epics is available, which requires a
new side branch for <ioctop>. The following steps are taken by
whoever is preparing it for the new epics release:
The main branch is checked out of the repository and the B1 changes
merged back into the main branch.
This is shown in the diagram as "B1 merged into main
Changes can be made to <ioctop> and committed without
affecting the B1 branch. During this step the major development work
needed is done on the main cvs branch.
When it is ready for use by other application developer's and/or
operations, a new side branch B2 is created. The R2 tag also created
below marks the file versions at the root of the new branch for later
cvs checkout -d <ioctop> ioc/<ioctop>
cvs update -d -j B1
cvs commit -m "B1 merged into main branch"
The new area is checked out via the commands:
cvs commit -m "changes for new epics release"
cvs tag R2
cvs tag -b B2
/bin/rm -rf <ioctop>
Changes can be made independently on either branch B1 or
B2, and iocs can be booted from either area. Once all iocs are
successfully using B2, the B1 cvs branch, operations 1 area, and any 1
private areas are "abandoned", i.e. they no longer need to be
maintained. The abandoned branch ends in a hollow box in the diagram.
cvs checkout -d 2 -r B2 ioc/<ioctop>
IMPORTANT WARNING. While B1 and B2 are both active, it is the
application developer and operations responsibility to make any changes
to both releases. This starts from the time step 2 is performed until
B1 is no longer active. The diagram identifies this period with "dual
changes". CVS can help with this task, but in most cases it will
probably be easier and just as quick to make and commit the same change
in both areas.
The diagram shows that the same steps are followed for branch B3.
Development is done on a branch so that when a new branch becomes the
operational branch, nothing has to be done to get the operational tree
on the latest release - it is already using it!
If development were done on the main cvs branch, then when new
development got to the point where it was desired to make this the new
operations release, a new release would have to be generated on a side
branch and the official area switched to use the new release. In
practice at APS this time has often only been known once the system was
already operational. The only easy way to prove that the new release is
really ready is to run it in an operational mode. By then however,
operations staff are already saying: "It works, please leave it alone."
<ioctop> developers may want to create a tag on a branch
before and after making extensive changes to the branch or before
merging a branch back onto the main tree. These tags should always be
of the form Rx-y where x is the branch and y is a minor release number.
This if the current branch is B2 then the tags are R2-1, R2-2, ....
These tags are NOT created for the purpose of creating operational
releases. They are only to tag snapshots of the cvs repository at key
- Have a standard place to do development. Each site may establish
guidelines. At APS/ASD it is recommended that each developer work in an
area under their home directory as follows:
- Checkout ONLY the <ioctop>s for which you are responsible.
- If cvs watch on has been set then all files will be
checked out read only. If you want to edit a file issue the command:
DO NOT just issue a chmod to change the file permissions.
cvs edit <filename>
- Do all development on the correct cvs branch. If it is time to
create a new cvs branch for an <ioctop> get help unless you understand
the procedures described previously.
- Be careful. Remember that the operations area is using the same
cvs repository as you.
- Whenever possible use the same config/RELEASE definitions as
where x is the branch desired. For example if B1 is the
current branch than x is 1.
Be Careful. Remember that when operations performs updates and
builds in the operations tree the operational system will be using your
modifications. During operational periods only changes really needed
for operations should be committed.
cvs checkout -d x -r Bx ioc/<ioctop>
all development is done here
Now prepare instructions for what operations should do.
The instructions should specify cvs updates and makes at the lowest
level possible. Do NOT just give instructions to go to <ioctop>/x
and update and rebuild.
Ask for detailed instructions from developer's about where to perform
cvs updates and makes.
Because <ioctop>s can be modified, it is possible that a
developer could perform cvs commits that will break an operational
system when a cvs update is done in the operations area. In order to
prevent a major problem, procedures should be in place to backup all
<ioctop>s on a regular schedule when the facility is on an
operational period. Procedures should also be in place to restore a
particular <ioctop> from the backup.
commit ONLY the files actually changed
Application developers must be aware that only minor changes should
be made during operational periods. When a developer makes a change
commits should be made only to the necessary files rather than commits
at the top level. He/she should instruct operations about what should
be updated and rebuilt, i.e. update just the files necessary and only
build the components affected by the updates.
Because application developers can accidentally commit changes that
would break the operational system care should be taken when performing
cvs updates on an <ioctop> in the operations area. A good idea
is to do the following before any updates:
Wherever you plan to issue a cvs update first issue the command
which will list the files that are liable to be changed.
Then on any files you want to update issue the command
cvs -n update
where Bx is the current branch tag, to see the
modifications that have been committed. Only if you are satisfied with
these changes should you perform the actual update.
This file should contain
cvs diff -r Bx <filename>
which makes cvs ignore all O.* files .