Experimental Physics and
Industrial Control System

Thomas Birke <[email protected]> · Wed, 15 Oct 2008 08:45:21 +0200

Tom Cumming wrote:

On Wed, 2008-10-15 at 05:23 +0200, Thomas Birke wrote:

If our index-files at BESSY approach the 2GB limit, I usually re-create
the index completely from scratch which typically produces a much
smaller index-file.

How do you you, "re-create the index completely from scratch"? The only
tool I know of is the, "RecoverIndex" tool that does it's thing by
making educated guesses by groking the data.

The current version of the ChannelArchiver also installs a tool named
"ArchiveIndexTool".

It has a "-reindex" option that will create a new index from an existing
one. Maybe that'll help a bit...
I forgot to mention, that we are creating weekly archives of several
different sets of PVs.

The index-files of these archives definitely never reach the 2GB limit
(not even 100MB), and every monday at 2am we start a new one.

These separate archives/index-files are then combined into a
"master_index" using the ArchiveIndexTool as well.
To do this, we create an xml-file (e.g. indexconfig.xml) containing a
list of index-files to combine into the so-called master_index.

This master_index is updated regularly and in our case after ~1-2 years
usually reaches the 2GB limit. All this is done automatically by scripts
run by cron.
I just recreated our master_index (although it was not yet necessary)
and it shrunk from ~1.5GB to ~1GB. It contains about 2.5 years of data
from ~32000 PVs summing up to ~520GB of data.
There are sample scripts bundled with the ChannelArchiver (directory
ExampleSetup) that do the job or may serve as a good starting point.
Thomas

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System