EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  <20082009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  <20082009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: How have folks been dealing with the Channel Archiver's 2GB indexfile limit?
From: "Kasemir, Kay" <[email protected]>
To: Thomas Birke <[email protected]>, Tom Cumming <[email protected]>
Cc: [email protected]
Date: Thu, 16 Oct 2008 09:08:27 -0400
Hi:

The basic limitation is in the "fseek" call,
based upon which the archiver code uses 32bit offset
pointers inside its files.

While one could spend time extending this
to 64bits, I guess that only creates bigger archives
which are even harder to handle than the current ones:
How do you delete older data for a channel?
How do you determine which channel contributes
the most samples?
How do you rename channels, scale data with changing units?
All that's better done in a relational database, which is
why the recent archiver developments went in that direction.

-Kay


-----Original Message-----
From: [email protected] on behalf of Thomas Birke
Sent: Wed 10/15/2008 02:45
To: Tom Cumming
Cc: [email protected]
Subject: Re: How have folks been dealing with the Channel Archiver's 2GB	indexfile limit?
 
Tom Cumming wrote:
> On Wed, 2008-10-15 at 05:23 +0200, Thomas Birke wrote:
>
>   
>> If our index-files at BESSY approach the 2GB limit, I usually re-create 
>> the index completely from scratch which typically produces a much 
>> smaller index-file.
>>     
>
> How do you you, "re-create the index completely from scratch"? The only
> tool I know of is the, "RecoverIndex" tool that does it's thing by
> making educated guesses by groking the data.
The current version of the ChannelArchiver also installs a tool named 
"ArchiveIndexTool".
It has a "-reindex" option that will create a new index from an existing 
one. Maybe that'll help a bit...

I forgot to mention, that we are creating weekly archives of several 
different sets of PVs.
The index-files of these archives definitely never reach the 2GB limit 
(not even 100MB), and every monday at 2am we start a new one.
These separate archives/index-files are then combined into a 
"master_index" using the ArchiveIndexTool as well.

To do this, we create an xml-file (e.g. indexconfig.xml) containing a 
list of index-files to combine into the so-called master_index.
This master_index is updated regularly and in our case after ~1-2 years 
usually reaches the 2GB limit. All this is done automatically by scripts 
run by cron.

I just recreated our master_index (although it was not yet necessary) 
and it shrunk from ~1.5GB to ~1GB. It contains about 2.5 years of data 
from ~32000 PVs summing up to ~520GB of data.

There are sample scripts bundled with the ChannelArchiver (directory 
ExampleSetup) that do the job or may serve as a good starting point.

Thomas




References:
How have folks been dealing with the Channel Archiver's 2GB index file limit? Tom Cumming
Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke
Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Tom Cumming
Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke

Navigate by Date:
Prev: Re: CA client application M.Babai
Next: Re: Building CAP5 under windows or cygwin Andrew Johnson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  <20082009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke
Next: How to save the x-y graph? 汪启胜
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  <20082009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·