Hi:
The basic limitation is in the "fseek" call,
based upon which the archiver code uses 32bit offset
pointers inside its files.
While one could spend time extending this
to 64bits, I guess that only creates bigger archives
which are even harder to handle than the current ones:
How do you delete older data for a channel?
How do you determine which channel contributes
the most samples?
How do you rename channels, scale data with changing units?
All that's better done in a relational database, which is
why the recent archiver developments went in that direction.
-Kay
-----Original Message-----
From: [email protected] on behalf of Thomas Birke
Sent: Wed 10/15/2008 02:45
To: Tom Cumming
Cc: [email protected]
Subject: Re: How have folks been dealing with the Channel Archiver's 2GB indexfile limit?
Tom Cumming wrote:
> On Wed, 2008-10-15 at 05:23 +0200, Thomas Birke wrote:
>
>
>> If our index-files at BESSY approach the 2GB limit, I usually re-create
>> the index completely from scratch which typically produces a much
>> smaller index-file.
>>
>
> How do you you, "re-create the index completely from scratch"? The only
> tool I know of is the, "RecoverIndex" tool that does it's thing by
> making educated guesses by groking the data.
The current version of the ChannelArchiver also installs a tool named
"ArchiveIndexTool".
It has a "-reindex" option that will create a new index from an existing
one. Maybe that'll help a bit...
I forgot to mention, that we are creating weekly archives of several
different sets of PVs.
The index-files of these archives definitely never reach the 2GB limit
(not even 100MB), and every monday at 2am we start a new one.
These separate archives/index-files are then combined into a
"master_index" using the ArchiveIndexTool as well.
To do this, we create an xml-file (e.g. indexconfig.xml) containing a
list of index-files to combine into the so-called master_index.
This master_index is updated regularly and in our case after ~1-2 years
usually reaches the 2GB limit. All this is done automatically by scripts
run by cron.
I just recreated our master_index (although it was not yet necessary)
and it shrunk from ~1.5GB to ~1GB. It contains about 2.5 years of data
from ~32000 PVs summing up to ~520GB of data.
There are sample scripts bundled with the ChannelArchiver (directory
ExampleSetup) that do the job or may serve as a good starting point.
Thomas
- References:
- How have folks been dealing with the Channel Archiver's 2GB index file limit? Tom Cumming
- Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke
- Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Tom Cumming
- Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke
- Navigate by Date:
- Prev:
Re: CA client application M.Babai
- Next:
Re: Building CAP5 under windows or cygwin Andrew Johnson
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
<2008>
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: How have folks been dealing with the Channel Archiver's 2GB index file limit? Thomas Birke
- Next:
How to save the x-y graph? 汪启胜
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
<2008>
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|