Re: Maximum archival rate

[Maren Purves <m.purves@jach.hawaii.edu>]

Hi Greg,

the short answer is: you don't.
As this is a telescope data acquisition happens at night
(well, unless you test stuff during the day, which sometimes
happens a lot and sometimes not at all).
Users (generally) don't access these data other than for
quick-look display (which is automated - and, data are
first written to disk raw and then read in again and
demultiplexed. The raw data are generally more than 100GB
per night per array).
The data get written to tape (BTW, we do have mirrored disks,
so the data is being accessed and copied as it is acquired)
by a cron job during the dat, and taking data at the time
the cron job runs is a bit dicy, but it can be done.
The cron job runs about 1/2 hour per day or so, so it
doesn't take very long.

We used to have RAID arrays for the data but found they were
too slow and throttling data acquisition at times.


Maren

On Thu, 29 Mar 2007, Greg Lawson wrote:

> Hi Maren,
>
> Just curious to know how fast you can get data from the disk to the LTO
> 2 drives while saving more data to the disk, and serving yesterdays data
> to users from the disk...
>
> Sounds like trouble to me.
>
> Greg Lawson
> SNS Controls
>
>
> On Thu, 2007-03-29 at 09:03 -1000, Maren Purves wrote:
> > Kay,
> >
> > (only responding to the data volume problem)
> >
> > On Thu, 29 Mar 2007, Kay-Uwe Kasemir wrote:
> > > On Mar 29, 2007, at 02:04 , Terry Cornall wrote:
> > > > I'm trying to dimension the archive requirements for all the 14 beamlines
> > > > for the Australian Synchrotron and am wondering if I can do it
> > > > with one PC or if I need many. Of course, sampling or monitoring rates will
> > > > play a big part, but also I need to know what is possible per archive
> > > > engine.
> > >
> > > Problem 2:
> > > The amount of data. One 'double' sample uses a little over
> > > 20 bytes for the timestamp, status, sev, value.
> > > For many values, the data file structure and index add
> > > relatively little to that, but you'll still get about 20GB
> > > per day. How do you intend to back that up?
> >
> > [...]
> >
> > > If somebody tells you that disks are cheap, please ask
> > > that person to take care of your archiving, then run
> > > as fast as you can.
> >
> > you got a point, but I'm not sure that's what Terry asked.
> >
> > And besides, with our wide field camera (motor control is
> > EPICS, array control and data acquisition isn't) that has
> > 4 arrays we get over 50GB per array in a good night with
> > short integrations. With all 4 arrays we can get well over
> > a Terabyte in a week.
> >
> > Yes, it's an archiving problem, but it's entirely doable.
> > LTO 2 drives in our case generally hold about 300 GB of
> > compressed data (depends on how compressible your data
> > are, that's why 'in our case').
> >
> > Aloha,
> > Maren