Experimental Physics and
Industrial Control System

the short answer is: you don't.
As this is a telescope data acquisition happens at night
(well, unless you test stuff during the day, which sometimes
happens a lot and sometimes not at all).
Users (generally) don't access these data other than for
quick-look display (which is automated - and, data are
first written to disk raw and then read in again and
demultiplexed. The raw data are generally more than 100GB
per night per array).
The data get written to tape (BTW, we do have mirrored disks,
so the data is being accessed and copied as it is acquired)
by a cron job during the dat, and taking data at the time
the cron job runs is a bit dicy, but it can be done.
The cron job runs about 1/2 hour per day or so, so it
doesn't take very long.

We used to have RAID arrays for the data but found they were
too slow and throttling data acquisition at times.

Hi Maren,

Just curious to know how fast you can get data from the disk to the LTO
2 drives while saving more data to the disk, and serving yesterdays data
to users from the disk...

Sounds like trouble to me.

Greg Lawson
SNS Controls

On Thu, 2007-03-29 at 09:03 -1000, Maren Purves wrote:

Kay,

(only responding to the data volume problem)

On Thu, 29 Mar 2007, Kay-Uwe Kasemir wrote:

On Mar 29, 2007, at 02:04 , Terry Cornall wrote:

I'm trying to dimension the archive requirements for all the 14 beamlines
for the Australian Synchrotron and am wondering if I can do it
with one PC or if I need many. Of course, sampling or monitoring rates will
play a big part, but also I need to know what is possible per archive
engine.

Problem 2:
The amount of data. One 'double' sample uses a little over
20 bytes for the timestamp, status, sev, value.
For many values, the data file structure and index add
relatively little to that, but you'll still get about 20GB
per day. How do you intend to back that up?

[...]

If somebody tells you that disks are cheap, please ask
that person to take care of your archiving, then run
as fast as you can.

you got a point, but I'm not sure that's what Terry asked.

And besides, with our wide field camera (motor control is
EPICS, array control and data acquisition isn't) that has
4 arrays we get over 50GB per array in a good night with
short integrations. With all 4 arrays we can get well over
a Terabyte in a week.

Yes, it's an archiving problem, but it's entirely doable.
LTO 2 drives in our case generally hold about 300 GB of
compressed data (depends on how compressible your data
are, that's why 'in our case').

Aloha,
Maren

Replies:

Re: Maximum archival rate Lawrence T. Hoff

References:

Maximum archival rate Terry Cornall

Re: Maximum archival rate Kay-Uwe Kasemir

Re: Maximum archival rate Maren Purves

Re: Maximum archival rate Greg Lawson

Navigate by Date:

Prev: Re: Maximum archival rate Greg Lawson

Next: Re: Maximum archival rate Lawrence T. Hoff

Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <2007>  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 

Navigate by Thread:

Prev: Re: Maximum archival rate Greg Lawson

Next: Re: Maximum archival rate Lawrence T. Hoff

Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <2007>  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024