1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 <2007> 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 <2007> 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Re: Maximum archival rate |
From: | Maren Purves <[email protected]> |
To: | Greg Lawson <[email protected]> |
Cc: | Maren Purves <[email protected]>, "[email protected]" <[email protected]> |
Date: | Thu, 29 Mar 2007 09:58:54 -1000 (HST) |
the short answer is: you don't. As this is a telescope data acquisition happens at night (well, unless you test stuff during the day, which sometimes happens a lot and sometimes not at all). Users (generally) don't access these data other than for quick-look display (which is automated - and, data are first written to disk raw and then read in again and demultiplexed. The raw data are generally more than 100GB per night per array). The data get written to tape (BTW, we do have mirrored disks, so the data is being accessed and copied as it is acquired) by a cron job during the dat, and taking data at the time the cron job runs is a bit dicy, but it can be done. The cron job runs about 1/2 hour per day or so, so it doesn't take very long.
We used to have RAID arrays for the data but found they were too slow and throttling data acquisition at times.
Hi Maren,
Just curious to know how fast you can get data from the disk to the LTO 2 drives while saving more data to the disk, and serving yesterdays data to users from the disk...
Sounds like trouble to me.
Greg Lawson SNS Controls
On Thu, 2007-03-29 at 09:03 -1000, Maren Purves wrote:Kay,
(only responding to the data volume problem)
On Thu, 29 Mar 2007, Kay-Uwe Kasemir wrote:On Mar 29, 2007, at 02:04 , Terry Cornall wrote:I'm trying to dimension the archive requirements for all the 14 beamlines for the Australian Synchrotron and am wondering if I can do it with one PC or if I need many. Of course, sampling or monitoring rates will play a big part, but also I need to know what is possible per archive engine.
Problem 2: The amount of data. One 'double' sample uses a little over 20 bytes for the timestamp, status, sev, value. For many values, the data file structure and index add relatively little to that, but you'll still get about 20GB per day. How do you intend to back that up?
[...]
If somebody tells you that disks are cheap, please ask that person to take care of your archiving, then run as fast as you can.
you got a point, but I'm not sure that's what Terry asked.
And besides, with our wide field camera (motor control is EPICS, array control and data acquisition isn't) that has 4 arrays we get over 50GB per array in a good night with short integrations. With all 4 arrays we can get well over a Terabyte in a week.
Yes, it's an archiving problem, but it's entirely doable. LTO 2 drives in our case generally hold about 300 GB of compressed data (depends on how compressible your data are, that's why 'in our case').
Aloha, Maren