EPICS Home

Experimental Physics and Industrial Control System


 
1994  1995  1996  1997  1998  <19992000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  <19992000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Channel Archiving
From: "Christopher A. Larrieu" <[email protected]>
Cc: [email protected]
Date: Wed, 17 Nov 1999 19:42:46 -0500
Hi all, 

  [the following is fairly long and perhaps tedious,
   so here's the executive summary]

  I'm working on a data logger which will run on an
  ioc, compress channel data in real-time, and stream 
  it over the network to a spooler.  I hope that the 
  resulting system can be generally useful to other 
  sites, and so need some small amount of information
  and/or comments.

  [end executive summary, but there's some more good 
   stuff at the bottom]

  (1) What is the fastest rate at which people want to
      log data from an ioc?

  (2) How much of this data is there?

  My preliminary investigation of how data behaves on
the ioc's here at JLAB indicates that, if we consider
each byte which comprises a signal's value as an
independent quantity, the probability that any particular
one of these will change in a ten minute period ranges
between 0.005 and 0.05 (this is sampling at 1 hz).  
Futhermore, when a byte does change, it does so in a 
remarkably structured manner  (see
http://www.sura.org/~larrieu/Info.html).  
>From an information theory standpoint, this means that 

  (1) the observation that a signal has not changed 
      carries very little information and can therefore 
      be conveyed with minimal overhead.

  (2) when a byte does change, it will probably change
      to one of the predictable values, so this too can 
      be encoded using very low bit rates.

  (3) when a byte changes to an unlikely value, then
      we need more bits to indicate such, though this
      happens relatively infrequently.

  Of course, when you know that a certain byte belongs
to a certain signal, then you know more about its 
behavior over time.  By monitoring this behavior we
can build a fairly accurate statistical model of it
so that we can further reduce the amount of information
required to indicate a change in value.

  The whole point here is that I believe we can achieve
dramatic results by compressing the channel data on the
ioc, then sending the result over the wire to a simple
spooling process.

  My intent is to:

  (1) implement this logging system
  (2) implement a client library for accessing the logged
      data, using Kay's C++ API.
  (3) build a new data extraction tool using (2)

  The benefits should be:

  (1) ability to store more information with less data
  (2) significantly reduced network traffic
  (3) circumvent CA size limit
  (4) compatible high-level interface, facilitating
      sharing of tools between participating labs.

  Trade-offs:

  (1) increased memory requirements on ioc
  (2) possibly significant CPU time on ioc
  (3) increased access time to archived data (decompress)
  (4) annoyance: another archiver?!
 
Chris
-- 
Christopher A. Larrieu
Jefferson Laboratory
(757) 269-5097


Replies:
RE: Channel Archiving Jeff Hill
References:
Re: Channel Archiving Bob Dalesio

Navigate by Date:
Prev: capfast symbol for transform and ANL motor record? Paul Stomski
Next: Re: capfast symbol for transform and ANL motor record? Rozelle Wright
Index: 1994  1995  1996  1997  1998  <19992000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Channel Archiving Bakul Banerjee
Next: RE: Channel Archiving Jeff Hill
Index: 1994  1995  1996  1997  1998  <19992000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024