Re: Channel Archiving

["Christopher A. Larrieu" <larrieu@jlab.org>]

Hi all, 

  [the following is fairly long and perhaps tedious,
   so here's the executive summary]

  I'm working on a data logger which will run on an
  ioc, compress channel data in real-time, and stream 
  it over the network to a spooler.  I hope that the 
  resulting system can be generally useful to other 
  sites, and so need some small amount of information
  and/or comments.

  [end executive summary, but there's some more good 
   stuff at the bottom]

  (1) What is the fastest rate at which people want to
      log data from an ioc?

  (2) How much of this data is there?

  My preliminary investigation of how data behaves on
the ioc's here at JLAB indicates that, if we consider
each byte which comprises a signal's value as an
independent quantity, the probability that any particular
one of these will change in a ten minute period ranges
between 0.005 and 0.05 (this is sampling at 1 hz).  
Futhermore, when a byte does change, it does so in a 
remarkably structured manner  (see
http://www.sura.org/~larrieu/Info.html).  
>From an information theory standpoint, this means that 

  (1) the observation that a signal has not changed 
      carries very little information and can therefore 
      be conveyed with minimal overhead.

  (2) when a byte does change, it will probably change
      to one of the predictable values, so this too can 
      be encoded using very low bit rates.

  (3) when a byte changes to an unlikely value, then
      we need more bits to indicate such, though this
      happens relatively infrequently.

  Of course, when you know that a certain byte belongs
to a certain signal, then you know more about its 
behavior over time.  By monitoring this behavior we
can build a fairly accurate statistical model of it
so that we can further reduce the amount of information
required to indicate a change in value.

  The whole point here is that I believe we can achieve
dramatic results by compressing the channel data on the
ioc, then sending the result over the wire to a simple
spooling process.

  My intent is to:

  (1) implement this logging system
  (2) implement a client library for accessing the logged
      data, using Kay's C++ API.
  (3) build a new data extraction tool using (2)

  The benefits should be:

  (1) ability to store more information with less data
  (2) significantly reduced network traffic
  (3) circumvent CA size limit
  (4) compatible high-level interface, facilitating
      sharing of tools between participating labs.

  Trade-offs:

  (1) increased memory requirements on ioc
  (2) possibly significant CPU time on ioc
  (3) increased access time to archived data (decompress)
  (4) annoyance: another archiver?!
 
Chris
-- 
Christopher A. Larrieu
Jefferson Laboratory
(757) 269-5097