RE: soft IOC string and array records

["Jeff Hill" <johill@lanl.gov>]

Hello Kurt,

With CA publish and subscribe we have a consumer producer situation. So if
the producer (the IOC) produces subscription updates at a sustained rate
that is higher than the rate at which the consumer (the client side
application) consumes them then we would need an infinite amount of memory
to save all of those updates in the server hoping that the client will
eventually consume at a higher rate. Of course, using an infinite amount of
memory could have a quite negative impact on the health of the IOC so we
implement some amount of buffering for bursts but choose to discard some
intermediate updates in situations where the buffer quota has been exceeded.
We of course ensure that a responsive client receives the most recent
update. There are actually five places where buffering can occur: there is
buffering in the event queue between records and the server, in the server's
protocol dispatch buffer, in the server's TCP socket, in the client
library's TCP socket, and in the client library's protocol dispatch buffer.

It's also true that the situation is subtly different for array and string
data types. For those types there is currently no buffering between the
record and the server. The implication is that if a higher priority record
processing thread calls db_post_events (if it posts a subscription update)
twice for these data types without giving up the CPU to the lower priority
server thread then an intermediate update will be discarded.

However, when one considers that when designing a reliable distributed
system we must always assume that client consumption rates might be less
than server production rates given the many loading variables in a system
maintained over a number of years on evolving hardware, and the bursty
nature of systems based on asynchronous network messages, then perhaps it is
prudent to not create a system that is independent of the depth of buffering
provided by the communication infrastructure.

Jeff

> -----Original Message-----
> From: Kurt Biery [mailto:biery@fnal.gov]
> Sent: Friday, January 05, 2007 1:06 PM
> To: tech-talk@aps.anl.gov
> Subject: soft IOC string and array records
> 
> At Fermilab, several of us are working on the data acquisition for NOvA,
> a future neutrino experiment.  We would like to use process variables in
> soft IOCs to pass string messages between processes in the system, and
> we are trying to understand the best way to accomplish this.
> 
> With EPICS Base R3.14.8.2, we have experimented with string and waveform
> records and have noticed that messages are not always delivered to
> monitor processes when the messages are quickly written to the PV.  I'll
> include some of the details of our tests below, but the central issue
> seems to be illustrated by the following snippet of code from
> db_post_single_event_private in dbEvent.c:
> 
>     /*
>      * if we have an event on the queue and we are
>      * not saving the current value (because this is a
>      * string or an array) then ignore duplicate
>      * events (saving them without the current value
>      * serves no purpose)
>      */
>     if (!event->valque && event->npend>0u) {
>         UNLOCKEVQUE(ev_que)
>         return;
>     }
> 
> I don't understand all of the ramifications of this code comment, but it
> certainly sounds like fast updates to string and array records are not
> all expected to be delivered.  So, the question becomes:  if we want
> updates to a string or waveform record (or something similar) to
> *always* be delivered to monitors, what are our options?  We have
> casually used CA servers when first learning EPICS, but have not yet
> investigated whether they have this same feature.  Are CA servers an
> option?  Or are there established ways to select the delivery of all
> updates in a soft IOC?
> 
> The tests that we ran used a soft IOC created with 'makeBaseApp.pl -t
> ioc' and database records like the following:
> 
> record(stringout, "test1/sampleString") {
>     field(DTYP, "Soft Channel")
> }
> 
> record(waveform,"test1/sampleWaveform") {
>     field(DTYP,"Soft Channel")
>     field(NELM,"32")
>     field(FTVL,"UCHAR")
> }
> 
> To update the PV value at a reasonably high rate, we simply made a copy
> of caput.c and added a loop around the writing of the data as shown
> below (the modification of the first character in the sbuf/pbuf array
> allowed us to check for missing or duplicate records when observing the
> updates with a camonitor client):
> 
>     for (idx = 0; idx < 100; idx++) {
>         sprintf(sbuf[0], "%d", idx);
>         result = ca_array_put (dbrType, count, pvs[0].chid, pbuf);
>         result = ca_pend_io(caTimeout);
>         if (result == ECA_TIMEOUT) {
>             fprintf(stderr, "Write operation timed out: Data was not
> written.\n");
>             return 1;
>         }
>     }
> 
> For the waveform record test, we started the soft IOC, ran 'camonitor
> test1/sampleWaveform', and then ran the modified caput several times
> with 'caput -a test1/sampleWaveform 5 2 4 6 8 10'.  Some of the times
> that we ran caput, we saw all 100 messages arrive at the camonitor.
> Other times, only one or two of them did.
> 
> Thanks in advance for any help that folks can provide with this,
> Kurt Biery