Re: monitors received out of order

[Tim Mooney <mooney@aps.anl.gov>]

I have some new data on this problem:

My client has 16 queue blocks.  The PVs whose monitors are getting time
reversed are assigned to blocks 4 and 7, with the PV showing up early
assigned to block 4.  This fits the scenario Ralph outlined below.  (By
the way, I've verified that flowControlMode was never on during event
posting or event reading.)

For concreteness, here's the apparent time sequence that's causing trouble:
    record A posts DATA=0 to queue block 7
    record A causes record B to process
    record B posts DATA=0 to queue block 4
    event_task gets the CPU and calls the monitor routine with record B.DATA,
        then it calls the routine with record A.DATA.  My client thinks A.DATA
        was posted after B.DATA.

Under vxWorks, this doesn't happen, I suppose because the event task
always gets the CPU in the time between the two postings.  ("Always" is
pretty strong, but the consequence of time-reversed monitors in this case
is obvious enough and serious enough that I would have been notified,
and the software has been running for many years, on many IOCs.)

I have a work-around, but it's not very robust.  If I call ca_add_event
(ca_create_subscription) for the time-sensitive PVs *last*, most of them
get assigned to the same event-queue block, and the problem goes away.
This is partly luck, of course, because it doesn't guarantee that all the
PVs will be in the same block; it just makes it more likely.

I could ensure that the PVs go in the same block by using inside information
from dbEvent.c, but that doesn't seem like a very good idea.  I could also ensure
it by using a separate CA context for the time-sensitive PV's, but I would lose the
time ordering between these PVs and the others.  I'm pretty sure that would cause
other problems.  I could implement a handshake between the monitor routine
and the record, so that record A waited for the monitor to arrive before causing
record B to process, but that would be a little messy, and also add overhead that
a good same-block solution would not add.  I'm not sure what's the best way to go.

Tim

On 11/1/2010 4:27 PM, Ralph Lange wrote:
>  On 01.11.2010 17:09, Tim Mooney wrote:
> > Dear folks,
> >
> > I have two records, and a separate task monitoring a field that both records
> > post, all in the same IOC.  Most of the time, my task receives monitors in the
> > order in which they were posted.  But sometimes, the task receives a monitor
> > from record B before it receives a previously posted monitor from record A.
> > (I know for sure which record is posting first, because record A posts before
> > causing record B to process.  Also, I've modified the record to set its time
> > stamp immediately before posting, and I get another time stamp on entry to
> > the monitor routine.  The time ordering of those stamps does not agree.)
> > I've seen this on solaris and Linux, but not on vxWorks.
> >
> > I have code that misbehaves when this happens, so I started digging around
> > and have convinced myself that I should not be relying on the time ordering of
> > monitors received from different records (even when those records are running
> > in the same task).  I now think I can rely on posts from a single record 
> > arriving in
> > time order, but not posts from different records.  I think this because 
> > events from
> > different record go into different queues, and there doesn't seem to be any code
> > in the vicinity that seems worried about time ordering across event queues.
> >
> > Am I right about this?
>
> Short answer: yes.
>
> The event queue is allocated in chunks. Each chunk has room for a number of 
> subscriptions, and holds the values that are on the queue for these 
> subscriptions.
> The event task (per client) simply loops over the event queue chunks for this 
> client (lines ~940 in dbEvent.c), and calls event_read() for each chunk, which 
> loops over the queue entries in a chunk (line ~815) and pushes everything it 
> finds there to the CA server (via user_sub, line ~865).
> Depending on which subscriptions from which records are held in which event 
> queue chunk, the order on the network may vary.
>
> You are almost correct: The only order that you can rely on is the order of 
> updates for a single subscription. (Two subscriptions for different fields of 
> the same record may end up in different event queue chunks and get sent 
> out-of-order.)
>
> Ralph

--
Tim Mooney (mooney@aps.anl.gov) (630)252-5417
Software Services Group, Advanced Photon Source, Argonne National Lab.