Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Wed, 18 Mar 2020 08:53:07 -0700

On 3/18/20 12:42 AM, sdcswd wrote:
> Hi Michael,
> 
>> Can you clarify "lost"?  Do you seen any messages being logged?
>> Are either the hardware or software FIFO overflow counters incrementing when this happens?
> 
> Actually I am logging all the event codes and timestamp using my EVR to diagnose the timing system. I did not check two overflow PV values before, but I think these two values did not change. The 1PPS input signal is used in EVG to send 125. 12 event codes are sent in 5 ms and then repeat every 20 ms.

So the event rate going into the EVR's FIFO is 2400 events/sec.
FIFO servicing should be able to keep up with this easily, though
it would normally exceed the software rate limit and discard some.
Previously you mention:

> (I added several functions to buffer the events in drvem.cpp)

Can this be made available for community use?  I know there are others
who would be interested in this feature.

> The minimal interval is 1 tick.

How many events have this separation?

If EVG configuration results in several sub-units needing to send events
on the same clock tick, a priority encoder adjudicates by delaying all
until an otherwise idle tick.  This delay is accomplished by a short
FIFO, which can overflow if a higher priority sub-unit is always sending.

The priority of sub-units (again as I recall) is fixed from highest to lowest as:

MXC0
MXC1
MXC2
MXC3
Seqeuncer0
Seqeuncer1
MXC4
MXC5
MXC6
MXC7

If is for this reason that I highly recommend using MXC7 for event 125
as this is (usually) least essential than the two sequencers.  Though
careful design, with the priority encoder behavior in mind, can avoid
the possibility of collisions leading to lost events.

> Anyway, since EVG works fine as well as other EVRs. I think we can exclude the hardware issue.

I'm not convinced of this.

> Here is the situation in my EVR which I am guessing:
> 
> When 125 is received, the two timestamp registers will be reset. During that period, next event code is received in FIFO and will call below statement to read timestamp. If two registers are still changing, READ32() might fail.
> 
> events[evt].last_sec=READ32(base, EvtFIFOSec); 
> events[evt].last_evt=READ32(base, EvtFIFOEvt);
> 
> So how long does it take to reset the register? Does this process time affected by the second_tick() function or high CPU load? During the reset, is there any lock to restrict the reading value? Or perhaps directly return a value 0? 

I do not think this code can fail in the way you describe.

https://github.com/epics-modules/mrfioc2/blob/7bee26590cb5fea8117d0a12920bc281ab1c7cd8/evrMrmApp/src/drvem.cpp#L1286-L1305

From the MRF manual concerning these registers:

> Note that reading the FIFO event code registers pulls the event code and timestamp/seconds value from the
> FIFO for access. The correct order to read an event from FIFO is to first read the event code register and
> after this the timestamp/seconds registers in any order. Every read access to the FIFO event register pulls a
> new event from the FIFO if it is not empty.

DCManual-180626.pdf section 3.8.13 Event FIFO

So values of the EvtFIFOSec and EvtFIFOEvt registers are stable after the earlier read
read of EvtFIFOCode.

Your concern and confusion is a good example of why registers with read side-effects
should imo. generally be avoided (to any firmware designers reading).

> Regards,
> Di WANG
> Linac Control Group, KEK
> 
>> On Reiwa 2 Mar 18, at 6:07, Michael Davidsaver <mdavidsaver at gmail.com> wrote:
>>
>> On 3/17/20 9:13 AM, Di Wang wrote:
>>> Hi Michael,
>>>
>>>   I found an event lost situation in my MRF VME-EVR-230RF,
>>
>> Can you clarify "lost"?  Do you seen any messages being logged?
>> Are either the hardware or software FIFO overflow counters incrementing
>> when this happens?
>>
>> Also, have you verified, by configuring an LED or output, that
>> event 125 is actually being received?
>>
>>> so I am wondering how will mrfioc2 handle the event received right after the timestamp reset event (i.e. 125) ?
>>
>> This should not make a difference as far as software is concerned.
>> The EVRMRM::seconds_tick method you link below is executed here:
>>
>> https://github.com/epics-modules/mrfioc2/blob/7bee26590cb5fea8117d0a12920bc281ab1c7cd8/evrMrmApp/src/drvem.cpp#L1317
>>
>> So it is executed by the thread servicing the hardware event FIFO.
>> However, it doesn't alter the entries being read.  It only acts to
>> cross-check the new seconds value to make sure that is has incremented
>> by one from the previous value.  Allowing corrupt timestamps into
>> generalTime has been known to cause strange issues.
>>
>>> We might send event code only 1 tick (i.e. 9 nanosecond) later than before one, so the 'right after' could be very fast. Here is a brief example.
>>>
>>> Event       Timestamp
>>> 1           953053925.999,917,620
>>> 125       953053925.999,977,626  <-----TS reset event
>>> 2 <-----lost, and the timestamp should be 953053926.999,977,650
>>> 3            953053926.000,100,043
>>> 4            953053926.000,450,210
>>>
>>>
>>>   I understand that when TS reset event is received the 32-bit seconds register will be reset, and below callback function would be called, but I still do not understand the process logic and the reason of event lost.
>>
>> Can I assume that the source of event 125 is asynchronous to the others?
>> That is, a real GPS receiver, as opposed to some kind of loopback for testing?
>>
>> Are there any other event codes which are going over the link?
>> "120 events" is mentioned below, though the more important question
>> is how frequently these are sent, and how many asynchronous sources
>> are involved.
>>
>> One explanation would be if the event link was too busy, and events
>> were being lost at the EVG due to the limited buffering at the
>> priority encoder.  Unfortunately, I don't believe there will be any
>> indication if this is happening other than missing events.
>>
>>
>>> https://github.com/epics-modules/mrfioc2/blob/7bee26590cb5fea8117d0a12920bc281ab1c7cd8/evrMrmApp/src/drvem.cpp#L1430
>>>
>>>   For you convenience, some information might help:
>>> epics 3.15.5, devlib 2-2.10, mrfioc 2-2.2.0 (I added several functions to buffer the events in drvem.cpp)
>>> event clock: 114 MHz, repetition rate: 50 Hz, more than 120 events are monitored
>>>
>>> Regards,
>>> Di WANG
>>> Linac Control Group, KEK
>>>
>>
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System