Re: Timestamp reset event in EVR

[sdcswd via Tech-talk <tech-talk at aps.anl.gov>]

Hi Michael,

> Can you clarify "lost"?  Do you seen any messages being logged?
> Are either the hardware or software FIFO overflow counters incrementing when this happens?

Actually I am logging all the event codes and timestamp using my EVR to diagnose the timing system. I did not check two overflow PV values before, but I think these two values did not change. The 1PPS input signal is used in EVG to send 125. 12 event codes are sent in 5 ms and then repeat every 20 ms. The minimal interval is 1 tick.

Anyway, since EVG works fine as well as other EVRs. I think we can exclude the hardware issue.

Here is the situation in my EVR which I am guessing:

When 125 is received, the two timestamp registers will be reset. During that period, next event code is received in FIFO and will call below statement to read timestamp. If two registers are still changing, READ32() might fail.

events[evt].last_sec=READ32(base, EvtFIFOSec); 
events[evt].last_evt=READ32(base, EvtFIFOEvt);

So how long does it take to reset the register? Does this process time affected by the second_tick() function or high CPU load? During the reset, is there any lock to restrict the reading value? Or perhaps directly return a value 0? 


Regards,
Di WANG
Linac Control Group, KEK

> On Reiwa 2 Mar 18, at 6:07, Michael Davidsaver <mdavidsaver at gmail.com> wrote:
> 
> On 3/17/20 9:13 AM, Di Wang wrote:
>> Hi Michael,
>> 
>>   I found an event lost situation in my MRF VME-EVR-230RF,
> 
> Can you clarify "lost"?  Do you seen any messages being logged?
> Are either the hardware or software FIFO overflow counters incrementing
> when this happens?
> 
> Also, have you verified, by configuring an LED or output, that
> event 125 is actually being received?
> 
>> so I am wondering how will mrfioc2 handle the event received right after the timestamp reset event (i.e. 125) ?
> 
> This should not make a difference as far as software is concerned.
> The EVRMRM::seconds_tick method you link below is executed here:
> 
> https://github.com/epics-modules/mrfioc2/blob/7bee26590cb5fea8117d0a12920bc281ab1c7cd8/evrMrmApp/src/drvem.cpp#L1317
> 
> So it is executed by the thread servicing the hardware event FIFO.
> However, it doesn't alter the entries being read.  It only acts to
> cross-check the new seconds value to make sure that is has incremented
> by one from the previous value.  Allowing corrupt timestamps into
> generalTime has been known to cause strange issues.
> 
>> We might send event code only 1 tick (i.e. 9 nanosecond) later than before one, so the 'right after' could be very fast. Here is a brief example.
>> 
>> Event       Timestamp
>> 1           953053925.999,917,620
>> 125       953053925.999,977,626  <-----TS reset event
>> 2 <-----lost, and the timestamp should be 953053926.999,977,650
>> 3            953053926.000,100,043
>> 4            953053926.000,450,210
>> 
>> 
>>   I understand that when TS reset event is received the 32-bit seconds register will be reset, and below callback function would be called, but I still do not understand the process logic and the reason of event lost.
> 
> Can I assume that the source of event 125 is asynchronous to the others?
> That is, a real GPS receiver, as opposed to some kind of loopback for testing?
> 
> Are there any other event codes which are going over the link?
> "120 events" is mentioned below, though the more important question
> is how frequently these are sent, and how many asynchronous sources
> are involved.
> 
> One explanation would be if the event link was too busy, and events
> were being lost at the EVG due to the limited buffering at the
> priority encoder.  Unfortunately, I don't believe there will be any
> indication if this is happening other than missing events.
> 
> 
>> https://github.com/epics-modules/mrfioc2/blob/7bee26590cb5fea8117d0a12920bc281ab1c7cd8/evrMrmApp/src/drvem.cpp#L1430
>> 
>>   For you convenience, some information might help:
>> epics 3.15.5, devlib 2-2.10, mrfioc 2-2.2.0 (I added several functions to buffer the events in drvem.cpp)
>> event clock: 114 MHz, repetition rate: 50 Hz, more than 120 events are monitored
>> 
>> Regards,
>> Di WANG
>> Linac Control Group, KEK
>> 
>