Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <[email protected]> · Mon, 24 Dec 2018 10:21:55 -0800

On 12/24/18 12:18 AM, Abdalla  Ahmad wrote:
> Hi Michael
> 
> Thank you that was quite helpful. I still did not understand the relation that time counter need to be monotonic.

This is a useful property as most uses of time in an IOC (and computers in general)
are only to compute a time interval.  eg. an inactivity timeout on a socket, or
a period scan rate.  There isn't much benefit to having such intervals include
administrator changes to the system time.

> You mentioned you are working on a true monotonic time function, doesn't that will have the same behavior? Or in other words, what do you mean by true monotonic time source?

The key point is that a monotonic time source won't be effected by administrator
changes to the system clock.

To quote from "man clock_gettime":

> CLOCK_REALTIME
>               System-wide clock that measures real (i.e., wall-clock) time.  Setting this clock requires appropriate privileges.  This clock
>               is affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the clock),  and  by
>               the incremental adjustments performed by adjtime(3) and NTP.

> CLOCK_MONOTONIC
>               Clock that cannot be set and represents monotonic time since some unspecified starting point.  This clock is not  affected  by
>               discontinuous  jumps in the system time (e.g., if the system administrator manually changes the clock), but is affected by the
>               incremental adjustments performed by adjtime(3) and NTP.

http://man7.org/linux/man-pages/man2/clock_gettime.2.html

On windows QueryPerformanceCounter() is used:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx

> Best Regards,
> Abdalla.
> 
> -----Original Message-----
> From: Michael Davidsaver [mailto:[email protected]] 
> Sent: Monday, December 24, 2018 6:08 AM
> To: Abdalla Ahmad <[email protected]>
> Cc: [email protected]
> Subject: Re: Weird IOC behavior with NTP time shift
> 
> On 12/23/18 5:55 AM, Abdalla Ahmad via Tech-talk wrote:
>> Hi
>>
>>  
>>
>> This issue happened with base 3.14.12.3 and 3.15.6. What happened is that the IT group upgraded their MS windows active directory servers, one of the included services in the update is the NTP. When all the servers rebooted, the NTP service did not start and caused a 3-hour time shift forward in time on all connected PCs and hosts.
>>
>>  
>>
>> On the IOC side, when the shift happened the IOC date command returns the host's date/time. But when the problem was resolved by the IT group, the time got synchronized and went 3-hour backward in time. At this moment the date command returns the same timestamp, i.e., a frozen time stamp. We had to reboot every IOC to get the machine working.
>>
>>  
>>
>> My question is, is there any explanation for this behavior and any 
>> potential way to avoid it? Also on daylight saving times, we can't 
>> recall this issue happening on a DST switch before most probably 
>> because the machine was OFF, will DST switch give the same effect? The 
>> issue is reproducible with any IOC, whenever the IOC's timestamp is 
>> ahead of the host's time stamp, the IOC's timestamp will get frozen,
> 
> What you describe is a known issue with IOC time handling which arises from the (imo contradictory) requirement that epicsTimeGetCurrent() be monotonic.
> The result is that after real time jumps backwards* calls to
> epicsTimeGetCurrent() will return the same value until the OS clock "catches up" with it's previous value.
> 
> https://github.com/epics-base/epics-base/blob/5f46d6dceed6c10c43ed5db0543979d696f1becd/src/libCom/osi/epicsGeneralTime.c#L162
> 
> As you've found, epicsTimeGetCurrent() is used internally in several places, most notably in timer queues.  So the effects are quite noticeable...
> 
> https://bugs.launchpad.net/epics-base/+bug/1392516
> 
> I have a (very) long term plan to switch the timer queue and similar to a true monotonic time source.  Such a time source is available as of Base 7.0.1 (cf. epicsMonotonicGet() ), so the next step is to replace usages of epicsTimeGetCurrent() and epicsTime::getCurrent().
> 
> 
> * Daylight savings time isn't implemented as a jump of the seconds counter.
>   Rather it is a change in the mapping between the seconds counter and
>   calendar/time of day.
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System