RE: Weird IOC behavior with NTP time shift

[Abdalla Ahmad via Tech-talk <tech-talk@aps.anl.gov>]

Hi Michael

Thank you that was quite helpful. I still did not understand the relation that time counter need to be monotonic. You mentioned you are working on a true monotonic time function, doesn't that will have the same behavior? Or in other words, what do you mean by true monotonic time source?

Best Regards,
Abdalla.

-----Original Message-----
From: Michael Davidsaver [mailto:mdavidsaver@gmail.com] 
Sent: Monday, December 24, 2018 6:08 AM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Cc: tech-talk@aps.anl.gov
Subject: Re: Weird IOC behavior with NTP time shift

On 12/23/18 5:55 AM, Abdalla Ahmad via Tech-talk wrote:
> Hi
> 
>  
> 
> This issue happened with base 3.14.12.3 and 3.15.6. What happened is that the IT group upgraded their MS windows active directory servers, one of the included services in the update is the NTP. When all the servers rebooted, the NTP service did not start and caused a 3-hour time shift forward in time on all connected PCs and hosts.
> 
>  
> 
> On the IOC side, when the shift happened the IOC date command returns the host's date/time. But when the problem was resolved by the IT group, the time got synchronized and went 3-hour backward in time. At this moment the date command returns the same timestamp, i.e., a frozen time stamp. We had to reboot every IOC to get the machine working.
> 
>  
> 
> My question is, is there any explanation for this behavior and any 
> potential way to avoid it? Also on daylight saving times, we can't 
> recall this issue happening on a DST switch before most probably 
> because the machine was OFF, will DST switch give the same effect? The 
> issue is reproducible with any IOC, whenever the IOC's timestamp is 
> ahead of the host's time stamp, the IOC's timestamp will get frozen,

What you describe is a known issue with IOC time handling which arises from the (imo contradictory) requirement that epicsTimeGetCurrent() be monotonic.
The result is that after real time jumps backwards* calls to
epicsTimeGetCurrent() will return the same value until the OS clock "catches up" with it's previous value.

https://github.com/epics-base/epics-base/blob/5f46d6dceed6c10c43ed5db0543979d696f1becd/src/libCom/osi/epicsGeneralTime.c#L162

As you've found, epicsTimeGetCurrent() is used internally in several places, most notably in timer queues.  So the effects are quite noticeable...

https://bugs.launchpad.net/epics-base/+bug/1392516

I have a (very) long term plan to switch the timer queue and similar to a true monotonic time source.  Such a time source is available as of Base 7.0.1 (cf. epicsMonotonicGet() ), so the next step is to replace usages of epicsTimeGetCurrent() and epicsTime::getCurrent().


* Daylight savings time isn't implemented as a jump of the seconds counter.
  Rather it is a change in the mapping between the seconds counter and
  calendar/time of day.