Thanks for everyone's response. 

Rebooting both of them ultimately got them synced again. We have plans of upgrading several of our IOCs to EPICS 3.15, so we will revisit this in the future. 

And yes Andrew, in exploring the many command line options, I ran "ClockTime_Shutdown" to see if stopping the thread would force a sync with NTP. 

From: [email protected] <[email protected]> on behalf of Johnson, Andrew N. via Tech-talk <[email protected]>
Sent: Wednesday, March 20, 2019 7:14 PM
To: [email protected]
Subject: Re: Change/Sync General Time Providers

Hi Kyle,

We have two IOCs, both mv2700s running EPICS r314123j0 that have OS Clock times in the future. Our NTP time is correct and it is set to a higher priority but is not syncing with the OS time. Is there a way to force synchronization of the two time providers at run-time? Or is there a way to forcibly set the current general time provider at run-time? Rebooting is an option, but I'm checking to see if there is a run-time solution. I've left some additional information below.

Unfortunately back in 3.14.12.3 the generalTime code had a forwards-only ratchet in its time synchronization routine, so if your OS clock runs fast it could never adjust it backwards with time from a higher priority NTP provider, which is obviously a problem. The fix for that particular bug landed 2¼ years ago in 3.14.12.6. I can't promise that an upgrade will completely fix your problem, but if your CPU's tick timer runs fast you're always going to have issues with it until you do upgrade.

In your case though your IOC isn't currently running the OS clock synchronization thread for some reason — did someone give it a "ClockTime_Shutdown" command? There is no way to restart that thread by hand so you'll have to reboot the IOC anyway, but you might want to investigate why it got stopped if you don't already know.

HTH,

- Andrew

IOC command line:

#####generalTimeReport#####

vxwrks@ioc -> generalTimeReport 5

Backwards time errors prevented 315503 times.

Current Time Providers:

    "NTP", priority = 100

Current Time is 2019-03-20 10:11:35.092046.

    "OS Clock", priority = 999

Current Time is 2064-12-01 07:21:34.754838.

Event Time Providers:

No Providers registered.

value = 0 = 0x0

#####NTPTime_Report#####

vxwrks@ioc ->  NTPTime_Report 2

NTP driver is synchronized with server

Syncronization interval = 60.0 seconds

Last synchronized at 2019-03-20 10:02:34.116603

OS tick rate = 60 Hz (nominal)

Measured tick rate = 59.997 Hz

NTP Server = XXX.XX.XXX.XX

value = 0 = 0x0

#####ClockTime_Report#####

vxwrks@ioc -> ClockTime_Report 2

OS Clock synchronization thread not running.

value = 0 = 0x0

PV Status: 

$(ioc):GTIM_CUR_SRC = OS Clock (***Is NTP on our functional IOCs***)

$(ioc):GTIM_HI_SRC = NTP

This is a strange one that seems to have come up before: https://epics.anl.gov/tech-talk/2017/msg01846.php

Any advice or suggestions would be much appreciated.

Thanks for your help,

Kyle

-- 
Arguing for surveillance because you have nothing to hide is no
different than making the claim, "I don't care about freedom of
speech because I have nothing to say." -- Edward Snowdon