Re: which time/timestamp format to use for an EPICS facility?

["Konrad, Martin" <konrad@frib.msu.edu>]

Emmanuel,

Thanks for sharing your thoughts.

> Possibly create a a small library/network service to convert from one to
> the other
I also thought about how to deploy the leap second information. All I 
need should be contained in the "Olson database" [1]. Right now I see 
multiple possible solutions:

1. Deploy and keep the tzdata package up to date via the package 
manager. This would be an easy task for our Debian machines but more 
difficult for other OS. This probably requires restarting the 
application after updating the tzdata package. I have to look into that.
2. Provide leap second and timezone information via a network service. 
Do the conversion on the IOCs.
3. Provide a service that converts timestamps before data is sent to GUI 
machines. Implementing this as part of the CA gateway would easily allow 
to use TAI on IOCs while using local time on GUI machines. But I'm still 
not sure this is a good idea: If someone writes down the time a problem 
occurred you might not know if this time is in TAI or local time. This 
probably should be avoided by using the same time for everything.

> For 2, you probably will have to use several timing sources.
Yes, for FRIB we will have
a) an Event Generator (Micro Research Finland EVG 230) + different EVRs 
for high precision applications
b) PTP for medium precision (jitter <=100us)
c) NTP for low precision (jitter <=50 ms)

> Do you know how this is solved in the archiver applications? (EPICS
> timestamps)
> Better yet, do you know how the archiver is used?
> I believe operators do not care so much about time accuracy.
Well I guess for a pulsed machine they might care. And we should not 
forget: This is not only about the operators but also about the 
scientists (EPICSv4 is moving into this direction). And they might care. 
But you're right: Usually people do not care about the absolute time of 
an event. Operators and physics users usually are interested in the 
order of events (operator: if failure B happened a short time after 
event A, A might be the cause of B; physicist: if a detector sees an 
event outside of a certain time window after a beam pulse hit the target 
it cannot be considered coincident and the event can be dropped).

> To look for events, operators use approximative time.
> "Give me the data for the  past 10 days"
> Then they find the event and zoom on it.
This works with TAI as well. So can I consider this sort of a vote for 
"TAI does the job as well"?

>  > But it will be difficult to tell which of two firmware files generated
>  > in different time zones is newer. And we probably do want to use the
>  > same time for everything...
>
> Given an infinite amount of money, time, and resource... yes. Otherwise,
> that's wishful thinking.
> In practice, you need to look at the timing requirement of each system.
Hardware and precision requirements might differ depending on the 
application. But I still want data acquisition, IOCs, and PLCs to be in 
sync. I definitely don't want the PLCs to use local time, the IOCs TAI, 
and firmware timestamps and logbook entries to be in UTC. Even if it is 
theoretically possible to convert times correctly I want to avoid it if 
it's not necessary (this might be error prone, some applications might 
convert while others do not, some might not say explicitly which time 
they are using). That's why I'm collecting thoughts about using TAI for 
everything.

> Are they issues with synchronization, accuracy,... what about jitter,
> drift, etc?
NTP and PTP will be synchronized to GPS, the EVG will be synchronized to 
GPS. Jitter and similar problems are perfectly under control. Our main 
problem right now seem to be leap seconds. Then PCs might be 1 s off, or 
correcting their clock speed at different rates. Some systems might have 
a second more than others [2]. I'm afraid we end up with a big mess and 
do not trust our timestamps...

> How reliable/trustworthy does you timing source needs to be?
> For accuracy and reliability, don't build an infrastructure on
> software/network based clocks, use hardware.
Using hardware clocks for all IOCs, FPGAs, and PLCs is way to expensive. 
As far as I can see there is nothing fundamentally wrong with a 
network/software based solution. It just needs to handle leap seconds right.

> GPS doesn't work indoor. How will you route your timing signals?
We are planning to use an outside GPS antenna for the timing master. 
Timestamps are distributed to our IOCs/PLCs from there via 
fiber/NTP/PTP. Signal routing is not a big issue for us since our 
requirements are not that strict. We do not need our time to be 
absolutely correct. We only need to make sure all systems are in sync 
(e.g. use the _same_ time +/- a few ms/us). But if we have some systems 
being off by a whole leap second that definitely has to be considered 
out of our specs...

> Are your operators interested in exact timing (where 16sec is an issue)?
They probably do not care much about _absolute_ time. I cannot see any 
real need for local time in the facility apart from shift hours. But 
using TAI/GPS time might still result in acceptance problems. Does 
anyone have experience with that?

> To me, this is a classic human computer interface issue.
> The high level software is designed for humans. (think productivity)
> The low level software is designed for machines. (think
> reliability/accuracy)
Unfortunately the POSIX timestamp (low level representation!) has been 
designed with the human machine interface in mind instead of making 
systems work reliably during leap seconds. Though it can be considered 
broken by design. Unfortunately EPICS has copied this behavior from 
POSIX systems. But it seems quite easy to fix that: Just replace "UTC" 
by "TAI" and you should be done since there will never be a leap second 
again. As far as I can see you do not even need to touch a single line 
of code. It's only about providing TAI via fiber/NTP/PTP.

> Why do you want your OS to have the same timing requirement as the data
> collected by your end-user applications?
> Use a standard NTP server, and when you collect your data use timestamps
> from another source (hardware?).
A good point. Using TAI (provided by fiber or PTP) only for the IOC 
application and sticking to good old NTP (UTC) for the OS might be a 
reasonable solution.

>  > What else do we need to keep in mind? Can anyone provide some experience
>  > with running a control system on a time that is different from local
>  > time/UTC?
>
> In our case, we have a control room which can "attach" to different
> machines in different time zones. (we are unique that way)
> Because our remote machines use their respective local time zone, we
> have to go through a few mental exercises of subtracting or adding hours
> in our remote archivers. It could have been implemented differently
> (force everyone to use a universal time), but we decided to put the
> burden on ourselves and not our users/clients/operators.
So the conversion of archive data is done by the database server? This 
means only the database server needs information about timezones. No 
need to have this information on all the clients. But where do you 
convert timestamps of data read directly from a PV?

Thanks a lot,

Martin

[1] http://en.wikipedia.org/wiki/Tz_database
[2] http://www.aps.anl.gov/epics/tech-talk/2012/msg01461.php

-- 
Martin Konrad
Control System Engineer
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: konrad@frib.msu.edu