Experimental Physics and
Industrial Control System

Di Maio Franck <[email protected]> · Mon, 9 Dec 2013 13:43:30 +0000

Hello Benoit

Actually, we are suppressing POSIX UTC from timing boards' drivers only & convert to UTC for applications (inc. EPICS records).
Here is the change from issue tracking system:
1) suppress conversion in driver, use only the 64b int representation in kernel mode and stay with time in TAI in the driver (and in libs when expressed as a 64b int).
2) implement conversions in user library between the POSIX struct in UTC and the 64b int in TAI.
Not sure it's the final solution but in our current Linux (RHEL 6.3), using something else that UTC is very confusing.
Maybe with newer kernels, it's already better (changes in Kernel 3.10 mentioned).

For conversion, we have imported the scitime library.

The leap seconds list is from one of the web sites providing it for NTP:
ftp://time.nist.gov/pub/leap-seconds.list
At the moment, it's in the library. We'll set-up a mechanism for pushing it from distribution servers to end-user systems.

For instantaneous conversion (we don't do that), the current leap second count is distributed by the PTP GMC.

Cheers,
Franck

From: [email protected] [mailto:[email protected]] On Behalf Of Benoit
Sent: 09 December 2013 12:15
To: Di Maio Franck
Cc: EPICS Tech Talk
Subject: Re: which time/timestamp format to use for an EPICS facility?

Hi Franck,

I am happy to see that we are doing almost the same.
. Could you give more details on how you proceed to distribute the leap seconds table on the network.
. You mention "conversion into POSIX struct in UTC in user library only": could you tell me exactly in which layer you mean by user library. We only do this in CSS/BOY layer and use timestamp PV in TAI format. 
Cheers,

--
Benoit RAT
www.neub.co.nr

On Mon, Dec 9, 2013 at 8:22 AM, Di Maio Franck <[email protected]> wrote:
Hi All

Short contribution from ITER...
We are using PTP for the timing network with time in TAI.
We import the leap seconds table from the network for conversions.
After some variations, we are standardizing on having time in TAI expressed as a 64 bit int (ns) in low-level software (ex: drivers) and conversion into POSIX struct in UTC in user library only.
System time is also set to UTC.
I am not aware of a POSIX format for TAI. (?)
Not using TAI in operational data will lead to not allowing scientific operation during a leap second.
Timing stamping the data in TAI seems the safer option with conversion at the last moment when required.

Franck

Franck DI MAIO
CODAC System Designer
CODAC Section

ITER Organization, Building 72/279, CHD, Control System Division
Route de Vinon sur Verdon - 13115 St Paul Lez Durance - France
Phone: +33 4 42 17 64 05
Get the latest ITER news on http://www.iter.org/newsline/latest
-----Original Message-----
From: Konrad, Martin [mailto:[email protected]]
Sent: 06 December 2013 17:15
To: EPICS Tech Talk
Subject: Re: which time/timestamp format to use for an EPICS facility?
Emmanuel,

Thanks for sharing your thoughts.

> Possibly create a a small library/network service to convert from one
> to the other
I also thought about how to deploy the leap second information. All I need should be contained in the "Olson database" [1]. Right now I see multiple possible solutions:

1. Deploy and keep the tzdata package up to date via the package manager. This would be an easy task for our Debian machines but more difficult for other OS. This probably requires restarting the application after updating the tzdata package. I have to look into that.
2. Provide leap second and timezone information via a network service.
Do the conversion on the IOCs.
3. Provide a service that converts timestamps before data is sent to GUI machines. Implementing this as part of the CA gateway would easily allow to use TAI on IOCs while using local time on GUI machines. But I'm still not sure this is a good idea: If someone writes down the time a problem occurred you might not know if this time is in TAI or local time. This probably should be avoided by using the same time for everything.

> For 2, you probably will have to use several timing sources.
Yes, for FRIB we will have
a) an Event Generator (Micro Research Finland EVG 230) + different EVRs for high precision applications
b) PTP for medium precision (jitter <=100us)
c) NTP for low precision (jitter <=50 ms)

> Do you know how this is solved in the archiver applications? (EPICS
> timestamps)
> Better yet, do you know how the archiver is used?
> I believe operators do not care so much about time accuracy.
Well I guess for a pulsed machine they might care. And we should not
forget: This is not only about the operators but also about the scientists (EPICSv4 is moving into this direction). And they might care.
But you're right: Usually people do not care about the absolute time of an event. Operators and physics users usually are interested in the order of events (operator: if failure B happened a short time after event A, A might be the cause of B; physicist: if a detector sees an event outside of a certain time window after a beam pulse hit the target it cannot be considered coincident and the event can be dropped).

> To look for events, operators use approximative time.
> "Give me the data for the  past 10 days"
> Then they find the event and zoom on it.
This works with TAI as well. So can I consider this sort of a vote for "TAI does the job as well"?

>  > But it will be difficult to tell which of two firmware files
> generated  > in different time zones is newer. And we probably do want
> to use the  > same time for everything...
>
> Given an infinite amount of money, time, and resource... yes.
> Otherwise, that's wishful thinking.
> In practice, you need to look at the timing requirement of each system.
Hardware and precision requirements might differ depending on the application. But I still want data acquisition, IOCs, and PLCs to be in sync. I definitely don't want the PLCs to use local time, the IOCs TAI, and firmware timestamps and logbook entries to be in UTC. Even if it is theoretically possible to convert times correctly I want to avoid it if it's not necessary (this might be error prone, some applications might convert while others do not, some might not say explicitly which time they are using). That's why I'm collecting thoughts about using TAI for everything.

> Are they issues with synchronization, accuracy,... what about jitter,
> drift, etc?
NTP and PTP will be synchronized to GPS, the EVG will be synchronized to GPS. Jitter and similar problems are perfectly under control. Our main problem right now seem to be leap seconds. Then PCs might be 1 s off, or correcting their clock speed at different rates. Some systems might have a second more than others [2]. I'm afraid we end up with a big mess and do not trust our timestamps...

> How reliable/trustworthy does you timing source needs to be?
> For accuracy and reliability, don't build an infrastructure on
> software/network based clocks, use hardware.
Using hardware clocks for all IOCs, FPGAs, and PLCs is way to expensive.
As far as I can see there is nothing fundamentally wrong with a network/software based solution. It just needs to handle leap seconds right.

> GPS doesn't work indoor. How will you route your timing signals?
We are planning to use an outside GPS antenna for the timing master.
Timestamps are distributed to our IOCs/PLCs from there via fiber/NTP/PTP. Signal routing is not a big issue for us since our requirements are not that strict. We do not need our time to be absolutely correct. We only need to make sure all systems are in sync (e.g. use the _same_ time +/- a few ms/us). But if we have some systems being off by a whole leap second that definitely has to be considered out of our specs...

> Are your operators interested in exact timing (where 16sec is an issue)?
They probably do not care much about _absolute_ time. I cannot see any real need for local time in the facility apart from shift hours. But using TAI/GPS time might still result in acceptance problems. Does anyone have experience with that?

> To me, this is a classic human computer interface issue.
> The high level software is designed for humans. (think productivity)
> The low level software is designed for machines. (think
> reliability/accuracy)
Unfortunately the POSIX timestamp (low level representation!) has been designed with the human machine interface in mind instead of making systems work reliably during leap seconds. Though it can be considered broken by design. Unfortunately EPICS has copied this behavior from POSIX systems. But it seems quite easy to fix that: Just replace "UTC"
by "TAI" and you should be done since there will never be a leap second again. As far as I can see you do not even need to touch a single line of code. It's only about providing TAI via fiber/NTP/PTP.

> Why do you want your OS to have the same timing requirement as the
> data collected by your end-user applications?
> Use a standard NTP server, and when you collect your data use
> timestamps from another source (hardware?).
A good point. Using TAI (provided by fiber or PTP) only for the IOC application and sticking to good old NTP (UTC) for the OS might be a reasonable solution.

>  > What else do we need to keep in mind? Can anyone provide some
> experience  > with running a control system on a time that is
> different from local  > time/UTC?
>
> In our case, we have a control room which can "attach" to different
> machines in different time zones. (we are unique that way) Because our
> remote machines use their respective local time zone, we have to go
> through a few mental exercises of subtracting or adding hours in our
> remote archivers. It could have been implemented differently (force
> everyone to use a universal time), but we decided to put the burden on
> ourselves and not our users/clients/operators.
So the conversion of archive data is done by the database server? This means only the database server needs information about timezones. No need to have this information on all the clients. But where do you convert timestamps of data read directly from a PV?

Thanks a lot,

Martin

[1] http://en.wikipedia.org/wiki/Tz_database
[2] http://www.aps.anl.gov/epics/tech-talk/2012/msg01461.php

--
Martin Konrad
Control System Engineer
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: [email protected]

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System