Re: RTEMS+zoneset seems to be broken

[Matt Rippa <mrippa@gemini.edu>]

Hi Michael,

We use RTEMS 4.10.2. with EPICS release 3.14.12.6.

EPICS is compliled with EPICS_TIMEZONE=HST::600:0:0

inside base/configure/CONFIG_SITE_ENV.

Our experience booting IOC's lately have shown the timezone

after iocInit to be arbitrarily set to UTC or to our current timezone,

HST. Calls to zoneset() are also arbitrarily ignored. To be clear,

the zoneset() routine is not causing the problem, but up until lately

we thought it always correctly adjusted the timezone to anything we

set inside our startup script. Our race condition still exists without the

use of zoneset().

As Ricardo mentions, we have a custom time provider, the Bancomm

635. This driver gets GPS time (which is almost UTC) from its board

registers and proceeds making calls to mktime() for example. So your

links to the RTEMS time calls help.  Our issue must be in our driver.

Thanks for the help.

-Matt

On Sun, Oct 29, 2017 at 6:01 PM, Michael Davidsaver <mdavidsaver@gmail.com> wrote:

Hi Ricardo,

Forgive me for being dense, but what is the unexpected behavior that you
see?  That the zoneset() has no effect?  That the effect come and then
goes?  That the time has some random offset?  What if you run zoneset()
again?

From a quick test, with RTEMS 4.9.6 I can use zoneset() to change the
output of generalTimeReport() after initialization.

> qemu> generalTimeReport 5
> Backwards time errors prevented 7 times.
>
> Current Time Providers:
>     "EVR", priority = 50
>         Current Time not available
>     "NTP", priority = 100
>         Current Time is 2017-10-29 21:56:33.953748.
>     "OS Clock", priority = 999
>         Current Time is 2017-10-29 21:56:33.964115.
>
> Event Time Providers:
>     "EVR", priority = 50
>
> qemu> zoneset HST10HST10
> qemu> generalTimeReport 5
> Backwards time errors prevented 7 times.
>
> Current Time Providers:
>     "EVR", priority = 50
>         Current Time not available
>     "NTP", priority = 100
>         Current Time is 2017-10-29 17:56:47.362454.
>     "OS Clock", priority = 999
>         Current Time is 2017-10-29 17:56:47.372182.
>
> Event Time Providers:
>     "EVR", priority = 50



I can find two candidates for wierdness.  One in newlib which I don't
think is relevant as it's only used by eg. strftime(), mktime(), and
similar.  And one in RTEMS relating to the NTP client which may be if
you use DHCP/BOOTP for address assignment.

The first is __gettzinfo() which gives access to the global variable
where zone info is stored (there is an associated lock).  This is set by
tzset() and used only when time is decomposed.

https://github.com/epicsdeb/rtems-gcc-newlib/search?utf8=%E2%9C%93&q=__gettzinfo&type=

The second is rtems_bsdnet_timeoffset which appears to be picked up from
a dhcp/bootp server.  It's used when shifting NTP server time to posix
time (I'm not clear on why this is desirable).  If you use dynamic
addressing you might check if your server returns the 'time-offset' (#2)
option.

https://github.com/epicsdeb/rtems/search?utf8=%E2%9C%93&q=rtems_bsdnet_timeoffset&type=

On 10/27/2017 07:45 PM, Ricardo Cardenes wrote:
>
> Hi all,
>
> A year ago, Matt Rippa reached to the list[1] because he was having
> trouble setting the time zone at runtime on EPICS+RTEMS.
>
> To remind you the problem: the firmware for some of our VME boards'
> firmware is PPCBug bootloeader, and for some others it is MOTLoad.
>
> In the MOTLoad-based CPUs there's not much of a problem: we could just
> set epics-tz in MOTLoad the environment, and that will be picked-up by
> RTEMS when reading info from the NVRAM, defining in turn the environment
> variable TZ. This is not possible (or at least it is not implemented)
> for the PPCBug boards, meaning that we have to set the time zone in some
> other way. epicsEnvSet() would let us set the appropriate variables from
> the startup script, but at this point in the execution, there was no way
> to call tzset() to make RTEMS/newlib change the system-wide timezone.
>
> Michael Davidsaver proposed a patch[2] introducing "zoneset" as an iocsh
> function, precisely to be able to do that (set TZ, then call tzet()).
>
> It worked for us, but we've noticed an inconsistency: sometimes we get
> an unexpected behaviour, that we won't see in the next reboot.
>
> Reading rtems_init.c:Init (the beginning of _main_ thread) I've realized
> that most probably this is due to a race condition. The startup sequence
> (simplified) goes like this:
>
>   1) Read config from NVRAM
>   2) Set the priority for _main_
>   3) Init the console
>   4) Start the network and initialize the remote filesystem
>   5) Use the BSP supplied time of the day, if available
>   6) Set TZ (if needed) and call tzset

>   7) osdTimeRegister() (*creates the NTPTimeSync and ClockTimeSync threads*)
>   8) Execute the startup script
>   9) Call main()
>
> Now, up until 7), there are only 3 threads in the system (_main_,
> ImsgDaemon, errlog), but this step creates two new threads to
> synchronize time sources.
>
> My conclusion is that there is a race condition where the call to
> zoneset() (which would be the first in the startup script), does
> sometimes execute /before/ the sync threads get to set anything in the

> system; but sometimes the sync threads manage to do some significant
> work before the time zone is changed to the desired one, screwing up the
> internal view that the EPICS/RTEMS/newlib system has of the time (we use
> also use Bancomm bc635 as an additional time source in some of those
> systems, which may, or may not, be further compounding the issue).
>
> I'm not 100% sure of the exact effects of such a race condition, as I'm
> still going through the code trying to figure out all the implications.
>
> Our fix so far has been to use instead
> the epicsRtemsInitPostSetBootConfigFromNVRAM hook, which is not as
> flexible (needs to be compiled in, where zoneset goes in the startup
> script), but is 100% stable.
>
> Now... Is there any reasonable solution to this?
>
> Regards,
> Ricardo
>
> [1] http://www.aps.anl.gov/epics/tech-talk/2016/msg00720.php
> [2] http://www.aps.anl.gov/epics/tech-talk/2016/msg00733.php