Re: Increasing scan rate to 10 kHz

[Till Straumann <strauman@slac.stanford.edu>]

Andrew.

I agree with you that fast EPICS scan rates are probably not
going to be reliable but I wanted to make the point that the
reason is not the underlying OS.

A RTOS (including linux/RT_PREEMPT) on a modern machine
can reasonably schedule multi-kHz tasks with worst-case
latency/jitter in the order of some low tens of microseconds.

If you have a hard real-time job then you can do this in a
dedicated high-priority thread and then e.g., feed cooked
and timestamped data into EPICS records.

I have used this approach for years now (and yes, I have
stuff running at several kHz) in multiple applications
and it works as expected.

- Till

On 01/08/2014 09:37 AM, Andrew Johnson wrote:
> Hi Till,
>
> The periodic scan threads use epicsEventWaitWithTimeout() to wait for
> their next tick — we need to be able to wake up those threads on demand
> so the IOC doesn't have to spend up to 10 seconds (or whatever is the
> longest scan period) waiting for those threads to shut down when the
> user types exit.
>
> On the Posix architectures the wait occurs in pthread_cond_timedwait(),
> on RTEMS in rtems_semaphore_obtain(), on VxWorks in semTake(), and on
> Windows in WaitForSingleObject(). The Posix API provides nanosecond
> resolution, and even the ability to choose which clock to use although
> we don't set that ourselves and that ability may not be supported on all
> architectures. Note that our Posix code runs on MacOS and Solaris as
> well as Linux.
>
> For RTEMS and VxWorks the wait time is expressed in system clock ticks.
> Increasing the tick rate to use a finer time resolution results in more
> CPU time spent handling the tick interrupt, as Dirk has shown; that is a
> trade-off the user can make though. On Windows the delay time is given
> in milliseconds so the maximum scan rate there would be 1KHz, assuming
> that the OS could cope.
>
> Personally I think that trying to support scan rates faster than about
> 1KHz at most is never going to be reliable and probably isn't worth
> spending much time on, but I'm always open to code contributions that
> prove me wrong (as long as they're sufficiently portable).
>
> - Andrew
>
> On 01/08/2014 10:36 AM, Till Straumann wrote:
> > It should be noted that with RT_PREEMPT the real-time threads
> > all execute in *user-mode*. They can run side-by-side with other
> > low-priority stuff in the same process and still get scheduled as
> > needed, preempting any other (lower-priority) thread or process
> > with low latency. Hence it is not necessary to play special tricks
> > with putting high-priority code into kernel modules etc.
> >
> > - Till
> >
> >
> > On 01/08/2014 08:20 AM, Till Straumann wrote:
> > > On 01/07/2014 02:19 PM, Andrew Johnson wrote:
> > > > Periodic scans are quite complicated, since they rely on the
> > > > underlying operating system to implement a delay timer and reschedule
> > > > the scan thread after the specific delay has elapsed. Most OSs don't
> > > > provide high accuracy delay scheduling, so I don't expect we will ever
> > > > be able to run periodic scan threads at 10KHz. - Andrew
> > > As a matter of fact I don't think that 10kHz periodic scans are a big
> > > problem for something like RTEMS
> > > or linux (with RT_PREEMPT) on a reasonably fast CPU.
> > >
> > > I just used a simple 'clock_nanosleep' based loop (based on code by
> > > windriver, modified by myself)
> > > which does something like
> > >
> > > while ( 1 ) {
> > >        /* get current time */
> > >          clock_gettime( CLOCK_MONOTONIC, &wakeup );
> > >        /* compute wakeup time 100us from now   */
> > >          timespec_add_100us( &wakeup );
> > >        /* sleep until wakeup time      */
> > >          clock_nanosleep( CLOCK_MONOTONIC, TIMER_ABSTIME, &wakeup, NULL );
> > >        /* see when we actually woke up */
> > >          clock_gettime( CLOCK_MONOTONIC, &now );
> > >        /* missed = now - wakeup        */
> > >          timespec_diff( &missed, &now, &wakeup );
> > >
> > >        compute_statistics( &missed );
> > > }
> > >
> > > for a bunch of threads (using the SCHED_FIFO scheduler).
> > >
> > > On my laptop (vanilla kernel, no RT_PREEMPT) I observe worst-case
> > > 'missed' deadline of several milli-seconds
> > > but even on that system the running average is between 5us..50us.
> > >
> > > On a kernel patched with RT_PREEMPT I get a worst-case missed deadline
> > > by ~10us (average around 3-4us)
> > > after running for a few minutes (on a not heavily loaded intel core
> > > i7-2655LE, 2.2 GHz).
> > >
> > > I would expect RTEMS/vxWorks to even perform a bit better.
> > >
> > > - Till