1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 <2014> 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 <2014> 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Re: Increasing scan rate to 10 kHz |
From: | Till Straumann <[email protected]> |
To: | Andrew Johnson <[email protected]> |
Cc: | EPICS Tech Talk <[email protected]> |
Date: | Wed, 08 Jan 2014 09:58:47 -0800 |
Andrew. I agree with you that fast EPICS scan rates are probably not going to be reliable but I wanted to make the point that the reason is not the underlying OS. A RTOS (including linux/RT_PREEMPT) on a modern machine can reasonably schedule multi-kHz tasks with worst-case latency/jitter in the order of some low tens of microseconds. If you have a hard real-time job then you can do this in a dedicated high-priority thread and then e.g., feed cooked and timestamped data into EPICS records. I have used this approach for years now (and yes, I have stuff running at several kHz) in multiple applications and it works as expected. - Till On 01/08/2014 09:37 AM, Andrew Johnson wrote:
Hi Till, The periodic scan threads use epicsEventWaitWithTimeout() to wait for their next tick — we need to be able to wake up those threads on demand so the IOC doesn't have to spend up to 10 seconds (or whatever is the longest scan period) waiting for those threads to shut down when the user types exit. On the Posix architectures the wait occurs in pthread_cond_timedwait(), on RTEMS in rtems_semaphore_obtain(), on VxWorks in semTake(), and on Windows in WaitForSingleObject(). The Posix API provides nanosecond resolution, and even the ability to choose which clock to use although we don't set that ourselves and that ability may not be supported on all architectures. Note that our Posix code runs on MacOS and Solaris as well as Linux. For RTEMS and VxWorks the wait time is expressed in system clock ticks. Increasing the tick rate to use a finer time resolution results in more CPU time spent handling the tick interrupt, as Dirk has shown; that is a trade-off the user can make though. On Windows the delay time is given in milliseconds so the maximum scan rate there would be 1KHz, assuming that the OS could cope. Personally I think that trying to support scan rates faster than about 1KHz at most is never going to be reliable and probably isn't worth spending much time on, but I'm always open to code contributions that prove me wrong (as long as they're sufficiently portable). - Andrew On 01/08/2014 10:36 AM, Till Straumann wrote:It should be noted that with RT_PREEMPT the real-time threads all execute in *user-mode*. They can run side-by-side with other low-priority stuff in the same process and still get scheduled as needed, preempting any other (lower-priority) thread or process with low latency. Hence it is not necessary to play special tricks with putting high-priority code into kernel modules etc. - Till On 01/08/2014 08:20 AM, Till Straumann wrote:On 01/07/2014 02:19 PM, Andrew Johnson wrote:Periodic scans are quite complicated, since they rely on the underlying operating system to implement a delay timer and reschedule the scan thread after the specific delay has elapsed. Most OSs don't provide high accuracy delay scheduling, so I don't expect we will ever be able to run periodic scan threads at 10KHz. - AndrewAs a matter of fact I don't think that 10kHz periodic scans are a big problem for something like RTEMS or linux (with RT_PREEMPT) on a reasonably fast CPU. I just used a simple 'clock_nanosleep' based loop (based on code by windriver, modified by myself) which does something like while ( 1 ) { /* get current time */ clock_gettime( CLOCK_MONOTONIC, &wakeup ); /* compute wakeup time 100us from now */ timespec_add_100us( &wakeup ); /* sleep until wakeup time */ clock_nanosleep( CLOCK_MONOTONIC, TIMER_ABSTIME, &wakeup, NULL ); /* see when we actually woke up */ clock_gettime( CLOCK_MONOTONIC, &now ); /* missed = now - wakeup */ timespec_diff( &missed, &now, &wakeup ); compute_statistics( &missed ); } for a bunch of threads (using the SCHED_FIFO scheduler). On my laptop (vanilla kernel, no RT_PREEMPT) I observe worst-case 'missed' deadline of several milli-seconds but even on that system the running average is between 5us..50us. On a kernel patched with RT_PREEMPT I get a worst-case missed deadline by ~10us (average around 3-4us) after running for a few minutes (on a not heavily loaded intel core i7-2655LE, 2.2 GHz). I would expect RTEMS/vxWorks to even perform a bit better. - Till