Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: RE: Novice EPICS questions re yet another port
From: David Kelly <Kelly@CCUR.COM.AU>
To: "'Marty Kraimer'" <>
Cc:, "'Andrew Johnson'" <>, "'Eric Norum'" <>, "'Jeff Hill'" <>
Date: Wed, 10 Dec 2003 22:34:28 +1100
Thanks Marty.

I've read all the follow-up correspondence, especially those from yourself,
Jeff Hill, Andrew Johnson, and Eric Norum. If I may reply to some of the
points raised:

1) I still intend to create a Redhawk-specific version of osdMutex and
osdThread, etc. This seems to be the most easily understandable and most
maintainable solution...though I'm open to better ideas. I intend that the
Redhawk-specific version of osdMutex will only contain one implementation of
a mutex.

2) Having read the discussion about mutexes and spinlocks, etc, I'm
convinced that the osdMutex code that I've built already, needs to be
revised. At present, the code is based on "true" spinlocks (i.e. initially,
an atomic memory operation instruction, followed by spinning using a
"standard" memory read operation and then retrying the atomic lock when the
"read" indicates that the lock is available), although it is cognisant of
whether it is running on a system which has 1 CPU active, or >1 CPU active
(including logical CPUs enabled by Hyperthreading). The present code only
spins if it's running on a multi-CPU system, otherwise it calls
"sched_yield()" to give up the CPU as after it tries once, and fails, to
acquire the lock. At present I do nothing special to cope with the Priority
Inheritance problem - I confess that I don't know enough about the EPICS
code to understand if Priority Inheritance is really required, or if the
code can never exhibit such problems. However, having said that, I suspect
that the code should somehow be made to cater for PI. (A forthcoming release
of Redhawk will provide kernel support for PI on mutexes, so in that case I
could spin for a while and then use a kernel mutex - but I don't want to
have to wait for it...) I'll let you know what I wind up building.

3) Re the difference between Redhawk and the new Linux 2.6 (presumably the
"3.6" in your email was a typo) kernel...  My understanding (not gospel) is
that the following Redhawk features will come as standard to all Linux
systems in the 2.6 kernel:
  - kernel preemption
  - low-latency kernel patches (though these are only a subset of what's in
  - the real-time scheduler
  - Posix timers (though only "low" resolution, not close to Redhawk's high
res timers)

This means that (at least) the following existing Redhawk features will not
be present in the standard Linux 2.6 kernel:

- rescheduling variables
- fast block/wake services
- usermap and /proc enhancements
- processor shielding and affinity
- frequency-based scheduler

Then there are new features planned for future releases of Redhawk and, as
mentioned, one of these is support for Priority Inheritance.

4) The "gmtime_r()" and "localtime_r()" functions are indeed present in
Redhawk - thanks to all for pointing out where these live.

5) I'm not sure what to do about the "must run as root" problem. I can't use
a wrapper as Eric suggested, because (for example) I can't set the
scheduling policy and priority for all threads once - at program
commencement - because this must be done as each thread is created or after
a thread is created. I'm not sure how to handle this problem - I *might*
have to really run stuff as root. I know Redhawk supports Pluggable
Authentication Modules (PAMs), but I've never looked at them in detail, so
I'm not sure if they provide the functionality required. Will have to take a
look... As always, other suggestions are welcome...

Thanks for everyone's help and advice - I really appreciate it. 

Kind regards,


-----Original Message-----
From: Marty Kraimer []
Sent: Tuesday, 9 December 2003 2:36 AM
To: David Kelly
Subject: Re: Novice EPICS questions re yet another port

This is a long answer to David's message so let me give the summary first.

Currently EPICS base 3.14 only supports thread priorities on vxWorks, RTEMS,
WIN32. What David is doing is certainly of interest.


First a few comments about the existing implementation of threads and
mutexes in 

The existing implementation works on solaris (at least 6,7,8,9), linux (at
several releases of redhat), and darwin.



The following statements appear:

/* Until these can be demonstrated to work leave them undefined*/

The big one is _POSIX_THREAD_PRIORITY_SCHEDULING. This means that priority 
scheduling is NOT being used on solaris, linux, or darwin.


The following statements appear
/* Until these can be demonstrated to work leave them undefined*/
/* On solaris 8 _POSIX_THREAD_PRIO_INHERIT fails*/

The big one is _POSIX_THREAD_PRIO_INHERIT. But, of course, it will only be

Also osdThread has THREE!! implementations of a mutex:

1) uses spin locks. I am sending a separate message about this to Jeff and
because I am confused about the implications.
2) Uses pthread_mutex_xxx IF the pthreads implemention support recursive
3) Uses a combination of a pthread_mutex and a pthread_cond if the pthreads 
implementation does not gurantee recursive mutexes.

It is certainly desirable to support at least
and _POSIX_THREAD_PRIO_INHERIT wherever possible. Who should we do this? I
two major problems: 1) Priority Scheduling, and 2) different platforms
different subsets of pthreads.

Priority Scheduling, at least implementing real-time priorities, gets us
the problem of privileges. Some kind of privilege (often root) is required.
will an EPICS IOC do this? Running everything as root is NOT a solution.

Different Platforms: Not only different platforms but different releases.
is a MAJOR problem. I have seen some strange behavior on redhat 8 that went
with redhat 9. I had big problems getting things to work on both solaris 6
solaris 8. If we turn on _POSIX_THREAD_PRIORITY_SCHEDULING for all
platforms, I 
bet that most (All?) will fail.

What should we do?

For Priority Scheduling work must be done to develop a good environment.
Norum's message could be a start.

With regards to threads and mutexes I see two approaches:

1) Keep all pthreads code in src/libCom/osi/os/posix and use #ifdefs to take

care of differences between platforms and releases of platforms.

2) Keep src/libCom/osi/os/posix but leave osdThread and osdMutex "dumbed
just like they are now and provide separate implementations for platforms
are carefully tested.

I say do 2). I suspect that 1) will lead to a set of code that will be
difficult to understand and maintain.

Even if we do 2) the problems of releases still arises.

Now I have a question for David.

When the 3.6 Linux kernel is available won't it provide most of what redhawk

linux provides?

Now I will answer some specifics:

David Kelly wrote:
> Hi all,
> I'm porting the EPICS R3.13.4 ioc base code to RedHawk Linux, and am
> some advice re several questions that I have.
> (But first, some background info may be helpful.... I work for Concurrent
> Computer Corp, who have a real-time version of Linux for x86 platforms,
> called RedHawk Linux. This is not a real-time kernel which runs underneath
> standard Linux, which is how RTLinux and L4-Linux work. Rather, it is an
> open source enhanced version of the Linux kernel which includes all the
> standard Linux stuff plus things like: true real-time priority scheduling,
> fully pre-emptible kernel, a plethora of fast interprocess synchronization
> and communication mechanisms, high resolution clocks/timers, extended
> mapping capabilities, and the ability to shield processors from interrupts
> and dedicate processes to specific processors in order to guarantee
> deterministic system response, etc. It is currently based on the 2.4.21
> kernel from It is installed by first installing Red Hat 8.0 on
> the target system, and then installing the RedHawk kernel and real-time
> libraries, etc. It is Red Hat 8.0 compatible. A propaganda page is here
> if anyone wants more
> info.)
> The work I've done so far includes:
> 1) Create a "redhawk-x86" build environment.
> 2) Moved a few elements of the "posix" source files to the new "redhawk"
> branch of the source tree and modify them to better take advantage of some
> RedHawk features. The work done does not affect a lot of code, and it
> remains largely based on the posix base. Most of the work is as follows,
> though there some additional bits and pieces:
>     - osdMutex: now uses RedHawk fast mutex mechanisms.
>     - osdThread: has now enabled the existing
> _POSIX_THREAD_PRIORITY_SCHEDULING code. This was not sufficient to enable
> priority scheduling under RedHawk, so a small amount of extra code was
> required (e.g. specifically enabling "SCHED_RR" [or SCHED-FIFO] on a
> per-thread basis).
>     - osdTime: Now uses CLOCK_REALTIME_HR for highest possible resolution.


 From what I answered above, I agree with the idea of creating a new 
subdirectory  libCom/osi/os/redhawk

Eric and I also mentioned the priviledge problem.

> 3) Compile the ioc base with all the changes and run the "epicsXXXXTest"
> programs.
> I would appreciate advice in two areas: some specific source code
> and then an initial much higher level question. (My biggest problem at the
> moment is that I'm "looking at the wood, not the trees" - that is, I'm
> hacking bits of low-level code without having much of an understanding of
> how epics works as a whole. I have, however, read through the EPICS I/O
> Controller Application Developer's Guide, some other EPICS training-type
> docs from the web, and some whitepapers/technical reports which give an
> overview of porting EPICS. Overall, these have been very helpful.....)
> Firstly, I'd appreciate some help with code-specific queries:
> a) The posix/osdThread.c routines uses "pthread_mutex_XXXX()" functions
> synchronization, rather than the platform specific mutexes in osdMutex.c.
> there any reason for this ? (I've changed osdThread.c to use the mutexes
> osdMutex.c and everything seems OK so far...)

I answered this above. I agree with your approach.

> b) The posix/osdTime.cpp file appears to contain current references to
> "gmtime_r()" and "localtime_r()" functions, which I gather are specific to
> VxWorks. These do not appear to be resolved anywhere in a build under
> (or RedHawk). Is this correct, or am I missing something ?

These are defined in the Single Unix Specification Version 3 (The successor
POSIX standards). See

> c) I don't yet understand the way the thread priorities are calculated in
> posix/osdThread.c, when _POSIX_THREAD_PRIORITY_SCHEDULING is enabled. Can
> someone please explain to me the effect that the getOssPriorityValue()
> function is intended to have ?

Given a epicsThread priority return a OS specific priority for it.

epicsThread priorities are from 0 to 99 with a higher number meaning higher 

On vxWorks priorities are from 0 to 255 with a lower number meaning higher 
priority. For the vxWorks implemenation we decided to map epics priorities 
0,...,99 to vxWorks priorities 199,...100.

> d) Under RedHawk, if a process desires to run with a real-time priority
> (i.e. use a real-time scheduling policy of SCHED_RR or SCHED_FIFO, where
> real-time priorities range from 99 down to 1 and are all therefore higher
> than the standard Linux SCHED_OTHER timesharing priority of zero), then
> process must be owned by, or run in the context of, the root user. I could
> do this by logging on as root and MAKEing the code as root. However, I
> prefer to MAKE the project in the context of my normal user login, which
> means that I then need to "chown root XXXX" and "chmod a+s XXXX" all the
> created binary executables so that they run as the root user. I'd like to
> able to automate the "chown" and "chmod" as part of the build (MAKE)
> process, but I haven't yet figured out how to modify the build environment
> to do this. Can anyone suggest how to do this, or at least where I should
> looking ?

I suggest that you use SCHED_FIFO. This is what we do for vxWorks.
The problem of root was discussed above and by Eric Norum's message.

> e) What other areas of the code should I be looking at, in order to
> the real-time performance (in a Linux-type environment) ?

Everywhere :-)

> Now for the "trees"-type question....
> i) I'm doing this work on my laptop computer, with some further testing on
> multi-processor system in our office. In this restricted environment,
> the simplest way to test that the resultant EPICS "application" runs
> correctly ? In the same vein, what's the simplest way to run a
> performance/response-time test of the EPICS/ioc code ? (I'd like to run
> performance/response measurements under standard Linux and then again
> RedHawk. Am thinking of doing something like that described in the
> EPICS to L4-Linux Based System" whitepaper, where a thread grabs the CPU
> counter, kicks an I/O module into interrupting, again grabs the CPU
> and then measures the difference in the counter values....)

Perhaps Odagiri-San ( could comment. He is the one
did the whitepaper.

> Thanks for any and all advice.
> Kind regards,
> David Kelly

Marty Kraimer

Re: Novice EPICS questions re yet another port Marty Kraimer

Navigate by Date:
Prev: RE: "assert (this->exitWait ( DBL_MAX ))" failed Jeff Hill
Next: Re: Novice EPICS questions re yet another port Marty Kraimer
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: Novice EPICS questions re yet another port Eric Norum
Next: Re: Novice EPICS questions re yet another port Marty Kraimer
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·