Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: Re: Novice EPICS questions re yet another port
From: Marty Kraimer <mrk@aps.anl.gov>
To: David Kelly <Kelly@CCUR.COM.AU>
Cc: tech-talk@aps.anl.gov
Date: Mon, 08 Dec 2003 09:35:33 -0600
This is a long answer to David's message so let me give the summary first.

SUMMARY:
Currently EPICS base 3.14 only supports thread priorities on vxWorks, RTEMS, and WIN32. What David is doing is certainly of interest.


END OF SUMMARY

First a few comments about the existing implementation of threads and mutexes in <base3_14>/src/libCom/osi/os/posix.

The existing implementation works on solaris (at least 6,7,8,9), linux (at least several releases of redhat), and darwin.

HOWEVER:

Threads:

The following statements appear:

/* Until these can be demonstrated to work leave them undefined*/
#undef _POSIX_THREAD_ATTR_STACKSIZE
#undef _POSIX_THREAD_PRIORITY_SCHEDULING

The big one is _POSIX_THREAD_PRIORITY_SCHEDULING. This means that priority scheduling is NOT being used on solaris, linux, or darwin.

Mutex:

The following statements appear
/* Until these can be demonstrated to work leave them undefined*/
/* On solaris 8 _POSIX_THREAD_PRIO_INHERIT fails*/
#undef _POSIX_THREAD_PROCESS_SHARED
#undef _POSIX_THREAD_PRIO_INHERIT

The big one is _POSIX_THREAD_PRIO_INHERIT. But, of course, it will only be of use when _POSIX_THREAD_PRIORITY_SCHEDULING is supported.

Also osdThread has THREE!! implementations of a mutex:

1) uses spin locks. I am sending a separate message about this to Jeff and David because I am confused about the implications.
2) Uses pthread_mutex_xxx IF the pthreads implemention support recursive mutexes.
3) Uses a combination of a pthread_mutex and a pthread_cond if the pthreads implementation does not gurantee recursive mutexes.


It is certainly desirable to support at least _POSIX_THREAD_PRIORITY_SCHEDULING and _POSIX_THREAD_PRIO_INHERIT wherever possible. Who should we do this? I see two major problems: 1) Priority Scheduling, and 2) different platforms implement different subsets of pthreads.

Priority Scheduling, at least implementing real-time priorities, gets us into the problem of privileges. Some kind of privilege (often root) is required. How will an EPICS IOC do this? Running everything as root is NOT a solution.

Different Platforms: Not only different platforms but different releases. This is a MAJOR problem. I have seen some strange behavior on redhat 8 that went away with redhat 9. I had big problems getting things to work on both solaris 6 and solaris 8. If we turn on _POSIX_THREAD_PRIORITY_SCHEDULING for all platforms, I bet that most (All?) will fail.

What should we do?

For Priority Scheduling work must be done to develop a good environment. Eric Norum's message could be a start.

With regards to threads and mutexes I see two approaches:

1) Keep all pthreads code in src/libCom/osi/os/posix and use #ifdefs to take care of differences between platforms and releases of platforms.

2) Keep src/libCom/osi/os/posix but leave osdThread and osdMutex "dumbed down" just like they are now and provide separate implementations for platforms that are carefully tested.

I say do 2). I suspect that 1) will lead to a set of code that will be extremely difficult to understand and maintain.

Even if we do 2) the problems of releases still arises.

Now I have a question for David.

When the 3.6 Linux kernel is available won't it provide most of what redhawk linux provides?

Now I will answer some specifics:


David Kelly wrote:
Hi all,

I'm porting the EPICS R3.13.4 ioc base code to RedHawk Linux, and am seeking
some advice re several questions that I have.

(But first, some background info may be helpful.... I work for Concurrent
Computer Corp, who have a real-time version of Linux for x86 platforms,
called RedHawk Linux. This is not a real-time kernel which runs underneath
standard Linux, which is how RTLinux and L4-Linux work. Rather, it is an
open source enhanced version of the Linux kernel which includes all the
standard Linux stuff plus things like: true real-time priority scheduling,
fully pre-emptible kernel, a plethora of fast interprocess synchronization
and communication mechanisms, high resolution clocks/timers, extended memory
mapping capabilities, and the ability to shield processors from interrupts
and dedicate processes to specific processors in order to guarantee
deterministic system response, etc. It is currently based on the 2.4.21
kernel from kernel.org. It is installed by first installing Red Hat 8.0 on
the target system, and then installing the RedHawk kernel and real-time
libraries, etc. It is Red Hat 8.0 compatible. A propaganda page is here
http://www.ccur.com/isd_solutions_redhawklinux.asp if anyone wants more
info.)

The work I've done so far includes:

1) Create a "redhawk-x86" build environment.

2) Moved a few elements of the "posix" source files to the new "redhawk"
branch of the source tree and modify them to better take advantage of some
RedHawk features. The work done does not affect a lot of code, and it
remains largely based on the posix base. Most of the work is as follows,
though there some additional bits and pieces:
- osdMutex: now uses RedHawk fast mutex mechanisms.
- osdThread: has now enabled the existing
_POSIX_THREAD_PRIORITY_SCHEDULING code. This was not sufficient to enable
priority scheduling under RedHawk, so a small amount of extra code was
required (e.g. specifically enabling "SCHED_RR" [or SCHED-FIFO] on a
per-thread basis).
- osdTime: Now uses CLOCK_REALTIME_HR for highest possible resolution.



From what I answered above, I agree with the idea of creating a new subdirectory libCom/osi/os/redhawk


Eric and I also mentioned the priviledge problem.

3) Compile the ioc base with all the changes and run the "epicsXXXXTest"
programs.

I would appreciate advice in two areas: some specific source code questions,
and then an initial much higher level question. (My biggest problem at the
moment is that I'm "looking at the wood, not the trees" - that is, I'm
hacking bits of low-level code without having much of an understanding of
how epics works as a whole. I have, however, read through the EPICS I/O
Controller Application Developer's Guide, some other EPICS training-type
docs from the web, and some whitepapers/technical reports which give an
overview of porting EPICS. Overall, these have been very helpful.....)

Firstly, I'd appreciate some help with code-specific queries:

a) The posix/osdThread.c routines uses "pthread_mutex_XXXX()" functions for
synchronization, rather than the platform specific mutexes in osdMutex.c. Is
there any reason for this ? (I've changed osdThread.c to use the mutexes in
osdMutex.c and everything seems OK so far...)


I answered this above. I agree with your approach.


b) The posix/osdTime.cpp file appears to contain current references to
"gmtime_r()" and "localtime_r()" functions, which I gather are specific to
VxWorks. These do not appear to be resolved anywhere in a build under Linux
(or RedHawk). Is this correct, or am I missing something ?


These are defined in the Single Unix Specification Version 3 (The successor to POSIX standards). See
http://www.unix.org/single_unix_specification/


c) I don't yet understand the way the thread priorities are calculated in
posix/osdThread.c, when _POSIX_THREAD_PRIORITY_SCHEDULING is enabled. Can
someone please explain to me the effect that the getOssPriorityValue()
function is intended to have ?

getOssPriorityValue()

Given a epicsThread priority return a OS specific priority for it.

epicsThread priorities are from 0 to 99 with a higher number meaning higher priority.

On vxWorks priorities are from 0 to 255 with a lower number meaning higher priority. For the vxWorks implemenation we decided to map epics priorities 0,...,99 to vxWorks priorities 199,...100.



d) Under RedHawk, if a process desires to run with a real-time priority
(i.e. use a real-time scheduling policy of SCHED_RR or SCHED_FIFO, where the
real-time priorities range from 99 down to 1 and are all therefore higher
than the standard Linux SCHED_OTHER timesharing priority of zero), then the
process must be owned by, or run in the context of, the root user. I could
do this by logging on as root and MAKEing the code as root. However, I
prefer to MAKE the project in the context of my normal user login, which
means that I then need to "chown root XXXX" and "chmod a+s XXXX" all the
created binary executables so that they run as the root user. I'd like to be
able to automate the "chown" and "chmod" as part of the build (MAKE)
process, but I haven't yet figured out how to modify the build environment
to do this. Can anyone suggest how to do this, or at least where I should be
looking ?


I suggest that you use SCHED_FIFO. This is what we do for vxWorks. The problem of root was discussed above and by Eric Norum's message.

e) What other areas of the code should I be looking at, in order to improve
the real-time performance (in a Linux-type environment) ?


Everywhere :-)


Now for the "trees"-type question....

i) I'm doing this work on my laptop computer, with some further testing on a
multi-processor system in our office. In this restricted environment, what's
the simplest way to test that the resultant EPICS "application" runs
correctly ? In the same vein, what's the simplest way to run a
performance/response-time test of the EPICS/ioc code ? (I'd like to run some
performance/response measurements under standard Linux and then again under
RedHawk. Am thinking of doing something like that described in the "Porting
EPICS to L4-Linux Based System" whitepaper, where a thread grabs the CPU
counter, kicks an I/O module into interrupting, again grabs the CPU counter,
and then measures the difference in the counter values....)


Perhaps Odagiri-San (jun-ichi.odagiri@kek.jp) could comment. He is the one who did the whitepaper.



Thanks for any and all advice.

Kind regards,

David Kelly

Marty Kraimer



Replies:
Re: Novice EPICS questions re yet another port Eric Norum
References:
Novice EPICS questions re yet another port David Kelly

Navigate by Date:
Prev: Re: scanRecord for 3.14 Marty Kraimer
Next: Re: Novice EPICS questions re yet another port Eric Norum
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: Novice EPICS questions re yet another port Eric Norum
Next: Re: Novice EPICS questions re yet another port Eric Norum
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  <20032004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·