Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: EPICS on RTEMS crashing on CA access, due to GNU readline?!?
From: Angus Gratton <angus.gratton@anu.edu.au>
To: tech-talk <tech-talk@aps.anl.gov>
Date: Tue, 03 May 2011 16:07:28 +1000
I have a _very_ unusual bug that I'm currently working through.

I'm running RTEMS 4.10 on PowerPC (MVME-3100) w/ EPICS base 3.14.11.

A short time ago I noticed my IOCs had started occasionally hanging
following a 'caput' - no console output, no iocsh, no ping or CA
responses, required a hard reboot. Strange.

Turns out I can reproduce it very easily from a command line client:

        while [ 1 ]; do caput test 0; caput test 1; done

Will hang my crate within 5-60 seconds of it running. It seems to
require the individual caput channel/socket init/teardowns to to run,
none of my long-running UI processes seem to cause it to trigger.


I went back back and bisected through my changes, and found that it was
when I recently enabled GNU Readline (!!) in Base that this started
happening.

Sure enough - if I just change COMMANDLINE_LIBRARY from READLINE back to
EPICS, I can run my test script for hours on end without hanging.


One better, if I edit src/libCom/osi/os/default/epicsReadline.c and add
a forced

#define EPICS_COMMANDLINE_LIBRARY EPICS_COMMANDLINE_LIBRARY_EPICS

... the problem goes away.



The problem doesn't seem to have any other mitigating factor that I can
find (I've disabled all my custom libraries, dbd includes, etc. and it
still happens.) Also, the use of readline in epicsReadline.c looks
innocuous - nothing there that jumps out at me as a red flag problem.

So my questions are:

1) Can anyone with a similar setup, running w/ readline, reproduce the
same problem if they run my test script?

2) Does anyone have any ideas on how I might continue to debug this? I'm
assuming it's some kind of memory corruption problem, may or may not
actually have anything to do with readline (it may just happen readline
triggers an access pattern that causes previously corrupted memory to
hang, maybe something w/ stdin/stdout on RTEMS.)

Any ideas?

Regards,


Angus


Replies:
Re: EPICS on RTEMS crashing on CA access, due to GNU readline?!? Angus Gratton

Navigate by Date:
Prev: Re: EPICS task watch dog Till Straumann
Next: Re: EPICS on RTEMS crashing on CA access, due to GNU readline?!? Angus Gratton
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: mca R7-0 Mark Rivers
Next: Re: EPICS on RTEMS crashing on CA access, due to GNU readline?!? Angus Gratton
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·