Re: IOC Crash with No Exception Generated

[Ricardo Cardenes via Tech-talk <tech-talk@aps.anl.gov>]

On Wed, Aug 8, 2018 at 1:07 PM Michael Davidsaver <mdavidsaver@gmail.com> wrote:

On 08/08/2018 03:16 AM, Ricardo Cardenes wrote:
>     When things are (apparently) working, what does epicsThreadShowAll() show?
>
>
> I can't really tell. At that point I can't interact with the iocsh console without making the system crash (see point #3), and CA is lost.

In this case, you could put epicsThreadShowAll() at the end of your startup script.

This is a capture from one of the times where the IOC booted all the way through:

tc2-sim-ioc> epicsThreadShowAll

            PRIORITY

    ID    EPICS RTEMS   STATE    WAIT         NAME

+--------+-----------+--------+--------+---------------------+

 0a010001   91 108         RUN          _main_

 0a010002   99 100        Wmsg 22010001 ImsgDaemon

 0a010003   10 189        Wmtx 1a010011 errlog

 0a010007   90 109        Wmtx 1a01002f NTPTimeSync

 0a010008   10 189        Wmtx 1a010038 taskwd

 0a010009   90 109        Wmtx 1a01003a ClockTimeSync

 0a01000a   70 129        Wmtx 1a010258 timerQueue

 0a01000b   59 140        Wmtx 1a01025d cbLow

 0a01000c   64 135        Wmtx 1a01025e cbMedium

 0a01000d   71 128        Wmtx 1a01025f cbHigh

 0a01000e   50 149        Wmtx 1a010262 dbCaLink

 0a01000f   58 141        Wmtx 1a010273 timerQueue

 0a010010   70 129        Wmtx 1a013619 scanOnce

 0a010011   60 139        Wmtx 1a01361b scan10

 0a010012   61 138        Wmtx 1a01361d scan5

 0a010013   62 137        Wmtx 1a01361f scan2

 0a010014   63 136        Wmtx 1a013621 scan1

 0a010015   64 135        Wmtx 1a013623 scan0.5

 0a010016   65 134        Wmtx 1a013625 scan0.2

 0a010017   66 133        Wmtx 1a013627 scan0.1

 0a010018   67 132        Wmtx 1a013629 scan0.05

 0a010019   18 181       Wevnt          CAS-TCP

 0a01001a   17 182       Delay          CAS-beacon

 0a01001b   16 183       Wevnt          CAS-UDP

 0a01001c   10 189       Delay          bcYearMonitor

 0a01001d   10 189        Wmtx 1a013657 ipToAsciiProxy

 0a01001e   51 148        Wmtx 1a013660 timerQueue

 0a01001f   52 147       Wevnt          CAC-UDP

 0a010020   10 189       Wevnt          CAC-repeater

 0a010021   51 148         RUN          CAC-event

 0a010022   50 149        Wmtx 1a0136c9 timeSeq

 0a010023   51 148        Wmtx 1a0136e0 CAC-event

 0a010024   50 149        Wmtx 1a0136e7 tcsSeqSysSimulate

 0a010025   50 149        Wmtx 1a0136f4 tcsSeqSysSimulate

 0a010026   50 149        Wmtx 1a013701 tcsSeqSysSimulate

 0a010027   50 149        Wmtx 1a01370e tcsSeqSysSimulate

 0a010028   50 149        Wmtx 1a01371b tcsSeqSysSimulate

 0a010029   50 149        Wmtx 1a013728 tcsSeqSysSimulate

 0a01002a   50 149        Wmtx 1a013735 tcsSeqSysSimulate

 0a01002b   50 149        Wmtx 1a013742 tcsSeqSysSimulate

 0a01002c   50 149        Wmtx 1a01374f tcsSeqSysSimulate

 0a01002d   50 149        Wmtx 1a01375c tcsSeqGsaoiSimulate

 0a01002e   50 149        Wmtx 1a013768 tcsSeqInit

 0a01002f   51 148        Wmtx 1a0136c0 timerQueue

 0a010030   52 147       Wevnt          CAC-UDP

 0a010031   50 149        Wmtx 1a01376a tcsSeqInit_1

 0a010032   50 149        Wmtx 1a01376c tcsSeqInit_2

 0a010033   50 149        Wmtx 1a01376e tcsSeqInit_3

 0a010034   50 149        Wmtx 1a013770 tcsSeqInit_4

 0a010035   50 149        Wmtx 1a013772 tcsSeqInit_5

 0a010036   50 149        Wmtx 1a013774 tcsSeqInit_6

 0a010037   50 149        Wmtx 1a013776 tcsSeqInit_7

 0a010038   50 149        Wmtx 1a013778 tcsSeqInit_8

 0a010039   50 149        Wmtx 1a01377a tcsSeqInit_9

 0a01003a   50 149        Wmtx 1a01377c tcsSeqInit_10

 0a01003b   50 149        Wmtx 1a01386f tcsReadHrwfs

On 08/08/2018 03:16 AM, Ricardo Cardenes wrote:
>     I can't explain this.  The "main" thread should be running with high priority (91).
>     It could hang if an iocsh function blocks locking a mutex held by a hung thread.
>     But simply pressing "return" shouldn't do this.
>
>
> I can't either, the high priority of the "_main_" thread flies in the face of the "something is hogging the CPU for everything low-priority". iocsh would need to stall itself in the same way for everything else to die, and I'm not sure what could cause this. I've had a look at the code that reads commands from the prompt and it's relatively straightforward.

I'm not sure if it is relevant, but which readline variant are you using?

> $ grep COMMANDLINE_LIBRARY configure/os/CONFIG_SITE.Common.RTEMS
> COMMANDLINE_LIBRARY = EPICS

It occurs to me that, while at NSLS2, I encountered a bad interaction between libtecla and RTEMS 4.10
which sounds similar.

$ grep COMMANDLINE_LIBRARY configure/os/CONFIG_SITE.Common.RTEMS

COMMANDLINE_LIBRARY = EPICS

#COMMANDLINE_LIBRARY = LIBTECLA

COMMANDLINE_LIBRARY = READLINE

Using the standard config, seems to me.