Debugging CAS-beacon crash

[Sue Witherspoon <withers@jlab.org>]

Dear folks,
  I have been trying to figure out why CAS-beacon thread is getting 
suspended after ~2hrs of running.  The application is
reading a serial Comport connected to a PC104. The simple driver does 
use interrupts. When I also include the devIocStats
application I did notice that the # CA Clients abruptly went from 3 to 
some number in excess of 2482020. Running the with
or without the devIocStats CAS-beacon eventually get suspended after ~2hrs.
 EPICS:  3.14.10
 IOC: pc104
 OS: RTEMS-pc386  w/CEXP
 SEQ: seq-2.0.12

Can someone give me hints on how to narrow down the source of the problem? 
Thanks in advance!
Sue Witherspoon
Accelerator Controls Software Group
Thomas Jefferson National Accelerator Facility


Here is a snippet of the errors I get after ~2hrs running.
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 32 bytes
> CA cast server: Unable to fetch N characters pending
> CAS: Sending a udp message of 16 bytes
> CAS: UDP recv error (errno=Bad file number)
> CAS: Sending a message of 72 bytes
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 40 bytes
> CA cast server: Unable to fetch N characters pending
> CAS: Sending a udp message of 16 bytes
> CAS: UDP recv error (errno=Bad file number)
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 32 bytes
> ----------------------------------------------------------
> Exception 13 caught at PC A26FA2 by thread 167837722
> ----------------------------------------------------------
> Processor execution context at time of the fault was  :
> ----------------------------------------------------------
>  EAX = 85       EBX = 7 ECX = 3246      EDX = 3212
>  ESI = FFFFFFFF EDI = 3BEAAC    EBP = 3BEB7C    ESP = 3BEA30
> ----------------------------------------------------------
> Error code pushed by processor itself (if not 0) = 0
> ----------------------------------------------------------
>  ************ FAULTY THREAD WILL BE DELETED **************
> ../online_notify.c: CA beacon routing (connect to 
> "255.255.255.255:65535") error was "Address family not supported by 
> protocol family"
> ../online_notify.c: CA beacon routing (connect to 
> "240.236.231.0:57856") error was "Address family not supported by 
> protocol family"
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 72 bytes
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 40 bytes
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 72 bytes
> CAS: Sending a message of 32 bytes
> CAS: Sending a message of 40 bytes
> Thread CAS-beacon (0xa01001a) suspended
Cexp>rtems_cpu_usage_report()
CPU Usage by thread
  ID            NAME         SECONDS   PERCENT
0x09010001   IDLE           7829.680000   99.534
0x0A010001   UI1              4.500000    0.057
0x0A010002   ntwk             0.380000    0.004
0x0A010003   FXPd             0.180000    0.002
0x0A010004   RPCd             0.020000    0.000
0x0A010005   PMAP             0.000000    0.000
0x0A010006   Imsg             0.000000    0.000
0x0A010007   NTPT             0.000000    0.000
0x0A010008   Cloc             0.000000    0.000
0x0A010009   task             0.000000    0.000
0x0A01000A   errl             0.020000    0.000
0x0A01000B   time             0.000000    0.000
0x0A01000C   cbLo             0.000000    0.000
0x0A01000D   cbMe             0.000000    0.000
0x0A01000E   cbHi             0.000000    0.000
0x0A01000F   dbCa             0.000000    0.000
0x0A010010   time             0.000000    0.000
0x0A010011   scan             0.000000    0.000
0x0A010012   scan             0.000000    0.000
0x0A010013   scan             0.000000    0.000
0x0A010014   scan             0.000000    0.000
0x0A010015   scan             0.000000    0.000
0x0A010016   scan             0.000000    0.000
0x0A010017   scan             0.000000    0.000
0x0A010018   scan             0.000000    0.000
0x0A010019   CAS-             0.000000    0.000
0x0A01001B   CAS-             0.240000    0.003
0x0A01001C   CAS-             0.000000    0.000
0x0A01001D   CAS-             0.000000    0.000
0x0A01001E   seqA            31.280000    0.397
0x0A01001F   bcmt             0.000000    0.000
0x0A010020   CAC-             0.000000    0.000
0x0A010021   CAS-             0.000000    0.000
0x0A010022   CAS-             0.000000    0.000
Time since last CPU Usage reset 7866.300000 seconds
0x00000000 (0)
Cexp>epicsThreadShowAll()
           PRIORITY
   ID    EPICS RTEMS   STATE    WAIT         NAME
+--------+-----------+--------+--------+---------------------+
0a010001   99 100         RUN          _main_
0a010006   99 100        Wmsg 22010004 ImsgDaemon
0a010007   90 109        Wmtx 1a01003f NTPTimeSync
0a010008   90 109        Wmtx 1a010045 ClockTimeSync
0a010009   10 189         RUN          taskwd
0a01000a   10 189        Wmtx 1a010054 errlog
0a01000b   70 129        Wmtx 1a010064 timerQueue
0a01000c   59 140        Wmtx 1a010069 cbLow
0a01000d   64 135        Wmtx 1a01006a cbMedium
0a01000e   71 128        Wmtx 1a01006b cbHigh
0a01000f   50 149        Wmtx 1a01006e dbCaLink
0a010010   60 139        Wmtx 1a01007c timerQueue
0a010011   70 129        Wmtx 1a0100ab scanOnce
0a010012   60 139        Wmtx 1a0100ad scan10
0a010013   61 138        Wmtx 1a0100af scan5
0a010014   62 137        Wmtx 1a0100b1 scan2
0a010015   63 136        Wmtx 1a0100b3 scan1
0a010016   64 135        Wmtx 1a0100b5 scan0.5
0a010017   65 134        Wmtx 1a0100b7 scan0.2
0a010018   66 133        Wmtx 1a0100b9 scan0.1
0a010019   18 181       Wevnt          CAS-TCP
0a01001a  *** RTEMS task gone! ***     CAS-beacon
0a01001b   16 183        SUSP          CAS-UDP
0a01001c   19 180        Wmtx 1a0100d8 CAS-event
0a01001d   20 179         RUN          CAS-client
0a01001e   51 148         RUN          seqAux
0a01001f   50 149         RUN          bcmtemp_seq
0a010020   51 148        Wmtx 1a0100ee CAC-event
0a010021   19 180        Wmtx 1a0100f8 CAS-event
0a010022   20 179         RUN          CAS-client
0x00000000 (0)
Cexp>