Re: 3.15.0.2: Shutdown issues

[Michael Davidsaver <mdavidsaver@bnl.gov>]

Hi Ben,

It looks like you have a thread exit handler which is calling epicsExit()?

Since you are running the latest, can you arrange to set the global
variable atExitDebug=1 (defined in epicsExit.c)?  This should give some
clue about where in the exit sequence things are crashing.

Michael


On 11/17/2014 01:57 PM, Benjamin Franksen wrote:
> Am Freitag, 14. November 2014, 10:32:45 schrieb Michael Davidsaver:
>> On 11/14/2014 04:16 AM, Ralph Lange wrote:
>> > ...
>> > Any ideas on what to do about rsrv shutdown?
>>
>> Yesterday I made a commit which I suspect will hide this issue (and many
>> other shutdown issues).
>>
>> http://bazaar.launchpad.net/~epics-core/epics-base/3.15/revision/12587
>>
>> This restores the DB shutdown sequence to what it was before the new DB
>> unittesting code was merged.  This will be necessary until RSRV (and may
>> other components) learn how to shutdown cleanly.
> 
> Funny, I have a problem that seems to related to shutdown, too. It may be
> related or not. This came up when I tested the sequencer (version 2.2, for
> which I promised Mark to make a new release ASAP). I updated my clone of the
> base repo immediately before the test, so I should have Michael's latest
> patch.
> 
> You can repat this by downloading http://www-csr.bessy.de/control/SoftDist/sequencer/releases/seq-2-2-snapshot-latest.tar.gz, building it against the head of base/3.15, and then running:
> 
> ben@sarun[1]: .../seq/branch-2-2 > cd test/validate/O.linux-x86_64
> ben@sarun[1]: .../validate/O.linux-x86_64 > ./bittypes -t -d ../bittypes.db
> Starting iocInit
> ############################################################################
> ## EPICS R3.15.0.2-DEV $$Date$$
> ## EPICS Base built Nov 17 2014
> ############################################################################
> iocRun: All initialization complete
> Sequencer release 2.2.0.3, compiled Mon Nov 17 19:16:07 2014
> Spawning sequencer program "bittypesTest", thread 0x143ce60: "bittypesTest"
> bittypesTest[0]: all channels connected & received 1st monitor
> 1..24
> ok  1 - sizeof((i8)) == 1
> ok  2 - sizeof((ui8)) == 1
> ok  3 - sizeof((i16)) == 2
> ok  4 - sizeof((ui16)) == 2
> ok  5 - sizeof((i32)) == 4
> ok  6 - sizeof((ui32)) == 4
> ok  7 - pvstat == pvStatOK
> ok  8 - pvstat == pvStatOK
> ok  9 - i8 == seqg_vars_test.seqg_vars_conversion.xi8
> ok 10 - pvstat == pvStatOK
> ok 11 - pvstat == pvStatOK
> ok 12 - ui8 == seqg_vars_test.seqg_vars_conversion.xui8
> ok 13 - pvstat == pvStatOK
> ok 14 - pvstat == pvStatOK
> ok 15 - i16 == seqg_vars_test.seqg_vars_conversion.xi16
> ok 16 - pvstat == pvStatOK
> ok 17 - pvstat == pvStatOK
> ok 18 - ui16 == seqg_vars_test.seqg_vars_conversion.xui16
> ok 19 - pvstat == pvStatOK
> ok 20 - pvstat == pvStatOK
> ok 21 - i32 != seqg_vars_test.seqg_vars_conversion.xi32
> ok 22 - pvstat == pvStatOK
> ok 23 - pvstat == pvStatOK
> ok 24 - ui32 != seqg_vars_test.seqg_vars_conversion.xui32
> 
>     Results
>     =======
>        Tests: 24
>       Passed:  24 = 100.00%
> Instance 0 of sequencer program "bittypesTest" terminated
> 
> 
> 
> A call to 'assert(ellCount(&pnotifyGlobal->freeList)==0)'
>     by thread 'bittypesTest' failed in ../../../src/ioc/db/dbNotify.c line
> 304.
> Dumping a stack trace of thread 'bittypesTest':
> [    0x7f589713447b]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(epicsStackTrace+0x4b)
> [    0x7f589712e1aa]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(epicsAssert+0x4a)
> [    0x7f589738c70c]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libdbCore.so.3.15.0(dbProcessNotifyExit+0x2c)
> [    0x7f5897125d1c]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(epicsExitCallAtExits+0x9c)
> [    0x7f58971260a8]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(epicsExit+0x8)
> [    0x7f5897125e34]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(epicsExitCallAtThreadExits+0x74)
> [    0x7f589712f191]: /mnt/extra/ben/Epics/R3.15/base/head/lib/linux-
> x86_64/libCom.so.3.15.0(start_routine+0x101)
> [    0x7f589639d182]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xc2)
> [    0x7f5896e21fbd]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)
> EPICS Release EPICS R3.15.0.2-DEV $$Date$$.
> Local time is 2014-11-17 19:18:35.822216163 CET
> Please E-mail this message to the author or to tech-talk@aps.anl.gov
> Calling epicsThreadSuspendSelf()
> 
> 
> I am using epicsAtThreadExit (in test/validate/testSupport.c) just before the
> state set thread executes the SNL program shutdown. When the thread exits, my
> atExit-callback does exit(EXIT_SUCCESS) in order to kill the whole process
> (the IOC). It is done in this apparently backward manner so that I can detect
> problems during the shutdown of the sequencer program, which I could not if I
> called exit directly.
> 
> Cheers
> Ben
> --
> "Make it so they have to reboot after every typo." -- Scott Adams
> 
> 
> ________________________________
> 
> Helmholtz-Zentrum Berlin für Materialien und Energie GmbH
> 
> Mitglied der Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren e.V.
> 
> Aufsichtsrat: Vorsitzender Prof. Dr. Dr. h.c. mult. Joachim Treusch, stv. Vorsitzende Dr. Beatrix Vierkorn-Rudolph
> Geschäftsführung: Prof. Dr. Anke Rita Kaysser-Pyzalla, Thomas Frederking
> 
> Sitz Berlin, AG Charlottenburg, 89 HRB 5583
> 
> Postadresse:
> Hahn-Meitner-Platz 1
> D-14109 Berlin
> 
> http://www.helmholtz-berlin.de
>