Experimental Physics and
Industrial Control System

Kay-Uwe Kasemir <[email protected]> · Wed, 18 Apr 2007 12:06:11 -0400

Hello:
We're having problems with IOCs deadlocking when re-loading the
Channel Access Security configuration files at runtime.
Maybe somebody else has seen this, too?
When doing this via the subroutine record AS routines,
a follow-up 'asDump' on the IOC shell will show that the file
was indeed loaded OK, but no more connections are allowed.
When instead running 'asInit' from the IOC shell, that hangs,
and a CTRL-C shows a stack trace that's analyzed below.
This happens with both R3.14.7 and R3.14.8.2,
on IOCs that do very different things.
After an IOC reboot, 'asInit' calls work fine,
and the access security files are of course OK as well.
By looking at the affected IOCs and their uptime, we assume
that this is caused by something that happened on the SNS
network between March 13 and April 4.
In the past, we suspected the 'pure Java CA client' code
to cause CA security related deadlocks. In principle,
we don't expect any more pure Java CA client code on the
SNS controls network, but we can't be 100% sure.
On the other hand, this deadlock could be completely unrelated
to Java CA, so forget all I said about Java CA.
Anyway, here's the stack trace with guesses based on
about 2 hours of comparing the stack offsets with the vxWorks 'l'
output and the 'gcc -S' PowerPC assembly code:
1e1a54 yystart        +96c: asInit ()
1b5b1a8 asInit         +10 : 1b5ab3c () = asInitCommon()
1b5ac28 asTrapWriteRegisterListener+384: asInitFile ()
1b59564 asInitFile     +7c : asInitFP (34a7)
1b545dc asInitFP       +11c: asInitialize ()
1b542b0 asInitialize   +31c: 1b55ec0 ()
  This is asAddMemberPvt()
1b56018 asDumpMemFP    +49c: 1b561cc ()
  This is asComputePvt()
1b563c8 asDumpMemFP    +84c: 1b650dc ()
  This must be the change-of-access-rights callback
  at the end of asComputePvt():
    if(pasgclient->pcallback && oldaccess!=access) {
        (*pasgclient->pcallback)(pasgclient,asClientCOAR);
1b6519c casUserNameInitiatingCurrentThread+1ebc: 1b64f5c ()

Based on a grep of the sources, the only code that registers for such

a callback is in rsrv/camessage.c, so this must be
casAccessRightsCB(),

which calls epicsMutexMustLock(pclient->eventqLock)
1b6502c casUserNameInitiatingCurrentThread+1d4c: epicsMutexLock ()
1a7a7c0 epicsMutexLock +24 : semTake ()
1f0c74 semTake        +13c: semMTake ()
  Deadlock...

-Kay

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System