Experimental Physics and
Industrial Control System

"Zimoch Dirk \(PSI\) via Core-talk" <core-talk at aps.anl.gov> · Tue, 8 Jun 2021 10:17:46 +0000

It seems that the problem is caused by the Atomic implementation.

Adding debug output to callbackInit() showed that it runs only once (as it should) and cbState is 0 (= cbInit). 
Nevertheless the Atomic CAS fails:

    if (epicsAtomicCmpAndSwapIntT(&cbState, cbInit, cbRun)!=cbInit) {
        fprintf(stderr, "Warning: callbackInit called again before callbackCleanup\n");
        return;
    }

Changing the version limit in vxWorks/epicsAtomicOSD.h from
#if _WRS_VXWORKS_MAJOR * 100 + _WRS_VXWORKS_MINOR >= 606
to
#if _WRS_VXWORKS_MAJOR * 100 + _WRS_VXWORKS_MINOR >= 609

Makes it work on my VxWorks 6.7 MV5100 System. No callbackInit warning any longer and no crash in scanIoRequest.

On Tue, 2021-06-08 at 11:30 +0200, Zimoch Dirk wrote:
> Hi Ben,
> 
> 
> I have applied your patch. Unfortunately it does not help with my problem. The out of bounds access you fixed is only
> a
> read access, unlikely to cause memory corruption.
> 
> After adding debug output, I see that the devIocStats function scan_time() runs several times without problems but
> then
> suddenly crashes in scanIoRequest.
> 
> I tested other drivers using I/O Intr scanning and got the same problem. So it seems devIocStats was innocent.
> 
> But I see another strange thing. Just after the iocInit message, I noticed "Warning: callbackInit called again before
> callbackCleanup". Looks like maybe something is messed up with the callbacks threads used by I/O Intr. Investigating
> that now...
> 
> Dirk
> 
> 
> On Tue, 2021-06-08 at 10:43 +0200, Ben Franksen via Core-talk wrote:
> > Am 07.06.21 um 14:53 schrieb Zimoch Dirk (PSI) via Core-talk:
> > > Further investigation showed that the bug is somehow related to devIocStats (not loading it does not trigger the
> > > problem).
> > > The problem seems to be caused by memory corruption. When printing spin->locked (which should be 0 or 1), I get
> > > 1919513701 = 0x7269746.
> > > Maybe one out our modifications to devIocStats was buggy.
> > 
> > The fix I have just now made a PR for
> > (https://github.com/epics-modules/iocStats/pull/40) may be related to this.
> > 
> > Cheers
> > Ben

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System