assert failed in db_post_single_event_private for CALC record

[Kay-Uwe Kasemir <kasemirk@ornl.gov>]

Hi:

One of my IOCs asked me to post the following to tech-talk.

scan0.5: A call to "assert ((epicsMutexLock(((ev_que)->writelock)) 
==epicsMutexLock OK))" failed in ../dbEvent.c at 665
..
EPICS Release EPICS R3.14.7 $R3-14-7$ $2004/12/06 22:31:52$.
Please E-mail this message and the output from "tt (0x14b8290)"
to the author or to tech-talk@aps.anl.gov

The "tt" dump shows a sequence of FLNK invocations.
scl-llrf-ioc15d> tt (0x14b8290)
1fa774 vxTaskEntry +68 : 1b60b68 ()
1b60bd8 epicsThreadPrivateGet+f8 : 1a9e118 ()
1a9e19c scanIoRequest +708: 1a9d1c4 ()
1a9d2d8 scanDelete +6b0: dbProcess ()
1a8c710 dbProcess +398: 1a59184 ()
1a59284 pvCaMonitorHandler+1060: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a5d5ec ()
1a5d6a0 pvCaMonitorHandler+547c: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a59184 ()
1a59284 pvCaMonitorHandler+1060: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a5d5ec ()
1a5d6a0 pvCaMonitorHandler+547c: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a59184 ()
1a59284 pvCaMonitorHandler+1060: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a59184 ()
1a59284 pvCaMonitorHandler+1060: recGblFwdLink ()
1aa6c08 recGblFwdLink +30 : dbScanFwdLink ()
1a8e538 dbScanFwdLink +74 : dbProcess ()
1a8c710 dbProcess +398: 1a5d5ec ()
1a5d68c pvCaMonitorHandler+5468: 1a5d3a0 ()
1a5d474 pvCaMonitorHandler+5250: db_post_events ()
1a9f804 db_post_events +c0 : 1a9ee14 ()
1a9ee7c db_cancel_event+46c: epicsAssert ()
1b5eef4 epicsAssert +154: epicsThreadSuspendSelf ()
1b60494 epicsThreadSuspendSelf+2c : taskSuspend ()
value = 0 = 0x0

The process routine 0x01a5d5ec() is in the
calcRSET = 0x1bb24f0: ...

d 0x1bb24f0, 10, 4
01bb24f0: 00000011 00000000 00000000 01a5d508 *................*
01bb2500: 01a5d5ec 01a5d6c0 00000000 00000000 *................*
01bb2510: 00000000 00000000 *................*

So I think it's a chain of FLNK'ed records, type CALC and
an unknown type.
By looking at all the records on the 0.5second scan thread,
I found one match for the pattern with CALC and AI records:
SCAN 0.5 AI -flnk-> CALC -> AI -> CALC -> AI -> AI -> CALC -> assert()

The calc record FLNK'ed from the AIs compare the value of the AI
with some setpoint.
There are actually a few more AI -> CALC -> AI -> CALC to follow,
and most of the time that works without running into the assert.

This is the second time the scan05 thread hung.
First time around it was at an earlier point of the chain,
but also inside a calc::process().

In principle, I don't see any reason for that to run into an assert,
except that something deleted the (ev_que)->writelock,
or general memory corruption.

Any idea, anybody?

Thanks,
-Kay