Re: recDynLink.c and swait.c error messaged during ioc start

["Gofron, Kazimierz via Tech-talk" <tech-talk@aps.anl.gov>]

Hi Mark,

I compiled sscan in, and recDynLinkQsize is included. However, increasing it even to 4096 produces error messages, until XF:10ID-CT{FbPid:06}PID.FBON->On, at least once. There must be something else related to initialization.

epics> var

CASDEBUG = 0

aCalcAsyncThreshold = 10000

aCalcLoopMax = 1000

aCalcMonitorMem_debug = 0

aCalcPerformDebug = 0

aCalcPostfixDebug = 0

aCalcoutRecordDebug = 0

arrayTestDebug = 0

asCaDebug = 0

asCheckClientIP = 0

atExitDebug = 0

boHIGHlimit = 100000

boHIGHprecision = 2

calcoutODLYlimit = 100000

calcoutODLYprecision = 2

callbackParallelThreadsDefault = 20

configMenuDebug = 0

dbAccessDebugPUTF = 0

dbBptNotMonotonic = 0

dbConvertStrict = 0

dbJLinkDebug = 0

dbQuietMacroWarnings = 0

dbRecordsAbcSorted = 0

dbRecordsOnceOnly = 0

dbTemplateMaxVars = 100

dbThreadRealtimeLock = 1

debugSubAve = 0

debug_saveData = 0

debug_saveDataMsg = 0

devaCalcoutSoftDebug = 0

devsCalcoutSoftDebug = 0

histogramSDELprecision = 2

interpDebug = 0

lnkDebug_debug = 0

logClientDebug = 0

pvHistoryDebug = 0

recDynLinkDebug = 0

recDynLinkQsize = 1024

reccastMaxHoldoff = 10

reccastTimeout = 20

sCalcLoopMax = 1000

sCalcPerformDebug = 0

sCalcPostfixDebug = 0

sCalcStackHW = 0

sCalcStackLW = 0

sCalcoutRecordDebug = 0

saveData_MessagePolicy = 0

save_restoreDatedBackupFiles = 1

save_restoreDebug = 0

save_restoreIncompleteSetsOk = 1

save_restoreNumSeqFiles = 3

save_restoreRemountThreshold = 10

save_restoreSeqPeriodInSeconds = 60

scanparmRecordDebug = 0

seqDLYlimit = 100000

seqDLYprecision = 2

sscanRecordConnectWaitSeconds = 1

sscanRecordDebug = 0

sscanRecordDontCheckLimits = 0

sscanRecordLookupTime = 10

sscanRecordViewPos = 0

sseqRecDebug = 0

swaitRecordDebug = 0

transformRecordDebug = 0

========================

My fb_epid.db record is initialized as follows:
```

epicsEnvSet("DEV", "{FbPid:01}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

epicsEnvSet("DEV", "{FbPid:02}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

epicsEnvSet("DEV", "{FbPid:03}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

epicsEnvSet("DEV", "{FbPid:04}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

epicsEnvSet("DEV", "{FbPid:05}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

epicsEnvSet("DEV", "{FbPid:06}")

dbLoadRecords("$(TOP)/db/fb_epid.db", "Sys=$(SYS),Dev=$(DEV),PID=PID,PREC=6,IN=$(SYS)$(DEV)PID:input_,OUT=$(SYS)$(DEV)PID:output_,MODE=PID,CALC=A,PERMIT1=,PERMIT2=,PERMIT3=,PERMIT4=")

```

Can you please post how you are initializing your fb_epid.db record.

---------------------------------

---------------------------------

Also, I wanted to use custom scan rates. In past I included a custom version of menuScan.dbd in the src directory and added it in Makefile. For newer version of EPICS (>3.15) compilation fails when sscan is included.

How to do custom scan rates?

```

kgofron@xf10idd-ioc1:/epics/iocs/pid/pidApp/src$ more menuScan.dbd

menu(menuScan) {

choice(menuScanPassive,"Passive")

choice(menuScanEvent,"Event")

choice(menuScanI_O_Intr,"I/O Intr")

# Periodic scans follow, ordered from slowest to fastest

choice(menuScan300_second,"300 second")

choice(menuScan180_second,"180 second")

choice(menuScan120_second,"120 second")

choice(menuScan60_second,"60 second")

choice(menuScan30_second,"30 second")

choice(menuScan10_second,"10 second")

choice(menuScan5_second,"5 second")

choice(menuScan2_second,"2 second")

choice(menuScan1_second,"1 second")

}

```

Thanks,

Kaz

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, May 5, 2022 11:51 AM
To: Gofron, Kazimierz <kgofron@bnl.gov>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

It appears that your application is not built with the sscan module.  That is where recDynLink comes from.  

sscan is needed to get the swait record support from the calc module.  It appears that calc was built with SSCAN defined (because swaitRecordDebug exists in your var list), but your application was not built with sscan.  Maybe that is the problem.

Mark

---

From: Gofron, Kazimierz <kgofron@bnl.gov>
Sent: Thursday, May 5, 2022 10:43 AM
To: Mark Rivers <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

I do not have recDynLinkQsize.

epics> var

CASDEBUG = 0

aCalcAsyncThreshold = 10000

aCalcLoopMax = 1000

aCalcMonitorMem_debug = 0

aCalcPerformDebug = 0

aCalcPostfixDebug = 0

aCalcoutRecordDebug = 0

arrayTestDebug = 0

asCaDebug = 0

asCheckClientIP = 0

atExitDebug = 0

boHIGHlimit = 100000

boHIGHprecision = 2

calcoutODLYlimit = 100000

calcoutODLYprecision = 2

callbackParallelThreadsDefault = 20

configMenuDebug = 0

dbAccessDebugPUTF = 0

dbBptNotMonotonic = 0

dbConvertStrict = 0

dbJLinkDebug = 0

dbQuietMacroWarnings = 0

dbRecordsAbcSorted = 0

dbRecordsOnceOnly = 0

dbTemplateMaxVars = 100

dbThreadRealtimeLock = 1

debugSubAve = 0

devaCalcoutSoftDebug = 0

devsCalcoutSoftDebug = 0

histogramSDELprecision = 2

interpDebug = 0

lnkDebug_debug = 0

logClientDebug = 0

pvHistoryDebug = 0

reccastMaxHoldoff = 10

reccastTimeout = 20

sCalcLoopMax = 1000

sCalcPerformDebug = 0

sCalcPostfixDebug = 0

sCalcStackHW = 0

sCalcStackLW = 0

sCalcoutRecordDebug = 0

save_restoreDatedBackupFiles = 1

save_restoreDebug = 0

save_restoreIncompleteSetsOk = 1

save_restoreNumSeqFiles = 3

save_restoreRemountThreshold = 10

save_restoreSeqPeriodInSeconds = 60

seqDLYlimit = 100000

seqDLYprecision = 2

sseqRecDebug = 0

swaitRecordDebug = 0

transformRecordDebug = 0

epics>

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, May 5, 2022 11:34 AM
To: Gofron, Kazimierz <kgofron@bnl.gov>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

The recDynLinkQsize variable exists for me on my IOCs.  This is the output of the "var" command:

epics> var

CASDEBUG = 0

PDBProviderDebug = 0

aCalcAsyncThreshold = 10000

aCalcLoopMax = 1000

aCalcMonitorMem_debug = 0

aCalcPerformDebug = 0

aCalcPostfixDebug = 0

aCalcoutRecordDebug = 0

aimDebug = 1

arrayTestDebug = 0

asCaDebug = 0

asCheckClientIP = 0

atExitDebug = 0

boHIGHlimit = 100000

boHIGHprecision = 2

calcoutODLYlimit = 100000

calcoutODLYprecision = 2

callbackParallelThreadsDefault = 20

configMenuDebug = 0

dbAccessDebugPUTF = 0

dbBptNotMonotonic = 0

dbConvertStrict = 0

dbJLinkDebug = 0

dbQuietMacroWarnings = 0

dbRecordsAbcSorted = 0

dbRecordsOnceOnly = 0

dbTemplateMaxVars = 100

dbThreadRealtimeLock = 1

debugSubAve = 0

debug_saveData = 0

debug_saveDataMsg = 0

devaCalcoutSoftDebug = 0

devsCalcoutSoftDebug = 0

drvMCDC2805debug = 0

drvPMNC87xxdebug = 0

eraseNDAttributes = 0

histogramSDELprecision = 2

icbDebug = 0

interpDebug = 0

lnkDebug_debug = 0

logClientDebug = 0

mcaRecordDebug = 0

pvHistoryDebug = 0

pvaLinkNWorkers = 1

recDigitelDebug = 0

recDynLinkDebug = 0

recDynLinkQsize = 256

sCalcLoopMax = 1000

sCalcPerformDebug = 0

sCalcPostfixDebug = 0

sCalcStackHW = 2

sCalcStackLW = 0

sCalcoutRecordDebug = 0

saveData_MessagePolicy = 0

save_restoreDatedBackupFiles = 1

save_restoreDebug = 0

save_restoreIncompleteSetsOk = 1

save_restoreNumSeqFiles = 3

save_restoreRemountThreshold = 10

save_restoreSeqPeriodInSeconds = 300

scalerRecordDebug = 0

scaler_wait_time = 10

scanparmRecordDebug = 0

seqDLYlimit = 100000

seqDLYprecision = 2

sscanRecordConnectWaitSeconds = 1

sscanRecordDebug = 0

sscanRecordDontCheckLimits = 0

sscanRecordLookupTime = 10

sscanRecordViewPos = 0

sseqRecDebug = 0

streamDebug = 0

streamError = 1

swaitRecordDebug = 0

tableRecordDebug = 0

transformRecordDebug = 0

I can change it:

epics> var recDynLinkQsize

recDynLinkQsize = 256

epics> var recDynLinkQsize 1024

epics> var recDynLinkQsize

recDynLinkQsize = 1024

Please send the output when you type the "var" command.

Mark

---

From: Gofron, Kazimierz <kgofron@bnl.gov>
Sent: Thursday, May 5, 2022 10:22 AM
To: Mark Rivers <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

Hi Mark,

The variable "Var recDynLinkQsize not found."  is not found during iocBoot

```

var recDynLinkQsize 1024

Var recDynLinkQsize not found.

iocInit()

```

Kaz,

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, May 5, 2022 10:37 AM
To: Gofron, Kazimierz <kgofron@bnl.gov>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

Hi Kaz,

recDynLinkQueueSize is a variable you can change from the IOC shell.  So you can do this:

var recDynLinkQueueSize 1024

I think you probably need to do that before iocInit.

Mark

---

From: Gofron, Kazimierz <kgofron@bnl.gov>
Sent: Thursday, May 5, 2022 9:29 AM
To: Mark Rivers <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; mooney@anl.gov <mooney@anl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

Hi Mark & Tim,

I added to st.cmd after envPaths, before iocInit:

```

errlogInit(20000)

callbackSetQueueSize(5000)

```

--------------------

Now there are only swaitRecord.c error messages:

```

epics> ioc_settings.sav: 1575 of 1575 PV's connected

filename="../swaitRecord.c" line number=870

recWait: rngBufGet error

filename="../swaitRecord.c" line number=872

recWait: rngBufGet error

```

These error messages stop after I start the feedback loop for the 1^st time

XF:10ID-CT{FbPid:06}PID:on -> On    {XF:10ID-CT{FbPid:06}PID.FBON->On}

---------------------

@Tim

./sscan/sscanApp/src/recDynLink.c:int   recDynLinkQsize = 256;

Are you suggesting to change it in sscan?

Or add recDynLinkQsize to the ioc, so it appears in 

static struct iocshVarDef vardefs[] = {... } 

Thank you 

Kazimierz

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Wednesday, May 4, 2022 5:36 PM
To: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>; Gofron, Kazimierz <kgofron@bnl.gov>
Subject: Re: recDynLink.c and swait.c error messaged during ioc start

 

Hi Kaz,

> errlog: 34 messages were discarded

That error indicates that your message queue is overflowing, so you may be missing important messages.  You should put this in st.cmd right after <envPaths

errlogInit(20000)

The error messages themselves might indicate that the callback queue not large enough.  I would suggest adding this line before iocInit:

callbackSetQueueSize(5000)

See if adding those fixes the problem.

Mark

---

From: Tech-talk <tech-talk-bounces@aps.anl.gov> on behalf of Gofron, Kazimierz via Tech-talk <tech-talk@aps.anl.gov>
Sent: Wednesday, May 4, 2022 4:09 PM
To: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>
Subject: recDynLink.c and swait.c error messaged during ioc start

 

Hi,

I've been working on EPICS pid ioc for temperature control, that uses Wiener power supply and Keithly 3706A PT1000 sensors scanner.

Recently we updated RHEL8.5 kernel, and I noticed that ioc shows errors since end of April 2022. Update might not be related to OS update. I might also be missing something in the dependency. Perhaps someone observed this before.

## EPICS R7.0.6
## Rev. R7.0.6

OS: RHEL8.5

During ioc startup, getting error messages from 

sscan: / recDynLink.c : 549

    precDynLink->onQueue++;

    if (epicsMessageQueueTrySend(recDynLinkOutMsgQ, (void *)&cmd, sizeof(cmd))) {

        errMessage(0,"recDynLinkPut: epicsMessageQueueTrySend error");

        status = RINGBUFF_PUT_ERROR;

        precDynLink->onQueue--;

    }

calc: / swait.c : 838/840

    recDynLinkGet(precDynLink, &monData, &nRequest, 0, 0, 0);

    if (swaitRecordDebug>5)

        errlogPrintf("swaitRecord(%s)inputChanged: queuing monitor on %d = %f\n",pwait->name,index,monData);

    if (epicsRingBytesPut(pcbst->monitorQ, (char *)&index, sizeof(char))

        != sizeof(char)) errMessage(0,"recWait rngBufPut error");

    if (epicsRingBytesPut(pcbst->monitorQ, (char *)&monData, sizeof(double))

        != sizeof(double)) errMessage(0,"recWait rngBufPut error");

    if (swaitRecordDebug>5) {

        errlogPrintf("swaitRecord(%s)inputChanged: %d entries left in monitorQ\n", pwait->name,

            (int)(epicsRingBytesFreeBytes(pcbst->monitorQ)/sizeof(struct qStruct)));

    }

calc: / swait.c: 870/872

    if (!swaitRecordCacheMode) {

        if (epicsRingBytesGet(pcbst->monitorQ, (char *)&inputIndex, sizeof(char))

            != sizeof(char)) errMessage(0, "recWait: rngBufGet error");

        if (epicsRingBytesGet(pcbst->monitorQ, (char *)&monData, sizeof(double))

            != sizeof(double)) errMessage(0, "recWait: rngBufGet error");

=========================

epics> filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

filename="../recDynLink.c" line number=549

recDynLinkPut: epicsMessageQueueTrySend error

ioc_settings.sav: 1575 of 1575 PV's connected

errlog: 34 messages were discarded

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=838

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=838

recWait rngBufPut error

filename="../swaitRecord.c" line number=840

recWait rngBufPut error

filename="../swaitRecord.c" line number=872

recWait: rngBufGet error

filename="../swaitRecord.c" line number=870

recWait: rngBufGet error

filename="../swaitRecord.c" line number=872

recWait: rngBufGet error

filename="../swaitRecord.c" line number=870

recWait: rngBufGet error

filename="../swaitRecord.c" line number=872

recWait: rngBufGet error

filename="../swaitRecord.c" line number=870

===================

After ioc boots, the messages stop.

Thanks for the help!

Kazimierz