RE: Question about two minor issues with IOCs

[Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>]

Ø  Is there a reason we shouldn't slightly increase the queue size in base to avoid this? Even around 2500 for example should solve this issue.

 

Increasing to 2500 would solve it for your specific IOC, but if you then decide to load 20 ROI plugins rather than 5 that might not be enough.

 

I think 2000 was chosen to keep memory footprint low for RTEMS and vxWorks, while being large enough that overflows are rare in most IOCs.

 

A better solution is to add callbackSetQueueSize(5000) to ADCore/iocBoot/EXAMPLE_commonPlugins.cmd.  I will do that. 

 

That will work with any version of base, and could be modified by the site or specific IOC.

 

This is the number of records of each type in a standard simDetector IOC that loads commonPlugins.cmd.

 

epics> dbnr

Records  Aliases  Record Type

    18        0    sseq

   272        0    calcout

     9        0    seq

   138        0    stringout

   428        0    bo

    34        0    dfanout

   279        0    mbbi

    67        0    busy

   208        0    mbbo

   379        0    ao

    39        0    asyn

   496        0    waveform

     3        0    fanout

  2073        0    longin

   370        0    bi

   780        0    subArray

   162        0    calc

   732        0    longout

     4        0    scalcout

     2        1    sub

   587        1    ai

   519        0    stringin

     5        0    sscan

Total 7604 records, 2 aliases

 

The input records are the ai, bi, mbbi, longin, stringin and some of the waveform records.  Most of these have SCAN=I/O Intr and so do callbacks at iocInit.  This is nearly 3900 records, which is why 2000 is sometimes not a large enough queue.  It depends on the speed at which the records can process compared to the rate at which the asynPortDrivers do callbacks at iocInit.  A queue size of 5000 seems like it should be enough.

 

Mark

 

 

 

From: Wlodek, Jakub <jwlodek@bnl.gov>
Sent: Thursday, January 16, 2020 12:33 PM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov
Subject: Re: Question about two minor issues with IOCs

 

Hi Mark,

 

I ran the callbackQueueShow command, and it does seem that the callbacks overflow by a little bit:

 

epics> callbackQueueShow

PRIORITY  HIGH-WATER MARK  ITEMS IN Q  Q SIZE  % USED  Q OVERFLOWS

   cbLow             2000           0    2000     0.0           64

cbMedium                0           0    2000     0.0            0

  cbHigh                0           0    2000     0.0            0

 

Meaning that the additional plugins are most likely the cause of this error message.

I can add the callbackSetQueueSize(5000), and the messages go away. Is there a reason we shouldn't slightly increase the

queue size in base to avoid this? Even around 2500 for example should solve this issue.

 

Jakub

 

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, January 16, 2020 1:18 PM
To: Wlodek, Jakub <jwlodek@bnl.gov>
Cc: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>
Subject: RE: Question about two minor issues with IOCs

 

Ø  In my case, I see at least a few printings of that error on every IOC boot,

Ø  though this may be because I have several additional plugins loading

Ø  PVs at startup, which means that I have more PVs than the example IOCs, however,

Ø  I had the same amount of plugins loading without this error on earlier releases.

 

I just typed the callbackQueueShow command on my simDetector IOC.  I see the following:\

 

epics> callbackQueueShow

PRIORITY  HIGH-WATER MARK  ITEMS IN Q  Q SIZE  % USED  Q OVERFLOWS

   cbLow             1875           0    2000     0.0            0

cbMedium                0           0    2000     0.0            0

  cbHigh                0           0    2000     0.0            0

 

Note that I did not overflow the queue, but it was close, 1875/2000.  I started the IOC 4 times and that high-water mark varied from 1580 to 1930.  So sometimes it will probably be over 2000.

 

What do you see?

 

You can probably eliminate the overflows by adding the following command to your startup script before iocInit()

 

callbackSetQueueSize(5000)

 

Mark

 

 

From: Wlodek, Jakub <jwlodek@bnl.gov>
Sent: Thursday, January 16, 2020 12:03 PM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov
Subject: Re: Question about two minor issues with IOCs

 

Hi Mark,

 

I am currently running the following versions of all modules:

ADCore - R3-8

base - R7.0.3.1

asyn - R4-37

 

I have also seen this buffer full error with ADCore R3-7 IOCs, though I don't remember seeing it on any earlier versions, and I don't remember

which base/asyn those were running.

 

In my case, I see at least a few printings of that error on every IOC boot, though this may be because I have several additional plugins loading

PVs at startup, which means that I have more PVs than the example IOCs, however, I had the same amount of plugins loading without this error on earlier

releases.

 

The other error also happens every IOC boot, but that one I haven't seen until recently, although it is possible it simply went unnoticed by me

in earlier versions.

 

Jakub

---

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Thursday, January 16, 2020 12:40 PM
To: Wlodek, Jakub <jwlodek@bnl.gov>
Cc: tech-talk@aps.anl.gov <tech-talk@aps.anl.gov>
Subject: RE: Question about two minor issues with IOCs

 

Hi Jakub,

 

I have been seeing both of those messages on my areaDetector IOCs as well.

 

Ø  Error (511,511) PV: 13SIM1::TIMEZONE devStringinEnvVar (init_record) Illegal INP parm field

 

That message happens every time the IOC boots for me.  I have been assuming is a problem with devIocStats, and have been meaning to track it down.

 

Ø  callbackRequest: cbLow ring buffer full

 

That message only very occasionally happens for me at the end of iocInit.  I believe it happens when the number and rate of driver callbacks for PVs are too large, so that the queue in EPICS base overflows.  We could probably eliminate them by increasing the size of that queue.  But before doing that I want to know why they just started happening, and I did not see them before.

 

I am running base 7.0.3.1 and the master branch of asyn.  The question is whether it was a change in either of those that caused the messages to start happening.

 

Note that this error can actually be serious, because it means that some records will not have their correct values, because the callback events were missed.

 

Do you see those cbLow messages every time you boot?  What version of base,  asyn and ADCore are you using?

 

Mark

 

 

From: Tech-talk <tech-talk-bounces@aps.anl.gov> On Behalf Of Wlodek, Jakub via Tech-talk
Sent: Thursday, January 16, 2020 9:53 AM
To: tech-talk@aps.anl.gov
Subject: Question about two minor issues with IOCs

 

Hi all,

 

I have been working on setting up some new USB cameras at a beamline recently, and I noticed two errors showing up in the IOC

shell, though the IOCs for the cameras actually work fine after startup. I wanted to ask if anyone knows why these show up, and how

to resolve them. To make sure it wasn't specific to the driver I had written, I compiled ADSimDetector against the same base, asyn etc,

and I still saw them showing up:

 

The first seems to be an issue with autosave, as this shows up right after reboot_restore starts:

----------------------------------------------------------------------------

reboot_restore: entry for file 'auto_settings.sav'

reboot_restore: Found filename 'auto_settings.sav' in restoreFileList.

*** restoring from './autosave/auto_settings.sav' at initHookState 6 (before record/device init) ***

reboot_restore: done with file 'auto_settings.sav'

 

Error (511,511) PV: 13SIM1::TIMEZONE devStringinEnvVar (init_record) Illegal INP parm field

 

 

The second issue happens after the auto_settings.req file is loaded:

------------------------------------------------------------------------------

create_monitor_set("auto_settings.req", 30, "P=13SIM1:")

callbackRequest: cbLow ring buffer full

callbackRequest: cbLow ring buffer full

callbackRequest: cbLow ring buffer full

 

And is buffer full message is repeated ~30 times.

 

After all of these messages, I get a message that all the PVs are connected, and the IOC runs with no issues, which is puzzling to me.

If anyone could give me some advice on the source and solution to these that would be great.

 

Regards,

Jakub Wlodek