1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 <2020> 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 <2020> 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | RE: Question about two minor issues with IOCs |
From: | Mark Rivers via Tech-talk <[email protected]> |
To: | "'Wlodek, Jakub'" <[email protected]> |
Cc: | "[email protected]" <[email protected]> |
Date: | Thu, 16 Jan 2020 20:25:15 +0000 |
Ø
I think that this is in fact a better solution. That way when adding plugins this number can be adjusted if necessary. I have added callbackSetQueueSize(5000) to EXAMPLE_commonPlugins.cmd and pushed to the master branch on Github. Mark From: Wlodek, Jakub <[email protected]>
I think that this is in fact a better solution. That way when adding plugins this number can be adjusted if necessary. I ran the dbnr command in my IOC and I got the following: epics> dbnr Records Aliases Record Type 2 1 sub 777 0 longout 312 0 mbbi 2296 0 longin 880 0 subArray 138 0 stringout 34 0 dfanout 662 1 ai 304 0 calcout 542 0 waveform 2 0 fanout 182 0 calc 78 0 busy 466 0 bo 7 0 seq 421 0 bi 431 0 ao 17 0 sseq 232 0 mbbo 44 0 asyn 601 0 stringin Total 8428 records, 2 aliases Meaning that I have several hundred more records due to the additional plugins, which is why I get the message on every IOC startup. Thanks for helping explain this issue, Jakub From: Mark Rivers <[email protected]> Ø
Is there a reason we shouldn't slightly increase the queue size in base to avoid this? Even around 2500 for example should solve this issue. Increasing to 2500 would solve it for your specific IOC, but if you then decide to load 20 ROI plugins rather than 5 that might not be enough. I think 2000 was chosen to keep memory footprint low for RTEMS and vxWorks, while being large enough that overflows are rare in most IOCs. A better solution is to add callbackSetQueueSize(5000) to ADCore/iocBoot/EXAMPLE_commonPlugins.cmd. I will do that.
That will work with any version of base, and could be modified by the site or specific IOC. This is the number of records of each type in a standard simDetector IOC that loads commonPlugins.cmd. epics> dbnr Records Aliases Record Type 18 0 sseq 272 0 calcout 9 0 seq 138 0 stringout 428 0 bo 34 0 dfanout 279 0 mbbi 67 0 busy 208 0 mbbo 379 0 ao 39 0 asyn 496 0 waveform 3 0 fanout 2073 0 longin 370 0 bi 780 0 subArray 162 0 calc 732 0 longout 4 0 scalcout 2 1 sub 587 1 ai 519 0 stringin 5 0 sscan Total 7604 records, 2 aliases The input records are the ai, bi, mbbi, longin, stringin and some of the waveform records. Most of these have SCAN=I/O Intr and so do callbacks at iocInit. This is nearly 3900 records, which is why 2000 is sometimes
not a large enough queue. It depends on the speed at which the records can process compared to the rate at which the asynPortDrivers do callbacks at iocInit. A queue size of 5000 seems like it should be enough. Mark From: Wlodek, Jakub <[email protected]>
Hi Mark, I ran the callbackQueueShow command, and it does seem that the callbacks overflow by a little bit: epics> callbackQueueShow PRIORITY HIGH-WATER MARK ITEMS IN Q Q SIZE % USED Q OVERFLOWS cbLow 2000 0 2000 0.0 64 cbMedium 0 0 2000 0.0 0 cbHigh 0 0 2000 0.0 0 Meaning that the additional plugins are most likely the cause of this error message. I can add the callbackSetQueueSize(5000), and the messages go away. Is there a reason we shouldn't slightly increase the queue size in base to avoid this? Even around 2500 for example should solve this issue. Jakub From: Mark Rivers <[email protected]> Ø
In my case, I see at least a few printings of that error on every IOC boot,
Ø
though this may be because I have several additional plugins loading Ø
PVs at startup, which means that I have more PVs than the example IOCs, however,
Ø
I had the same amount of plugins loading without this error on earlier releases. I just typed the callbackQueueShow command on my simDetector IOC. I see the following:\ epics> callbackQueueShow PRIORITY HIGH-WATER MARK ITEMS IN Q Q SIZE % USED Q OVERFLOWS cbLow 1875 0 2000 0.0 0 cbMedium 0 0 2000 0.0 0 cbHigh 0 0 2000 0.0 0 Note that I did not overflow the queue, but it was close, 1875/2000. I started the IOC 4 times and that high-water mark varied from 1580 to 1930. So sometimes it will probably be over 2000. What do you see? You can probably eliminate the overflows by adding the following command to your startup script before iocInit() callbackSetQueueSize(5000) Mark From: Wlodek, Jakub <[email protected]>
Hi Mark, I am currently running the following versions of all modules: ADCore - R3-8 base - R7.0.3.1 asyn - R4-37 I have also seen this buffer full error with ADCore R3-7 IOCs, though I don't remember seeing it on any earlier versions, and I don't remember which base/asyn those were running. In my case, I see at least a few printings of that error on every IOC boot, though this may be because I have several additional plugins loading PVs at startup, which means that I have more PVs than the example IOCs, however, I had the same amount of plugins loading without this error on earlier
releases. The other error also happens every IOC boot, but that one I haven't seen until recently, although it is possible it simply went unnoticed by me in earlier versions. Jakub From: Mark Rivers <[email protected]> Hi Jakub, I have been seeing both of those messages on my areaDetector IOCs as well. Ø
Error (511,511) PV: 13SIM1::TIMEZONE devStringinEnvVar (init_record) Illegal INP parm field That message happens every time the IOC boots for me. I have been assuming is a problem with devIocStats, and have been meaning to track it down. Ø
callbackRequest: cbLow ring buffer full That message only very occasionally happens for me at the end of iocInit. I believe it happens when the number and rate of driver callbacks for PVs are too large, so that the queue in EPICS base overflows. We could probably eliminate
them by increasing the size of that queue. But before doing that I want to know why they just started happening, and I did not see them before. I am running base 7.0.3.1 and the master branch of asyn. The question is whether it was a change in either of those that caused the messages to start happening. Note that this error can actually be serious, because it means that some records will not have their correct values, because the callback events were missed. Do you see those cbLow messages every time you boot? What version of base, asyn and ADCore are you using? Mark From: Tech-talk <[email protected]>
On Behalf Of Wlodek, Jakub via Tech-talk Hi all, I have been working on setting up some new USB cameras at a beamline recently, and I noticed two errors showing up in the IOC shell, though the IOCs for the cameras actually work fine after startup. I wanted to ask if anyone knows why these show up, and how to resolve them. To make sure it wasn't specific to the driver I had written, I compiled ADSimDetector against the same base, asyn etc, and I still saw them showing up: The first seems to be an issue with autosave, as this shows up right after reboot_restore starts: ---------------------------------------------------------------------------- reboot_restore: entry for file 'auto_settings.sav' reboot_restore: Found filename 'auto_settings.sav' in restoreFileList. *** restoring from './autosave/auto_settings.sav' at initHookState 6 (before record/device init) *** reboot_restore: done with file 'auto_settings.sav' Error (511,511) PV: 13SIM1::TIMEZONE devStringinEnvVar (init_record) Illegal INP parm field The second issue happens after the auto_settings.req file is loaded: ------------------------------------------------------------------------------ create_monitor_set("auto_settings.req", 30, "P=13SIM1:") callbackRequest: cbLow ring buffer full callbackRequest: cbLow ring buffer full callbackRequest: cbLow ring buffer full And is buffer full message is repeated ~30 times. After all of these messages, I get a message that all the PVs are connected, and the IOC runs with no issues, which is puzzling to me. If anyone could give me some advice on the source and solution to these that would be great. Regards, Jakub Wlodek |