Re: Excessive scan times for periodic scans

["Johnson, Andrew N. via Core-talk" <core-talk@aps.anl.gov>]

Hi Martin,

On 4/3/19 11:58 AM, Konrad, Martin via Core-talk wrote:

dbScan warning from '1 second' scan thread:
        Scan processing averages 1.405 seconds (1.028 .. 2.559).
        Over-runs have now happened 10 times in a row.
        To fix this, move some records to a slower scan rate.

The IOC is talking to 132 devices using StreamDevice. "scanppl" reports
11408 records on the '1 second' scan list. The VM hosting the IOC is
running happily with 20-30% CPU load on each of its 8 cores. "top -H"
reports 10-15% for the "scan-1" thread.

How heavily loaded are the callback threads cb{Low,Medium,High}?

Naively, I would expect the "scan-1" thread to use 100% of one core (at
least most of the time) when the warning above gets printed. However,
the records in question are being scanned asynchronously which might put
part of the burden on other threads. Nevertheless it's unclear to me why
the "scan-1" thread can't keep up and if there is anything I can do to
improve performance.

Do you have chains of records that are linked together?

What other resources do your devices share that might limit how fast the IOC can communicate with them? How long does it take to "talk" to each device?

I'm also wondering if the work of scanning can be spread out over
multiple threads as suggested by Ralph in [1] - it's unclear to me if
this ever got implemented.

Multiple callback threads was certainly added, and I suspect it might help your situation because the second-half asynchronous processing is probably happening there. For example:

tux% bin/linux-x86_64/softIoc
epics> help callbackParallelThreads
callbackParallelThreads 'no of threads' priority
epics> callbackParallelThreads 2 LOW
epics> callbackParallelThreads 2 HIGH
epics> dbLoadRecords db/softIocExit.db IOC=anj
epics> iocInit
Starting iocInit
############################################################################
## EPICS R7.0.2.1-DEV
## EPICS Base built Mar 20 2019
############################################################################
iocRun: All initialization complete
epics> epicsThreadShowAll
            NAME       EPICS ID   LWP ID   OSIPRI  OSSPRI  STATE
          _main_      0x2037450    13485      0       0       OK
          errlog      0x20facd0    14140     10       0       OK
          taskwd      0x20fb2e0    14141     10       0       OK
      timerQueue      0x20fbd40    14142     70       0       OK
         cbLow-0      0x20fc180    14143     59       0       OK
         cbLow-1      0x20fc290    14144     59       0       OK
        cbMedium      0x20fc6c0    14145     64       0       OK
        cbHigh-0      0x20fcaf0    14146     71       0       OK
        cbHigh-1      0x20fce50    14147     71       0       OK
        dbCaLink      0x20fd3f0    14148     50       0       OK
        scanOnce      0x2111050    14149     67       0       OK
         scan-10      0x21113b0    14150     60       0       OK
          scan-5      0x2111710    14151     61       0       OK
          scan-2      0x2111a70    14152     62       0       OK
          scan-1      0x2111dd0    14153     63       0       OK
        scan-0.5      0x2112130    14154     64       0       OK
        scan-0.2      0x2112490    14155     65       0       OK
        scan-0.1      0x21127f0    14156     66       0       OK
         CAS-TCP      0x211b2b0    14157     18       0       OK
         CAS-UDP      0x211b610    14158     16       0       OK
      CAS-beacon      0x211b970    14159     17       0       OK
epics>

Note there are two cbLow and cbHigh threads above but only one cbMedium because I didn't ask for two there. You can set the default number of threads used for all priorities at once by omitting the second argument to the callbackParallelThreads command.

We have not tried to implement parallel periodic scan threads yet, IIRC we suspected there would be more likely to be issues with synchronization or deadlocks in those cases which could affect device or driver support. That's also why we default to having a single callback thread, but allow you to add more in your startup script.

HTH,

- Andrew

-- 
Arguing for surveillance because you have nothing to hide is no
different than making the claim, "I don't care about freedom of
speech because I have nothing to say." -- Edward Snowdon