Experimental Physics and
Industrial Control System

dbScan warning from '1 second' scan thread:
        Scan processing averages 1.405 seconds (1.028 .. 2.559).
        Over-runs have now happened 10 times in a row.
        To fix this, move some records to a slower scan rate.

The IOC is talking to 132 devices using StreamDevice. "scanppl" reports
11408 records on the '1 second' scan list. The VM hosting the IOC is
running happily with 20-30% CPU load on each of its 8 cores. "top -H"
reports 10-15% for the "scan-1" thread.

Naively, I would expect the "scan-1" thread to use 100% of one core (at
least most of the time) when the warning above gets printed. However,
the records in question are being scanned asynchronously which might put
part of the burden on other threads. Nevertheless it's unclear to me why
the "scan-1" thread can't keep up and if there is anything I can do to
improve performance.

I'm also wondering if the work of scanning can be spread out over
multiple threads as suggested by Ralph in [1] - it's unclear to me if
this ever got implemented.

tux% bin/linux-x86_64/softIoc
epics> help callbackParallelThreads
callbackParallelThreads 'no of threads' priority
epics> callbackParallelThreads 2 LOW
epics> callbackParallelThreads 2 HIGH
epics> dbLoadRecords db/softIocExit.db IOC=anj
epics> iocInit
Starting iocInit
############################################################################
## EPICS R7.0.2.1-DEV
## EPICS Base built Mar 20 2019
############################################################################
iocRun: All initialization complete
epics> epicsThreadShowAll
            NAME       EPICS ID   LWP ID   OSIPRI  OSSPRI  STATE
          _main_      0x2037450    13485      0       0       OK
          errlog      0x20facd0    14140     10       0       OK
          taskwd      0x20fb2e0    14141     10       0       OK
      timerQueue      0x20fbd40    14142     70       0       OK
         cbLow-0      0x20fc180    14143     59       0       OK
         cbLow-1      0x20fc290    14144     59       0       OK
        cbMedium      0x20fc6c0    14145     64       0       OK
        cbHigh-0      0x20fcaf0    14146     71       0       OK
        cbHigh-1      0x20fce50    14147     71       0       OK
        dbCaLink      0x20fd3f0    14148     50       0       OK
        scanOnce      0x2111050    14149     67       0       OK
         scan-10      0x21113b0    14150     60       0       OK
          scan-5      0x2111710    14151     61       0       OK
          scan-2      0x2111a70    14152     62       0       OK
          scan-1      0x2111dd0    14153     63       0       OK
        scan-0.5      0x2112130    14154     64       0       OK
        scan-0.2      0x2112490    14155     65       0       OK
        scan-0.1      0x21127f0    14156     66       0       OK
         CAS-TCP      0x211b2b0    14157     18       0       OK
         CAS-UDP      0x211b610    14158     16       0       OK
      CAS-beacon      0x211b970    14159     17       0       OK
epics>

-- 
Arguing for surveillance because you have nothing to hide is no
different than making the claim, "I don't care about freedom of
speech because I have nothing to say." -- Edward Snowdon