Excessive scan times for periodic scans

["Konrad, Martin via Core-talk" <core-talk@aps.anl.gov>]

Hi,
On one of our IOCs I'm seeing the following warning:

dbScan warning from '1 second' scan thread:
        Scan processing averages 1.405 seconds (1.028 .. 2.559).
        Over-runs have now happened 10 times in a row.
        To fix this, move some records to a slower scan rate.

The IOC is talking to 132 devices using StreamDevice. "scanppl" reports
11408 records on the '1 second' scan list. The VM hosting the IOC is
running happily with 20-30% CPU load on each of its 8 cores. "top -H"
reports 10-15% for the "scan-1" thread.

Naively, I would expect the "scan-1" thread to use 100% of one core (at
least most of the time) when the warning above gets printed. However,
the records in question are being scanned asynchronously which might put
part of the burden on other threads. Nevertheless it's unclear to me why
the "scan-1" thread can't keep up and if there is anything I can do to
improve performance.

I'm also wondering if the work of scanning can be spread out over
multiple threads as suggested by Ralph in [1] - it's unclear to me if
this ever got implemented.

Thanks a lot,

Martin

[1] http://accelconf.web.cern.ch/AccelConf/ICALEPCS2013/papers/moppc124.pdf

-- 
Martin Konrad
High Performance Controls Team Leader
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: konrad@frib.msu.edu