Using all the cores available on modern processors

[<nick.rees@diamond.ac.uk>]

This is probably aimed at Mark Rivers (sorry Mark!) but I am hoping other people may have seen the same symptoms. As I mentioned at the last EPICS meeting, on a number of development projects over the past year (mostly with area detector systems, but with some others) we have run into what superficially looks like a common problem.

When we configure a system simply (e.g. an areaDetector configuration which just reads out data and writes it to disk and does nothing else apart from scalar status callbacks) then the main CPU load is on a single core and it can use up to 90% of that core quite happily. 

When we make things more complicated and add processing plugins, for example, the data throughput to disk drops off dramatically and frames start buffering up (and some may be dropped). The CPU load is distributed across all cores, but they are only loaded at the 20-30% level, so the system seems largely idle with plenty of processing power available, but it can't be utilised.

Typically we have seen these problems on dual socket NUMA architecture Intel systems, with a non pre-emptively scheduled Windows or Linux OS.

In theorising about this we have conjectured a number of scenarios (in some sort of priority order):

1. AreaDetector has a locking issue and processing is held up by tasks holding locks that they shouldn't have.
2. Because this has happened on non-preemptively scheduled systems, the high priority tasks aren't getting CPU for some reason (despite there being CPU available). This may be related to (1) so maybe the system isn't overcoming priority inversions properly.
3. There is some other resource bottleneck - such as a bottleneck in the QPI because of the NUMA architecture.
4. There is a problem with top or the Windows performance monitor which doesn't account for overheads properly in a busy multi-core system.

So, at this stage we have not looked at this problem in any detail, we have just noticed the symptoms. Has anyone else come across it and have any pointers before we invest a serious amount of effort looking at it. I suspect that if we are going to understand this it may take some time, and may have some design implications for EPICS on modern architectures.

Cheers,

Nick Rees
Principal Software Engineer           Phone: +44 (0)1235-778430
Diamond Light Source                  Fax:   +44 (0)1235-446713


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom