Multiple Prosilica GIGE cameras contention

["Zhang, Dehong via Tech-talk" <tech-talk at aps.anl.gov>]

Hi EPICS Friends!

In some beamlines, we would like to run >= 5 separate areaDetector IOCs,

on one server, for Mako G125B GIGE cameras, one IOC for each camera.

It looks like we are experiencing some kind of resource contention.  The

symptom is like:

one IOC (one camera) running                -- good, caput/caget and PyDM GUI (old and new) all working

start another IOC for another camera -- old GUI (already established CA connections)

                                                                                    continue to work for the previous camera;

                                                                                    but new caput/caget and PyDM GUI can only work

                                                                                    with the new IOC/camera

start another IOC for another camera -- old GUI (already established CA connections)

                                                                                    continue to work for the previous cameras;

                                                                                    but new caput/caget and PyDM GUI can only work

                                                                                    with the new IOC/camera

...

It is like a new IOC pushes the old ones out of the CA world.  The old IOCs will

continue to serve the already-established connections, but will not respond to

new CA requests, and some record instances even become Invalid.   Only the

newest IOC is healthy.

In the first beamline with 3.14.12, asyn 4.31, ADCore 2.6 running on CentOS 7,

the issue disappeared after we installed a 10G switch, set the Acquire Mode to

"Fixed Rate", AcquirePeriod to 0.5 (2 Hz), and set PSByteRate to 10M

In the second beamline with 3.15.5, asyn 4.31, ADCore 2.6 running on CentOS 9,

the issue persists with the same HW and SW configuration, even after reducing

PSByteRate to 5M.

These cameras have about 1.3 M pixels, at 12 bits mono.  So each frame is only

like 2.5 MB. 

We even killed the caRepeator, and let the CA system start a new one when we

try caget.  The caget still can not reach PVs from the old IOCs, it can only reach

the newest IOC.

We connect the cameras to the Giga-Bit ports, but connect the computer to

the 10G port.  Does the switch have some settings affecting this?

Have you experienced a situation like this?  How did you solve it?

Thank you very much, best regards,

Dehong