RE: Multiple Prosilica GIGE cameras contention

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Dehong,

 

There are potentially 2 completely separate issues here:

 

1. The issue with only be able to reach one of the IOCs with CA clients is the problem Ralph addressed.  This is independent of whether those IOCs are running GigE cameras or something else.  If all CA clients are on the same subnet as the camera IOC computer then there will not be a problem as long as you use the default CA settings that effectively has EPICS_CA_ADDR_LIST = broadcast address of that subnet.  You will not be able to reach multiple IOCs if you use EPICS_CA_ADDR_LIST=IP address of camera IOC server. If the clients are on another subnet then Ralph’s iptables solution is one option.  Another is to get your network people to configure the router to allow “directed broadcasts”, so clients on subnet A can broadcast messages to the IOC server on subnet B.

 

2. There can be an issue with the total network bandwidth if you run multiple cameras on the same network, and the network or the NIC in the server are not fast enough.  If this is an issue, then ADVimba lets you restrict the bandwidth that each camera can use so they can coexist without problems.

 

Mark

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Zhang, Dehong via Tech-talk
Sent: Wednesday, April 23, 2025 2:26 PM
To: tech-talk at aps.anl.gov
Subject: Multiple Prosilica GIGE cameras contention

 

Hi EPICS Friends!

 

In some beamlines, we would like to run >= 5 separate areaDetector IOCs,

on one server, for Mako G125B GIGE cameras, one IOC for each camera.

 

It looks like we are experiencing some kind of resource contention.  The

symptom is like:

one IOC (one camera) running                -- good, caput/caget and PyDM GUI (old and new) all working

start another IOC for another camera -- old GUI (already established CA connections)

                                                                                    continue to work for the previous camera;

                                                                                    but new caput/caget and PyDM GUI can only work

                                                                                    with the new IOC/camera

start another IOC for another camera -- old GUI (already established CA connections)

                                                                                    continue to work for the previous cameras;

                                                                                    but new caput/caget and PyDM GUI can only work

                                                                                    with the new IOC/camera

...

 

It is like a new IOC pushes the old ones out of the CA world.  The old IOCs will

continue to serve the already-established connections, but will not respond to

new CA requests, and some record instances even become Invalid.   Only the

newest IOC is healthy.

 

In the first beamline with 3.14.12, asyn 4.31, ADCore 2.6 running on CentOS 7,

the issue disappeared after we installed a 10G switch, set the Acquire Mode to

"Fixed Rate", AcquirePeriod to 0.5 (2 Hz), and set PSByteRate to 10M

 

In the second beamline with 3.15.5, asyn 4.31, ADCore 2.6 running on CentOS 9,

the issue persists with the same HW and SW configuration, even after reducing

PSByteRate to 5M.

 

These cameras have about 1.3 M pixels, at 12 bits mono.  So each frame is only

like 2.5 MB. 

 

We even killed the caRepeator, and let the CA system start a new one when we

try caget.  The caget still can not reach PVs from the old IOCs, it can only reach

the newest IOC.

 

We connect the cameras to the Giga-Bit ports, but connect the computer to

the 10G port.  Does the switch have some settings affecting this?

 

Have you experienced a situation like this?  How did you solve it?

 

Thank you very much, best regards,

Dehong