Re: Asyn device support does not reconnect

[Miroslaw Dach via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

Many thanks for checking the issue with the Vimba driver.

Our case in ALS is that from time to time the cameras (due to exposure to radiation or whatever other reason) simply hang and we have to power cycle them. Such a camera is powered cycled but the ioc does not reconnect automatically with the vimba driver. When using the legacy prosilica driver the ioc was reconnecting so it would be very much desirable to have the same behaviour with vimba driver.

Best Regards

Mirek

On Fri, Nov 8, 2024 at 4:22 PM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Mirek,

 

• 2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

 

You need to be specific about what you mean by “connection with the camera is lost”.  There are at least 2 cases:

1. The Ethernet connection is interrupted, either by unplugging the cable, restarting a switch, etc.
2. The camera is power-cycled

 

I just tested ADVimba for case 1, unplugging the Ethernet cable for about 15 seconds.  After a few seconds, these messages appear on the IOC console:

 

2024/11/08 18:04:53.889 ADVimba::connectionCallback calling disconnect()

2024/11/08 18:04:54.139 13VMB1:cam1:PoolPollStats devAsynInt32::reportQueueRequestStatus queueRequest error port VMB1 not connected

 

When this happens the green Connected message on the OPI screen changes to red Disconnected.  This alerts the user to the problem.

 

After reconnecting the Ethernet cable the following message appears on the IOC console:

 

2024/11/08 18:05:05.896 ADVimba::connectionCallback calling connect()

 

The Disconnected message on the OPI screen changes back to Connected.

 

I am then able to operate the camera again normally.  Do you see the same thing?

 

If the camera is power-cycled (case 2) then the same messages appear on the console and on the OPI screen.  I find that I can still communicate with the camera.  However, it does not stream images correctly.

 

I think that power-cycling the camera leads to the same issues that Matt raised for the Galil controller.  When the camera is power cycled its settings (readout region, exposure time, trigger mode, etc.) will be back to power-up defaults and completely out of sync with the EPICS settings.  How do you propose that it should work in this case?  The simplest thing to do it to restart the IOC. 

 

In ADCore R3-12 I added the following:

 

ADAutoSaveMenu.req

• New file to be used with the configMenu utility in autosave. It saves all of the camera and plugin settings defined in iocBoot/iocXXX/auto_settings.req. This allows manually saving and loading different configurations for different setups with the same IOC. It also allows saving and then loading the settings when a camera is reset or restarted, without restarting the IOC.

 

In principle this should allow you to download the last autosaved settings after the camera is power-cycled.  However, this has not been extensively tested, and when I just tried saving with configMenu I got a failure.  I need to track that down.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Miroslaw Dach via Tech-talk
Sent: Thursday, November 7, 2024 12:02 PM
To: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Asyn device support does not reconnect

 

Hi All,

 

We are using in ALS various drivers which are based on Asyn.

Some of them do not auto reconnect. I wanted to consult with you 

what could be done in order to make them auto reconnect.

 

1. Galil (Galil-4-0-02) motion controller driver. When the connection is lost it reconnects and crashes straight after with the message:

free(): invalid size

or

munmap_chunk(): invalid pointer

 

example:

Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
connected set false
Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
2024/10/30 15:17:58.330 GALILSYNC0 -1 autoConnect could not connect: Can't connect to 131.243.196.71:23 TCP: Connection timed out
2024/10/30 15:20:09.411 GALILSYNC0 -1 port is now connected
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
free(): invalid size
Aborted (core dumped)

 

2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

We use the driver for the Manta G-235B camera (with the Allied Vision 1800 U-501 NIR  cheap) and also older cameras Manta_G-145 . We used before the Prosilica driver what was doing the auto connect but Vimba does not do that. 

 

3. Custom stream device support driver which uses the call:

drvAsynIPPortConfigure("custom_device", "HW_dev:8003",0,0,0)

The ioc does not reconnect when the underlying hardware is off for a longer time (2-3 hours) and powered back.

 

Best Regards

Mirek