RE: Asyn device support does not reconnect

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mirek,

 

I have fixed ADVimba so it now handles reconnecting to the camera after network outages or power-cycling the camera.  This is described in the release notes for the upcoming R1-5 release:

 

• Added support for reconnecting the camera to the driver after network interruption or power-cycling the camera.
• Added code to receive callbacks when Vimba SDK detects connects or disconnects.
• Added code to process manual connect and disconnect events, for example using asynRecord.
• Call pasynManager->exceptionConnect or exceptionDisconnect when the camera connection state changes. This causes the current connection state to be displayed in OPI screens and prevents device support calls to the driver when the camera is disconnected.
• Changed VimbaFeature::isAvailble to reconnect to the feature after camera power cycle.
• Added disconnectCamera() method.

 

You can manually disconnect and connect the camera to the driver using the Connect and Disconnect buttons in the upper left of the ADVimba OPI screen (attached).

 

If you disconnect the network or power-cycle the camera the connection status above those buttons will change to Disconnected.  Once the network is restored or the camera is powered back on that status should automatically go back to Connected after a few seconds. 

 

I have found that when it reconnects after a power-cycle there will sometimes be errors when you try to start acquiring.  This seems to always be fixed by manually Disconnecting and then Connecting again.

 

If you power-cycle the camera then the actual camera settings will almost certainly not agree with the EPICS values.  This can be fixed by using the autosave configMenu utility, which was added to ADCore in R3-12.  I have attached a screenshot that shows that I have saved 2 configurations of the ADVimba IOC, one for the full camera readout, and one for an ROI.  Once the camera power-cycle is complete and it is reconnected, I can press Load on one of those configurations, and all the settings will be restored to EPICS and pushed to the camera.

 

Please test the master branch of ADVimba and let me know if it works for you.

 

Cheers,

Mark

 

 

 

From: Miroslaw Dach <mdach at lbl.gov>
Sent: Friday, November 8, 2024 7:29 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Re: Asyn device support does not reconnect

 

Hi Mark,

 

Many thanks for checking the issue with the Vimba driver.

Our case in ALS is that from time to time the cameras (due to exposure to radiation or whatever other reason) simply hang and we have to power cycle them. Such a camera is powered cycled but the ioc does not reconnect automatically with the vimba driver. When using the legacy prosilica driver the ioc was reconnecting so it would be very much desirable to have the same behaviour with vimba driver.

 

Best Regards

Mirek

 

 

 

 

On Fri, Nov 8, 2024 at 4:22 PM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Mirek,

 

• 2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

 

You need to be specific about what you mean by “connection with the camera is lost”.  There are at least 2 cases:

1. The Ethernet connection is interrupted, either by unplugging the cable, restarting a switch, etc.
2. The camera is power-cycled

 

I just tested ADVimba for case 1, unplugging the Ethernet cable for about 15 seconds.  After a few seconds, these messages appear on the IOC console:

 

2024/11/08 18:04:53.889 ADVimba::connectionCallback calling disconnect()

2024/11/08 18:04:54.139 13VMB1:cam1:PoolPollStats devAsynInt32::reportQueueRequestStatus queueRequest error port VMB1 not connected

 

When this happens the green Connected message on the OPI screen changes to red Disconnected.  This alerts the user to the problem.

 

After reconnecting the Ethernet cable the following message appears on the IOC console:

 

2024/11/08 18:05:05.896 ADVimba::connectionCallback calling connect()

 

The Disconnected message on the OPI screen changes back to Connected.

 

I am then able to operate the camera again normally.  Do you see the same thing?

 

If the camera is power-cycled (case 2) then the same messages appear on the console and on the OPI screen.  I find that I can still communicate with the camera.  However, it does not stream images correctly.

 

I think that power-cycling the camera leads to the same issues that Matt raised for the Galil controller.  When the camera is power cycled its settings (readout region, exposure time, trigger mode, etc.) will be back to power-up defaults and completely out of sync with the EPICS settings.  How do you propose that it should work in this case?  The simplest thing to do it to restart the IOC. 

 

In ADCore R3-12 I added the following:

 

ADAutoSaveMenu.req

• New file to be used with the configMenu utility in autosave. It saves all of the camera and plugin settings defined in iocBoot/iocXXX/auto_settings.req. This allows manually saving and loading different configurations for different setups with the same IOC. It also allows saving and then loading the settings when a camera is reset or restarted, without restarting the IOC.

 

In principle this should allow you to download the last autosaved settings after the camera is power-cycled.  However, this has not been extensively tested, and when I just tried saving with configMenu I got a failure.  I need to track that down.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Miroslaw Dach via Tech-talk
Sent: Thursday, November 7, 2024 12:02 PM
To: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Asyn device support does not reconnect

 

Hi All,

 

We are using in ALS various drivers which are based on Asyn.

Some of them do not auto reconnect. I wanted to consult with you 

what could be done in order to make them auto reconnect.

 

1. Galil (Galil-4-0-02) motion controller driver. When the connection is lost it reconnects and crashes straight after with the message:

free(): invalid size

or

munmap_chunk(): invalid pointer

 

example:

Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
connected set false
Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
2024/10/30 15:17:58.330 GALILSYNC0 -1 autoConnect could not connect: Can't connect to 131.243.196.71:23 TCP: Connection timed out
2024/10/30 15:20:09.411 GALILSYNC0 -1 port is now connected
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
free(): invalid size
Aborted (core dumped)

 

2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

We use the driver for the Manta G-235B camera (with the Allied Vision 1800 U-501 NIR  cheap) and also older cameras Manta_G-145 . We used before the Prosilica driver what was doing the auto connect but Vimba does not do that. 

 

3. Custom stream device support driver which uses the call:

drvAsynIPPortConfigure("custom_device", "HW_dev:8003",0,0,0)

The ioc does not reconnect when the underlying hardware is off for a longer time (2-3 hours) and powered back.

 

Best Regards

Mirek

 

Attachment: ADVimba.png
Description: ADVimba.png

Attachment: configMenu.png
Description: configMenu.png