Re: [EXTERNAL] Re: Asyn device support does not reconnect

[Miroslaw Dach via Tech-talk <tech-talk at aps.anl.gov>]

Hi Matt,

Thanks for all of your hints on that. 

>The controller may have been reconfigured while it was disconnected. Then the database is out of sync with the controller. You can move in the wrong direction, too fast, burn out a motor, etc.

The controller may have been replaced and at factory default or some other random configuration. Same problem as 1).
For the Galil in particular, if the settings were not saved to memory via ‘BN’, and the controller is rebooted, same problem as1).
We restore the position via autosave before IOC init. So if the controller reboots, and the positions revert to zero, then the software reconnects, and someone attempts a move, it’s easy to move a greater distance than you intended.
I’m sure there’s more ways it can fail too…

These are valid points.

We have some critical systems where the reconnection is desired. 

The configuration after the reconnection should not be an issue for thoses systems.  

Best Regards

Mirek

On Thu, Nov 7, 2024 at 11:58 AM Pearson, Matthew <pearsonmr at ornl.gov> wrote:

Hi,

 

This is an aside that you may not find useful, but we go out of our way to not auto-reconnect motion controllers. Too many things can go wrong in terms of controller configuration (see below). So we force the user to reboot the IOC to reestablish communications.  

 

Since we rely on autosave to correctly configure a controller, these things can go wrong:

 

1. The controller may have been reconfigured while it was disconnected. Then the database is out of sync with the controller. You can move in the wrong direction, too fast, burn out a motor, etc.
2. The controller may have been replaced and at factory default or some other random configuration. Same problem as 1).
3. For the Galil in particular, if the settings were not saved to memory via ‘BN’, and the controller is rebooted, same problem as1).
4. We restore the position via autosave before IOC init. So if the controller reboots, and the positions revert to zero, then the software reconnects, and someone attempts a move, it’s easy to move a greater distance than you intended.
5. I’m sure there’s more ways it can fail too…

 

I know with additional effort many of those issues could be mitigated, but forcing an IOC reboot is a safe option in the absence of those measures.

 

Cheers,

Matt

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Miroslaw Dach via Tech-talk
Sent: Thursday, November 7, 2024 2:35 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: [EXTERNAL] Re: Asyn device support does not reconnect

 

Hi Mark,

 

Many thanks for your quick reply. 

 

>I am using Galil 3-6 and I don’t see this problem with my DMC-4183 controllers.  When I power-cycle the controller the IOC reconnects with no problem.  These are the messages I see when I power it off for about 10 seconds.

We have observed the same problem using Galil-4-0-02 and Galil-3-0-V3-6 drivers. If the driver works for you well with reconnection then we have to check the startup script  if something is not missing.

 

>Have you added this line to your startup script?

asynSetOption("custom_device", 0, "disconnectOnReadTimeout", "Y")

No, we do not have this entry. I will add it to our software and see how it works.

Many thanks again for your feedback and expertise.

 

Best Regards

Mirek

 

 

 

 

 

 

On Thu, Nov 7, 2024 at 10:42 AM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Mirek,

 

·  1. Galil (Galil-4-0-02) motion controller driver. When the connection is lost it reconnects and crashes straight after with the message:

 

I am using Galil 3-6 and I don’t see this problem with my DMC-4183 controllers.  When I power-cycle the controller the IOC reconnects with no problem.  These are the messages I see when I power it off for about 10 seconds.

 

Disconnected from DMC4183 Rev 1.3k at 10.54.160.161

2024/11/07 12:33:43.977 GALILSYNC2 -1 autoConnect could not connect: Can't connect to 10.54.160.161:23 TCP: No route to host

Connected to DMC4183 Rev 1.3k at 10.54.160.161

Code started successfully on model DMC4183 Rev 1.3k, address 10.54.160.161

 

·  2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

 

That is an issue that we should look at for ADVimba, ADSpinnaker, and other areaDetector drivers with Ethernet connections.  It would be nice to solve it in a centralized place, perhaps in ADGenICam.

 

·  3. Custom stream device support driver which uses the call:

·  drvAsynIPPortConfigure("custom_device", "HW_dev:8003",0,0,0)

·  The ioc does not reconnect when the underlying hardware is off for a longer time (2-3 hours) and powered back.

 

Have you added this line to your startup script?

 

asynSetOption("custom_device", 0, "disconnectOnReadTimeout", "Y")

 

That will close the socket on the IOC when there is a timeout.  If autoConnect is true, then it will then continuously attempt to reconnect every few seconds.

 

Mark

 

 

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Miroslaw Dach via Tech-talk
Sent: Thursday, November 7, 2024 12:02 PM
To: EPICS Tech-Talk <tech-talk at aps.anl.gov>
Subject: Asyn device support does not reconnect

 

Hi All,

 

We are using in ALS various drivers which are based on Asyn.

Some of them do not auto reconnect. I wanted to consult with you 

what could be done in order to make them auto reconnect.

 

1. Galil (Galil-4-0-02) motion controller driver. When the connection is lost it reconnects and crashes straight after with the message:

free(): invalid size

or

munmap_chunk(): invalid pointer

 

example:

Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
connected set false
Disconnected from DMC4040 Rev 1.3i-SER at 131.243.196.71
2024/10/30 15:17:58.330 GALILSYNC0 -1 autoConnect could not connect: Can't connect to 131.243.196.71:23 TCP: Connection timed out
2024/10/30 15:20:09.411 GALILSYNC0 -1 port is now connected
connected_=1
Connected to DMC4040 Rev 1.3i-SER at 131.243.196.71
free(): invalid size
Aborted (core dumped)

 

2. Vimba ccd camera driver simply does not reconnect when the connection with the camera  is lost.

We use the driver for the Manta G-235B camera (with the Allied Vision 1800 U-501 NIR  cheap) and also older cameras Manta_G-145 . We used before the Prosilica driver what was doing the auto connect but Vimba does not do that. 

 

3. Custom stream device support driver which uses the call:

drvAsynIPPortConfigure("custom_device", "HW_dev:8003",0,0,0)

The ioc does not reconnect when the underlying hardware is off for a longer time (2-3 hours) and powered back.

 

Best Regards

Mirek