RE: Asyn not automatically reconnecting

["Daykin, Evan" <daykin@frib.msu.edu>]

>When your client shuts down it does so "cleanly", closing the socket?  It looks like it, since drvAsynIPPort detected the closed connection immediately, without first getting any timeouts, etc.

Yes, I just verified this with wireshark. The TCP connection is closed gracefully. Netstat on both ends shows there is no connection as well.

> When the client comes back up do you see any different messages?
No, the "...read." message just repeats extremely quickly.

> When the client comes back up you can trying using the asynRecord GUI to try to manually reconnect.  Does that work?
I am not familiar with this. Can you show me where to find that?

Something else interesting is that rebooting results in the correct behavior; the connection is re-established. However, if the controlling process on the client device is killed first, this issue persists.

-Evan

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu] 
Sent: Friday, February 23, 2018 10:49 AM
To: Daykin, Evan <daykin@frib.msu.edu>; tech-talk@aps.anl.gov
Subject: Re: Asyn not automatically reconnecting

Hi Evan,


When your client shuts down it does so "cleanly", closing the socket?  It looks like it, since drvAsynIPPort detected the closed connection immediately, without first getting any timeouts, etc.


When the client comes back up do you see any different messages?


When the client comes back up you can trying using the asynRecord GUI to try to manually reconnect.  Does that work?


Mark



________________________________
From: Daykin, Evan <daykin@frib.msu.edu>
Sent: Friday, February 23, 2018 8:55 AM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Asyn not automatically reconnecting


Here is the output from the moment I killed the connection:



2018/02/23 09:51:47.086 TEST-RF-DCU-N0001 read

VALUE AMP_WATER_FLOW AMP 2 7.0049e-168\n

2018/02/23 09:51:47.087 TEST-RF-DCU-N0001 set Eos 0

2018/02/23 09:51:47.087 asynManager::portThread port=TEST-RF-DCU-N0001 callback

2018/02/23 09:51:47.087 TEST-RF-DCU-N0001 addr -1 queueRequest priority 0 not lockHolder

2018/02/23 09:51:47.088 TEST-RF-DCU-N0001 schedule queueRequest timeout

2018/02/23 09:51:47.088 asynManager::portThread port=TEST-RF-DCU-N0001 callback

2018/02/23 09:51:47.088 test-rf-dcu-n0001.cts:5001 read.

2018/02/23 09:51:47.088 test-rf-dcu-n0001.cts:5001 read 0

2018/02/23 09:51:47.088 Close test-rf-dcu-n0001.cts:5001 connection (fd 18): Read from broken connection

2018/02/23 09:51:47.088 _TEST-RF-DCU-N0001:asyn: exception 0, asynExceptionConnect

2018/02/23 09:51:47.088 TEST-RF-DCU-N0001 read

2018/02/23 09:51:47.089 test-rf-dcu-n0001.cts:5001 read.

2018/02/23 09:51:47.089 test-rf-dcu-n0001.cts:5001 read.



The last line then repeats rapidly forever.





From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Friday, February 23, 2018 9:20 AM
To: Daykin, Evan <daykin@frib.msu.edu>; tech-talk@aps.anl.gov
Subject: RE: Asyn not automatically reconnecting



Can you set the asynTraceIOMask to 2 and the asynTraceMask to 0xff on that port just before you shut down the client?



From: Daykin, Evan [mailto:daykin@frib.msu.edu]
Sent: Friday, February 23, 2018 8:18 AM
To: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: RE: Asyn not automatically reconnecting



Hi Mark,



Thanks for the reply. We are using asyn 4.33-2 and Stream 2.6-3. The protocol file has a ReplyTimeout of 2000ms.



From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Thursday, February 22, 2018 6:20 PM
To: Daykin, Evan <daykin@frib.msu.edu<mailto:daykin@frib.msu.edu>>; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: RE: Asyn not automatically reconnecting



Hi Evan,



Ø  We have client devices that, when remotely updated and rebooted, the IOC does not attempt to reconnect automatically

Ø  TEST-RF-DCU-N0001 multiDevice:No canBlock:Yes autoConnect:Yes

Ø  enabled:Yes connected:No numberConnects 1

Ø  nDevices 0 nQueued 114 blocked:No



The above information says that indeed autoConnect is set to Yes, and it is currently not connected.  The 114 entries in the queue seems strange.



What version of asyn and what version of StreamDevice are you using?  What you are observing does not seem correct.



What timeouts if any have you specified in your protocol file?



Mark









From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Daykin, Evan
Sent: Thursday, February 22, 2018 4:32 PM
To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Asyn not automatically reconnecting



Hello,



Apologies if this has already been asked and answered. We have client devices that, when remotely updated and rebooted, the IOC does not attempt to reconnect automatically. Any records tied to it simply enter a TIMEOUT_ALARM state, and no asyn or StreamDevice error messages are printed.  autoConnect is not specified in drvAsynIPPortconfigure, which, according to the documentation, implicitly enables autoConnect/attempting to reconnect every 20 seconds.  Does anything else need to be done?



The devices are configured to communicate using Stream Device.

Relevant asyn configuration in the startup script:

-drvAsynIPPortConfigure("TEST-RF-DCU-N0001",  "test-rf-dcu-n0001.cts:5001")



asynReport after client is disconnected and brought back up:

TEST-RF-DCU-N0001 multiDevice:No canBlock:Yes autoConnect:Yes

    enabled:Yes connected:No numberConnects 1

    nDevices 0 nQueued 114 blocked:No

    asynManagerLock:No synchronousLock:Yes

    exceptionActive:No exceptionUsers 2 exceptionNotifys 0

    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1

    interposeInterfaceList

        asynOctet pinterface 0x7f0a07506900 drvPvt 0x6c3720

    interfaceList

        asynCommon pinterface 0x7f0a07502650 drvPvt 0x6bfe30

        asynOption pinterface 0x7f0a07502670 drvPvt 0x6bfe30

        asynOctet pinterface 0x6bff50 drvPvt 0x6bfe30

    Port test-rf-dcu-n0001.cts:5001: Disconnected

                    fd: -1

    Characters written: 127036

       Characters read: 138000



-Evan