EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: PMAC asyn autoconnect issue
From: Bruno Martins <[email protected]>
To: Mark Rivers <[email protected]>
Cc: "[email protected]" <[email protected]>, "[email protected]" <[email protected]>
Date: Mon, 12 Feb 2018 14:45:19 -0500
Hi Mark,

We're using tpmac-asyn 3.10.12, asyn R4-32.

Regarding pmacAsynIPConfigure, it immediately calls drvAsynIPPortConfigure: https://github.com/epicsdeb/pmacasyn/blob/master/pmacApp/pmacAsynIPPortSrc/pmacAsynIPPort.c#L184


You're right that the return code is asynError (from queueLockPort [asynManager.c:1832]) but queueRequest returns asynDisconnected [asynManager.c:1505]. I added more debugging.


I noticed two things:

1. If the IOC starts with the controller powered off, the above messages are printed. asyn/tpmac correctly knows the device is disconnected, but there's no attempt at a reconnection, so powering it on doesn't do anything. Here's the report in this case:

asynManager::queueLockPort queueRequest failed: port P0 not connected
2018/02/12 14:05:56.201 pmacController::lowLevelWriteRead: Error from pasynOctetSyncIO->writeRead. command: #2 ? F P
2018/02/12 14:05:56.201 drvPmacAxisGetStatus: not all status values returned. Status: 3

epics> asynReport 5 P0
P0 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:No numberConnects 0
    nDevices 0 nQueued 18 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 11 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7fcf1ca2c900 drvPvt 0xf5c060
    interfaceList
        asynCommon pinterface 0x7fcf1ca29650 drvPvt 0xf579e0
        asynOption pinterface 0x7fcf1ca29670 drvPvt 0xf579e0
        asynOctet pinterface 0xf57b00 drvPvt 0xf579e0
    Port 10.41.104.10:1025: Disconnected
                    fd: -1
    Characters written: 0
       Characters read: 0


2. If the IOC starts with the controller powered on, everything works as expected:

epics> asynReport 5 P0
P0 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 11 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7fafe57dc900 drvPvt 0x1aa6090
    interfaceList
        asynCommon pinterface 0x7fafe57d9650 drvPvt 0x1aa1a10
        asynOption pinterface 0x7fafe57d9670 drvPvt 0x1aa1a10
        asynOctet pinterface 0x1aa1b30 drvPvt 0x1aa1a10
    Port 10.41.104.10:1025: Connected
                    fd: 7
    Characters written: 9787
       Characters read: 5345


Then when we yank the power from the controller:

asynManager::queueLockPort queueRequest timed out
2018/02/12 14:33:03.069 pmacController::lowLevelWriteRead: Error from pasynOctetSyncIO->writeRead. command: #1 ? F P
2018/02/12 14:33:03.069 drvPmacAxisGetStatus: not all status values returned. Status: 3

epics> asynReport 5 P0
P0 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 18 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 11 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7fafe57dc900 drvPvt 0x1aa6090
    interfaceList
        asynCommon pinterface 0x7fafe57d9650 drvPvt 0x1aa1a10
        asynOption pinterface 0x7fafe57d9670 drvPvt 0x1aa1a10
        asynOctet pinterface 0x1aa1b30 drvPvt 0x1aa1a10
    Port 10.41.104.10:1025: Connected
                    fd: 7
    Characters written: 26671
       Characters read: 11150


So it still thinks it is connected, even after quite some time.



I tried enabling the disconnectOnReadTimeout option, it yielded a weird behavior: with the motor controller powered ON, the IOC seems to start fine but then seems to disconnect from the controller by itself:

asynManager::queueLockPort queueRequest failed: port P0 not connected
2018/02/12 14:40:38.972 pmacController::lowLevelWriteRead: Error from pasynOctetSyncIO->writeRead. command: #1 ? F P
2018/02/12 14:40:38.972 drvPmacAxisGetStatus: not all status values returned. Status: 3

epics> asynReport 5 P0
P0 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:No numberConnects 1
    nDevices 0 nQueued 21 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 11 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7f5c7df24900 drvPvt 0x8f21a0
    interfaceList
        asynCommon pinterface 0x7f5c7df21650 drvPvt 0x8edb20
        asynOption pinterface 0x7f5c7df21670 drvPvt 0x8edb20
        asynOctet pinterface 0x8edc40 drvPvt 0x8edb20
    Port 10.41.104.10:1025: Disconnected
                    fd: -1
    Characters written: 848
       Characters read: 470



Bruno

On Mon, Feb 12, 2018 at 1:32 PM, Mark Rivers <[email protected]> wrote:

Hi Bruno,

 

I think your startup script contains this command, correct?

 

pmacAsynIPConfigure(portName, hostInfo)

 

This means that the pmac driver is not directly communicating with the drvAsynIPPort driver supplied with asyn.  It is talking to the port created with the above command, which implements the asynInterposeInterface between the pmac motor driver and the underlying drvAsynIPPort driver.  I have not studied that code carefully, but it is possible that it is doing something to prevent the reconnect feature from working correctly.

 

I have tested the asyn autoreconnect feature quite a bit recently, when I was trying to reduce the number of error messages that are printed when a device is powered off and then back on again.  For example, look at the Modbus R2-10 release notes:

 

http://cars9.uchicago.edu/software/epics/modbusReleaseNotes.html

 

Mark

 

 

From: [email protected] [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Mark Rivers
Sent: Monday, February 12, 2018 12:10 PM
To: 'Bruno Martins' <[email protected]>; [email protected]
Cc: [email protected]
Subject: RE: PMAC asyn autoconnect issue

 

Hi Bruno,

 

What version of asyn are you using?

 

What version of tpmac are you using?

 

Ø  2018/02/12 12:00:14.826 pmacController::lowLevelWriteRead: Error from pasynOctetSyncIO->writeRead. command: #2 ? F P
2018/02/12 12:00:14.826 drvPmacAxisGetStatus: not all status values returned. Status: 3
Command :#2 ? F P

 

Note that Status: 3 is asynError, not asynDisconnected.

 

Are you sure that asyn really thinks the port is disconnected at this time?  You should run asynReport on that port to see whether asyn thinks it is disconnected or not.

 

One problem is that when you power-off an IP device on Linux it takes quite a while before the OS decides the socket is disconnected.  Before that time it returns an error like “Resource temporarily unavailable”.  During this time asyn will not try to reconnect, because it thinks it is still connected.

 

This behavior is one reason that the drvAsynIPPort driver was enhanced in R4-29 to support the disconnectOnReadTimeout option.  If this option is enabled then the port will automatically disconnect whenever there is a read timeout.

 

Mark

 

 

From: [email protected] [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Bruno Martins
Sent: Monday, February 12, 2018 11:34 AM
To: [email protected]
Cc: [email protected]
Subject: PMAC asyn autoconnect issue

 

Hi,

I've been investigating an issue with the PMAC driver regarding reconnection via Ethernet. When the controller is powered off the IOC starts emitting messages like the following (as expected):

2018/02/12 12:00:14.826 pmacController::lowLevelWriteRead: Error from pasynOctetSyncIO->writeRead. command: #2 ? F P
2018/02/12 12:00:14.826 drvPmacAxisGetStatus: not all status values returned. Status: 3
Command :#2 ? F P

 

 

However, when the controller is powered on again, the IOC does not re-establish the connection (even though noAutoConnect is set to zero).

This is the sequence of calls that lead to these messages:

lowLevelWriteRead calls pasynOctetSyncIO->writeRead
pasynOctetSyncIO->writeRead calls pasynManager->queueLockPort

pasynManager->queueLockPort calls pasynManager->queueRequest

queueRequest sets checkPortConnect = TRUE

 

If checkPortConnected was FALSE autoConnectDevice would be called later on and a new connection would be attempted. But for this particular call chain checkPortConnected is always set to TRUE.

Now, since queueRequest fails a request is never queued and pasynManager->portThread's autoConnectDevice is also not reached.

So my question is how to properly configure/trigger an auto connect?

 

 

PS.: I found this old thread that might be related:

https://epics.anl.gov/tech-talk/2012/msg01216.php

 

Thanks!



Bruno



Replies:
RE: PMAC asyn autoconnect issue Mark Rivers
Re: PMAC asyn autoconnect issue Torsten Bögershausen
References:
PMAC asyn autoconnect issue Bruno Martins
RE: PMAC asyn autoconnect issue Mark Rivers
RE: PMAC asyn autoconnect issue Mark Rivers

Navigate by Date:
Prev: RE: PMAC asyn autoconnect issue Mark Rivers
Next: RE: PMAC asyn autoconnect issue Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: PMAC asyn autoconnect issue Mark Rivers
Next: RE: PMAC asyn autoconnect issue Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
ANJ, 15 Feb 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·