I am noticing something for when it is stuck vs when it is not stuck. When it gets stuck the asynReport says "synchronousLock:Yes", but when it is not stuck it says "synchronousLock:No".
Does this mean anything?
When it is stuck epicsMutexShowAll gives me the following:
epics> epicsMutexShowAll, 1
ellCount(&mutexList) 157 ellCount(&freeList) 0
epicsMutexId 0x74ae00 source ../../asyn/asynDriver/asynManager.c line 1911
But when it is not stuck I just get:
epics> epicsMutexShowAll, 1
ellCount(&mutexList) 157 ellCount(&freeList) 0
The PV got stuck again today for several minutes so I will post the asynReport message here (to me it looks the same as what I posted yesterday except the higher number of characters read):
c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
enabled:Yes connected:Yes numberConnects 1
nDevices 0 nQueued 1 blocked:No
asynManagerLock:No synchronousLock:Yes
exceptionActive:No exceptionUsers 1 exceptionNotifys 0
traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
interposeInterfaceList
asynOctet pinterface 0x7f0e81723a40 drvPvt 0x108da20
interfaceList
asynCommon pinterface 0x7f0e81720d20 drvPvt 0x108cb10
asynOctet pinterface 0x108cbf8 drvPvt 0x108cb10
Port 10.17.2.60:4016: Connected
fd: 5
Characters written: 66
Characters read: 4650
Here is a message from when it is not stuck:
epics> asynReport 10 c3-tsrv1-p16
c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
enabled:Yes connected:Yes numberConnects 1
nDevices 0 nQueued 0 blocked:No
asynManagerLock:No synchronousLock:No
exceptionActive:No exceptionUsers 1 exceptionNotifys 0
traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
interposeInterfaceList
asynOctet pinterface 0x7f6015129a40 drvPvt 0x765910
interfaceList
asynCommon pinterface 0x7f6015126d20 drvPvt 0x764a00
asynOctet pinterface 0x764ae8 drvPvt 0x764a00
Port 10.17.2.60:4016: Connected
fd: 5
Characters written: 1833
Characters read: 4159
-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 6, 2020 9:14 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: Re: StreamDevice, prevent records from getting "stuck"
When it is really "hung up" run the asynReport. You can also use the asynRecord to manually try to communicate and see if you get a response.
________________________________
From: Sobhani, Bayan <bsobhani at bnl.gov>
Sent: Monday, April 6, 2020 6:48 PM
To: Mark Rivers
Cc: tech-talk
Subject: Re: StreamDevice, prevent records from getting "stuck"
Maybe the reason for the nQueue being 5 is because there are multiple PVs associated with each device.
These are RS-485 temperature controllers that are connected to a moxa. I am not sure if "noise" is the correct word but some of these devices seem to be prone to sending random characters at times. For some of these devices changing the cabling seems to help. Other times it seems to be the device itself.
Get Outlook for Android<https://urldefense.com/v3/__https://aka.ms/ghei36__;!!P4SdNyxKAPE!W2RJdAuBmGW3k6OG6ix29cxOZIknDYbiP6yckLyoJ-QP7VreJelqZIHkUllXCRYI$ >
________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 6, 2020 7:35:29 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: RE: StreamDevice, prevent records from getting "stuck"
Note that asynReport said "nQueued 5". That suggests the port was busy while 5 more I/O requests were queued up.
This looks like a device that is not responding.
Your original message said
`sometimes when the signal is noisy, the PV gets "stuck"`
What do you mean by "signal is noisy". The TCP messages should not be "noisy".
Mark
-----Original Message-----
From: Sobhani, Bayan <bsobhani at bnl.gov>
Sent: Monday, April 6, 2020 5:57 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: RE: StreamDevice, prevent records from getting "stuck"
The command I use to configure the port is drvAsynIPPortConfigure. For some reason I had some trouble reproducing the problems but after a few IOC restarts I got the PVs stuck again.
asynReport 10 PortName gives the following:
c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
enabled:Yes connected:Yes numberConnects 1
nDevices 0 nQueued 5 blocked:No
asynManagerLock:No synchronousLock:Yes
exceptionActive:No exceptionUsers 1 exceptionNotifys 0
traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
interposeInterfaceList
asynOctet pinterface 0x7f3dff3a9a40 drvPvt 0xe99ad0
interfaceList
asynCommon pinterface 0x7f3dff3a6d20 drvPvt 0xe98bc0
asynOctet pinterface 0xe98ca8 drvPvt 0xe98bc0
Port 10.17.2.60:4016: Connected
fd: 5
Characters written: 117
Characters read: 227
After running this, a few seconds later I got the message:
"2020/04/06 18:45:23.072801 c3-tsrv1-p16 XF:17ID-CT{RG:C3}T:1-SP: Timeout after reading 176 bytes "...<ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff>""
And then the PV started reading data again. I am not sure if the asynReport command is somehow fixing this or if it is just a coincidence.
Alex
-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 6, 2020 6:10 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: Re: StreamDevice, prevent records from getting "stuck"
What asyn driver are you using (drvAsynIPPort, drvAsynSerialPort, etc.)?
You should run the following to see if the problem is in the asyn driver
asynReport 10 PortName
where PortName is the name of the asyn port. That will tell you if the port is connected, etc.
Mark
________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Sobhani, Bayan via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, April 6, 2020 4:57 PM
To: tech-talk at aps.anl.gov
Subject: StreamDevice, prevent records from getting "stuck"
For StreamDevice I notice that sometimes when the signal is noisy, the PV gets "stuck" if it makes one too many failed requests to the device. Getting "stuck" means that no matter how many times I process the PV with a StreamDevice OUT field, it does not seem to attempt to send anything to the device until I restart the IOC.
I see no technical reason why this must happen so I think this is probably something intentional in StreamDevice. Can I turn this off? I want the PV to always try to send a signal to the device, and if it can't I want to see the red letters saying there was a mismatch.
Is there any way to prevent the PVs from getting "stuck"?
Alex
- Replies:
- RE: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- References:
- StreamDevice, prevent records from getting "stuck" Sobhani, Bayan via Tech-talk
- Re: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- RE: StreamDevice, prevent records from getting "stuck" Sobhani, Bayan via Tech-talk
- RE: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- Re: StreamDevice, prevent records from getting "stuck" Sobhani, Bayan via Tech-talk
- Re: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- Navigate by Date:
- Prev:
RE: Packages to build EPICS base, synApps, and areaDetector on Centos 8 Mark Rivers via Tech-talk
- Next:
RE: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
<2020>
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- Next:
RE: StreamDevice, prevent records from getting "stuck" Mark Rivers via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
<2020>
2021
2022
2023
2024
|