Experimental Physics and
Industrial Control System

"Sobhani, Bayan via Tech-talk" <tech-talk at aps.anl.gov> · Tue, 7 Apr 2020 14:33:39 +0000

I am noticing something for when it is stuck vs when it is not stuck. When it gets stuck the asynReport says "synchronousLock:Yes", but when it is not stuck it says "synchronousLock:No".

Does this mean anything?

When it is stuck epicsMutexShowAll gives me the following:

epics> epicsMutexShowAll, 1
ellCount(&mutexList) 157 ellCount(&freeList) 0
epicsMutexId 0x74ae00 source ../../asyn/asynDriver/asynManager.c line 1911

But when it is not stuck I just get:

epics> epicsMutexShowAll, 1
ellCount(&mutexList) 157 ellCount(&freeList) 0

The PV got stuck again today for several minutes so I will post the asynReport message here (to me it looks the same as what I posted yesterday except the higher number of characters read):

c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 1 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 1 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7f0e81723a40 drvPvt 0x108da20
    interfaceList
        asynCommon pinterface 0x7f0e81720d20 drvPvt 0x108cb10
        asynOctet pinterface 0x108cbf8 drvPvt 0x108cb10
    Port 10.17.2.60:4016: Connected
                    fd: 5
    Characters written: 66
       Characters read: 4650

Here is a message from when it is not stuck:

epics> asynReport 10 c3-tsrv1-p16
c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 0 blocked:No
    asynManagerLock:No synchronousLock:No
    exceptionActive:No exceptionUsers 1 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7f6015129a40 drvPvt 0x765910
    interfaceList
        asynCommon pinterface 0x7f6015126d20 drvPvt 0x764a00
        asynOctet pinterface 0x764ae8 drvPvt 0x764a00
    Port 10.17.2.60:4016: Connected
                    fd: 5
    Characters written: 1833
       Characters read: 4159

-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu> 
Sent: Monday, April 6, 2020 9:14 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: Re: StreamDevice, prevent records from getting "stuck"

When it is really "hung up" run the asynReport.  You can also use the asynRecord to manually try to communicate and see if you get a response.

________________________________
From: Sobhani, Bayan <bsobhani at bnl.gov>
Sent: Monday, April 6, 2020 6:48 PM
To: Mark Rivers
Cc: tech-talk
Subject: Re: StreamDevice, prevent records from getting "stuck"

Maybe the reason for the nQueue being 5 is because there are multiple PVs associated with each device.

These are RS-485 temperature controllers that are connected to a moxa. I am not sure if "noise" is the correct word but some of these devices seem to be prone to sending random characters at times. For some of these devices changing the cabling seems to help. Other times it seems to be the device itself.

Get Outlook for Android<https://urldefense.com/v3/__https://aka.ms/ghei36__;!!P4SdNyxKAPE!W2RJdAuBmGW3k6OG6ix29cxOZIknDYbiP6yckLyoJ-QP7VreJelqZIHkUllXCRYI$ >

________________________________
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 6, 2020 7:35:29 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: RE: StreamDevice, prevent records from getting "stuck"

Note that asynReport said "nQueued 5".  That suggests the port was busy while 5 more I/O requests were queued up.

This looks like a device that is not responding.

Your original message said
    `sometimes when the signal is noisy, the PV gets "stuck"`

What do you mean by "signal is noisy".  The TCP messages should not be "noisy".

Mark

-----Original Message-----
From: Sobhani, Bayan <bsobhani at bnl.gov>
Sent: Monday, April 6, 2020 5:57 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: RE: StreamDevice, prevent records from getting "stuck"

The command I use to configure the port is drvAsynIPPortConfigure. For some reason I had some trouble reproducing the problems but after a few IOC restarts I got the PVs stuck again.

asynReport 10 PortName gives the following:

c3-tsrv1-p16 multiDevice:No canBlock:Yes autoConnect:Yes
    enabled:Yes connected:Yes numberConnects 1
    nDevices 0 nQueued 5 blocked:No
    asynManagerLock:No synchronousLock:Yes
    exceptionActive:No exceptionUsers 1 exceptionNotifys 0
    traceMask:0x1 traceIOMask:0x0 traceInfoMask:0x1
    interposeInterfaceList
        asynOctet pinterface 0x7f3dff3a9a40 drvPvt 0xe99ad0
    interfaceList
        asynCommon pinterface 0x7f3dff3a6d20 drvPvt 0xe98bc0
        asynOctet pinterface 0xe98ca8 drvPvt 0xe98bc0
    Port 10.17.2.60:4016: Connected
                    fd: 5
    Characters written: 117
       Characters read: 227

After running this, a few seconds later I got the message:

"2020/04/06 18:45:23.072801 c3-tsrv1-p16 XF:17ID-CT{RG:C3}T:1-SP: Timeout after reading 176 bytes "...<ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff><ff>""

And then the PV started reading data again. I am not sure if the asynReport command is somehow fixing this or if it is just a coincidence.

Alex

-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Monday, April 6, 2020 6:10 PM
To: Sobhani, Bayan <bsobhani at bnl.gov>
Cc: tech-talk <tech-talk at aps.anl.gov>
Subject: Re: StreamDevice, prevent records from getting "stuck"

What asyn driver are you using (drvAsynIPPort, drvAsynSerialPort, etc.)?

You should run the following to see if the problem is in the asyn driver

asynReport 10 PortName

where PortName is the name of the asyn port.  That will tell you if the port is connected, etc.

Mark

________________________________
From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Sobhani, Bayan via Tech-talk <tech-talk at aps.anl.gov>
Sent: Monday, April 6, 2020 4:57 PM
To: tech-talk at aps.anl.gov
Subject: StreamDevice, prevent records from getting "stuck"

For StreamDevice I notice that sometimes when the signal is noisy, the PV gets "stuck" if it makes one too many failed requests to the device. Getting "stuck" means that no matter how many times I process the PV with a StreamDevice OUT field, it does not seem to attempt to send anything to the device until I restart the IOC.

I see no technical reason why this must happen so I think this is probably something intentional in StreamDevice. Can I turn this off? I want the PV to always try to send a signal to the device, and if it can't I want to see the red letters saying there was a mismatch.

Is there any way to prevent the PVs from getting "stuck"?

Alex

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System