Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
<== Date ==> <== Thread ==>

Subject: RE: Weird stream device behavior when using the IOC shell's exit function
From: Abdalla Ahmad via Tech-talk <tech-talk@aps.anl.gov>
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Tue, 5 Feb 2019 12:34:05 +0000
Hi Mark

I attached the help output in help.txt. For the database nothing is I/O scanned, standard 1 second rate as in the attached database. I also checked the protocol file but nothing seems strange.

While writing the email I thought of increasing the scan rate to 5 seconds and it works. I still get the same asynError in write but the IOC eventually exits. Even if it exits gracefully now, it is still a problem because we need the scan rate to match the IMG ones. As I told you before it used to work on the old setup without any problems. Is there anything we can investigate?

Best Regards,
Abdalla.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu] 
Sent: Tuesday, February 05, 2019 2:20 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Cc: tech-talk@aps.anl.gov
Subject: Re: Weird stream device behavior when using the IOC shell's exit function

Hi Abdalla,


Your IOC is clearly built OK with asyn because you have threads that are in asyn functions.


Thread 13 (Thread 0x7fffed943700 (LWP 29173)):
#0  0x00007ffff6034f0d in poll () from /lib64/libc.so.6
#1  0x00007ffff7926c14 in readIt (drvPvt=0x70eb30, pasynUser=0x79e4a8, data=0x70fd40 ">", maxchars=2048, nbytesTransfered=0x7fffed942ba0, gotEom=0x7fffed942b90)
    at ../../asyn/drvAsynSerial/drvAsynIPPort.c:726
#2  0x00007ffff7930886 in readIt (drvPvt=0x70fbb0, pasynUser=0x79e4a8, data=0x70fd40 ">", maxchars=<optimized out>, nbytesTransfered=0x7fffed942ba0, eomReason=0x7fffed942b90)
    at ../../asyn/interfaces/asynOctetBase.c:233
#3  0x00007ffff793a563 in readIt (ppvt=0x70fcb0, pasynUser=0x79e4a8, data=0x79e3a8 ">", maxchars=63, nbytesTransfered=0x7fffed942ca0, eomReason=0x7fffed942c70)
    at ../../asyn/miscellaneous/asynInterposeEos.c:231
#4  0x00007ffff7bab5f5 in AsynDriverInterface::readHandler (this=this@entry=0x79e2e0) at ../AsynDriverInterface.cc:960
#5  0x00007ffff7bacd08 in handleRequest (pasynUser=<optimized out>) at ../AsynDriverInterface.cc:1503
#6  0x00007ffff791c9f3 in portThread (pport=0x70ece0) at ../../asyn/asynDriver/asynManager.c:902
#7  0x00007ffff6b6f7fc in start_routine (arg=0x70f290) at ../../../src/libCom/osi/os/posix/osdThread.c:403
#8  0x00007ffff5d2ce25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff603fbad in clone () from /lib64/libc.so.6



Thread 12 (Thread 0x7fffeda44700 (LWP 29172)):
#0  0x00007ffff5d30995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff6b71f3b in epicsEventWait (pevent=0x70d520) at ../../../src/libCom/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6b6b029 in epicsEventMustWait (id=<optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:125
#3  0x00007ffff791c40c in portThread (pport=0x70d050) at ../../asyn/asynDriver/asynManager.c:788
#4  0x00007ffff6b6f7fc in start_routine (arg=0x70d590) at ../../../src/libCom/osi/os/posix/osdThread.c:403
#5  0x00007ffff5d2ce25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff603fbad in clone () from /lib64/libc.so.6

I don't understand why the asyn iocsh commands would not be available.


Please send the complete output of the iocsh "help" command.


The hang at shutdown appears to be caused by this thread:


Thread 1 (Thread 0x7ffff7fda740 (LWP 29161)):
#0  0x00007ffff5d3351d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff5d2ee36 in _L_lock_870 () from /lib64/libpthread.so.0
#2  0x00007ffff5d2ed2f in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ffff6b71a76 in mutexLock (id=0x70ef60) at ../../../src/libCom/osi/os/posix/osdMutex.c:46
#4  epicsMutexOsdLock (pmutex=0x70ef60) at ../../../src/libCom/osi/os/posix/osdMutex.c:130
#5  0x00007ffff7917e9b in lockPort (pasynUser=0x710668) at ../../asyn/asynDriver/asynManager.c:1741
#6  0x00007ffff7925876 in cleanup (arg=0x70eb30) at ../../asyn/drvAsynSerial/drvAsynIPPort.c:246
#7  0x00007ffff6b65ce3 in epicsExitCallAtExitsPvt (pep=<optimized out>) at ../../../src/libCom/misc/epicsExit.c:95
#8  epicsExitCallAtExits () at ../../../src/libCom/misc/epicsExit.c:113
#9  0x00007ffff6b66088 in epicsExit (status=0) at ../../../src/libCom/misc/epicsExit.c:181
#10 0x000000000040544d in main (argc=<optimized out>, argv=<optimized out>) at ../iocMain.cpp:21


So the epicsExit function is hung up in the drvAsynIPPort::cleanup function.  It has called lockPort, but cannot get that mutex.


No other thread is waiting for a mutex, so it is not a traditional deadlock.  But it seems that some other thread does have that mutex and is blocking the epicsExit thread from continuing.


It is probably this thread, which is in StreamDevice.


Thread 13 (Thread 0x7fffed943700 (LWP 29173)):
#0  0x00007ffff6034f0d in poll () from /lib64/libc.so.6
#1  0x00007ffff7926c14 in readIt (drvPvt=0x70eb30, pasynUser=0x79e4a8, data=0x70fd40 ">", maxchars=2048, nbytesTransfered=0x7fffed942ba0, gotEom=0x7fffed942b90)
    at ../../asyn/drvAsynSerial/drvAsynIPPort.c:726
#2  0x00007ffff7930886 in readIt (drvPvt=0x70fbb0, pasynUser=0x79e4a8, data=0x70fd40 ">", maxchars=<optimized out>, nbytesTransfered=0x7fffed942ba0, eomReason=0x7fffed942b90)
    at ../../asyn/interfaces/asynOctetBase.c:233
#3  0x00007ffff793a563 in readIt (ppvt=0x70fcb0, pasynUser=0x79e4a8, data=0x79e3a8 ">", maxchars=63, nbytesTransfered=0x7fffed942ca0, eomReason=0x7fffed942c70)
    at ../../asyn/miscellaneous/asynInterposeEos.c:231
#4  0x00007ffff7bab5f5 in AsynDriverInterface::readHandler (this=this@entry=0x79e2e0) at ../AsynDriverInterface.cc:960
#5  0x00007ffff7bacd08 in handleRequest (pasynUser=<optimized out>) at ../AsynDriverInterface.cc:1503
#6  0x00007ffff791c9f3 in portThread (pport=0x70ece0) at ../../asyn/asynDriver/asynManager.c:902
#7  0x00007ffff6b6f7fc in start_routine (arg=0x70f290) at ../../../src/libCom/osi/os/posix/osdThread.c:403
#8  0x00007ffff5d2ce25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff603fbad in clone () from /lib64/libc.so.6


Do you have StreamDevice records that are I/O Intr scanned?


Mark




________________________________
From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Tuesday, February 5, 2019 5:53 AM
To: Mark Rivers
Cc: tech-talk@aps.anl.gov
Subject: RE: Weird stream device behavior when using the IOC shell's exit function


Hi Mark



No command available which starts with asyn. I cloned the latest asyn from github with the same behavior.

For the gdb part, attached is the stack trace from gdb for all pending threads.



Best Regards,

Abdalla.



From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Monday, February 04, 2019 4:52 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Cc: tech-talk@aps.anl.gov
Subject: RE: Weird stream device behavior when using the IOC shell's exit function



At the iocsh prompt when the IOC is still running type the command



help



It should show a complete list of commands that the iocsh understands.  See which ones start with "asyn".



Mark





From: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo<mailto:abdalla.ahmad@sesame.org.jo>>
Sent: Monday, February 4, 2019 8:44 AM
To: Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>>
Cc: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Re: Weird stream device behavior when using the IOC shell's exit function



Hello Mark

I will apply the gdb tip and get back to you. For the asyn commands, I don't see any asyn command when I type exit.

Get Outlook for Android<https://aka.ms/ghei36>





On Mon, Feb 4, 2019 at 4:33 PM +0200, "Mark Rivers" <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>> wrote:

Hi Abdalla,



> asynError in write. Asyn driver says: device:port disconnected.



During IOC shutdown asyn does close all TCP ports.  However, record processing should have already been shut down, so I don't understand why you are getting that message from Stream.



Another problem we are facing with this new setup is that I can't find some asyn IOC shell function like asynTraceMask for example. The IOC is configured properly in RELEASE and src/Makefile. Is there anything we miss in the new setup?



> But eventually the IOC exits. For the gamma controllers we get something really strange.

> There a point in the database where the IOC never exits, the exit command just freezes and Ctrl-C is the only way to shut down the IOC.



I don't think I have seen that with Stream on any version of Stream/asyn/base that I have used.  That includes base 3.14.12, 3.15.5, 7.0.2.



If you run the IOC with gdb then when you type exit and it hangs do the following:



- Type Ctrl-C

- Enter the gdb command

thread apply all bt



That will show you the current stack trace for all threads.  You can then see what is blocking the threads.



> Another problem we are facing with this new setup is that I can't find some asyn IOC shell function like asynTraceMask for example.

> The IOC is configured properly in RELEASE and src/Makefile. Is there anything we miss in the new setup?



The command is not "asynTraceMask" it is "asynSetTraceMask" or "asynSetTraceIOMask".



What asyn commands do you see if you type "help" at the iocsh prompt?



Mark



________________________________________

From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov>  on behalf of Abdalla Ahmad via Tech-talk

Sent: Monday, February 4, 2019 1:41 AM

To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>

Subject: Weird stream device behavior when using the IOC shell's exit function



Hi



We are using the following setup to test control of the agilent XGS gauge controllers and Gamma ion pump controllers:

1.       EPICS Base 3.15.6

2.       Asyn R4-33

3.       Stream R2-7-7c



For agilent controllers we get the following error:



asynError in write. Asyn driver says: device:port disconnected.



But eventually the IOC exits. For the gamma controllers we get something really strange. There a point in the database where the IOC never exits, the exit command just freezes and Ctrl-C is the only way to shut down the IOC. For now I can see that this behavior occurs because more DB substitutions are configured which means more PVs and more controllers. But that was not the case when we had:

1.       EPICS Base 3.14.12.3

2.       Asyn R4-18

3.       Stream R2-5-1



Where the IOC exits with no errors or freezing. Should we upgrade our support modules or change the EPICS base?



Another problem we are facing with this new setup is that I can't find some asyn IOC shell function like asynTraceMask for example. The IOC is configured properly in RELEASE and src/Makefile. Is there anything we miss in the new setup?



Best Regards,



Abdalla Ahmad

Control Engineer

SESAME

Allan, Jordan.

Tel: (+962-5) 3511348 , ext. 265

Fax: (+962-5) 3511423

Mob: (+962-7)88183296

http://www.sesame.org.jo/


#               ClockTime_Report                ClockTime_Shutdown
appendToFile    asDumpHash      asInit          asSetFilename   
asSetSubstitutions              asVerify        ascar           asdbdump
asphag          aspmem          asprules        aspuag          astac
autosaveBuild   callbackParallelThreads         callbackSetQueueSize
casr            cd              coreRelease     create_manual_set
create_monitor_set              create_periodic_set             
create_triggered_set            date            dbDumpBreaktable
dbDumpDevice    dbDumpDriver    dbDumpField     dbDumpFunction  dbDumpMenu
dbDumpPath      dbDumpRecord    dbDumpRecordType                
dbDumpRegistrar dbDumpVariable  dbLoadDatabase  dbLoadRecords   dbLoadTemplate
dbLockShowLocked                dbNotifyDump    dbPutAttribute  dbPvdDump
dbPvdTableSize  dbReportDeviceConfig            dbStateClear    dbStateCreate
dbStateSet      dbStateShow     dbStateShowAll  dba             dbap
dbb             dbc             dbcar           dbd             dbel
dbgf            dbgrep          dbhcr           dbior           dbl
dbla            dblsr           dbnr            dbp             dbpf
dbpr            dbrestoreShow   dbs             dbsr            dbstat
dbtgf           dbtpf           dbtpn           dbtr            
drvAsynIPPortConfigure          drvAsynIPServerPortConfigure    
drvAsynSerialPortConfigure      echo            eltc            epicsEnvSet
epicsEnvShow    epicsEnvUnset   epicsMutexShowAll               epicsParamShow
epicsPrtEnvParams               epicsThreadResume               
epicsThreadShow epicsThreadShowAll              epicsThreadSleep
eraseFile       errlog          errlogInit      errlogInit2     exit
fdbrestore      fdbrestoreX     generalTimeReport               gft
help            installLastResortEventProvider  iocBuild        iocInit
iocLogInit      iocLogPrefix    iocLogShow      iocPause        iocRun
ioc_registerRecordDeviceDriver  iocshCmd        iocshLoad       iocshRun
makeAutosaveFileFromDbInfo      makeAutosaveFiles               manual_save
pft             postEvent       pwd             registryDeviceSupportFind
registryDriverSupportFind       registryDump    registryFunctionFind
registryRecordTypeFind          reload_manual_set               
reload_monitor_set              reload_periodic_set             
reload_triggered_set            remove_data_set save_restoreSet_CAReconnect
save_restoreSet_CallbackTimeout save_restoreSet_DatedBackupFiles
save_restoreSet_Debug           save_restoreSet_FilePermissions 
save_restoreSet_IncompleteSetsOk                save_restoreSet_NFSHost
save_restoreSet_NumSeqFiles     save_restoreSet_RetrySeconds    
save_restoreSet_SeqPeriodInSeconds              save_restoreSet_UseStatusPVs
save_restoreSet_status_prefix   save_restoreShow                
scanOnceSetQueueSize            scanpel         scanpiol        scanppl
setIocLogDisable                set_pass0_restoreFile           
set_pass1_restoreFile           set_requestfile_path            
set_saveTask_priority           set_savefile_name               
set_savefile_path               streamReload    taskwdShow      tpn
var

Attachment: gamma-ion-pump.template
Description: gamma-ion-pump.template

Attachment: gamma-ion-pump.substitutions
Description: gamma-ion-pump.substitutions

Attachment: gamma-qpc.proto
Description: gamma-qpc.proto


Replies:
Re: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk
References:
Weird stream device behavior when using the IOC shell's exit function Abdalla Ahmad via Tech-talk
Re: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk
Re: Weird stream device behavior when using the IOC shell's exit function Abdalla Ahmad via Tech-talk
RE: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk
RE: Weird stream device behavior when using the IOC shell's exit function Abdalla Ahmad via Tech-talk
Re: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk

Navigate by Date:
Prev: Re: AsynDriver with delayed and async communication Mark Rivers via Tech-talk
Next: RE: AsynDriver with delayed and async communication Joao Afonso via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
Navigate by Thread:
Prev: Re: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk
Next: Re: Weird stream device behavior when using the IOC shell's exit function Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020 
ANJ, 05 Feb 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·