Tektronic Scope DPO7254 over VXI-11 crashes IOC

[Benjamin Franksen <benjamin.franksen@bessy.de>]

Hi,

this is something for the experts.

I'm trying to talk to the mentioned scope via asyn/stream and get mysterious 
crashes the first time the IOC tries to communicate over the link. Below is 
a trace of the IOC shell up to the crash. I am using the latest and greatest 
versions of everything except EPICS base: base-3.14.8.2, asyn-4.11a 
(supposedly the last version that works with base-3.14.8), StreamDevice-2-
snapshot20081007. The crash happens on the latest ubuntu version as well as 
on a debian stable system. I should point out that with the exact same setup 
I can talk fine with e.g. a GPIB controller.

I ran a debugger over the produced core file but I could not find any 
indication what went wrong. The crash happens inside the RPC clnt_call(), 
this is the backtrace from ddd:

Reading symbols from /net/csr/srv/projects/ctl/franksen/epics/ioc/MLS-
Controls/work-scopes/bin/linux-x86/SIOC3CP...done.
(gdb) core-file /net/csr/srv/projects/ctl/franksen/epics/ioc/MLS-
Controls/work-scopes/core
warning: core file may not match specified executable file.
[New Thread 29415]
[New Thread 29409]
[New Thread 29421]
[New Thread 29411]
[New Thread 29416]
[New Thread 29413]
[New Thread 29417]
[New Thread 29414]
[New Thread 29418]
[New Thread 29419]
[New Thread 29420]
warning: Can't read pathname for load map: Input/output error.
Core was generated by `../../bin/linux-x86/SIOC3CP ./st.cmd'.
Program terminated with signal 11, Segmentation fault.
#0  0x0041450d in clnttcp_call (h=0x98828f0, proc=11, xdr_args=0x80a2670 
<xdr_Device_WriteParms>, args_ptr=0xb774a0cc "", xdr_results=0x80a23e0 
<xdr_Device_WriteResp>, results_ptr=0xb774a0e4 "", timeout=...) at 
clnt_tcp.c:258
        in clnt_tcp.c
Current language:  auto
The current source language is "auto; currently c".
(gdb) frame 1
#1  0x080987eb in clientIoCall (pvxiPort=0x9869350, pasynUser=<value 
optimized out>, req=11, proc1=0x80a2670 <xdr_Device_WriteParms>, 
addr1=0xb774a0cc "", proc2=0x80a23e0 <xdr_Device_WriteResp>, 
addr2=0xb774a0e4 "") at ../../asyn/vxi11/drvVxi11.c:558
(gdb) frame 2
#2  0x080989e9 in vxiWrite (drvPvt=0x9869350, pasynUser=0x98825bc, 
data=0x98819b4 "HDR 0;VERB ON;\r\n", numchars=16, 
nbytesTransfered=0xb774a13c) at ../../asyn/vxi11/drvVxi11.c:1211
(gdb) frame 3
#3  0x0809b093 in writeIt (drvPvt=0x9871de0, pasynUser=0x98825bc, 
data=0x98819b4 "HDR 0;VERB ON;\r\n", numchars=16, 
nbytesTransfered=0xb774a1d8) at ../../asyn/asynGpib/asynGpib.c:396
(gdb) frame 4
#4  0x080a0080 in writeIt (drvPvt=0x9877d98, pasynUser=0x98825bc, 
data=0x98819b4 "HDR 0;VERB ON;\r\n", numchars=16, 
nbytesTransfered=0xb774a1d8) at ../../asyn/interfaces/asynOctetBase.c:223
(gdb) frame 5
#5  0x0806d712 in AsynDriverInterface::writeHandler (this=0x9882498) at 
../AsynDriverInterface.cc:624
Current language:  auto
The current source language is "auto; currently c++".
(gdb) frame 6
#6  0x0806db20 in handleRequest (pasynUser=0x98825bc) at 
../AsynDriverInterface.cc:1299
(gdb) frame 7
#7  0x0807fa0b in portThread (pport=0x9877320) at 
../../asyn/asynDriver/asynManager.c:799
/net/csr/srv/projects/ctl/franksen/epics/support/asyn/4-11/asyn/asynDriver/asynManager.c:799:28962:beg:0x807fa0b
Current language:  auto
The current source language is "auto; currently c".
(gdb) frame 8
#8  0x08146d6f in start_routine (arg=0x9877c28) at 
../../../src/libCom/osi/os/posix/osdThread.c:320
	in ../../../src/libCom/osi/os/posix/osdThread.c
(gdb) 


I can't see anything obviously wrong, here, no NULL pointers, etc. Could 
this be a bug in the RPC implementation? One that gets triggered by bad data 
sent back from the server?

Any other ideas?


This is the IOC shell log:

franksen@tiber:~/ctl/MLS-Controls/work-scopes ______________________________ 
14:09:06
> cd iocBoot/iocSIOC3CP && ./st.cmd    
#!../../bin/linux-x86/SIOC3CP
# Change to top directory
cd ../..
########################################################### Set up 
Environment
epicsEnvSet IOC SIOC3CP
#epicsEnvSet LOG_DIR /opt/IOC/log
epicsEnvSet LOG_DIR /projects/ctl/franksen/epics/ioc/MLS-Controls/work-
scopes/log
#epicsEnvSet EPICS_CA_PUT_LOG_INET 192.168.48.97
epicsEnvSet EPICS_CA_PUT_LOG_INET 193.149.12.189
epicsEnvSet EPICS_CA_SERVER_PORT 12345
epicsEnvSet VXI11_IO_TIMEOUT 1
epicsEnvSet VXI11_RPC_TIMEOUT 10
#VXI11_RECOVER_W_IFC 0 : no
#                    1 : yes
epicsEnvSet VXI11_RECOVER_W_IFC 1
epicsEnvSet GPIB_ADDR 1
epicsEnvSet GPIB_LINK L1
epicsEnvSet GPIB_VXI11_NAME gpib0
epicsEnvSet GPIB_GATEWAY gpibc3cp.mlscs.bessy.de
epicsEnvSet SCOPEC2S1GP_INET scopec2s1gp.mlscs.bessy.de
epicsEnvSet SCOPEC2S1GP_LINK L2
epicsEnvSet SCOPEC2S1GP_ADDR -1
epicsEnvSet SCOPEC2S1GP_VXI11_NAME inst0
epicsEnvSet STREAM_PROTOCOL_PATH db
############################################## Register all support 
components
dbLoadDatabase dbd/SIOC3CP.dbd 0 0
SIOC3CP_registerRecordDeviceDriver pdbbase
############################################################ Configure 
drivers
# Configure asyn vxi-11 driver
#vxi11Configure ${GPIB_LINK} ${GPIB_GATEWAY} ${VXI11_RECOVER_W_IFC} 
${VXI11_IO_TIMEOUT} ${GPIB_VXI11_NAME}
vxi11Configure L2 scopec2s1gp.mlscs.bessy.de 0 1 inst0 0 1
#asynSetTraceMask ${GPIB_LINK} ${GPIB_ADDR} 0x0010
#asynSetTraceIOMask ${GPIB_LINK} ${GPIB_ADDR} 0x0003
#asynSetOption ${GPIB_LINK} -1 rpctimeout ${VXI11_RPC_TIMEOUT}
asynSetTraceMask L2 -1 0x001F
asynSetTraceIOMask L2 -1 0x0003
#asynSetOption ${SCOPEC2S1GP_LINK} -1 rpctimeout ${VXI11_RPC_TIMEOUT}
####################################################### Autosave 
configuration
set_savefile_path /projects/ctl/franksen/epics/ioc/MLS-Controls/work-
scopes/log/autoSaveRestore
save_restoreSet_DatedBackupFiles 1
save_restoreSet_UseStatusPVs 0
######################################################## Load record 
instances
dbLoadRecords db/SIOC-stats.db IOC=SIOC3CP
filename="../dbLexRoutines.c" line number=238
No such file or directory dbRead opening file db/SIOC-stats.db#dbLoadRecords 
db/ScopeMeas.db  "DEVN=SCOPE1ZCP,ADDR=${GPIB_ADDR},LINK=${GPIB_LINK}"
dbLoadRecords db/ScopeMeas.db  "DEVN=SCOPEC2S1GP,ADDR=-1,LINK=L2"
######################################################## Restore record 
values
set_pass0_restoreFile ScopeMeas.sav
########################################################### Configure IOC 
Core
# IOC Log Server Connection  0=enabled 1=disabled
setIocLogDisable 0
# Configure Access Security
asSetFilename db/security.acf
#################################################################### Start 
IOC
iocInit
Starting iocInit
############################################################################
###  EPICS IOC CORE built on Mar  3 2010
###  EPICS R3.14.8.2 $R3-14-8-2$ $2006/01/06 15:55:13$
############################################################################
save_restore: Can't figure out which seq file is most recent,
save_restore: so I'm just going to start with 
'/projects/ctl/franksen/epics/ioc/MLS-Controls/work-
scopes/log/autoSaveRestore/ScopeMeas.sav0'.
save_restore: Can't find a file to restore from...save_restore: ...last 
tried '/projects/ctl/franksen/epics/ioc/MLS-Controls/work-
scopes/log/autoSaveRestore/ScopeMeas.sav2'. I give up.
2010/03/16 14:09:14.864 L2 -1 registerInterruptUser
2010/03/16 14:09:14.865 L2 addr -1 queueRequest priority 3 not lockHolder
2010/03/16 14:09:14.865 asynManager connect queueCallback port:L2
2010/03/16 14:09:14.865 L2 addr -1 vxiConnect
2010/03/16 14:09:14.865 L2 vxiConnectPort
2010/03/16 14:09:16.020 L2SRQ read error: Connection reset by peer
2010/03/16 14:09:16.020 L2SRQ vxiSrqThread terminating
2010/03/16 14:09:16.020 L2 vxiCreateIrqChannel VXI: channel not established 
(create_intr_chan)
2010/03/16 14:09:16.020 L2 Warning -- SRQ not operational.
2010/03/16 14:09:16.020 L2 asynGpib:srqHappened
2010/03/16 14:09:16.020 L2 addr -1 queueRequest priority 1 not lockHolder
2010/03/16 14:09:16.020 L2 addr -1 queueRequest priority 0 not lockHolder
2010/03/16 14:09:16.020 L2 schedule queueRequest timeout
2010/03/16 14:09:16.020 L2 callback
2010/03/16 14:09:16.020 L2 callback
2010/03/16 14:09:16.020 L2 -1 vxiWrite numchars 16
[1]    25241 segmentation fault (core dumped)  ./st.cmd
franksen@tiber:~/ctl/MLS-Controls/work-scopes/iocBoot/iocSIOC3CP ___________ 
14:09:16


Cheers
Ben