RE: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

My recent tests have been done with the following:

 

- A very "thin" startup script that only loads the ip330 and dac128V drivers and records.

- A "thick" application that is built with most of the synApps modules. 

 

I just stripped it down so the application is only built with base, asyn, ipac, ip330, dac128v.  I made fresh git clones of asyn, ipac, ip330, dac128v.

 

The application Makefile is very simple:

 

**********************

corvette:CARS/CARSApp/testSrc>more Makefile

TOP=../..

 

include $(TOP)/configure/CONFIG

#----------------------------------------

#  ADD MACRO DEFINITIONS AFTER THIS LINE

 

PROD_NAME = CARSTest

PROD_IOC_vxWorks = $(PROD_NAME)

 

DBD_NAME = $(PROD_NAME)VX

DBD  += $(DBD_NAME).dbd

 

$(DBD_NAME)_DBD += base.dbd

 

$(DBD_NAME)_DBD += asyn.dbd

$(PROD_NAME)_LIBS := asyn $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += dac128VSupport.dbd

$(PROD_NAME)_LIBS := dac128V $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += ip330Support.dbd

$(PROD_NAME)_LIBS := ip330 $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += drvIpac.dbd

$(PROD_NAME)_LIBS := Ipac $($(PROD_NAME)_LIBS)

 

$(PROD_NAME)_LIBS += $(EPICS_BASE_IOC_LIBS)

 

$(PROD_NAME)_SRCS  += $(DBD_NAME)_registerRecordDeviceDriver.cpp

 

$(PROD_NAME)_OBJS_vxWorks += $(EPICS_BASE_BIN)/vxComLibrary

 

#===========================

 

include $(TOP)/configure/RULES

#----------------------------------------

#  ADD RULES AFTER THIS LINE

**********************

 

These are the commands that build the dbd file and link the application.

 

Creating dbd file CARSTestVX.dbd

perl -CSD /corvette/usr/local/epics-devel/base-7.0.6/bin/linux-x86_64/dbdExpand.pl   -I. -I.. -I../O.Common -I../../../dbd -I/corvette/home/epics/devel/asyn-4-42/dbd -I/corvette/home/epics/devel/CARS/dbd -I/corvette/home/epics/devel/dac128V-2-10-1/dbd -I/corvette/home/epics/devel/ipac-2-16/dbd -I/corvette/home/epics/devel/ip330-2-10/dbd -I/corvette/usr/local/epics-devel/base-7.0.6/dbd -o CARSTestVX.dbd base.dbd asyn.dbd dac128VSupport.dbd ip330Support.dbd drvIpac.dbd

Installing created dbd file ../../../dbd/CARSTestVX.dbd

 

/usr/local/vw/vxWorks-6.9/gnu/4.3.3-vxworks-6.9/x86-linux2/bin/ldppc -r -o CARSTest  -L/corvette/home/epics/devel/asyn-4-42/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/dac128V-2-10-1/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/ip330-2-10/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/ipac-2-16/lib/vxWorks-ppc32/ -L/corvette/usr/local/epics-devel/base-7.0.6/lib/vxWorks-ppc32/                   /corvette/usr/local/epics-devel/base-7.0.6/bin/vxWorks-ppc32/vxComLibrary.o CARSTestVX_registerRecordDeviceDriver.o    -lIpac -lip330 -ldac128V -lasyn -ldbRecStd -ldbCore –lca

 

I have a script that is writing new values to DAC channel 0 at 100 Hz.  That causes all 16 ADC records to post monitors at about 10 Hz.

 

I closed the medm screen and stopped camonitor on the ADC channels.  I started the IOC and it ran fine for 5 minutes with the script writing to the DAC.

 

I then opened the medm screen and started the camonitor on the ADC channels.  It failed in a few seconds with the A32 address error:

 

ioc13lab2>

VME Bus Error accessing A32: 0xbff99994

machine check

Exception next instruction address: 0x0329b0e0

Machine Status Register: 0x0008b032

Condition Register: 0x48000884

Task: 0x3560930 "CAS-event"

0x3560930 (CAS-event): task 0x3560930 has had a failure and has been stopped.

0x3560930 (CAS-event): The task has been terminated because it triggered an exception that raised the signal 10.

 

Mark

 

 

 

-----Original Message-----
From: Michael Davidsaver <mdavidsaver at gmail.com>
Sent: Friday, May 27, 2022 1:49 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

 

> On 5/27/22 06:17, Mark Rivers wrote:

>> Hi Michael,

>>

>> I commented out all lines in recGbl.c that reference the new amsg and namsg fields.  That did not fix the problem.

 

I take this to mean both db_post_events() and epicsVsnprintf() calls, which means that both AMSG and NAMSG should always be an empty string.

(aka. char[40] all nils)

 

https://github.com/epics-base/epics-base/commit/892a361de7002bf2b3f375f24bc5bf61858de2e5

 

> On 5/27/22 10:33, Mark Rivers wrote:

>> I am now using the unmodified 7.0.6 release.

...

> Before I build in that directory I run

 

I don't know how difficult it is with vxWorks @APS, but it might be a good idea to go further and start from fresh git-clone of all modules (base and synapps and application).  This should rule out build leftovers entirely.

 

 

On 5/27/22 10:33, Mark Rivers wrote:

> What could possibly cause this?

 

This is a very good question...

 

 

As I think about it, adding/removing AMSG+NAMSG would have exposed some partial rebuild problems with Base and parts of synapps.  The generated record type headers include a check that the number of fields at load time is the same as the number at build time.

 

 

If not a build/ABI issue, I don't see an easy way forward in troubleshooting.

 

We've passed the point where I would be trying to setup a (remote) debugger to inspect the stack of a faulting thread.  This should identify one of the records involved, and a dump of that record struct may show evidence of corruption.

 

For that matter, "dbpr <name> 5" could show evidence, if you can find one of the records.