RE: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Ø  I have rebuilt the example application with soft records with base 7.0.5 with the DAC128V enabled.  It has currently run for 2500 seconds without failing, while base 7.0.6 always failed within 138 seconds.  I will let it run overnight.

 

Base 7.0.5 ran fine overnight, no failures in over 15 hours.

 

I previously reported that when I removed the AMSG and PAMSG fields from dbCommon in base 7.0.6 it did not fail when I was testing with IPAC hardware.  However, removing those fields did NOT fix the problem when running with records with soft device support.

 

I thus decided to run “git bisect” again, testing with the soft device support records.  I bisected from R7.0.6 to R7.0.5.  The results were [good, bad, bad, bad].  The bisect completed successfully with this output:

 

corvette:local/epics-devel/base-7.0.6>git bisect bad

The merge base f9ea6a5bff695c5f88bb95dce38a3fd349738907 is bad.

This means the bug has been fixed between f9ea6a5bff695c5f88bb95dce38a3fd349738907 and [1c6b02b9be2a1c778f6395dd677ef425b04410be 7f142e03f5cef74b5516e76494f29c3a0f143f4f].

 

This looks like a merge commit, so I am not sure how to proceed.

 

Mark

 

 

From: Mark Rivers
Sent: Monday, May 30, 2022 3:02 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov
Subject: RE: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

 

Hi Michael,

 

I have now reproduced the problem with no VME bus activity, i.e. no VME bus drivers initialized and no records using the VME bus.  This is the configuration:

 

-          Base 7.0.6, only change is to configure/CONFIG_SITE to build for vxWorks-ppc32

-          asyn, ipac, ip330, and dac128V modules built from new clones, master branch

-          IOC application newly created using makeBaseApp.pl from base 7.0.6

 

These are the changes to the example application created with makeBaseApp.

 

configure/RELEASE has these changes:

 

# Variables and paths to dependent modules:

MODULES = /corvette/home/epics/devel_test

#MYMODULE = $(MODULES)/my-module

 

# If using the sequencer, point SNCSEQ at its top directory:

#SNCSEQ = $(MODULES)/seq-ver

 

ASYN = $(MODULES)/asyn-4-42

IPAC = $(MODULES)/ipac-2-16

IP330 = $(MODULES)/ip330-2-10

DAC128V = $(MODULES)/dac128V-2-10-1

 

exampleApp/src/Makefile was changed to this.  Modules can be added and removed from the application by editing configure/RELEASE.  I removed the example records, etc. that is normally built in this file.

 

TOP=../..

 

include $(TOP)/configure/CONFIG

#----------------------------------------

#  ADD MACRO DEFINITIONS BELOW HERE

 

# Build the IOC application

PROD_IOC = example

 

# example.dbd will be created and installed

DBD += example.dbd

 

# example.dbd will include these files:

example_DBD += base.dbd

 

# example_registerRecordDeviceDriver.cpp derives from example.dbd

example_SRCS += example_registerRecordDeviceDriver.cpp

 

# Build the main IOC entry point where needed

example_SRCS_DEFAULT += exampleMain.cpp

example_SRCS_vxWorks += -nil-

 

ifdef IPAC

    example_LIBS += Ipac

    example_DBD += drvIpac.dbd

endif

 

ifdef IP330

    example_LIBS += ip330

    example_DBD += ip330Support.dbd

endif

 

ifdef DAC128V

    example_LIBS += dac128V

    example_DBD += dac128VSupport.dbd

endif

 

ifdef ASYN

    example_LIBS += asyn

    example_DBD += asyn.dbd

endif

 

# Finally link IOC to the EPICS Base libraries

example_LIBS += $(EPICS_BASE_IOC_LIBS)

 

include $(TOP)/configure/RULES

#----------------------------------------

#  ADD EXTRA GNUMAKE RULES BELOW HERE

 

In the iocBoot/iocexample directory I added 2 new database files containing records with soft device support only. 

 

aoTest.template contains 10 ao records:

 

corvette:example/iocBoot/iocexample>more aoTest.template

record(ao,"testAo_0") {}

record(ao,"testAo_1") {}

record(ao,"testAo_2") {}

record(ao,"testAo_3") {}

record(ao,"testAo_4") {}

record(ao,"testAo_5") {}

record(ao,"testAo_6") {}

record(ao,"testAo_7") {}

record(ao,"testAo_8") {}

record(ao,"testAo_9") {}

 

aiTest.template contains 10 ai records with an INP link to the corresponding ao record, using CP

 

corvette:example/iocBoot/iocexample>more aiTest.template

record(ai,"testAi_0") {field(INP, "testAo_0 CP") }

record(ai,"testAi_1") {field(INP, "testAo_1 CP") }

record(ai,"testAi_2") {field(INP, "testAo_2 CP") }

record(ai,"testAi_3") {field(INP, "testAo_3 CP") }

record(ai,"testAi_4") {field(INP, "testAo_4 CP") }

record(ai,"testAi_5") {field(INP, "testAo_5 CP") }

record(ai,"testAi_6") {field(INP, "testAo_6 CP") }

record(ai,"testAi_7") {field(INP, "testAo_7 CP") }

record(ai,"testAi_8") {field(INP, "testAo_8 CP") }

record(ai,"testAi_9") {field(INP, "testAo_9 CP") }

 

The st.cmd file just loads these 2 databases, it does not initialize any VME or IPAC drivers.

 

corvette:example/iocBoot/iocexample>more st.cmd

#- Example vxWorks startup file

 

#- The following is needed if your board support package doesn't at boot time

#- automatically cd to the directory containing its startup script

#cd "/home/epics/devel_test/example/iocBoot/iocexample"

 

< cdCommands

 

# Mount drives with NFS

nfsAuthUnixSet("corvette", 849601092, 849600513, 0, 0)

nfsMount("corvette","/home","/corvette/home")

nfsMount("corvette","/home","/home")

 

cd topbin

load("example.munch")

cd startup

 

dbLoadDatabase "$(TOP)/dbd/example.dbd"

example_registerRecordDeviceDriver pdbbase

 

## Load record instances

dbLoadRecords "./aoTest.template"

dbLoadRecords "./aiTest.template"

 

iocInit

 

When the IOC boots this is the output of dbior and asynReport.  This indicates that these drivers are present, but there are no IPAC carrier cards or asyn port drivers initialized.

 

iocexample> dbior

Driver: drvIpac

Driver: drvAsyn

value = 0 = 0x0

 

iocexample> asynReport

value = 0 = 0x0

 

Once the IOC is running I run the following scripts.

 

This script runs camonitor on all 10 ai records and all 10 ao records.

 

corvette:example/iocBoot/iocexample>more test_monitor

camonitor -tcr \

testAi_0 \

testAi_1 \

testAi_2 \

testAi_3 \

testAi_4 \

testAi_5 \

testAi_6 \

testAi_7 \

testAi_8 \

testAi_9 \

testAo_0 \

testAo_1 \

testAo_2 \

testAo_3 \

testAo_4 \

testAo_5 \

testAo_6 \

testAo_7 \

testAo_8 \

testAo_9 \

 

I run 2 instances of that script on a Linux machine to cause lots of CA monitor events.

 

I run this IDL script that writes new values to all 10 ao records in a loop at 100 Hz.  That causes the ai records to process since they have CP links.  There will be 20 records * 100 Hz = 2000 new record values per second.  Since there are 2 camonitor clients that is 4000 monitor events/s.

 

corvette:example/iocBoot/iocexample>more test_ao.pro

out = -10.

while 1 do begin

  t = caput('testAo_0', out)

  t = caput('testAo_1', out)

  t = caput('testAo_2', out)

  t = caput('testAo_3', out)

  t = caput('testAo_4', out)

  t = caput('testAo_5', out)

  t = caput('testAo_6', out)

  t = caput('testAo_7', out)

  t = caput('testAo_8', out)

  t = caput('testAo_9', out)

  wait, .01

  out = out + 0.1

  if (out gt 10.0) then out = -10.0

endwhile

end

 

If the example application is built with configure/RELEASE as shown above, defining ASYN, IPAC, IP330, and DAC128V, then it fails.  In 4 tests it failed after running for [42, 132, 32, 138] seconds respectively.

 

VME Bus Error accessing A32: 0xbff66664

machine check

Exception next instruction address: 0x0332133c

Machine Status Register: 0x0008b032

Condition Register: 0x48000844

Task: 0x34d4950 "CAC-event"

0x34d4950 (CAC-event): task 0x34d4950 has had a failure and has been stopped.

0x34d4950 (CAC-event): The task has been terminated because it triggered an exception that raised the signal 10.

 

Note that the task in error is CAC-event.  In the tests I did previously when using the VME hardware it was failing in CAS-event.

 

This is the task trace:

 

iocexample> tt  0x34d4950

0x0012489c vxTaskEntry  +0x48 : epicsThreadEntry ()

0x0333fe9c epicsThreadEntry+0x80 : 0x0329b884 ()

0x0329bb5c db_start_events+0x458: db_delete_field_log ()

0x0329b06c db_delete_field_log+0x54 : freeListFree ()

value = 0 = 0x0

 

I have found that if I build the application with DAC128V commented out in configure/RELEASE then the application does not crash within the time I tested for.  I ran it twice and it ran for [630, 1400] seconds before I killed it.

 

I cannot explain why building without DAC128V allows the application to run, since the DAC128V driver is never being initialized.  It could just be a memory layout artifact if there are uninitialized pointers or something.

 

When I was doing previous tests with the dac128V and ip330 hardware enabled and running I made some changes to base 7.0.6 and observed the following:

 

-          If I comment out all lines referencing dbCommon.amsg and dbCommon.namsg in dbAccess.c, dbDbLink.c, dbEvent.c, and recGbl.c then it did fail, but in 2 tests it took [4317, 22400] seconds to fail.

-          If I also comment out the AMSG and NAMSG fields in dbCommon.dbd.pod then it did not fail in 18000 seconds.

 

However, with the soft record example described here the application failed in [3, 171] seconds in CAC-event even with all of those changes to base.

 

I have rebuilt the example application with soft records with base 7.0.5 with the DAC128V enabled.  It has currently run for 2500 seconds without failing, while base 7.0.6 always failed within 138 seconds.  I will let it run overnight.

 

Thanks,

Mark

 

 

 

From: Mark Rivers
Sent: Friday, May 27, 2022 3:17 PM
To: Michael Davidsaver <mdavidsaver at gmail.com>
Cc: tech-talk at aps.anl.gov
Subject: RE: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

 

My recent tests have been done with the following:

 

- A very "thin" startup script that only loads the ip330 and dac128V drivers and records.

- A "thick" application that is built with most of the synApps modules. 

 

I just stripped it down so the application is only built with base, asyn, ipac, ip330, dac128v.  I made fresh git clones of asyn, ipac, ip330, dac128v.

 

The application Makefile is very simple:

 

**********************

corvette:CARS/CARSApp/testSrc>more Makefile

TOP=../..

 

include $(TOP)/configure/CONFIG

#----------------------------------------

#  ADD MACRO DEFINITIONS AFTER THIS LINE

 

PROD_NAME = CARSTest

PROD_IOC_vxWorks = $(PROD_NAME)

 

DBD_NAME = $(PROD_NAME)VX

DBD  += $(DBD_NAME).dbd

 

$(DBD_NAME)_DBD += base.dbd

 

$(DBD_NAME)_DBD += asyn.dbd

$(PROD_NAME)_LIBS := asyn $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += dac128VSupport.dbd

$(PROD_NAME)_LIBS := dac128V $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += ip330Support.dbd

$(PROD_NAME)_LIBS := ip330 $($(PROD_NAME)_LIBS)

 

$(DBD_NAME)_DBD += drvIpac.dbd

$(PROD_NAME)_LIBS := Ipac $($(PROD_NAME)_LIBS)

 

$(PROD_NAME)_LIBS += $(EPICS_BASE_IOC_LIBS)

 

$(PROD_NAME)_SRCS  += $(DBD_NAME)_registerRecordDeviceDriver.cpp

 

$(PROD_NAME)_OBJS_vxWorks += $(EPICS_BASE_BIN)/vxComLibrary

 

#===========================

 

include $(TOP)/configure/RULES

#----------------------------------------

#  ADD RULES AFTER THIS LINE

**********************

 

These are the commands that build the dbd file and link the application.

 

Creating dbd file CARSTestVX.dbd

perl -CSD /corvette/usr/local/epics-devel/base-7.0.6/bin/linux-x86_64/dbdExpand.pl   -I. -I.. -I../O.Common -I../../../dbd -I/corvette/home/epics/devel/asyn-4-42/dbd -I/corvette/home/epics/devel/CARS/dbd -I/corvette/home/epics/devel/dac128V-2-10-1/dbd -I/corvette/home/epics/devel/ipac-2-16/dbd -I/corvette/home/epics/devel/ip330-2-10/dbd -I/corvette/usr/local/epics-devel/base-7.0.6/dbd -o CARSTestVX.dbd base.dbd asyn.dbd dac128VSupport.dbd ip330Support.dbd drvIpac.dbd

Installing created dbd file ../../../dbd/CARSTestVX.dbd

 

/usr/local/vw/vxWorks-6.9/gnu/4.3.3-vxworks-6.9/x86-linux2/bin/ldppc -r -o CARSTest  -L/corvette/home/epics/devel/asyn-4-42/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/dac128V-2-10-1/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/ip330-2-10/lib/vxWorks-ppc32/ -L/corvette/home/epics/devel/ipac-2-16/lib/vxWorks-ppc32/ -L/corvette/usr/local/epics-devel/base-7.0.6/lib/vxWorks-ppc32/                   /corvette/usr/local/epics-devel/base-7.0.6/bin/vxWorks-ppc32/vxComLibrary.o CARSTestVX_registerRecordDeviceDriver.o    -lIpac -lip330 -ldac128V -lasyn -ldbRecStd -ldbCore –lca

 

I have a script that is writing new values to DAC channel 0 at 100 Hz.  That causes all 16 ADC records to post monitors at about 10 Hz.

 

I closed the medm screen and stopped camonitor on the ADC channels.  I started the IOC and it ran fine for 5 minutes with the script writing to the DAC.

 

I then opened the medm screen and started the camonitor on the ADC channels.  It failed in a few seconds with the A32 address error:

 

ioc13lab2>

VME Bus Error accessing A32: 0xbff99994

machine check

Exception next instruction address: 0x0329b0e0

Machine Status Register: 0x0008b032

Condition Register: 0x48000884

Task: 0x3560930 "CAS-event"

0x3560930 (CAS-event): task 0x3560930 has had a failure and has been stopped.

0x3560930 (CAS-event): The task has been terminated because it triggered an exception that raised the signal 10.

 

Mark

 

 

 

-----Original Message-----
From: Michael Davidsaver <mdavidsaver at gmail.com>
Sent: Friday, May 27, 2022 1:49 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: Bus errors accessing VME with base 7.0.6.1 and latest synApps modules

 

> On 5/27/22 06:17, Mark Rivers wrote:

>> Hi Michael,

>>

>> I commented out all lines in recGbl.c that reference the new amsg and namsg fields.  That did not fix the problem.

 

I take this to mean both db_post_events() and epicsVsnprintf() calls, which means that both AMSG and NAMSG should always be an empty string.

(aka. char[40] all nils)

 

https://github.com/epics-base/epics-base/commit/892a361de7002bf2b3f375f24bc5bf61858de2e5

 

> On 5/27/22 10:33, Mark Rivers wrote:

>> I am now using the unmodified 7.0.6 release.

...

> Before I build in that directory I run

 

I don't know how difficult it is with vxWorks @APS, but it might be a good idea to go further and start from fresh git-clone of all modules (base and synapps and application).  This should rule out build leftovers entirely.

 

 

On 5/27/22 10:33, Mark Rivers wrote:

> What could possibly cause this?

 

This is a very good question...

 

 

As I think about it, adding/removing AMSG+NAMSG would have exposed some partial rebuild problems with Base and parts of synapps.  The generated record type headers include a check that the number of fields at load time is the same as the number at build time.

 

 

If not a build/ABI issue, I don't see an easy way forward in troubleshooting.

 

We've passed the point where I would be trying to setup a (remote) debugger to inspect the stack of a faulting thread.  This should identify one of the records involved, and a dump of that record struct may show evidence of corruption.

 

For that matter, "dbpr <name> 5" could show evidence, if you can find one of the records.