Experimental Physics and
Industrial Control System

Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov> · Fri, 1 Jul 2022 11:29:46 -0700

On 6/30/22 13:07, Mooney, Tim M. wrote:

Hi Pete,

If you're using /dev/mem to read/write memory implemented in the FPGA using an AXI 4 peripheral, you may not have to write a driver, but may be able to use the generic-uio driver.  This allows access from a non-root account, and it allows access by name (which occurs in the device tree) instead of by address, which might change a become a pain in the neck.

I'll second this.  If you are only after memory mapped register access,
then using the "generic-uio" platform driver should be sufficient.
The main obsticle is understanding, and handling, linux device tree files.

I tried to lay out some of this back in 2020:

https://epics.anl.gov/tech-talk/2020/msg02201.php

Things do get more complicated if you want to make use of functionality
like interrupts or DMA.

Tim Mooney (mooney at anl.gov) (630)252-5417
Beamline Controls Group (www.aps.anl.gov)
Advanced Photon Source, Argonne National Lab

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*From:* Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Siddons, David via Tech-talk <tech-talk at aps.anl.gov>
*Sent:* Thursday, June 30, 2022 2:25 PM
*To:* Rivers, Mark L. <rivers at cars.uchicago.edu>; Michael Davidsaver <mdavidsaver at gmail.com>
*Cc:* tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
*Subject:* Re: Timeout when running soft IOC as root
Hi Michael,
     You are right, of course. I am looking into getting a driver written, but it takes time.
Pete.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*From:* Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Michael Davidsaver via Tech-talk <tech-talk at aps.anl.gov>
*Sent:* Monday, June 27, 2022 9:12 PM
*To:* Mark Rivers <rivers at cars.uchicago.edu>
*Cc:* tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
*Subject:* Re: Timeout when running soft IOC as root
On 6/27/22 10:24, Mark Rivers via Tech-talk wrote:

Folks,

I am trying to help Pete Siddons with a problem with an IOC that uses the quadEM module for a BPM he is developing.  It runs EPICS on a Zynq processor.  He says it was working on Debian 7, but is not working on Debian 11.

Which Linux kernel variant/version?

  Because the IOC talks to /dev/mem it needs to run as root.

ick...

I would very much recommend that Pete look at creating a real Linux driver.
mmap()ing "/dev/mem" into an userspace process is playing with fire.

Alternately, adjust file permissions and/or Linux kernel options to bypass
some /dev/mem safety checks (eg. STRICT_DEVMEM and IO_STRICT_DEVMEM)

https://urldefense.com/v3/__https://elixir.bootlin.com/linux/latest/source/lib/Kconfig.debug*L1804__;Iw!!P4SdNyxKAPE!FUgNGKi4r71uFnnc8bN6dWdAsVE564Lgzf5o-sG6kwN9Sen6WKDXYFqb3EbEpUN7LozrwhsWJrOZ-VqJllx9ig$ <https://urldefense.com/v3/__https://elixir.bootlin.com/linux/latest/source/lib/Kconfig.debug*L1804__;Iw!!P4SdNyxKAPE!FUgNGKi4r71uFnnc8bN6dWdAsVE564Lgzf5o-sG6kwN9Sen6WKDXYFqb3EbEpUN7LozrwhsWJrOZ-VqJllx9ig$>

The driver can be built in simulation mode, so it does not talk to the hardware at all.  In this mode the IOC runs fine when run as a non-privileged user, and takes a total of 3.5 seconds to start up.  This includes the time to create many areaDetector plugins.

However, when running as root there are timeout errors waiting for each of the plugin threads to be created.  The timeout is 2 seconds, and increasing it to 10 seconds does not fix the problem.

The quickest way to an answer will probably be gdb (or gdbserver).

Run with a break point set where the "timeout" message is printed[1].
When this is reached, dump all of the thread stacks. (run
"thread apply all backtrace")

This will show if the plugin threads are being created, and if so,
what they are doing.

[1] https://github.com/areaDetector/ADCore/blob/bcaced94ad48b9bb5fdbd1a7f6d9d8be5c416003/ADApp/pluginSrc/NDPluginDriver.cpp#L912 <https://github.com/areaDetector/ADCore/blob/bcaced94ad48b9bb5fdbd1a7f6d9d8be5c416003/ADApp/pluginSrc/NDPluginDriver.cpp#L912>

If you haven't encountered gdbserver before, there are two parts.
On the target run something like:

$ gdbserver 0.0.0.0:3333 ./bin/linux-arm/myioc st.cmd

Then on the (Debian) host run:

$ gdb-multiarch
(gdb) target extended-remote <target-ip>:3333
(gdb) b NDPluginDriver.cpp:912
(gdb) run
... wait for breakpoint
(gdb) thread apply all backtrace
... lots of output ...
(gdb) quit

The TCP port number can be anything.  3333 is an arbitrary choice.

This the abbreviated IOC startup output when not running as root.

*****************************

date

2022/06/27 16:03:39.620634

errlogInit(5000)

< envPaths

epicsEnvSet("IOC","iocNSLS2_IC")

epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")

…

dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")

quadEMTestApp_registerRecordDeviceDriver(pdbbase)

# The search path for database files

# Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows

date

2022/06/27 16:03:39.953518

drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)

Memory mapped

Poller thread created

Creating parameters 0

Poll thread starts

Creating parameters 1

Creating parameters 2

Creating parameters 3

Creating parameters 4

Opening DACS

Getting firmware version

fpgabase=0x569a60

FPGA version=1001

Calling callbacks

date

2022/06/27 16:03:40.129864

# Create 11 statistics plugins

NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template",     "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")

NDTimeSeriesConfigure("NSLS2_IC_STATS1_TS", 20, 0, "NSLS2_IC_STATS1", 1, 23)

…

NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=PosY")

#< $(QUADEM)/iocBoot/saveRestore.cmd

date

2022/06/27 16:03:43.046352

iocInit()

############################################################################

## EPICS R7.0.6.1

## Rev. 2022-03-25T16:17+0000

###########################

iocRun: All initialization complete

#################################################

Gain: 0

Setting bias voltage to 0.000000

date

2022/06/27 16:03:46.982984

# save settings every thirty seconds

#create_monitor_set("auto_settings.req",30,"P=$(PREFIX), R=$(RECORD)")

*****************************

The IOC starts up fine.  The total time to start is about 3.5 seconds.  There are many plugin threads created during this time.

This the abbreviated IOC startup output when running as root.

*****************************

date

2022/06/27 14:20:48.298214

errlogInit(5000)

< envPaths

epicsEnvSet("IOC","iocNSLS2_IC")

epicsEnvSet("TOP","/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4")

…

dbLoadDatabase("/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/dbd/quadEMTestApp.dbd")

quadEMTestApp_registerRecordDeviceDriver(pdbbase)

# The search path for database files

# Note: the separator between the path entries needs to be changed to a semicolon (;) on Windows

epicsEnvSet("EPICS_DB_INCLUDE_PATH", "/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db:/home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/db")

< /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/iocNSLS2_IC/NSLS2_IC.cmd

date

2022/06/27 14:20:48.626216

…

drvNSLS2_ICConfigure("NSLS2_IC", 0, 10000)

Memory mapped

Poller thread created

Creating parameters 0

Poll thread starts

Creating parameters 1

Creating parameters 2

Creating parameters 3

Creating parameters 4

Opening DACS

Getting firmware version

fpgabase=0x5809b8

FPGA version=1001

Calling callbacks

…

date

2022/06/27 14:20:48.802463

< /home/peter/epics/extensions/synApps/support-master/synApps/support/quadEM-R9-4/iocBoot/commonPlugins.cmd

…

# Create 11 statistics plugins

NDStatsConfigure("NSLS2_IC_STATS1", 20, 0, "NSLS2_IC", 0, 0, 0)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDStats.template",     "P=quadEMTest:,R=NSLS2_IC:Current1:, PORT=NSLS2_IC_STATS1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC,NDARRAY_ADDR=0, HIST_SIZE=256,XSIZE=10000,YSIZE=0,NCHANS=1000,ENABLED=1,")

…

# FFT plugins

NDFFTConfigure("NSLS2_IC_FFT1", 20, 0, "NSLS2_IC_TS1", 0)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current1:, PORT=NSLS2_IC_FFT1, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=0, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=Current1")

NDFFTConfigure("NSLS2_IC_FFT2", 20, 0, "NSLS2_IC_TS1", 1)

dbLoadRecords("/home/peter/epics/extensions/synApps/support2022/06/27 14:21:19.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

2022/06/27 14:21:23.270 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

2022/06/27 14:21:26.247 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

2022/06/27 14:21:28.278 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

Starting iocInit

-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current2:, PORT=NSLS2_IC_FFT2, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=1, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV CP  MS,ENABLED=1,NAME=Current2")

NDFFTConfigure("NSLS2_IC_FFT3", 20, 0, "NSLS2_IC_TS1", 2

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current3:, PORT=NSLS2_IC_FFT3, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=2, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=Current3")

NDFFTConfigure("NSLS2_IC_FFT4", 20, 0, "NSLS2_IC_TS1", 3)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:Current4:, PORT=NSLS2_IC_FFT4, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=3, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=Current4")

NDFFTConfigure("NSLS2_IC_FFT5", 20, 0, "NSLS2_IC_TS1", 4)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumX:, PORT=NSLS2_IC_FFT5, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=4, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=SumX")

NDFFTConfigure("NSLS2_IC_FFT6", 20, 0, "NSLS2_IC_TS1", 5)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumY:, PORT=NSLS2_IC_FFT6, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=5, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=SumY")

NDFFTConfigure("NSLS2_IC_FFT7", 20, 0, "NSLS2_IC_TS1", 6)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:SumAll:, PORT=NSLS2_IC_FFT7, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=6, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=SumAll")

NDFFTConfigure("NSLS2_IC_FFT8", 20, 0, "NSLS2_IC_TS1", 7)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffX:, PORT=NSLS2_IC_FFT8, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=7, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=DiffX")

NDFFTConfigure("NSLS2_IC_FFT9", 20, 0, "NSLS2_IC_TS1", 8)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:DiffY:, PORT=NSLS2_IC_FFT9, ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=8, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=DiffY")

NDFFTConfigure("NSLS2_IC_FFT10", 20, 0, "NSLS2_IC_TS1", 9)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosX:,PORT=NSLS2_IC_FFT10,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=9, NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=PosX")

NDFFTConfigure("NSLS2_IC_FFT11", 20, 0, "NSLS2_IC_TS1", 10)

dbLoadRecords("/home/peter/epics/extensions/synApps/support-master/synApps/support/areaDetector-R3-11/ADCore/db/NDFFT.template","P=quadEMTest:,R=NSLS2_IC:FFT:PosY:,PORT=NSLS2_IC_FFT11,ADDR=0,TIMEOUT=1,NDARRAY_PORT=NSLS2_IC_TS1,NDARRAY_ADDR=10,NCHANS=1000,TIME_LINK=quadEMTest:NSLS2_IC:TS:TSAveragingTime_RBV  CP MS,ENABLED=1,NAME=PosY")

#< $(QUADEM)/iocBoot/saveRestore.cmd

date

2022/06/27 14:21:29.253141

iocInit()

############################################################################

## EPICS R7.0.6.1

## Rev. 2022-03-25T16:17+0000

###########################2022/06/27 14:23:25.241 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

2022/06/27 14:23:25.241 NDPluginDriver::writeInt32 ERROR, status=3, function=67, paramName=NUM_THREADS, value=1, connectedToArrayPort_=1

2022/06/27 14:23:32.243 NDPluginDriver::startCallbackThreads timeout waiting for plugin thread 0 start message

2022/06/27 14:23:32.244 NDPluginDriver::writeInt32 ERROR, status=3, function=64, paramName=QUEUE_SIZE, value=20, connectedToArrayPort_=1

2022/06/27 14:23:34.244 NDPluginDriver::deleteCallbackThreads timeout waiting for plugin thread 0 exit message

epicsThread::~epicsThread(): "_main_" blocking for thread "NSLS2_IC_STATS2_Plugin_1" to exit