Experimental Physics and
Industrial Control System

Hello all, good evening. 

We are experimenting several segfaults with motor IOCs here. Behavior is unpredictable, but i suspect common cause. Thanks in advance for anyone who has tips on how to debug.

Host machine: Linux - debian 9.11. Kernel version 4.9.0-11-amd64.

It runs a variable number of EPICS IOCs with procServ, between 20 and 30 depending on the day. Segfaults were noticed with dmesg after some clients got virtual circuit disconnect messages from camonitors or instantiated PV objects in pyepics.

In this machine, dmesg -T shows:

[qui jan 26 11:43:12 2023] motorPoller[2714]: segfault at 55d4ff3a2e68 ip 00007f38dac4f77b sp 00007f38d61e6e80 error 4 in libpmacAsynMotorPort.so[7f38dac30000+6a000] [qui jan 26 12:22:06 2023] motorPoller[5494]: segfault at 55eaaf933da8 ip 00007ff3a6cbe77b sp 00007ff3a2255e80 error 4 in libpmacAsynMotorPort.so[7ff3a6c9f000+6a000]

dmesg version is 2.29.2.

i have no core dump, possibly because sigfault is in thread and not in main IOC process. Main IOC process can take up to 15 seconds after kernel segfault message to actually terminate and be automatically restarted by procServ.

Segfault is noticeably frequent. Frequency varies, possibly being higher when clients are using IOC. I have seen whole days without a single segfault and days in which it happens once every 10 minutes.

Problematic iocs are from Pmac module (pulled from here: https://github.com/dls-controls/pmac), but i  have observed similar behavior in NPT NewFocus module IOCs.

I tried running

gdb --args /path/to/ioc/binary/executable /path/to/st.cmd/file

with procServ so that when IOC sigfaulted i could telnet into the gdb console and debug, but the only IOC in which i could do that didnt segfault until now.
In Pmac IOCs, what i can observe until now is several "fail to parse axis status" messages, but i also observe these in non-problematic IOCs:

2023/01/25 17:20:51.692 getAxisStatus: Failed to parse position. Key: #33P  Value:  $0000400000000000
2023/01/25 17:20:51.693 pmacCSAxis::GetAxisStatus: Failed to parse position. Key: &1Q89  Value:  $0000f80000000000

Im probably going to get more info on that but for now its all i have.
Any tips on how to debug this?