Re: Problems with Dante (XGLab) Driver

[Dariush Hampai via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark
Hi Andew
with a carefully read of "(gdb) thread apply all bt" I found a strange Thread almost at the end...

Thread 256 (Thread 0x7fff669a2640 (LWP 1412432) "save_restore"):
#0  0x00007ffff58884da in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007ffff588acaf in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007ffff6c8578d in epicsEventWaitWithTimeout (pevent=0x7fff78020330, timeout=<optimized out>) at ../osi/os/posix/osdEvent.c:131
#3  0x00007ffff6c86c27 in myReceive (timeout=<optimized out>, size=512, message=0x200, pmsg=0xa724a0) at ../osi/os/default/osdMessageQueue.cpp:369
#4  epicsMessageQueueReceiveWithTimeout (pmsg=0xa724a0, message=message@entry=0x7fff669a1a40, size=size@entry=512, timeout=1) at ../osi/os/default/osdMessageQueue.cpp:404
#5  0x00007ffff73d5796 in save_restore () at ../save_restore.c:1226
#6  0x00007ffff6c82795 in start_routine (arg=0x3384670) at ../osi/os/posix/osdThread.c:442
#7  0x00007ffff588b4f9 in start_thread () from /lib64/libc.so.6
#8  0x00007ffff59106e0 in clone3 () from /lib64/libc.so.6

only here it was called the Thread "save_restore" and only here the "epicsMessageQueueReceiveWithTimeout"
maybe is this the problem?

Dariush

Il 23/06/2026 14:05, Dariush Hampai via Tech-talk ha scritto:

Hi Mark,
Hi Andrew,
any idea?

Dariush

Il 22/06/2026 11:25, Dariush Hampai ha scritto:

Dear Mark
Dear Andrew,

I replied the crash and it seems the same.

#0  0x00007ffff5904b92 in pselect () from /lib64/libc.so.6
#1  0x00007ffff6b143bb in rl_getc () from /lib64/libreadline.so.8
#2  0x00007ffff6b13cd1 in rl_read_key () from /lib64/libreadline.so.8
#3  0x00007ffff6af8497 in readline_internal_char () from /lib64/libreadline.so.8
#4  0x00007ffff6b01535 in readline () from /lib64/libreadline.so.8
#5  0x00007ffff6c85cd2 in osdReadline (context=0x444dc0, prompt=0x7ffff6c9c183 "epics> ") at ../osi/os/default/gnuReadline.c:70
#6  epicsReadline (prompt=0x7ffff6c9c183 "epics> ", context=0x444dc0) at ../osi/epicsReadline.c:68
#7  0x00007ffff6c77aea in iocshBody (pathname=<optimized out>, commandLine=0x0, macros=0x0) at ../iocsh/iocsh.cpp:1143
#8  0x000000000040a616 in main (argc=<optimized out>, argv=<optimized out>) at ../mcaDanteAppMain.cpp:20

In attach I'll put the output of "thread apply all backtrace​"

awaiting your replies...

Dariush

Il 19/06/2026 17:13, Johnson, Andrew N. ha scritto:

Hi Dariush,

Please also include any messages output just before and announcing the crash, and instead of just the gdb command  backtrace​  first run  set height 0​  to disable the pager and then  thread apply all backtrace​  which will produce lots of output that may help Mark diagnose the problem.

- Andrew

-- 

Complexity comes for free, Simplicity you have to work for.

On 6/19/26, 9:50 AM, "Tech-talk" <tech-talk-bounces at aps.anl.gov> wrote:

What did you do that triggered the crash this time?

Please continue to run the IOC using gdb.  Each time it crashes save the output of backtrace.  We need to see if it is always crashing in the readline library.

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Friday, June 19, 2026 9:45 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

Dear Mark,

I don't know if it is the same for all the previous crashes... however the effects are the same...

Dariush

Il 19/06/2026 16:42, Mark Rivers ha scritto:

Hi Dariush,

The gdb backtrace says that the crash is actually in the Linux readline library.  Was this crash caused by the same sequence of events as previous crashes you observed?

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Friday, June 19, 2026 9:17 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

 Hi Mark,

Are you using a Dante1 or a Dante8?
I'm using Dante8

Does this happen every time you start, or just occasionally.  If it is occasionally, then how frequently does it happen?
Occasionally, more often when I two records are executed very closely

Are there any error messages on the IOC?
no, however some records are in "acquire" exit (as $(P)$(M).ACQG in mca window)

Are you running on Linux or Windows? 
Linux (Centos 9)

If you are running on Linux then please run the IOC in the GNU debugger. You can do that with the following commands from the iocDante1 directory:

gdb ../../bin/linux-x86_64/mcaDanteApp
run st.cmd

When it crashes type this command at the debugger prompt:

backtrace

(gdb) backtrace
#0  0x00007ffff5904b92 in pselect () from /lib64/libc.so.6
#1  0x00007ffff6b143bb in rl_getc () from /lib64/libreadline.so.8
#2  0x00007ffff6b13cd1 in rl_read_key () from /lib64/libreadline.so.8
#3  0x00007ffff6af8497 in readline_internal_char () from /lib64/libreadline.so.8
#4  0x00007ffff6b01535 in readline () from /lib64/libreadline.so.8
#5  0x00007ffff6c85cd2 in osdReadline (context=0x444dc0, prompt=0x7ffff6c9c183 "epics> ") at ../osi/os/default/gnuReadline.c:70
#6  epicsReadline (prompt=0x7ffff6c9c183 "epics> ", context=0x444dc0) at ../osi/epicsReadline.c:68
#7  0x00007ffff6c77aea in iocshBody (pathname=<optimized out>, commandLine=0x0, macros=0x0) at ../iocsh/iocsh.cpp:1143
#8  0x000000000040a616 in main (argc=<optimized out>, argv=<optimized out>) at ../mcaDanteAppMain.cpp:20

Thank you in advance
Dariush e Maurizio

Il 16/06/2026 18:18, Mark Rivers ha scritto:

Hi Dariush,

Are you using a Dante1 or a Dante8?

Does this happen every time you start, or just occasionally.  If it is occasionally, then how frequently does it happen?

Are there any error messages on the IOC?

Are you running on Linux or Windows? 

If you are running on Linux then please run the IOC in the GNU debugger. You can do that with the following commands from the iocDante1 directory:

gdb ../../bin/linux-x86_64/mcaDanteApp

run st.cmd

When it crashes type this command at the debugger prompt:

backtrace

Send me the output.

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Tuesday, June 16, 2026 10:23 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Problems with Dante (XGLab) Driver

Hi Community,
Hi Mark,

I'm almost finish the implementation of Dante EPICS Drivers in our
system, however I have a big problem (maybe a bug?).
When I start few acquisitions (from caput command or from Phebus), the
system seems that crash.
The Dante:dante2:ElapsedRealTime stops (not on target). Moreover on
Phebus the text is in "Collecting" mode, The Acquire Busy is in
"Acquiring" mode and the IOC do not respond to any command that I send.
Up to now, the only solution is to stop the IOC and restart it.
What's the problem?
Is there a possibility to force a reset/reinitialize the driver without
stop and restart it?

awaiting your (precious) help

Dariush

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room:	+39.06.9403.5248
Lab.:	+39.06.9403.2286
Mob.:	+39.06.9403.8025
Fax.:	+39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room:	+39.06.9403.5248
Lab.:	+39.06.9403.2286
Mob.:	+39.06.9403.8025
Fax.:	+39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room:	+39.06.9403.5248
Lab.:	+39.06.9403.2286
Mob.:	+39.06.9403.8025
Fax.:	+39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room:	+39.06.9403.5248
Lab.:	+39.06.9403.2286
Mob.:	+39.06.9403.8025
Fax.:	+39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room:	+39.06.9403.5248
Lab.:	+39.06.9403.2286
Mob.:	+39.06.9403.8025
Fax.:	+39.06.9403.2597

************************************