Hi Rea,

• Is it even possible to use the Model 3 framework to communicate with something that uses such a specific and non-standard byte stream, or will I have to find a different way? Kind regards,

• Enable asynTraceIODriver and ASYN_TRACEIO_ESCAPE for the underlying asyn communication port.  That will let you see the complete communication with the device.  Put these lines in your startup script right after you create the asyn communication port, which I have called $(PORT) here:

• Put debugging printf or asynPrint statements in your driver to see what is happening.

Any thoughts or updates on Linux signalling for EPICS from the devs?

I don't think there have been any updates, but I remember there were implementation suggestions for my idea. For epics-base, it would consist of launching a thread listening on an epicsEvent, which, when signaled, would call epicsExit. For a module with platform specific code, it would consist of registering signal handlers (possibly automatically or via iocsh commands) and having those signal the previously mentioned epicsEvent.

If you're keen on getting your hands a little dirty, how about we collaborate on moving this forward? Maybe start implementing the epics-base interface so we can check its suitability for the support module?

As far as APIs.  If epicsAtExit() is not enough, or if the ordering gets too complex, I would suggest looking at the "de-init" hooks.  See "initHookAtShutdown" in initHooks.h .

also fyi. https://github.com/epics-base/epics-base/pull/673

Hi Mark,
I'm trying to fix the racecondition that I have, and the file dante.cpp confirms a hypothesis more precise than the initial one.
As you said correctly waitReply does NOT block forever — it has a 20-second timeout (MESSAGE_TIMEOUT). However, the "infinite" hang I'm observing is likely the cumulative sum of many cascading 20s timeouts, caused by a very specific design flaw I've identified:

***
else if (function == DanteKeepAlive) {
    // This just periodically reads the firmware version of the first board
    // to keep the socket alive
    callId_ = getFirmware(danteIdentifier_, 0);
    waitReply(callId_, danteReply_, "getFirmware");
}
***

It seems as:
1) callId_ is a single shared member variable (dante.h: uint32_t callId_;), neither local nor protected by a robust matching mechanism.
2) Board responses arrive asynchronously via danteCallback(), invoked by internal SDK threads (AsyncTask, the ones you saw in today's backtrace) — this callback is not subject to asynPortDriver's lock, because it originates from the SDK world, not the asyn world.
3) If a request (e.g., the getFirmware call from DanteKeepAlive, or the one invoked inside acquisitionTask for stop() in list mode, line 1268) times out (20s) before the response arrives, that response still arrives afterward, and ends up in the msgQ_ queue.
4) The next call to waitReply (perhaps for a completely different command) reads that "old" response from the queue, finds a non-matching call_id (lines 406-410), discards it, and returns an error — but meanwhile, the legitimate response for that new request is still on its way, and will end up misaligning the subsequent call as well.

The result seems a progressive and permanent queue desynchronization — from that point on, every waitReply call uselessly waits the full 20 seconds (or misinterprets a message), because the queue is now "shifted" by one position. With long scans, as more operations are performed, it becomes increasingly likely that this desynchronization occurs and worsens, until some critical state transition (e.g., detecting the end of an acquisition) never receives the correct response — and at that point, the system appears permanently hung.

Instead of giving up at the first non-matching message, waitReply should continue draining the queue until it finds the right message or the total time expires, as:

***
asynStatus Dante::waitReply(uint32_t callId, char *reply, const char *caller) {
    struct danteMessage message;
    int numRecv;
    static const char *functionName = "waitReply";
    epicsTimeStamp start, now;
    double elapsed;
    double remaining;

    epicsTimeGetCurrent(&start);

    while (1) {
        epicsTimeGetCurrent(&now);
        elapsed = epicsTimeDiffInSeconds(&now, &start);
        remaining = MESSAGE_TIMEOUT - elapsed;
        if (remaining <= 0) {
            asynPrint(pasynUserSelf, ASYN_TRACE_ERROR,
                "%s::%s timeout waiting for matching reply, caller=%s expected callId=%u\n",
                driverName, functionName, caller, callId);
            return asynError;
        }

        numRecv = msgQ_->receive(&message, sizeof(message), remaining);
        if (numRecv != sizeof(message)) {
            asynPrint(pasynUserSelf, ASYN_TRACE_ERROR,
                "%s::%s error receiving message in %f seconds, caller=%s numRecv=%d\n",
                driverName, functionName, remaining, caller, numRecv);
            return asynError;
        }

        if (message.call_id != callId) {
            asynPrint(pasynUserSelf, ASYN_TRACE_WARNING,
                "%s::%s stale/mismatched reply discarded, caller=%s expected=%u actual=%u (retrying)\n",
                driverName, functionName, caller, callId, message.call_id);
            continue;
        }

        for (uint32_t i=0; i<message.length; i++) {
            reply[i]=message.data[i];
        }
        return asynSuccess;
    }
}
***

changing in:
1) The 20s timeout is now total, not per individual receive() call — if 3 misaligned messages arrive in rapid succession, you don't have 3x20s of waiting, but the time still expires within the overall 20s window
2) Non-matching messages are silently discarded but the queue continues to be drained within the same loop, so the next caller won't find a "toxic" discarded message waiting for them
3) Added a warning log (not error) for intermediate mismatches, useful for future diagnostics without clogging logs with false errors

Dariush

Hi Darius,

• That reply never arrives, so waitReply() blocks indefinitely — while still holding the asynPortDriver lock, since the port thread typically holds it for the duration of the I/O operation.

I don't believe it is possible for waitReply() to block indefinitely.  The line where it actually waits is here:

https://github.com/epics-modules/Dante/blob/2ef206c6c2c2b0349218c8329bc4c7cb15999c93/danteApp/danteSrc/dante.cpp#L399

This is the code:

    numRecv = msgQ_->receive(&message, sizeof(message), MESSAGE_TIMEOUT);

    if (numRecv != sizeof(message)) {

        asynPrint(pasynUserSelf, ASYN_TRACE_ERROR,

            "%s::%s error receiving message in %f seconds, caller=%s numRecv=%d\n",

            driverName, functionName, MESSAGE_TIMEOUT, caller, numRecv);

        return asynError;

    }

It is waiting for a message to arrive on an epicsMessageQueue.  It will timeout after MESSAGE_TIMEOUT, which is defined to be 20 seconds.

If it times out you should see an error message on the IOC console, which is printed in that asynPrint line.

You could add debugging asynPrint(pasynUserSelf, ASYN_TRACE_ERROR, ...) calls before and after the call to msgQ_->receive() to see if it is really waiting for more than 20 seconds.

• Possible fix directions:
  1) Add a timeout to waitReply() so a missing reply from the board doesn't block forever
  
  

There is already a timeout as shown above.

• 2) Avoid issuing getFirmware (or other non-acquisition commands) while an acquisition is in progress, or ensure they don't compete for the same lock held by acquisitionTask

  
  

The getFirmware command is being sent because the Dante SDK requires that "keep-alive" messages be sent periodically or it will close the socket.  The following is the KeepAlive record that is used to do this.

https://github.com/epics-modules/Dante/blob/2ef206c6c2c2b0349218c8329bc4c7cb15999c93/danteApp/Db/dante.template#L203-L212

# This record scans periodically when acquisition is not active to keep the socket connection alive

record(bo,"$(P)$(R)KeepAlive") {

    field(DTYP, "asynInt32")

    field(OUT, "@asyn($(PORT))DanteKeepAlive")

    field(SCAN,"10 second")

    field(DISV,"1")

    field(SDIS,"$(P)$(R)MCAAcquireBusy NPP NMS")

    field(ZNAM,"Done")

    field(ONAM,"Poll")

}

Note that it is processed every 10 seconds, and that this record is disabled if MCAAcquireBusy is 1.  Thus, it should not be calling getFirmware if the Dante is currently acquiring.

Please open an issue for this on the Dante Github page.  That is a more appropriate location than tech-talk for this.  Please provide full background for the issue there, including all information that I have requested but that you have not provided, like versions of all packages.

https://github.com/epics-modules/Dante/issues

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Thursday, June 25, 2026 5:36 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Johnson, Andrew N. <anj at anl.gov>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

Hi Mark,
I'm running into a recurring IOC lockup with the Dante driver during long scans, and I managed to capture a full backtrace (thread apply all bt via gdb attached to the running process) right at the moment of the freeze. I believe this is a genuine deadlock in the driver. Here's what I found:
Thread 137 "DANTE1" (the asyn portThread):
#5  Dante::waitReply(callId=244, caller="getFirmware") at dante.cpp:399
#6  Dante::writeInt32(value=0) at dante.cpp:567
#7  asynPortDriver::writeInt32
#8  processCallbackOutput (devAsynInt32.c)
#9  portThread (asynManager.c)
Thread 129 "acquisitionTask":
#2  epicsMutexLock
#3  asynPortDriver::lock() at asynPortDriver.cpp:949
#4  Dante::acquisitionTask() at dante.cpp:1290
What appears to be happening:

The dedicated asyn portThread (Thread 137) is executing a getFirmware command via writeInt32(), which internally calls waitReply() waiting for a response from the board (callId=244).
That reply never arrives, so waitReply() blocks indefinitely — while still holding the asynPortDriver lock, since the port thread typically holds it for the duration of the I/O operation.
Meanwhile, acquisitionTask (Thread 129) needs the same lock to process newly acquired data at dante.cpp:1290, and blocks forever waiting for it.
Since the portThread is the single thread serializing all asyn requests on that port, the entire IOC becomes unresponsive to any further command (including StopAll/Reset) — only a full IOC restart recovers it.

Trigger condition:
This appears to be a race condition: it happened after ~300 acquisitions on one run, then again after only 21 and 33 acquisitions on subsequent runs — at variable scan steps, not a fixed one — which points to a timing-dependent race rather than a resource leak. We suspect the getFirmware command is being triggered by a periodic PV poll from a Phoebus/CS-Studio diagnostic screen running concurrently with the scan, colliding with an in-progress acquisition cycle.

Possible fix directions:
1) Add a timeout to waitReply() so a missing reply from the board doesn't block forever
2) Avoid issuing getFirmware (or other non-acquisition commands) while an acquisition is in progress, or ensure they don't compete for the same lock held by acquisitionTask

Happy to provide the full backtrace file, the exact PollTime settings, channel/trigger configuration (4096 channels, single mode, free-running, no trigger), or any other diagnostic info that would help track this down.
Thanks,
Dariush

p.s.: thanks also to Claude that help me in debug

Il 24/06/2026 16:19, Mark Rivers ha scritto:

All of my testing was done with PollTime=0.01 and I did not have a problem.  What type of computer are you using, i.e. how many cores and what clock speed.

When you acquire with PollTime=0.01 and you run "top -H" on the Linux machine what do you see.

PollTime=0.1 will reduce performance if you are trying to acquire quickly.  You could see if 0.02 is sufficient to fix the problem.

Please send a screenshot of the Dante screen in the configuration where you saw the problem.

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Wednesday, June 24, 2026 9:06 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Johnson, Andrew N. <anj at anl.gov>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

Hi Mark,

the IOC doesn’t exit… simply it remains in “acquiring status” without exit (even StopAll command doesn’t work).

I tried to increase the poll (from 0.01 to 0.1) as Claude suggested, and apparently works.

Tomorrow I’ll start a complete session (so there will be many acquisition) in order to see if the problem remain.

Two days ago I’ll attach a complete (and full) report about threads…

Dariush

************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:

XLab-Frascati

LNF-INFN

Casella Postale 13

Frascati (RM)

Italy

Room: +39.06.9403.5248
Lab.: +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.: +39.06.9403.2597

************************************

Il giorno 24 giu 2026, alle ore 15:47, Mark Rivers <rivers at cars.uchicago.edu> ha scritto:

Hi Dariush,

I am a bit confused.  In your original message you said:

• When I start few acquisitions (from caput command or from Phebus), the system seems that crash.

That led me to understand that the IOC was crashing, i.e. the exiting with an error. 

I then asked you to run the IOC under gdb and send me the backtrace when it crashed.  You did that, and it showed the error was in the readline library.

But now I am wondering if it really crashed.  What did you do to get to the gdb prompt and type "backtrace".  Did you type ^C, i.e. did you force it to crash?

You need to be clear on whether it is "crashing" or "hanging".

To make progress on this you need to do the following:

• Tell us exactly what versions of each module you are using (base, asyn, mca, dante).
• Send a screenshot of the Dante screen when it crashes so we know how you have configured it.
• Run the IOC under gdb and send the complete output from when you first start gdb to when it crashes and you type "backtrace".

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Wednesday, June 24, 2026 4:10 AM
To: Johnson, Andrew N. <anj at anl.gov>; Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

Hi Mark,
Hi Andrew,
using Claude maybe I found the problem (but not the solution...). Following what Claude ask, I use Dante in 4096 channels mode, not in mapping mode and without trigger.

Claude answer to me:

There are two key threads:
Thread 17 "acquisitionTask": 
#2 epicsThreadSleep 
#3 Dante::acquisitionTask (this=0x594560) at ../dante.cpp:1289 
Thread 8 "DANTE1": 
#2 epicsThreadSleep 
#3 Dante::writeInt32 (this=0x594560, value=1) at ../dante.cpp:461 
#4 asynPortDriver::writeInt32 
#5 processCallbackOutput (devAsynInt32.c) 
#6 portThread (asynManager.c)

1) Thread 8 "DANTE1" is the dedicated thread for the asyn port (portThread in asynManager.c) — the one EPICS uses to serialize all read/write requests on that asyn device. It is blocked inside Dante::writeInt32 with value=1, which with very high probability corresponds exactly to the EraseStart command you sent. This function internally calls epicsThreadSleep — so writeInt32 is waiting in a blocking, synchronous manner for something to happen (most likely for the acquisition to signal "done").

2) Thread 17 "acquisitionTask" is the internal thread of the Dante driver that should handle the acquisition loop, also stopped on epicsThreadSleep inside a loop at dante.cpp:1289. This is almost certainly a polling loop that periodically checks the acquisition status (while acquiring → sleep → check status).

The critical point: if acquisitionTask never sees the "acquisition completed" condition — due to a lost event, a race condition, or a status flag not being correctly updated by the firmware/board — it remains stuck in this loop forever. And since writeInt32 (called by the EraseStart command) most likely waits for this very completion before returning, the asyn port thread also remains blocked.

Why the IOC appears "frozen" and no commands work:

This is the most important detail: in asyn, a single thread (portThread) serializes all requests on that port. If that thread is blocked inside writeInt32, no other command can be processed on that device — not even StopAll or Reset, because they end up queued behind the blocked request. This is why the system appears completely frozen and unresponsive to anything until you restart the IOC.

Confirmation from official documentation:

This matches exactly what I found in the Performance page: in mapping mode with 2048 and 1024 channels, above a certain trigger frequency spectra are lost and the EPICS IOC must be restarted. It's very likely you're running into a variant of the same problem — a race condition in the driver where an acquisition completion event is lost, leaving acquisitionTask (and therefore writeInt32/EraseStart) waiting indefinitely. Paulscherrerinstitute

What you can do:

1) There is no documented clean software reset for this scenario, because the thread that should process any reset command is precisely the one that is blocked. Restarting the IOC remains, according to the documentation itself, the official solution.

2) Preventive mitigation — since the problem seems related to a lost event/state during acquisition, try to:
    a) Slightly increase PollTime (e.g., from 0.01 to 0.02-0.05s) to give the firmware more margin to communicate the status
    b) If you are in mapping mode with 1024/2048 channels and high trigger frequencies, evaluate whether you can switch to 4096 channels, where (according to the Performance table) the limits are much higher
    c) Verify that the Dante board firmware is at the latest version — bugs of this type are often fixed on the firmware side

3) If the problem persists, it's worth reporting it directly to EPICS Community or to Dante/XGLab support — the backtrace you produced is exactly the kind of evidence needed for a bug report, because it precisely identifies the lines (dante.cpp:1289 and dante.cpp:461) where the driver gets stuck.

Dariush

Il 23/06/2026 14:51, Dariush Hampai via Tech-talk ha scritto:

Hi Mark
Hi Andew
with a carefully read of "(gdb) thread apply all bt" I found a strange Thread almost at the end...

Thread 256 (Thread 0x7fff669a2640 (LWP 1412432) "save_restore"):
#0  0x00007ffff58884da in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007ffff588acaf in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007ffff6c8578d in epicsEventWaitWithTimeout (pevent=0x7fff78020330, timeout=<optimized out>) at ../osi/os/posix/osdEvent.c:131
#3  0x00007ffff6c86c27 in myReceive (timeout=<optimized out>, size=512, message=0x200, pmsg=0xa724a0) at ../osi/os/default/osdMessageQueue.cpp:369
#4  epicsMessageQueueReceiveWithTimeout (pmsg=0xa724a0, message=message@entry=0x7fff669a1a40, size=size@entry=512, timeout=1) at ../osi/os/default/osdMessageQueue.cpp:404
#5  0x00007ffff73d5796 in save_restore () at ../save_restore.c:1226
#6  0x00007ffff6c82795 in start_routine (arg=0x3384670) at ../osi/os/posix/osdThread.c:442
#7  0x00007ffff588b4f9 in start_thread () from /lib64/libc.so.6
#8  0x00007ffff59106e0 in clone3 () from /lib64/libc.so.6

only here it was called the Thread "save_restore" and only here the "epicsMessageQueueReceiveWithTimeout"
maybe is this the problem?

Dariush

Il 23/06/2026 14:05, Dariush Hampai via Tech-talk ha scritto:

Hi Mark,
Hi Andrew,
any idea?

Dariush

Il 22/06/2026 11:25, Dariush Hampai ha scritto:

Dear Mark
Dear Andrew,

I replied the crash and it seems the same.

#0  0x00007ffff5904b92 in pselect () from /lib64/libc.so.6
#1  0x00007ffff6b143bb in rl_getc () from /lib64/libreadline.so.8
#2  0x00007ffff6b13cd1 in rl_read_key () from /lib64/libreadline.so.8
#3  0x00007ffff6af8497 in readline_internal_char () from /lib64/libreadline.so.8
#4  0x00007ffff6b01535 in readline () from /lib64/libreadline.so.8
#5  0x00007ffff6c85cd2 in osdReadline (context=0x444dc0, prompt=0x7ffff6c9c183 "epics> ") at ../osi/os/default/gnuReadline.c:70
#6  epicsReadline (prompt=0x7ffff6c9c183 "epics> ", context=0x444dc0) at ../osi/epicsReadline.c:68
#7  0x00007ffff6c77aea in iocshBody (pathname=<optimized out>, commandLine=0x0, macros=0x0) at ../iocsh/iocsh.cpp:1143
#8  0x000000000040a616 in main (argc=<optimized out>, argv=<optimized out>) at ../mcaDanteAppMain.cpp:20

In attach I'll put the output of "thread apply all backtrace​"

awaiting your replies...

Dariush

Il 19/06/2026 17:13, Johnson, Andrew N. ha scritto:

Hi Dariush,

Please also include any messages output just before and announcing the crash, and instead of just the gdb command  backtrace​  first run  set height 0​  to disable the pager and then  thread apply all backtrace​  which will produce lots of output that may help Mark diagnose the problem.

- Andrew

-- 

Complexity comes for free, Simplicity you have to work for.

On 6/19/26, 9:50 AM, "Tech-talk" <tech-talk-bounces at aps.anl.gov> wrote:

What did you do that triggered the crash this time?

Please continue to run the IOC using gdb.  Each time it crashes save the output of backtrace.  We need to see if it is always crashing in the readline library.

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Friday, June 19, 2026 9:45 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

Dear Mark,

I don't know if it is the same for all the previous crashes... however the effects are the same...

Dariush

Il 19/06/2026 16:42, Mark Rivers ha scritto:

Hi Dariush,

The gdb backtrace says that the crash is actually in the Linux readline library.  Was this crash caused by the same sequence of events as previous crashes you observed?

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Friday, June 19, 2026 9:17 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Problems with Dante (XGLab) Driver

 

 Hi Mark,

Are you using a Dante1 or a Dante8?
I'm using Dante8

Does this happen every time you start, or just occasionally.  If it is occasionally, then how frequently does it happen?
Occasionally, more often when I two records are executed very closely

Are there any error messages on the IOC?
no, however some records are in "acquire" exit (as $(P)$(M).ACQG in mca window)

Are you running on Linux or Windows? 
Linux (Centos 9)

If you are running on Linux then please run the IOC in the GNU debugger. You can do that with the following commands from the iocDante1 directory:

gdb ../../bin/linux-x86_64/mcaDanteApp
run st.cmd

When it crashes type this command at the debugger prompt:

backtrace

(gdb) backtrace
#0  0x00007ffff5904b92 in pselect () from /lib64/libc.so.6
#1  0x00007ffff6b143bb in rl_getc () from /lib64/libreadline.so.8
#2  0x00007ffff6b13cd1 in rl_read_key () from /lib64/libreadline.so.8
#3  0x00007ffff6af8497 in readline_internal_char () from /lib64/libreadline.so.8
#4  0x00007ffff6b01535 in readline () from /lib64/libreadline.so.8
#5  0x00007ffff6c85cd2 in osdReadline (context=0x444dc0, prompt=0x7ffff6c9c183 "epics> ") at ../osi/os/default/gnuReadline.c:70
#6  epicsReadline (prompt=0x7ffff6c9c183 "epics> ", context=0x444dc0) at ../osi/epicsReadline.c:68
#7  0x00007ffff6c77aea in iocshBody (pathname=<optimized out>, commandLine=0x0, macros=0x0) at ../iocsh/iocsh.cpp:1143
#8  0x000000000040a616 in main (argc=<optimized out>, argv=<optimized out>) at ../mcaDanteAppMain.cpp:20

Thank you in advance
Dariush e Maurizio

Il 16/06/2026 18:18, Mark Rivers ha scritto:

Hi Dariush,

Are you using a Dante1 or a Dante8?

Does this happen every time you start, or just occasionally.  If it is occasionally, then how frequently does it happen?

Are there any error messages on the IOC?

Are you running on Linux or Windows? 

If you are running on Linux then please run the IOC in the GNU debugger. You can do that with the following commands from the iocDante1 directory:

gdb ../../bin/linux-x86_64/mcaDanteApp

run st.cmd

When it crashes type this command at the debugger prompt:

backtrace

Send me the output.

Mark

---

From: Dariush Hampai <dariush.hampai at lnf.infn.it>
Sent: Tuesday, June 16, 2026 10:23 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Problems with Dante (XGLab) Driver

Hi Community,
Hi Mark,

I'm almost finish the implementation of Dante EPICS Drivers in our
system, however I have a big problem (maybe a bug?).
When I start few acquisitions (from caput command or from Phebus), the
system seems that crash.
The Dante:dante2:ElapsedRealTime stops (not on target). Moreover on
Phebus the text is in "Collecting" mode, The Acquire Busy is in
"Acquiring" mode and the IOC do not respond to any command that I send.
Up to now, the only solution is to stop the IOC and restart it.
What's the problem?
Is there a possibility to force a reset/reinitialize the driver without
stop and restart it?

awaiting your (precious) help

Dariush

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

-- 
************************************

Dr. Dariush Hampai, PhD

INFN - LNF
X-Lab Frascati
Via E. Fermi, 54 (ex 40)
I-00044 Frascati (RM)
Italy

Mail Address:
XLab-Frascati
LNF-INFN
Casella Postale 13
Frascati (RM)
Italy

Room: +39.06.9403.5248
Lab.:     +39.06.9403.2286
Mob.: +39.06.9403.8025
Fax.:     +39.06.9403.2597

************************************

Dear colleagues,
I am one of the authors of the paper to which you refer. Thank you for your interest in our work. I would like to offer a few clarifications that I hope may be useful.

None of the authors belong to the DONES consortium. Our positions are purely academic. We have no corporate interest, preference, or external support of any kind. Despite the long tradition of some of the authors in open software and hardware communities, we have endeavored to provide unbiased results while avoiding oversimplification of the many challenges addressed in this contribution.

Our work was developed within the scope of WPENS under the EUROfusion framework and, as such, is limited to research, comparative studies, proposals, feasibility and proof-of-concept tests in areas of possible interest to IFMIF-DONES. These kinds of activities are complex, technically and methodically speaking but valuable discussions to continue the advancement of science and technology. Their outputs can offer nuanced analysis that the engineers responsible for the final design of the DONES control system may or may not find useful, as they see fit.

The context of the comparison is important. Using the control systems of that facility as baseline case, this study focuses particularly on the requirements and needs of the central CODAC system, which in that case comprise services such as monitoring/HMI, alarm and data management, etc. We fully agree that the knowledge gained and experience in this project and in LIPAc are fundamental. This was taken into account, and precisely this is one of the reasons why this study concludes by suggesting a hybrid architecture of control systems, using an industrial SCADA system for the central monitoring and control, while keeping EPICS at the local control level in order to benefit from the best of both worlds. This hybridisation of EPICS with industrial SCADA systems is, in fact, one of our main lines of research, and it is illustrated in some of the publications referenced in the article.

The conclusions were not reached by the authors alone. The evaluation reflects the opinions and experiences of vendors, development companies, and users consulted across a wide range of expertise, including people with long experience in both EPICS and industrial SCADAs. A voting system was used for all criteria to mitigate individual bias. The scores were not binary; they expressed assessments based on availability of options, ease of implementation, maturity of native support, and similar factors. It is also worth noting that most of the scores do not concern EPICS Base itself, but rather other components of the ecosystem and integration considerations.

A few brief technical clarifications in response to specific comments:
-    Regarding OPC-UA: the EPICS Device Support module for OPC-UA is an excellent tool that has, in fact, inspired much of our own work. The criterion in the study assessed whether a framework provides native support for this protocol without requiring a separate bridge, gateway, or custom development. In fact, what the point of the considered rq list really assessed was the possibility of using industrial and market standards in general; OPC UA being one of them.
-    Regarding the IOC restarts comment: this referred to structural or configuration changes, not to property value updates, which we are aware can be handled through the mentioned mechanisms. In any case, this point carries limited weight in the overall conclusions.
-    Most of our laboratory setups are built on EPICS 7 and pvAccess, and we are well aware of the significant progress they represent. The a ssessments in some questions concern not the technical quality of the framework, but the practical implications of maintaining custom developments over time, including challenges related to updates, support, and dependency on specific individuals or software versions. We are aware that customizations are needed in these scientific and particular scopes and, as said, one of our lines of research is directed precisely at integrating such custom solutions with industry standards, to facilitate the adoption of new technologies; key specially when we talk of small-medium facility with limited engineering resources.

Finally, as a personal opinion: the fact that some industrial solutions score higher in certain functional areas does not diminish EPICS. If anything, I think that studies of this kind can help identify areas where new lines of work — on interoperability, maintainability, or total cost— may be worth pursuing. These are dimensions that purely technical comparisons do not always address adequately.

Thank you again for your interest and for the constructive nature of the discussion.

Kind regards,
   Javi.

---------- Forwarded message ---------
De: Diego Omitto via Tech-talk <tech-talk at aps.anl.gov>
Date: mié, 24 jun 2026 a las 20:20
Subject: Re: 2026 white paper on EPICS vs commercial SCADA software
To: Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>

Subject: Re: Comparative SCADA evaluation - IFMIF-DONES paper

Hi Rich,

I read the paper and have a few specific concerns.

I noticed a factual error in the operational critique: the authors claim that changing a variable property in EPICS requires restarting the IOC database. Runtime field modification via caput, dbpf, and autosave/restore is standard practice. This is not a minor point. If the evaluators did not know this, the scoring in Table 2 reflects unfamiliarity with the framework, not its actual capabilities.

This also undermines the paper's characterization of EPICS as a legacy framework. EPICS 7 introduced PVAccess as a first-class protocol with structured data types, high-throughput streaming, and normative types that go well beyond what OPC-UA offers in accelerator contexts. The ecosystem around it is actively developed.

More damaging to the paper's credibility: it cites the ITER OPC-UA/EPICS gatewa y as prior art for the CODAC-MPS integration problem, then penalizes EPICS in the scoring for limited OPC-UA support. The solution they acknowledge exists was excluded from the evaluation.

Finally, the paper does not address the most obvious question: LIPAc, the direct operational prototype of IFMIF-DONES, runs EPICS. I don't see an explanation for why a framework adequate for the prototype is suddenly inadequate for the full facility.

That omission says more about the motivation behind this study than the scoring does.

Best,
Diego

 

On Wed, Jun 24, 2026 at 1:29 PM Evans, Richard K. (GRC-H000) via Tech-talk <tech-talk at aps.anl.gov> wrote:

Dear EPICS community,

I recently came across a new whitepaper that present a comparative evaluation of EPICS against various commercial SCADA platform alternatives. The evaluation uses a set of requirements that they state are representative of the IFMIF-DONES plant currently being constructed in Escúzar, Granada, Spain.  The paper is linked below.  If you have a moment and are inclined, please take a look at Table 2 and Figure 3 and let me know if you feel that EPICS has been rated accurately/fairly.

"A comparative study of industrial and open-source SCADAs to optimize the design of control systems for the IFMIF-DONES plant"
https://urldefense.us/v3/__https://www.researchgate.net/publication/403928088_A_comparative_study_of_industrial_and_open-source_SCADAs_to_optimize_the_design_of_control_systems_for_the_IFMIF-DONES_plant__;!!G_uCfscf7eWS!dLrszc_F4-MNU6rvR6_tBzD-oxGJWgrejzopanOX1p9BNsJxyEPjxFevbkdTX0hdTvbTDWos84abzN6ViuHnARMNORgosr0r$

Note - Figure 3 shows EPICS as seriously deficient as compared to commercial SCADA software, but then I saw what they said on Table 2 requirement #17 wrt video and it doesn't seem like they fully understood EPICS well enough to give it an honest rating. 

The paper is being circulated within my organization as strong evidence that EPICS is a poor option as a control system SCADA for a facility.  Any insights this community can provide in explaining the possible motivations of  the authors and/or highlighting any clear bias or incorrect assessments that the paper may contain would be much appreciated.

Thanks in advance.
- Rich Evans
  NASA Armstrong Test Facility
  Sandusky, Ohio

 

 

Hi Sue,

I am presently working on this project.

I have a working IOC that is based around G-P's Windows-only SDK.

I firmly intend to provide a cross-platform solution; this is dependent on the company enabling me to do so by means on an NDA for their existing C# source code.
I have an ongoing negotiation with them, on this question. 

I refer also to http://www.aps.anl.gov/epics/tech-talk/2016/msg00794.php.

Hope this helps,

Peter.

-----Original Message-----
From: tech-talk-bounces at aps.anl.gov [mailto:tech-talk-bounces at aps.anl.gov] On Behalf Of Sue Witherspoon
Sent: 13 October 2016 17:00
To: Tech-talk
Subject: Granville-Phillips 830 Vacuum Quality Monitor

Greetings,

One of our experiments has chosen to use the Granville-Phillips 830 VQM for monitoring the scattering chamber vacuum. Has anyone developed an EPICS driver for this device. It appears to come with a USB 2.0 windows UI. But maybe, someone has a linux solution?


--
Sue Witherspoon
Staff Computer Scientist
Accelerator Controls Software
757-269-7579
Thomas Jefferson National Accelerator Facility

• Add rules support - all widgets now can have properties dynamically changed;
• New widgets: GraphXY, Histogram, Spinner;
• Add alarm summary panel with severity and PV names filtering;
• A standalone web Alarm Handling service with a panel, latching alarms and acknowledgment is underway;
• Include Display Format property for all text and value monitoring widgets (string, hex, engineering, scientific or timestamp);
• Add PV snapshot save/restore filterable by OPI;
• Standalone S&R service is planned for next releases;
• Major performance upgrade on React rendering layer;
• Update documentation - add tutorial page with animations and update contents, see it here: https://docs.weiss-controls.org/getting_started/tutorial.html
• Ensure support for Podman build through CI pipeline;
• Add "Distribute horizontally", "Distribute vertically", "Match width" and "Match height" functions to Toolbar;
• Several other bug fixes and UX improvements (see release notes).