EPICS/RTEMS tracing and scheduler monitoring

[Ricardo Cardenes <rcardenes@gemini.edu>]

Hi all,

Trying to get some insight on some weird behavior that we've seen in one of our systems, I looked around for some tools that could help us better understand the low-level system behavior, down to the task scheduler.

[Context note: we're using RTEMS 4.10.2 on EPICS 3.14.12.x]

On the RTEMS side I've found the Capture Engine (but this I found too late, and I was already working on my own extension - I may migrate my code to make use of CAPE at a later point), but when it comes to the EPICS level, I haven't been able to find anything similar. For example, I haven't found a way to figure out what record on a periodic scan list is being processed at an arbitrary point in time, as all the info seems to be private to db/dbScan.c. Correct me if I'm wrong (I'm quite a beginner when it comes to EPICS), but EPICS does not seem to be instrumented for this kind of debugging, not on version 3.x, nor on version 7.x. Please feel free to point me to the right documentation/examples otherwise.

I've started working on a support module to extract this kind of information[1] (don't be fooled by its name, it's a tracing tool - the module started as something else I still haven't come to rename it), and I have a few questions:

• Is there a tool out there doing what I'm trying to achieve? As I said, I haven't found one, but my Google-fu may have failed me. I'd rather use/augment something else instead of reinventing the wheel. Assuming that I'm on my own:
• What would be the preferred way to send opaque data in bulk? (arrays of struct, in this case). I decided to take advantage of CA and so far I'm using an customized aSub, that among other things is offering through some of its outputs a number of arrays of LONG (to send over 16kB in one go without changing the maximum message size), but I wonder if I shouldn't be using asyn or something similar.
• If I were to make the module more EPICS-aware, I'd need access to internals. I can get around this using the RTEMS Trace Linker to selectively instrument certain function calls, but maybe there's interest out of our group and I should just add callback points to the EPICS code and contribute them?

[1] https://github.com/gemini-rtsw/rtemsStats

Regards,

Ricardo