Re: Asyn + Streamdevice: Gamma Vacuum Digtial QPC, missmatch after some time

["Dr. Simon Friederich via Tech-talk" <tech-talk at aps.anl.gov>]

Dear Mark and Ralph,

Thanks for your replies!

I got response from Dirk concerning StreamDevice (Thanks Dirk!). StreamDevice sends the errlogs by default to stderr and he thinks that for VxWorks(*) this is also sent to errlog.

He checked, but unfortunately it seems to require a lot of coding to change the behaviour so that StreamDevice also sends its logs via the file descriptor of errlog.

Maybe I can store the StreamDevice errors in a file which I can read with my filebeat-elasticsearch-setup, yet, I am not very familiar with this.

Best regards
Simon

(*: We don't use VxWorks)

Am 14.10.2024 um 15:49 schrieb Mark Rivers:

Hi Simon,

 

This is the low-level code where asyn prints error messages.

 

https://github.com/epics-modules/asyn/blob/9519475a97613bacc55af9368fcfbe44979c8236/asyn/asynDriver/asynManager.c#L2843

 

If a file has been specified (asynSetTraceFile) it calls fprintf to that file, else it calls errLogPrintf.  So those message should go to your logger.

 

I am not sure whether StreamDevice calls errLogPrintf by default, or if it can be made to do so.

 

Mark

 

 

From: Dr. Simon Friederich <s.friederich at uni-mainz.de>
Sent: Monday, October 14, 2024 8:31 AM
To: Mark Rivers <rivers at cars.uchicago.edu>; tech-talk at aps.anl.gov
Subject: Re: Asyn + Streamdevice: Gamma Vacuum Digtial QPC, missmatch after some time

 

Dear Mark,

Thank you for your fast response. I have set up a log server based on elasticsearch, see Nikos answer, https://epics.anl.gov/tech-talk/2024/msg00160.php.

Unfortunately I can't see the messages in my iocLog:

Am I missing something in my IOC configuration?
My st.cmd looks like this:

    #!../../bin/linux-x86_64/docker
    < envPaths
    epicsEnvSet ("STREAM_PROTOCOL_PATH", "${TOP}/db")
    epicsEnvSet("EPICS_IOC_LOG_PORT", "1514")
    epicsEnvSet("EPICS_IOC_LOG_INET", "10.32.200.15")
    cd "${TOP}"
    ...
    iocInit
    iocLogPrefix "[${HOSTNAME}] "
    iocLogInit
    errlog "starting ${IOC}"
    caPutLogInit 10.32.200.15:1515 0

Best regards
Simon

Am 14.10.2024 um 14:58 schrieb Mark Rivers:

Hi Simon,

 

1. Are these error messages from ASYN or from IOC itself?

 

They are actually from StreamDevice.

 

2. Can I retrieve these errors also in my iocLog?

 

Yes, they should be there.

 

When I have problems like this the first thing I do is enable asynTrace to see all of the messages between the IOC and the device. 

 

asynSetTraceIOMask($(PORT), 0, ESCAPE)

asynSetTraceMask($(PORT), 0, ERROR|DRIVER)

 

Change $(PORT) to the name of your underlying asyn port.

 

That way you can if what Ralph suggested is happening, or if it is garbled messages, etc.  There will be a lot of messages on the console.  You can send them to a file instead with this command:

 

asynSetTraceFile($(PORT), 0, “filename”)

 

The following command can also be useful for devices that get out of sync.

 

asynSetOption($(PORT), 0, "disconnectOnReadTimeout", "Y")

 

That causes asyn to close the socket when there is a read timeout.  It will attempt to re-open the socket the next time an operation is queued.  This only works for socket connections, not serial ports.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Dr. Simon Friederich via Tech-talk
Sent: Monday, October 14, 2024 5:33 AM
To: tech-talk at aps.anl.gov
Subject: Re: Asyn + Streamdevice: Gamma Vacuum Digtial QPC, missmatch after some time

 

Maybe one follow-up question:

Are these error messages from ASYN or from IOC itself?

Can I retrieve these errors also in my iocLog?

Best regards
Simon

Am 14.10.2024 um 12:19 schrieb Dr. Simon Friederich:

Dear Ralph and Marco,

Your argumentation sounds reasonable. Thank you for the advice, I'll give Marco's replytimeout a try :-)

(regardless of whether this solution works for me, this sounds like something that could be collected on a tips & tricks subpage of https://epics-controls.org/ :-))

Best regards,
Simon

Am 14.10.2024 um 11:30 schrieb Dr. Simon Friederich:

Dear EPICS-community,

I am struggeling with our implementation of the IGP controller from Gamma Vacuum Digital QPC that we use.
Out of the sudden or at a certain point, which I can neither explain nor reproduce, the IOC complains that the controllers' responses to the protocol no longer fit.

Here is an example output that I've retrieved in the morning, before that the IOC ran for weeks without error:

2024/10/08 08:15:41.970931 igpctrlmodh2071_port melba060:pump:i_get_raw: No reply within 2000 ms to "cmd 0A 1<0d>"
2024/10/08 08:15:43.974231 igpctrlmodh2071_port melba060:pump:p_get_raw: No reply within 2000 ms to "cmd 0B 1<0d>"
2024/10/08 08:15:45.977001 igpctrlmodh2071_port melba060:pump:u_get: No reply within 2000 ms to "cmd 0C 1<0d>"
2024/10/08 08:15:46.722205 igpctrlmodh2071_port melba040:diffp_down:on_get: Input "1.3E-11 MBAR" does not match format "%{NO|YES}"
2024/10/08 08:15:46.842189 igpctrlmodh2071_port melba060:pump:stat_get: Input "13" does not match format "%{STANDBY|RUNNING 00|SAFE-CONN 00|INTERLOCK 00|PUMP ERROR 00|CALIBRATION 00}"
2024/10/08 08:15:46.967906 igpctrlmodh2071_port melba040:diffp_down:stat_get: Input "18" does not match format "%{STANDBY|RUNNING 00|SAFE-CONN 00|INTERLOCK 00|PUMP ERROR 00|CALIBRATION 00}"
2024/10/08 08:15:47.087939 igpctrlmodh2071_port melba040:diffp_down:i_get_raw: Input "NO" does not match format "%e"
2024/10/08 08:15:47.207884 igpctrlmodh2071_port melba040:diffp_down:p_get_raw: Input "STANDBY" does not match format "%e"
2024/10/08 08:15:47.327939 igpctrlmodh2071_port melba060:pump:on_get: Input "PUMP ERROR 00" does not match format "%{NO|YES}"
2024/10/08 08:15:47.693674 igpctrlmodh2071_port melba060:pump:u_get: Input "NO" does not match format "%i"
2024/10/08 08:15:48.059375 igpctrlmodh2071_port melba040:diffp_down:i_get_raw: Input "OK 00 14" too short.
2024/10/08 08:15:48.059449 igpctrlmodh2071_port No match for " AMPS"
2024/10/08 08:15:48.179663 igpctrlmodh2071_port melba040:diffp_down:p_get_raw: Input "OK 00 7.5E-11 AMPS"
2024/10/08 08:15:48.179734 igpctrlmodh2071_port melba040:diffp_down:p_get_raw: mismatch after 14 bytes "...0 7.5E-11 "
2024/10/08 08:15:48.179764 igpctrlmodh2071_port melba040:diffp_down:p_get_raw: got "AMPS" where "MBAR" was expected
2024/10/08 08:15:48.299339 igpctrlmodh2071_port melba060:pump:on_get: Input "1.3E-11 MBAR" does not match format "%{NO|YES}"
2024/10/08 08:15:48.419330 igpctrlmodh2071_port melba060:pump:i_get_raw: Input "NO" does not match format "%e"
2024/10/08 08:15:48.545033 igpctrlmodh2071_port melba060:pump:p_get_raw: Input "OK 00 1.9E-10 AMPS"
2024/10/08 08:15:48.545094 igpctrlmodh2071_port melba060:pump:p_get_raw: mismatch after 14 bytes "...0 1.9E-10 "
2024/10/08 08:15:48.545132 igpctrlmodh2071_port melba060:pump:p_get_raw: got "AMPS" where "MBAR" was expected
2024/10/08 08:16:35.567653 igpctrlmodh2071_port melba040:diffp_down:p_get_raw: No reply within 2000 ms to "cmd 0B 3<0d>"
2024/10/08 08:16:36.821755 igpctrlmodh2071_port melba060:pump:on_get: Input "1.3E-11 MBAR" does not match format "%{NO|YES}"

I attach the proto and database files we've written. Maybe you can spot any obvious error or have any advice for me to further debug this issue.

Best regards,
Simon

-- 

Dr. Simon Friederich

Institut für Kernphysik

Kollaboration B | MESA

Johannes Gutenberg-Universität Mainz

Johann-Joachim-Becher-Weg 45

D 55128 Mainz

Tel.: +49 (0)6131 39-23160

E-Mail: s.friederich at uni-mainz.de