Re: asyn IMAX data transfer exceeding EPICS_CA_MAX_ARRAY_BYTES

["Marco A. Barra Montevechi Filho via Tech-talk" <tech-talk at aps.anl.gov>]

The output is the following:

epics> epicsPrtEnvParams

EPICS_CA_ADDR_LIST: 10.0.38.46:60000 10.0.38.59:60000 10.10.10.76 s-mgn-vtx01-l 10.20.46.17 10.20.46.18 10.32.4.27 s-mgn-a-rio01 s-mgn-a-pb02 mgn-fanless1  s-mgn-a-rio02

EPICS_CA_CONN_TMO: 30.0

EPICS_CA_AUTO_ADDR_LIST: YES

EPICS_CA_REPEATER_PORT: 5065

EPICS_CA_SERVER_PORT: 5064

EPICS_CA_MAX_ARRAY_BYTES: 15728645

EPICS_CA_MAX_SEARCH_PERIOD: 300.0

EPICS_CA_NAME_SERVERS is undefined

EPICS_CAS_INTF_ADDR_LIST is undefined

EPICS_CAS_IGNORE_ADDR_LIST is undefined

EPICS_CAS_AUTO_BEACON_ADDR_LIST is undefined

EPICS_CAS_BEACON_ADDR_LIST is undefined

EPICS_CAS_SERVER_PORT is undefined

EPICS_CA_BEACON_PERIOD: 15.0

EPICS_CAS_BEACON_PERIOD is undefined

EPICS_CAS_BEACON_PORT is undefined

EPICS_BUILD_COMPILER_CLASS: gcc

EPICS_BUILD_OS_CLASS: Linux

EPICS_BUILD_TARGET_ARCH: linux-x86_64

EPICS_TIMEZONE: CUS::360:031102:110402

EPICS_TS_NTP_INET is undefined

EPICS_IOC_LOG_PORT: 7004

EPICS_IOC_LOG_INET is undefined

EPICS_IOC_LOG_FILE_LIMIT: 1000000

EPICS_IOC_LOG_FILE_NAME is undefined

EPICS_IOC_LOG_FILE_COMMAND is undefined

EPICS_CMD_PROTO_PORT is undefined

EPICS_AR_PORT: 7002

IOCSH_PS1: epics>

IOCSH_HISTSIZE: 50

So i suppose the environment variable is set correctly(?)
I will investigate it with gdb then and post the results when found. Thanks for the suggestion!

Marco

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: 16 March 2022 15:11
To: Marco A. Barra Montevechi Filho <marco.filho at lnls.br>
Cc: EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: Re: asyn IMAX data transfer exceeding EPICS_CA_MAX_ARRAY_BYTES

 

Please send the output of the following command at the IOC prompt:

epicsPrtEnvParams

That will show if the environment variable is set correctly in the IOC.

The seg fault is definitely happening in the IOC, not the equipment controller. You can use gdb to get the traceback from the seg fault to determine what line of code it is failing on.

Mark

Sent from my iPhone

On Mar 16, 2022, at 8:06 AM, Marco A. Barra Montevechi Filho <marco.filho at lnls.br> wrote:



Hello, Mark! Good morning.

Unfortunately it didnt. Im not sure if i understood the question correctly, but if i did, the answer is the following:

The IOC runs in a virtual machine (host) and is accessed by another virtual machine (client) which is configured via EPICS_CA_ADDR_LIST to search in the host's IP. This design works well here in LNLS, so it should not be the problem. EPICS_CA_MAX_ARRAY_BYTES is defined both in the host and in the client and set to the same values.
We also tried running caget in a jupyter notebook which also had EPICS_CA_MAX_ARRAY_BYTES defined on it as an environment variable. Neither strategy (runnning caget on the machine or in the notebook) worked.

For some reason, the error which i sent here earlier ("data transfer bigger than available memory or EPICS_CA_MAX_ARRAY_BYTES") is not showing up again. Instead, we get a direct segmentation fault that kills the IOC. Thus, i raise the following doubts:

-Does it makes sense to think of an IOC segmentation fault? Or is it just reporting a segmentation fault that actually happened inside the equipment controller and terminating for some reason? Does it makes sense to think that the data transfer is bigger than whats set in IMAX or in EPICS_CA_MAX_ARRAY_BYTES and so the IOC sigfaults and dies?
-If the answer to the first question is no, then does it makes sense to think of the IOC just propagating the segmentation fault message that comes from the equipment controller?
- if the answer to both previous questions is no, then should i only believe that sigfault comes from machine memory error?
-Is asyn prepared to deal with IMAX as big as 524288? (im not sure if this last question matters, though, since requesting large arrays of data with IMAX set to 256 also resulted in error sometimes).

If it helps, the field in which we are trying to caget is .BINP. .IFMT is configured to hybrid.

Thanks for your attention!

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: 16 March 2022 14:11
To: Siddons, David <siddons at bnl.gov>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>; Marco A. Barra Montevechi Filho <marco.filho at lnls.br>
Subject: Re: asyn IMAX data transfer exceeding EPICS_CA_MAX_ARRAY_BYTES

 

Did Pete’s suggestion fix the problem?

Sent from my iPhone

On Mar 15, 2022, at 9:42 AM, Siddons, David via Tech-talk <tech-talk at aps.anl.gov> wrote:



Hi Marco,

   Is EPICS_CA_MAX_ARRAY_BYTES set on both ends of the link?

Pete.

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Marco A. Barra Montevechi Filho via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, March 15, 2022 3:39 PM
To: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: asyn IMAX data transfer exceeding EPICS_CA_MAX_ARRAY_BYTES

 

Hello all

Im debugging an EPICS asyn IOC which was working fine until now. The st.cmd of the IOC has a line which loaded asynRecord.db:

dbLoadRecords("$(ASYN)/db/asynRecord.db","P=$(DEV),R=asynEth,PORT=E873_ETH,ADDR=0,OMAX=256,IMAX=256")

and caput caget BL:H:EQ:asynEth.IFMT 1 (Hybrid)

caget BL:H:EQ:asynEth.BINP

in the IOC machine returned an array of numbers without any problem. However, it is recovering an array from a file much alrger than 256, so i suppose it just threw parts of the file away.
We wanted to get more of the file, so we raised IMAX to 350000 and defined the env variable EPICS_CA_MAX_ARRAY_BYTES=15728645. SO we got the error

caget MGN:A:E873:DIAG:A:asynEth.BINP

CA.Client.Exception...............................................

    Warning: "The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES"

    Context: "op=0, channel=MGN:A:E873:DIAG:A:asynEth.BINP, type=DBR_TIME_CHAR, count=16406, ctx="server unable to load read (or subscription update) response into protocol buffer PV="MGN:A:E873:DIAG:A:asynEth" max bytes=16408""

    Source File: ../getCopy.cpp line 92

    Current Time: Tue Mar 15 2022 16:32:11.936961493

..................................................................

Read operation timed out: some PV data was not read.

MGN:A:E873:DIAG:A:asynEth.BINP 350000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  (the zeroes continue)

So we tried lowering IMAX to 16406 but still the error persisted:

CA.Client.Exception...............................................

    Warning: "The requested data transfer is greater than available memory or EPICS_CA_MAX_ARRAY_BYTES"

    Context: "op=0, channel=MGN:A:E873:DIAG:A:asynEth.BINP, type=DBR_TIME_CHAR, count=16406, ctx="server unable to load read (or subscription update) response into protocol buffer PV="MGN:A:E873:DIAG:A:asynEth" max bytes=16408""

    Source File: ../getCopy.cpp line 92

    Current Time: Tue Mar 15 2022 16:32:11.936961493

..................................................................

Read operation timed out: some PV data was not read.

MGN:A:E873:DIAG:A:asynEth.BINP 16406 0 0 0 0 0 0

does anyone know if we are doing something wrong or forgetting something? I dont think memory should be the problem here... Where do these "max bytes=16408" come from?

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.