Re: [Ext] RE: ADEiger Stream stop zmq?

[Jesse Hopkins via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

Yup, that fixes the disk free error.

All the best.

- Jesse

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source

On Fri, Dec 10, 2021 at 11:44 AM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Ø  I tried the fix_disk_free, and while it's not returning a negative number anymore, I think it's still not correct.

 

I understand the problem.  This is the record definition:

 

# FileWriter free buffer space

record(ai, "$(P)$(R)FWFree_RBV")

{

    field(DTYP, "asynInt32")

    field(INP,  "@asyn($(PORT),$(ADDR),$(TIMEOUT))FW_FREE")

    field(DESC, "Free disk space")

    field(EGU,  "GB")

    field(PREC, "3")

    field(ASLO, "9.5367431640625e-07")

    field(SCAN, "I/O Intr")

}

 

Note that the ASLO (adjustment slope) is 9.54e-7 which is 1/(1024*1024).  So that will convert to MB, but the EGU on the record is GB, so it is inconsistent, at least for the 1.8 API.

 

Either this is a mistake, or perhaps1.6 API actually returns KB over the RestAPI, rather than bytes?

 

I have changed ASLO to 1e-9 so the units will be the base 10 version of GB.  I have merged the fix_disk_free branch into fix_stream, so now there is only one test branch.  Please try fix_stream again and see if the GB are correct.

 

I will think some more about how to get the behavior you want with zmq.

 

Mark

 

 

From: Jesse Hopkins <jhopkins1 at iit.edu>
Sent: Friday, December 10, 2021 11:01 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: [Ext] RE: ADEiger Stream stop zmq?

 

Hi Mark,

 

Thanks for making the bug reports and the test fixs.

 

I tried the fix_disk_free, and while it's not returning a negative number anymore, I think it's still not correct. At the moment the detector API returns:

{"access_mode":"r","unit":"B","value":84988747776,"value_type":"uint"}

 

If I convert bytes to GB (the unit in caQtDM) I get ~84.98 GB (using the conventional 1000^3 bytes, or ~79.15 if you use 1024^3 bytes). The caQtDM window shows 81367.617 GB, so off by about three orders of magnitude.

 

I also tried the fix_stream branch. It kind of works. If I set StreamEnable to 0, I can indeed get all the expected messages on my own zmq client. However, the issue is that I have to go into the detector API and manually re-enable the stream, since StreamEnable is setting the stream api config/mode to disabled. Ideally what I'd like to be able to do is just disable the zmq client in AreaDetector, without actually turning off the stream on the detector computer. E.g. instead of changing StreamEnable, if I change the AreaDetector Data Source from Stream to None or FileWriter it turns off the zmq client but leaves the steam on the detector API enabled.  As it is, I could work with it, but I'd have to add something into my zmq client that watched the stream status on the detector and changed it to enabled if it became disabled, such as if I restart the IOC and autosave resets that value.

All the best.

- Jesse

 

P.S. Resending to include tech-talk.

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source

 

 

On Fri, Dec 10, 2021 at 8:28 AM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Jesse,

 

I have created a fix_stream branch on Github which I hope will fix the issue you are having with the zmq interface.  The zmq socket will now be created when StreamEnable is set to 1, and deleted when StreamEnable is set to 0.  It would be good if you can test the following:

-          If you do not enable the Stream interface then your zmq client should receive all of the messages

-          If you do enable the Stream interface then ADEiger should get the images.  You can check with NDPluginStats or other plugins.

-          If the Stream interface is disabled and you try to change DataSource to Stream then you should get an error message and it should not enable the Stream interface, which it previously would have done.

 

Thanks,

Mark

 

From: Mark Rivers
Sent: Thursday, December 9, 2021 5:05 PM
To: Jesse Hopkins <jhopkins1 at iit.edu>
Cc: tech-talk at aps.anl.gov
Subject: RE: ADEiger Stream stop zmq?

 

Hi Jesse,

 

I suspect you have already found an issue I was afraid of when I fixed the race condition in ADEiger R3-1.  Previously the zmq socket in ADEiger was only created when acquisition was started and the Stream interface was enabled.  That had a race condition, because the socket was only created after the Eiger might start sending data over the zmq interface.  The fix was to create the zmq socket in the thread that reads the zmq socket before acquisition starts, and whether or not the Stream interface is enabled.  We thought that would be OK, but I think it could be causing your problem with your client only getting every other zmq message.

 

Can you try rolling back to ADEiger R3-0 and see if the problem goes away?  If so then I will need to change the code so it only opens the zmq socket when the Stream interface is enabled, and closes it when it is disabled.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Jesse Hopkins via Tech-talk
Sent: Thursday, December 9, 2021 3:32 PM
To: EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: ADEiger Stream stop zmq?

 

Hi folks,

 

We recently got an Eiger2 XE 9M and I've been using AreaDetector to control it. So far, if I'm just using AreaDetector by itself everything works great (thanks for that!). In the long run what I would like to do is use AreaDetector to control the exposures, but use my own program to read the Eiger Stream over zmq and process the data in real time. According to the documentation this should be possible.

 

However, when I run the Eiger from AreaDetector with DataSource set to None, as specified in the documentation, my zmq client only receives half of the messages, which is what I would expect with two PULL clients connected to a single PUSH client. For example, if I take two images, I see only frame 0 and the dseries-end messages but not the dheader or frame 1 messages. If I collect images under the same conditions without the IOC running (e.g. just stop the IOC and send an arm and trigger command directly to the api with all settings left the same) I get all of the expected messages.

 

I'm wondering if there's some setting I've missed (see attached screenshot of the caQtDM window) or if this is a bug.

 

Also, as a side note, you'll see in the caQtDM window that the DCU RAM Disk Free is reporting a very wrong number, I suspect it's an overflow error. If I query buffer_free (which I assume is that that is), I get the following:

{"access_mode":"r","unit":"B","value":85402976256,"value_type":"uint"}

 

I'm using:

ADCore: 3.11.0

ADEiger: 3.1.0

SIMPLON API: 1.8.0

Eiger firmware: release-2020.2.4

EPICS: 7.0.4.1

And the IOC is running on Debian 11

 

All the best.

- Jesse

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source