RE: [Ext] RE: ADEiger Stream stop zmq?

[Mark Rivers via Tech-talk <tech-talk at aps.anl.gov>]

Just to confirm, that was working when FWCompression was set to Enable and CompressionAlgo was set to “BS LZ4”?

 

Mark

 

 

From: Jesse Hopkins <jhopkins1 at iit.edu>
Sent: Tuesday, December 14, 2021 11:38 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: [Ext] RE: ADEiger Stream stop zmq?

 

Hi Mark,

 

The fix_filewriter branch seems to work as expected. With data source set to filewriter I can collect data without getting the error, and I was able to get statistics (I just turned on stats 1 to check), so it seems to be reading the image.

All the best.

- Jesse

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source

 

 

On Mon, Dec 13, 2021 at 7:08 PM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Jesse,

 

Ø  I just tested it, and it seems to perform as expected and desired! If the data source is set to stream, messages end up split between my client and AreaDetector. If data source is set to None, my client gets all the messages. Thanks for that!

 

Great, thanks for testing.  I have merged fix_stream into master.

 

Ø  Next question, when I set the datasource to FileWriter I get an error message which says that "Driver can't decode BSLZ4 HDF5 files". I think ADSupport built successfully and I've set the HDF5_PLUGIN_PATH, but maybe I've missed something?

 

That error message reflects a previous limitation of the FileWriter interface which should no longer exist providing you have set HD5_PLUGIN_PATH appropriately.

 

I have commented out that test and pushed to the new fix_filewriter branch on Github.  Please see if it will now read HDF5 files with BSLZ4 compression.

 

Mark

 

 

 

From: Jesse Hopkins <jhopkins1 at iit.edu>
Sent: Monday, December 13, 2021 6:18 PM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: [Ext] RE: ADEiger Stream stop zmq?

 

Hi Mark,

 

I just tested it, and it seems to perform as expected and desired! If the data source is set to stream, messages end up split between my client and AreaDetector. If data source is set to None, my client gets all the messages. Thanks for that!

 

Next question, when I set the datasource to FileWriter I get an error message which says that "Driver can't decode BSLZ4 HDF5 files". I think ADSupport built successfully and I've set the HDF5_PLUGIN_PATH, but maybe I've missed something?

All the best.

- Jesse

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source

 

 

On Sat, Dec 11, 2021 at 1:38 PM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Jesse,

 

I have changed the logic so I think it does what you want, and is more backwards compatible with the master branch than my previous attempt.

 

The logic is now:

-          When DataSource is set to Stream it creates the StreamAPI object which create the zmq socket.

-          When DataSource is set to anything except Stream is deletes the StreamAPI object, which deletes the zmq socket.

-          The behavior of StreamEnable is unchanged from the master branch, it can be enabled or disabled without affecting the zmq socket.

 

Please give it a try when you get a chance.

 

Mark

 

 

 

From: Jesse Hopkins <jhopkins1 at iit.edu>
Sent: Friday, December 10, 2021 11:01 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: tech-talk at aps.anl.gov
Subject: Re: [Ext] RE: ADEiger Stream stop zmq?

 

Hi Mark,

 

Thanks for making the bug reports and the test fixs.

 

I tried the fix_disk_free, and while it's not returning a negative number anymore, I think it's still not correct. At the moment the detector API returns:

{"access_mode":"r","unit":"B","value":84988747776,"value_type":"uint"}

 

If I convert bytes to GB (the unit in caQtDM) I get ~84.98 GB (using the conventional 1000^3 bytes, or ~79.15 if you use 1024^3 bytes). The caQtDM window shows 81367.617 GB, so off by about three orders of magnitude.

 

I also tried the fix_stream branch. It kind of works. If I set StreamEnable to 0, I can indeed get all the expected messages on my own zmq client. However, the issue is that I have to go into the detector API and manually re-enable the stream, since StreamEnable is setting the stream api config/mode to disabled. Ideally what I'd like to be able to do is just disable the zmq client in AreaDetector, without actually turning off the stream on the detector computer. E.g. instead of changing StreamEnable, if I change the AreaDetector Data Source from Stream to None or FileWriter it turns off the zmq client but leaves the steam on the detector API enabled.  As it is, I could work with it, but I'd have to add something into my zmq client that watched the stream status on the detector and changed it to enabled if it became disabled, such as if I restart the IOC and autosave resets that value.

All the best.

- Jesse

 

P.S. Resending to include tech-talk.

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source

 

 

On Fri, Dec 10, 2021 at 8:28 AM Mark Rivers <rivers at cars.uchicago.edu> wrote:

Hi Jesse,

 

I have created a fix_stream branch on Github which I hope will fix the issue you are having with the zmq interface.  The zmq socket will now be created when StreamEnable is set to 1, and deleted when StreamEnable is set to 0.  It would be good if you can test the following:

-          If you do not enable the Stream interface then your zmq client should receive all of the messages

-          If you do enable the Stream interface then ADEiger should get the images.  You can check with NDPluginStats or other plugins.

-          If the Stream interface is disabled and you try to change DataSource to Stream then you should get an error message and it should not enable the Stream interface, which it previously would have done.

 

Thanks,

Mark

 

From: Mark Rivers
Sent: Thursday, December 9, 2021 5:05 PM
To: Jesse Hopkins <jhopkins1 at iit.edu>
Cc: tech-talk at aps.anl.gov
Subject: RE: ADEiger Stream stop zmq?

 

Hi Jesse,

 

I suspect you have already found an issue I was afraid of when I fixed the race condition in ADEiger R3-1.  Previously the zmq socket in ADEiger was only created when acquisition was started and the Stream interface was enabled.  That had a race condition, because the socket was only created after the Eiger might start sending data over the zmq interface.  The fix was to create the zmq socket in the thread that reads the zmq socket before acquisition starts, and whether or not the Stream interface is enabled.  We thought that would be OK, but I think it could be causing your problem with your client only getting every other zmq message.

 

Can you try rolling back to ADEiger R3-0 and see if the problem goes away?  If so then I will need to change the code so it only opens the zmq socket when the Stream interface is enabled, and closes it when it is disabled.

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Jesse Hopkins via Tech-talk
Sent: Thursday, December 9, 2021 3:32 PM
To: EPICS Tech Talk <tech-talk at aps.anl.gov>
Subject: ADEiger Stream stop zmq?

 

Hi folks,

 

We recently got an Eiger2 XE 9M and I've been using AreaDetector to control it. So far, if I'm just using AreaDetector by itself everything works great (thanks for that!). In the long run what I would like to do is use AreaDetector to control the exposures, but use my own program to read the Eiger Stream over zmq and process the data in real time. According to the documentation this should be possible.

 

However, when I run the Eiger from AreaDetector with DataSource set to None, as specified in the documentation, my zmq client only receives half of the messages, which is what I would expect with two PULL clients connected to a single PUSH client. For example, if I take two images, I see only frame 0 and the dseries-end messages but not the dheader or frame 1 messages. If I collect images under the same conditions without the IOC running (e.g. just stop the IOC and send an arm and trigger command directly to the api with all settings left the same) I get all of the expected messages.

 

I'm wondering if there's some setting I've missed (see attached screenshot of the caQtDM window) or if this is a bug.

 

Also, as a side note, you'll see in the caQtDM window that the DCU RAM Disk Free is reporting a very wrong number, I suspect it's an overflow error. If I query buffer_free (which I assume is that that is), I get the following:

{"access_mode":"r","unit":"B","value":85402976256,"value_type":"uint"}

 

I'm using:

ADCore: 3.11.0

ADEiger: 3.1.0

SIMPLON API: 1.8.0

Eiger firmware: release-2020.2.4

EPICS: 7.0.4.1

And the IOC is running on Debian 11

 

All the best.

- Jesse

 

----
Jesse Hopkins, PhD
Deputy Director
BioCAT, Sector 18
Advanced Photon Source