Hi Matt,
Thank you for these numbers. I do not even dare to publicly state what kind of data volumes some of our (accelerator) users are asking for… I think we need to seriously consider binning.
Cheers,
Timo
From:
"Pearson, Matthew R." <pearsonmr at ornl.gov>
Date: Monday, 1 November 2021 at 22:27
To: Mark L Rivers <rivers at cars.uchicago.edu>, Timo Korhonen <Timo.Korhonen at ess.eu>, Ralph Lange <ralph.lange at gmx.de>
Cc: EPICS Core Talk <core-talk at aps.anl.gov>
Subject: RE: [EXTERNAL] CA gateway chaining
Hi Timo,
Yes, what Mark said.
For example, a typical machine vision camera or astronomy CCD might be 2048*2048*2 bytes = 8.4MB. So I might choose to use 2x2 or 4x4 binning on this, via the areaDetector ROI plugin, to get down to either 2MB
or 0.5MB (depending on how large the image widget will be on the OPI). And, in addition, I could use the processing plugin to scale to 8-bit greyscale data, so now we get down to 1MB or 0.25MB.
In practice, a monitor will only display up to 8-bit greyscale anyway, and humans can’t see a full 16-bit range of greyscale either.
Similarly, our neutron time-of-flight 1-D spectrums are multiples of 160,000 * 4 bytes = 0.64MB. So these are binned by a factor 10 to get to 64KB.
Cheers,
Matt
From: Mark L Rivers <rivers at cars.uchicago.edu>
Sent: Monday, November 1, 2021 5:06 PM
To: Timo Korhonen <Timo.Korhonen at ess.eu>; Pearson, Matthew R. <pearsonmr at ornl.gov>; Ralph Lange <ralph.lange at gmx.de>
Subject: RE: [EXTERNAL] CA gateway chaining
Hi Timo,
If your arrays are coming from areaDetector then binning can easily be done in the IOC using NDPluginROI.
If the arrays are images then you may also be able to use NDPluginCodec to compress them, for example with the JPEG compressor. areaDetector
provides Linux and Windows decompressors that can be called from your PVA client. Those are used from the ImageJ plugin, for example.
Mark
Do you do the binning in the IOC? This is something we have thought of but not implemented yet.
Timo
Hi,
We have seen similar issues and one way we dealt with it was to provide a heavily re-binned array for use via gateways. Inside the beamline network we provide both a full un-binned array, hidden behind ‘detailed’
buttons, as well as the re-binned array screens that users will first encounter on higher level screens. We also limited the array size that the gateway will transmit, so some of the larger arrays don’t work which forces people to view the re-binned arrays.
Also, depending on the original data type and software, you can ‘compress’ the array data from 16 or 32-bit down to 8-bit data by re-scaling it to 0-255, like a JPEG image. Then the waveform data type can be
set to UCHAR.
The above only works if the waveforms are simply for visualization.
Cheers,
Matt
Right, this is the way we are going for now. It is not exactly trivial to manage but should be doable.
Thanks,
TImo
Just a short follow-up.
We found out that the delays are due to large arrays being monitored over the gateway. Not a surprise, even if we have tried to remind users about the effects with big arrays.
One way to mitigate this - I think this approach was implemented at some point at the SLS - is to route the large array channels through a dedicated separate gateway instance.
(Ease of configuration depends on your naming convention.)