Re: Question regarding ADPluginFile driver

[Jörn Dreyer via Tech-talk <tech-talk at aps.anl.gov>]

Hi Mark,

thanks for the detailed answer. I am not using NDPluginPVA. Due to an in-house request I had to use openPMD (see http//www.openpmd.org or
https://github.com/openPMD/openPMD-api ) for that. It uses internally ADIOS2 (https://csmd.ornl.gov/software/adios2) as storage back-end.
And additionally can store the data in XML, YAML and toml format. We want to use the sst back-end to transfer the data over a 10GB link to our HPC cluster for analysis.
I will take a look at the NDPluginFile and try to implement the changes you suggested. We will do a test once the system is up and running again.
Currently it is in maintenance.

Regards,

Jörn

Am Dienstag, 10. Oktober 2023, 19:17:17 CEST schrieb Mark Rivers:
> Hi Jörn,
> 
> I am curious whether you might use NDPluginPva for sending the data via a stream over the network?  That is a common thing to do.
> 
> NDPluginFile does not currently have a mechanism for continuous streaming.  A common workaround is to set NumCapture to a very large number, but that will eventually fail if the number of frames streamed is >2^31.
> 
> From a quick look at the NDPluginFile code it seems that we could change the logic so that if NumCapture is 0 it will stream forever.  It would require changes in the following lines so that the comparison with numCapture is only done if it is non-zero.
> 
> https://github.com/areaDetector/ADCore/blob/076bd4543827798537fa486c00e334722e9889f4/ADApp/pluginSrc/NDPluginFile.cpp#L743
> https://github.com/areaDetector/ADCore/blob/076bd4543827798537fa486c00e334722e9889f4/ADApp/pluginSrc/NDPluginFile.cpp#L750
> https://github.com/areaDetector/ADCore/blob/076bd4543827798537fa486c00e334722e9889f4/ADApp/pluginSrc/NDPluginFile.cpp#L767
> 
> I think that would be backwards compatible because NumCapture=0 currently does not do anything useful.  If the user sets NumCapture=0 that might cause existing plugins to fail.  But there is nothing preventing users from doing that now, e.g. the NumCapture record does not set DRVL=1.
> 
> If you want to test this and issue a Pull Request I'll be happy to look at it.
> 
> Mark
> 
> -----Original Message-----
> From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Jörn Dreyer via Tech-talk
> Sent: Tuesday, October 10, 2023 4:49 AM
> To: tech-talk at aps.anl.gov
> Subject: Question regarding ADPluginFile driver
> 
> Hi,
> 
> I am currently developing an ADPluginFile driver that will  send the data via a stream over the network to a remote process for fast analysis. It can also store the data locally in various formats.
> For the network transfer I would like to be able to do continuous streaming without automatic limit by a defined number of events.
> Is that possible? Or can it be implemented within my code?
> 
> Regards,
> 
> Jörn
> 
> 
> 
> 


-- 
Dr. Jörn Dreyer
HIBEF DAQ + Controls
Institut für Strahlenpysik
Abt. FWKX
Tel: +49 351 260 3263
Fax: +49 351 260 123263

Helmholtz-Zentrum Dresden - Rossendorf e.V. (HZDR)
Bautzner Landstr. 400 | 01328 Dresden | Germany
Tel: +49 (0) 351 260 1234 
http://www.hzdr.de 
Vorstand: Prof. Dr. Sebastian M. Schmidt, Dr. Diana Stiller
Vereinsregister: VR 1693 beim Amtsgericht Dresden