Re: Compression in pvAccess

[Marty Kraimer via Core-talk <core-talk@aps.anl.gov>]

On 11/19/18 4:09 PM, Mark Rivers via Core-talk wrote:
> > It might be better to implement the whole chain through with at least imagej before we start talking about inclusion in lower level libraries.
> I agree, and that's what I am suggesting as well.  I'd just like to get it on the radar screen as something that we might want to do in the not too distant future.

Looking briefly at
https://imagej.nih.gov/ij/docs/guide/146-7.html#toc-Section-7
It looks like it should not be hard for the areaDetector ImageJ plugin 
to support the JPEG NDPluginCodec.
By looking at 3rd party ImageJ plugins there should be a good chance to 
find support for Blocxs

Marty

> Mark
>
>
> -----Original Message-----
> From: Michael Davidsaver <mdavidsaver@gmail.com>
> Sent: Monday, November 19, 2018 2:20 PM
> To: Mark Rivers <rivers@cars.uchicago.edu>
> Cc: 'core-talk@aps.anl.gov' <core-talk@aps.anl.gov>
> Subject: Re: Compression in pvAccess
>
> On 11/19/18 12:05 PM, Mark Rivers wrote:
> > I should add that I also need the Java Normative Types library to implement this, not just normativeTypesCPP.  In fact Java is higher priority for now, since that is what ImageJ and CSS use.
> What has been prototyped so far?  Only the NDPluginCodec piece?
>
> I hope you'll forgive me.  The history of (imo.) premature standardization in PVA world has left me a bit gun shy about adding utility code before it has actually been used.
> It might be better to implement the whole chain through with at least imagej before we start talking about inclusion in lower level libraries.
>
>
>
> > -----Original Message-----
> > From: Mark Rivers
> > Sent: Monday, November 19, 2018 12:48 PM
> > To: 'Michael Davidsaver' <mdavidsaver@gmail.com>
> > Cc: core-talk@aps.anl.gov
> > Subject: RE: Compression in pvAccess
> >
> > > Can you be more specific about what is being suggested?  "pvAccess"
> > > is a bit generic, at least on the
> > > c++ side (eg. pvAccessCPP vs. normativeTypesCPP vs. pvDatabaseCPP)?
> > I'm not sure what the best level to implement this is.  My particular application would be met by having normativeTypesCPP implement compression and decompression methods for NTNDArrays according to the compression algorithm specified in the existing ".codec" field.  But perhaps there are other large data objects that people use with pvAccess which would also benefit from compression, and so this should not be implemented only for NTNDArrays?
> >
> > I think I will take a stab at doing this in the ImageJ client, just to see if it works.  But longer term it would be nice to have it implemented in the pvAccess libraries, not in the client code.
> >
> > Mark
> >
> >
> > -----Original Message-----
> > From: Michael Davidsaver <mdavidsaver@gmail.com>
> > Sent: Monday, November 19, 2018 12:41 PM
> > To: Mark Rivers <rivers@cars.uchicago.edu>
> > Cc: core-talk@aps.anl.gov
> > Subject: Re: Compression in pvAccess
> >
> > > it would be much nicer to have the decompression library and
> > > compress/decompress methods as part of pvAccess
> > Can you be more specific about what is being suggested?  "pvAccess" is
> > a bit generic, at least on the
> > c++ side (eg. pvAccessCPP vs. normativeTypesCPP vs. pvDatabaseCPP)?
> >
> >
> > On 11/16/18 12:25 PM, Mark Rivers wrote:
> > > Folks,
> > >
> > >   
> > >
> > > Bruno has written a new plugin for areaDetector called NDPluginCodec.  It compresses and decompresses NDArrays using JPEG (lossy) and Blosc (lossless) compression.  NDArrays now have a .codec field, just as NTNDArrays do.  He implemented these fields in the areaDetector converters for NDArrays <-> NTNDArrays.
> > >
> > >   
> > >
> > > Using these new capabilities I just set up the following configuration.  The simDetector IOC is running on one machine (happens to be Linux), and the receiving IOC and ImageJ are running on another machine (happens to be Windows).
> > >
> > >   
> > >
> > >    simDetector detector   =>  NDPluginCodec (compress)   =>
> > > NDPluginPVA   =>   pvaAccess over Ethernet  =>  pvaDetector  =>
> > > NDPluginCodec (decompress)  =>  NDPluginPva  =>  pvaAccess localhost
> > > =>  ImageJ pvAccess viewer
> > >
> > >   
> > >
> > > By enabling the compression I can reduce the network bandwidth >100x for highly compressible images.
> > >
> > >   
> > >
> > > The next step would be to eliminate the need for the IOC on the viewer machine, i.e. the viewer can directly decompress the NTNDArray that comes from pvAccess.
> > >
> > >   
> > >
> > > One way to implement this is to require the decompression code be in
> > > the client (ImageJ, Display Builder, etc.).  But it would be much nicer to have the decompression library and compress/decompress methods as part of pvAccess, so then each client does not need to implement it.
> > >
> > >   
> > >
> > > In my opinion the top priority should be Java, since Phoebus/CSS/Display Builder and ImageJ are both written in Java.  If we get this working then we can transport images between camera/detector IOCs and remote viewing clients with much lower network bandwidth than is currently required.  We can do both lossless and lossy compression.
> > >
> > >   
> > >
> > > There is already a Blosc implementation for Java:
> > >
> > > https://github.com/Blosc/JBlosc
> > >
> > >   
> > >
> > > So far I have only found a JNI implementation for JPEG, but hopefully there is a pure Java implementation as well.
> > >
> > >   
> > >
> > > Thoughts?
> > >
> > >   
> > >
> > > Mark
> > >
> > >