Re: PVD/PVA: Examples for multi-dimensional arrays?

[Mark Rivers via Core-talk <core-talk@aps.anl.gov>]

NTNDArrays have always had a .codec field, which implies support for compression.  It has never been implemented until recently.  Now we support codec = "blosc" or "jpeg".  It does not involve any changes to the network protocol, it just changes the size of the "value".


What I was suggesting was that somewhere in EPICS 7 there be compress() and decompress() methods that can be invoked on the NTNDArray value field, so that each client and server does not need to implement this logic.


Mark



________________________________
From: core-talk-bounces@aps.anl.gov <core-talk-bounces@aps.anl.gov> on behalf of Marty Kraimer via Core-talk <core-talk@aps.anl.gov>
Sent: Wednesday, November 21, 2018 2:32 PM
To: core-talk@aps.anl.gov
Subject: Re: PVD/PVA: Examples for multi-dimensional arrays?

On 11/21/18 11:10 AM, Ralph Lange via Core-talk wrote:
Hi Mark,

On Wed, 21 Nov 2018 at 16:37, Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>> wrote:
What do you mean by "support"?  NTNDArrays are certainly N-dimensional.  Do you want the API to allow extracting rows/columns, etc.?

Hm.
The problem with leaving n-dim support to the application is that every application using n-dim arrays will have to do its own implementation of the accessors. Also, because of that, these n-dim arrays will likely not be compatible between applications.

Does that sound like I am asking for a new normative type?

Cheers,
~Ralph

At the very beginning of PVData development, i. e. the JavaIOC, a deliberate decision was made that pvData would only support 1 dim arrays.
The problem with supporting more dimensions is how many and how to define each dimension.
Just consider a 2d array. It it a 2d matrix? Is it a sparse  matrix? Is it a table?


I also remember a request for pvData defining a map type.
Where do we stop?

At the very beginning it was decided that the following types would be supported

1) scalars of numeric , boolean, and string.
2) 1d arrays of each scalar type
3) structures
4) 1d structure arrays


This was defined in 2006.

The only types that were added later are:

5) union
6) 1d union array.

It is ONLY data of  above types that need to be supported by the pva network protocol.

Also from the beginning it was assumed that more complex types would be supported via structure definitions.
This later became the normative types.
Note that normative types already has definiintions for NTTable, NTMatrix, and NTHistogram.
Each of these is a different type of 2d array.
NTNDArray includes a multi dimensional array plus lots more.

No normative type required any changes to the network protocol.


Recently there was a discussion of pvAccess directly supporting  compression.
Although it was talking about pvAccess it would probably have to involve pvData.
For the same reason for not supporting multi deminsional array compression should NOT be directly supported by pvData/pvAccess.
What kind of compression?
lossless?, JPEG?, etc, etc.
Would each additional type require changes to the pva network protocol.
Note that areaDetecter already supports a few compression types.

Marty