Re: double filters

["Johnson, Andrew N. via Core-talk" <core-talk at aps.anl.gov>]

Hi Ben,

On Mar 29, 2020, at 9:32 AM, Ben Franksen via Core-talk <core-talk at aps.anl.gov> wrote:

I understand that some JSON parsers won’t handle duplicate keys, is
that what concerns you?

I don't expect any concrete difficulties arising from that, though I
must say it gives me an uneasy feeling. We are deviating from the
"normal" way a JSON object is interpreted without any concrete benefit
to justify that.

We’re also requiring that the order of keys be preserved, which some JSON parsers can’t handle either. The YAJL library that we include in Base supports both for both parsing and generation because it’s a stream parser.

Repeating the same kind of filter seems less necessary, but
given that we can support it at no cost I don’t see why we should try
to prevent it.

The main reasons are: (1) double filters are redundant, and (2) they are
problematic to support for put filters (so there is a (hidden) cost to
them).

I suspect there are things we can do with multiple sync filters that can’t be replicated otherwise, see below.

What can a filter do? Generally speaking, it is restricted to change
what's in a db_field_log: time, status, severity, field size and type,
number of elements, and the data itself; it can also drop the whole
thing. Each filter plugin limits itself to changing a subset of these 6
items in a specific way and according to the parameters is was given.

In some of its configurations the sync filter stores an incoming db_field_log and doesn’t pass it on until the next one arrives, to implement “last” filtering for example. I haven’t tried using it with arrays so it might be broken there, but for scalar values at least I suspect we can chain two filters together to get behaviors that can’t be achieved with a single sync filter (give me the last-but-one value before the “green” state).

I gets worse though. If we add put/write filters to the picture, then
the feature is no longer "no cost". In fact, supporting it is not
possible without a lot of overhead.

I agree that put filters are probably completely different to get/monitor filters, and there will probably be only a few put filters.

For get or monitor events, it is easy to get an operational intuition

for what applying two array filters means: we first get the slice/stride
of the original field, then get another slice/stride of the resulting array.

It is unclear how to (operationally) reverse this process.

Your example introduced an undefined value to flag elements that shouldn’t be written to. I don’t see that as practical, and I don’t think you did either. If we want an array put filter be able to overwrite a subset of the elements of an array I would prefer a slightly different approach.

I’m also thinking of a put filter where you can give individual array index values to be written to, so to create a filter that only overwrites specific elements of the target array you might want to specify a channel filter as

pvname.{“subset”:[0,1,2,4,6,10,12,16,18]} 

The idea is that there might not be an arithmetic slice/stride pattern to the index values (there is a pattern to the above index list though, in case anyone wants a little puzzle). To implement this, the filter would have to convert the incoming data array into a data structure that contains a list of index/value pairs. It then becomes easy to chain multiple subset filters, since a downstream filter that gets one of these data structures instead of a flat array can just delete entries from the list that came from upstream.

Note that I wouldn't expect this to be efficient, and I’m still not completely sure who really wants/needs single put filters let alone chained ones, but I’m happy to explore these concepts in case they do prove useful.

A very interesting idea that I’ve just had for a get and put filter is one that transports JSON objects, so a client can atomically read from and write to multiple fields of the addressed record through it say. The filter would change the native data type of the channel into a long string (i.e. a large char array), and allow the user to specify a set of fields to be read during get/monitor. The put JSON object would be allowed to specify a different set of fields to write to, something like this perhaps:

caput -S 'ai.{"json":{"fields":["RVAL","ESLO","EOFF","LINR","VAL"]}}’ \

  ‘{“LINR”:”LINEAR”, "EGUL":-10, "EGUF":10, "EGU":"Volts”, "PROC”:true}'

This could also allow writing subsets of an array because JSON supports null as a first-class value, so:

caput -S ‘wf.{"json":{}}’ ‘{“VAL”:[null, null, 3, 4, 5, null, null]}'

or maybe even something like:

caput -S ‘wf.{"json":{}}’ ‘{“VAL[1:2:9]”:[1, 3, 5, null, 9]}'

but that might be a little tricky to implement.

Before going too far with this we should probably consider whether it would make sense for any internal APIs to use pvData objects, and how this might interface with QSRV. I suspect not, but the answer isn’t obvious to me yet.

It doesn’t seem useful to be able to chain anything with this “json” filter.

- Andrew

-- 

Complexity comes for free, simplicity you have to work for.