Experimental Physics and
Industrial Control System

Ben Franksen via Core-talk <core-talk at aps.anl.gov> · Sun, 29 Mar 2020 16:32:12 +0200

Am 29.03.20 um 01:26 schrieb Johnson, Andrew N.:
> Allowing multiple filters to be used chained together is most
> definitely something we want to support, although in most cases they
> would probably be different filters, say array to extract one value
> followed by a deadband filter say, possibly with a time stamp filter
> as well.

Of course.

[reordered:]
> I understand that some JSON parsers won’t handle duplicate keys, is
> that what concerns you?
I don't expect any concrete difficulties arising from that, though I
must say it gives me an uneasy feeling. We are deviating from the
"normal" way a JSON object is interpreted without any concrete benefit
to justify that.

> Repeating the same kind of filter seems less necessary, but
> given that we can support it at no cost I don’t see why we should try
> to prevent it.

The main reasons are: (1) double filters are redundant, and (2) they are
problematic to support for put filters (so there is a (hidden) cost to
them).

What can a filter do? Generally speaking, it is restricted to change
what's in a db_field_log: time, status, severity, field size and type,
number of elements, and the data itself; it can also drop the whole
thing. Each filter plugin limits itself to changing a subset of these 6
items in a specific way and according to the parameters is was given.
Applying two instances of the same filter means we successively change
the same subset of items twice. It is easy to see that this can always
be reduced to a single application of that filter, at least for the
existing plugins:

 * the timestamp filter is idempotent so adding a second one does nothing
 * two decimate filters are equivalent to one with the decimation
factors multiplied
 * two array filters can be collapsed into one with a bit of index
arithmetic
 * etc

So actually using the same sort of filter twice is just a waste of CPU
and memory.

I gets worse though. If we add put/write filters to the picture, then
the feature is no longer "no cost". In fact, supporting it is not
possible without a lot of overhead.

Let me restrict the consideration to array filters because I cannot
conceive of another useful put filter ATM.

The first problem is that to define what two such filters in succession
should mean for a put operation. The only useful definition I could come
up with is that it should do the reverse of what it does on get. In
other words, if we get an array field with two filters f1, f2 applied in
succession, then put the result back using the same two filters, then
this should be a no-op (assuming no processing happens in between).

For get or monitor events, it is easy to get an operational intuition
for what applying two array filters means: we first get the slice/stride
of the original field, then get another slice/stride of the resulting array.

It is unclear how to (operationally) reverse this process.

Here is an example. f1=[1:2:5], f2=[-2]. When we apply that in a get
operation to the array [0,1,2,3,4,5] we get first [1,3,5], then [3]. Now
suppose we use the same two filters for put, in reverse: we start with
[3] and write that to the second to last element of... what? Imagine we
could get at the "final number of elements" of the filter f1, then we
could write to a fictitions [_,_,_] to get [_,3,_] using f2. Next write
the result via filter f1 to [0,1,2,3,4,5] which gives us [0,_,2,3,4,_].
The mysterious "_" values here have the meaning of "undefined value",
and we need a convention that writing "undefined" to a position with a
"defined" value means: don't actually write. So the actual result is
[0,1,2,3,4,5] as expected. But doing this in C/EPICS is not so easy,
since we cannot simply add an "undefined value" to the existing
primitive types. For float and double we could use NaN but what about
the other data types? One possible idea could be to allocate a second
array, possibly compressed to one bit per element, that indicates
"definedness" of the corresponding index.

So while it is possible to define a meaning for double array filtered
puts, it is difficult (though not impossible) to conceive of an
implementation along the lines of how the existing (get/event) filters
are "chained". However, all the extra effort needed to pull this off is
wasted on a feature which is redundant and for which nobody found a
valid use case yet.

Cheers
Ben
--
...it is impossible to speak of a depoliticized economy as the liberals
do, or of a separation between economic exploitation and political
oppression as the Marxists articulate. The basic distinction, rather, is
between power and creativity...
             -- Jonathan Nitzan and Shimshon Bichler: Capital as Power

________________________________

Helmholtz-Zentrum Berlin für Materialien und Energie GmbH

Mitglied der Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren e.V.

Aufsichtsrat: Vorsitzender Dr. Volkmar Dietz, stv. Vorsitzende Dr. Jutta Koch-Unterseher
Geschäftsführung: Prof. Dr. Bernd Rech (Sprecher), Prof. Dr. Jan Lüning, Thomas Frederking

Sitz Berlin, AG Charlottenburg, 89 HRB 5583

Postadresse:
Hahn-Meitner-Platz 1
D-14109 Berlin

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System