Re: Parse a variable-length, multi-column waveform in stream device

[Dirk Zimoch <dirk.zimoch@psi.ch>]

Hello Ahmad,

I have tested a solution using regexp to isolate the columns. But while 
doing that I found a bug in the regexp converter. Thus to make it work 
you need to use the latest commit from 
git@github.com:paulscherrerinstitute/StreamDevice.git

What I do is the following:
8 waveform records read the 8 columns. One is active, the other 7 read 
the data with "I/O Intr".

8 protocols use regsub (%#/regexp/subst/) to remove the 7 values not 
needed for each waveform.

Process waveform 1 and all 8 will get their column.

I had to assume some details, e.g. line terminator "\n" and space after 
each value, including the last column. If that is wrong, the protocols 
need to me modified a bit.

As I can't see any special marker for the last line, I do NOT use an 
InTerminator, because I need to read the whole table as one string. In 
case there is a marker, e.g. an empty line, you may set inTerminator 
accordingly ("\n\n") and remove the "\n" at the end of the 'in' commands.

See attached files.

Dirk


On 12.08.2018 08:22, Abdalla  Ahmad wrote:
> Hello Dirk
>
> Thank you for your reply, looking forward for your results. I would like to point out some notes:
> 1. Regarding variable size, you can expect for sure way larger buffer sizes than 1000.
> 2. Why the protocol parse you mentioned won't work? I may have understood you wrong but In a single line you can parse any number you want by skipping the others and I managed to do that by the same format you provided. The first "%i" will get the first number. But the problem is if I don't use "ExtraInput = Ignore" the IOC shell will be flooded with "x bytes surplus input" error.
> 3. Will you implement new parser in stream device to parse interlaced arrays? After many trials with stream device, it's either this or to develop a new EPICS driver for the device.
>
> Best Regards,
> Abdalla
>
> -----Original Message-----
> From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch]
> Sent: Wednesday, August 08, 2018 12:37 PM
> To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov
> Subject: Re: Parse a variable-length, multi-column waveform in stream device
>
> Hello Ahmad,
>
> StreamDevice can read variable length arrays. You need to set NELM to the maximum number of elements. StreamDevice will set NORD to the number of elements found. Modern EPICS versions are able to handle variable length arrays when the client sets up a monitor using 0 for the requested array length. It should then use the current NORD value.
>
> However StreamDevice reads arrays in rows, not columns. If you have a waveform record and use { separator=" "; in "%i"; }, stream device will repeat the "%i" up to NELM times (or until end of line, until the separator is not found any more or until the conversion fails).
>
> But in your case it seems you have interlaced arrays:
>
> a[0] b[0] c[0] d[0] e[0] f[0] g[0] h[0]
> a[1] b[1] c[1] d[1] e[1] f[1] g[1] h[1]
> a[2] b[2] c[2] d[2] e[2] f[2] g[2] h[2]
>
> So StreamDevice will see the first elements of each array, not all elements of one array.
> StreamDevice would be able to read rows with variable length:
>
> a[0] a[1] a[2] ...
> b[0] b[1] b[2] ...
> c[0] c[1] c[2] ...
> d[0] d[1] d[2] ...
> e[0] e[1] e[2] ...
> f[0] f[1] f[2] ...
> g[0] g[1] g[2] ...
> h[0] h[1] h[2] ...
>
> Also this will not help:
> in "%i %*i %*i %*i %*i %*i %*i %*i";
> Because the first %i reads all the elements in the row before even parsing the first %*i.
>
> I need a new syntax to specify interlaces arrays.
>
> The only way I can think to at the moment is to pre-process the input with a regsub which removed the "other" array elements from the input, but I need to test that first. It may be quite inefficient. Thus if <N> becomes huge (say >1000) parsing may be quite slow.
>
> I will tell you soon if I can find a solution...
>
> Dirk
>
>
> On 08.08.2018 11:09, Abdalla  Ahmad wrote:
> > Hi
> >
> > We are trying to implement an IOC for the Libera Spark BPM using
> > stream device, one of the tricky commands for example is TBT_IQ <N>
> > which returns a variable-length buffer of turn-by-turn data. An example:
> >
> > TBT_IQ 3
> >
> > 22 -36  19 -38  16 -31  -3  -31
> >
> > 39 -212 29 -212 26 -163 -37 -150
> >
> > 39 -175 36 -176 27 -133 -21 -129
> >
> > As you can see the output is an array of fixed columns (IQ parameters)
> > and variable rows (waveforms length). In a single line you can easily
> > parse and skip any value, but in this format we have two problems:
> >
> > 1.Is it possible to parse variable number of IN commands in the
> > protocol file?
> >
> > 2.Variable "IN" commands mean variable waveform length, is this
> > possible in EPICS?
> >
> > Best Regards,
> >
> > Abdalla Ahmad
> >
> > Control Engineer
> >
> > SESAME
> >
> > Allan, Jordan.
> >
> > Tel: (+962-5) 3511348 , ext. 265
> >
> > Fax: (+962-5) 3511423
> >
> > Mob: (+962-7)88183296
> >
> > www.sesame.org.jo <http://www.sesame.org.jo/>
record (waveform, "IQ1")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ1 terminal")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ2")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ2 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ3")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ3 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ4")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ4 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ5")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ5 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ6")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ6 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ7")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ7 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}
record (waveform, "IQ8")
{
    field(DTYP, "stream")
    field(INP, "@bpm.proto TBT_IQ8 terminal")
    field(SCAN, "I/O Intr")
    field(FTVL, "SHORT")
    field(NELM, "$(NEML)")
}

OutTerminator = NL;
TBT_IQ1 {
    out "TBT_IQ 3";
    in "%#/([-0-9]+ *)([-0-9]+ *){7}/\1/%i \n";
}

TBT_IQ2 {
    in "%#/([-0-9]+ *)([-0-9]+ *)([-0-9]+ *){6}/\2/%i\n";
}

TBT_IQ3 {
    in "%#/([-0-9]+ *){2}([-0-9]+ *)([-0-9]+ *){5}/\2/%i \n";
}

TBT_IQ4 {
    in "%#/([-0-9]+ *){3}([-0-9]+ *)([-0-9]+ *){4}/\2/%i \n";
}

TBT_IQ5 {
    in "%#/([-0-9]+ *){4}([-0-9]+ *)([-0-9]+ *){3}/\2/%i \n";
}

TBT_IQ6 {
    in "%#/([-0-9]+ *){5}([-0-9]+ *)([-0-9]+ *){2}/\2/%i \n";
}

TBT_IQ7 {
    in "%#/([-0-9]+ *){6}([-0-9]+ *)([-0-9]+ *)/\2/ %i\n";
}

TBT_IQ8 {
    in "%#/([-0-9]+ *){7}([-0-9]+ *)/\2/%i \n";
}