Re: Parse a variable-length, multi-column waveform in stream device

[Mark Rivers <rivers@cars.uchicago.edu>]

Your Makefile does not seem to contain calcSupport.dbd.  Since you defined CALC in StreamDevice/configure/RELEASE then StreamDevice is built with support for the records defined in CALC.  You need to include calcSupport.dbd when building your IOC application.  You also need to add "calc" and "sscan" to your Ioc_LIBS.


 Mark



________________________________
From: tech-talk-bounces@aps.anl.gov <tech-talk-bounces@aps.anl.gov> on behalf of Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>
Sent: Tuesday, August 14, 2018 2:57 AM
To: Dirk Zimoch; tech-talk@aps.anl.gov; Ralph Lange
Subject: RE: Parse a variable-length, multi-column waveform in stream device

Hello Dirk

I used PCRE_INCLUDE and StreamDevice build succeeded. I built it using synApps packages collection from the script "assembly_synApps.sh" and base 3.15.5, on my IOC I get this strange error regarding the DBD

Declaration of recordtype(scalcout) preceeded full definition.
Error at or befor "}" in file dbd/ioc.dbd line 12380
Ioc_registerRecordDeviceDriver pdbbase
registerRecordDeviceDriver failed aSub
registerRecordDeviceDriver failed ai
registerRecordDeviceDriver failed stringin
registerRecordDeviceDriver failed sub

And then I get the error "record is of unknown type ai". In my iocApp/src/Makefile I have

Ioc_DBD += base.dbd
Ioc_DBD += stream.dbd
Ioc_DBD += drvAsynSerialPort.dbd
Ioc_DBD += drvAsynIPPort.dbd

Ioc_LIBS += asyn
Ioc_LIBS += stream

And in StreamDevice/configure/RELEASE I have asy R4-33 and calc R3-7-1

Thank you very much
Abdalla.

-----Original Message-----
From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch]
Sent: Monday, August 13, 2018 4:42 PM
To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo>; tech-talk@aps.anl.gov; Ralph Lange <ralph.lange@gmx.de>
Subject: Re: Parse a variable-length, multi-column waveform in stream device



On 13.08.2018 14:41, Abdalla  Ahmad wrote:
> Dear Ralph
> I edited configure/Release and ran make inside stream-master, I got:
> Makefile: No such a file or directory
> No rule to make target Makefile. Stop
>
> Dear Dirk
> I commented PCRE from the configure/RELEASE but I got undefined reference errors despite the location of PCRE libraries is defined in /etc/ld.so.conf.d/. "rpm -qa | grep pcre" gave me the same libraries except version 8.32 and for x86_64, no i686.
>

I have both, 32 and 64 bit libraries installed, because I build EPICS for both. You don't need both.

StreamDevice uses the host's native PRCE if either PCRE_INCLUDE (directory of pcre.h) or PCRE_LIB (location of libpcre.so) is defined in RELEASE (or both). The problem is that the locations differ between Linux distros. It uses the EPICS module PRCE if PCRE defined in RELEASE.
It does not use PRCE if neither is defined.

I don't know if anything is different in PCRE version 8.32. What exactly are your undefined reference errors?

To build PCRE as an EPICS module, I had downloaded the source code from https://www.pcre.org and added this Makefile:
http://epics.web.psi.ch/software/streamdevice/pcre/Makefile

Worked with PRRE version 7.2.



> Thank you all.
> Abdalla.
>
>
> -----Original Message-----
> From: tech-talk-bounces@aps.anl.gov
> [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Dirk Zimoch
> Sent: Monday, August 13, 2018 2:39 PM
> To: tech-talk@aps.anl.gov
> Subject: Re: Parse a variable-length, multi-column waveform in stream
> device
>
>
>
> On 13.08.2018 13:12, Ralph Lange wrote:
>> If I remember correctly,
>> the epics-modules/stream does not require you to run configure in the
>> pcre directory.
>> Have you tried just building it?
>
> Exactly. The EPICS Makefile replaces the ./configure stuff.
>
> If you have PCRE already installed on your system, you should not need to build it. PCRE should be available for all Linux systems and for Windows as well. Install the development package (the once with the header files).
>
> For example, I have this for my Linux system:
> $ rpm -qa | grep pcre
> pcre-devel-7.8-7.el6.i686
> pcre-devel-7.8-7.el6.x86_64
> pcre-7.8-7.el6.x86_64
> pcre-7.8-7.el6.i686
>
>
>>
>> Cheers,
>> ~Ralph
>>
>>
>> On Mon, 13 Aug 2018 at 12:17, Abdalla Ahmad
>> <abdalla.ahmad@sesame.org.jo <mailto:abdalla.ahmad@sesame.org.jo>> wrote:
>>
>>      Hello Dirk
>>
>>      No the device does not echo any command.
>>
>>      I downloaded the latest stream device from github but It needs PCRE,
>>      which I got from epics-modules/stream , I took the folder "pcre",
>>      did "./configure" and make and I get the error
>>
>>      No rule to make target doc/html/NON-AUTOTOOL-Build.txt needed by all-am.
>>
>>      Could there be an option to pass to the configure script do disable
>>      doc files? Or maybe I got PCRE module? There is a yum package
>>      installed called pcre so I added the libraries to ldconfig but still
>>      I get undefined reference errors.
>>
>>      Best Regards,
>>      Abdalla.
>>
>>      -----Original Message-----
>>      From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch
>>      <mailto:dirk.zimoch@psi.ch>]
>>      Sent: Monday, August 13, 2018 1:05 PM
>>      To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo
>>      <mailto:abdalla.ahmad@sesame.org.jo>>; tech-talk@aps.anl.gov
>>      <mailto:tech-talk@aps.anl.gov>
>>      Subject: Re: Parse a variable-length, multi-column waveform in
>>      stream device
>>
>>
>>
>>      On 13.08.2018 11:33, Abdalla  Ahmad wrote:
>>       > Hello Dirk
>>       >
>>       > That is really impressive, thank you very much. I will integrate
>>      your protocol in the IOC and provide you with feedback.
>>       > Regarding your assumptions:
>>       > 1. We used the LF terminator with setpoints and readings and it
>>      works fine so I suppose that is the correct terminator.
>>       > 2. Will this work with variable buffer length? I mean instead of
>>      "3" for example we pass the PV for buffer length.
>>
>>      Yes. Just make sure that NELM is big enough for your longest
>>      possible reply.
>>
>>       > 3. Since the rest are passive scan, If I understand correctly
>>      processing the first one will process the others right?
>>
>>      The input from the first one will make the "I/O Intr" records process.
>>      However, any input from any other record will be given to the "I/O
>>      Intr"
>>      records. Thus if you have other data that matches the format, those
>>      records will be processed falsely.
>>
>>      Does the device echo the command "TBT_IQ <n>"? If not, there is no
>>      way for the "I/O Intr" records to find out if they should parse
>>      input or not. Thus they will try to parse everything and if they
>>      succeed, they will process the record.
>>
>>       >
>>       > Best Regards,
>>       > Abdalla.
>>       >
>>       > -----Original Message-----
>>       > From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch
>>      <mailto:dirk.zimoch@psi.ch>]
>>       > Sent: Monday, August 13, 2018 12:15 PM
>>       > To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo
>>      <mailto:abdalla.ahmad@sesame.org.jo>>; tech-talk@aps.anl.gov
>>      <mailto:tech-talk@aps.anl.gov>
>>       > Subject: Re: Parse a variable-length, multi-column waveform in
>>      stream device
>>       >
>>       > Hello Ahmad,
>>       >
>>       > I have tested a solution using regexp to isolate the columns. But
>>      while doing that I found a bug in the regexp converter. Thus to make
>>      it work you need to use the latest commit from
>>      git@github.com:paulscherrerinstitute/StreamDevice.git
>>       >
>>       > What I do is the following:
>>       > 8 waveform records read the 8 columns. One is active, the other 7
>>      read the data with "I/O Intr".
>>       >
>>       > 8 protocols use regsub (%#/regexp/subst/) to remove the 7 values
>>      not needed for each waveform.
>>       >
>>       > Process waveform 1 and all 8 will get their column.
>>       >
>>       > I had to assume some details, e.g. line terminator "\n" and space
>>      after each value, including the last column. If that is wrong, the
>>      protocols need to me modified a bit.
>>       >
>>       > As I can't see any special marker for the last line, I do NOT use
>>      an InTerminator, because I need to read the whole table as one
>>      string. In case there is a marker, e.g. an empty line, you may set
>>      inTerminator accordingly ("\n\n") and remove the "\n" at the end of
>>      the 'in' commands.
>>       >
>>       > See attached files.
>>       >
>>       > Dirk
>>       >
>>       >
>>       > On 12.08.2018 08:22, Abdalla  Ahmad wrote:
>>       >> Hello Dirk
>>       >>
>>       >> Thank you for your reply, looking forward for your results. I
>>      would like to point out some notes:
>>       >> 1. Regarding variable size, you can expect for sure way larger
>>      buffer sizes than 1000.
>>       >> 2. Why the protocol parse you mentioned won't work? I may have
>>      understood you wrong but In a single line you can parse any number
>>      you want by skipping the others and I managed to do that by the same
>>      format you provided. The first "%i" will get the first number. But
>>      the problem is if I don't use "ExtraInput = Ignore" the IOC shell
>>      will be flooded with "x bytes surplus input" error.
>>       >> 3. Will you implement new parser in stream device to parse
>>      interlaced arrays? After many trials with stream device, it's either
>>      this or to develop a new EPICS driver for the device.
>>       >>
>>       >> Best Regards,
>>       >> Abdalla
>>       >>
>>       >> -----Original Message-----
>>       >> From: Dirk Zimoch [mailto:dirk.zimoch@psi.ch
>>      <mailto:dirk.zimoch@psi.ch>]
>>       >> Sent: Wednesday, August 08, 2018 12:37 PM
>>       >> To: Abdalla Ahmad <abdalla.ahmad@sesame.org.jo
>>      <mailto:abdalla.ahmad@sesame.org.jo>>; tech-talk@aps.anl.gov
>>      <mailto:tech-talk@aps.anl.gov>
>>       >> Subject: Re: Parse a variable-length, multi-column waveform in
>>      stream
>>       >> device
>>       >>
>>       >> Hello Ahmad,
>>       >>
>>       >> StreamDevice can read variable length arrays. You need to set
>>      NELM to the maximum number of elements. StreamDevice will set NORD
>>      to the number of elements found. Modern EPICS versions are able to
>>      handle variable length arrays when the client sets up a monitor
>>      using 0 for the requested array length. It should then use the
>>      current NORD value.
>>       >>
>>       >> However StreamDevice reads arrays in rows, not columns. If you
>>      have a waveform record and use { separator=" "; in "%i"; }, stream
>>      device will repeat the "%i" up to NELM times (or until end of line,
>>      until the separator is not found any more or until the conversion
>>      fails).
>>       >>
>>       >> But in your case it seems you have interlaced arrays:
>>       >>
>>       >> a[0] b[0] c[0] d[0] e[0] f[0] g[0] h[0] a[1] b[1] c[1] d[1] e[1]
>>      f[1]
>>       >> g[1] h[1] a[2] b[2] c[2] d[2] e[2] f[2] g[2] h[2]
>>       >>
>>       >> So StreamDevice will see the first elements of each array, not
>>      all elements of one array.
>>       >> StreamDevice would be able to read rows with variable length:
>>       >>
>>       >> a[0] a[1] a[2] ...
>>       >> b[0] b[1] b[2] ...
>>       >> c[0] c[1] c[2] ...
>>       >> d[0] d[1] d[2] ...
>>       >> e[0] e[1] e[2] ...
>>       >> f[0] f[1] f[2] ...
>>       >> g[0] g[1] g[2] ...
>>       >> h[0] h[1] h[2] ...
>>       >>
>>       >> Also this will not help:
>>       >> in "%i %*i %*i %*i %*i %*i %*i %*i";
>>       >> Because the first %i reads all the elements in the row before
>>      even parsing the first %*i.
>>       >>
>>       >> I need a new syntax to specify interlaces arrays.
>>       >>
>>       >> The only way I can think to at the moment is to pre-process the
>>      input with a regsub which removed the "other" array elements from
>>      the input, but I need to test that first. It may be quite
>>      inefficient. Thus if <N> becomes huge (say >1000) parsing may be
>>      quite slow.
>>       >>
>>       >> I will tell you soon if I can find a solution...
>>       >>
>>       >> Dirk
>>       >>
>>       >>
>>       >> On 08.08.2018 11:09, Abdalla  Ahmad wrote:
>>       >>> Hi
>>       >>>
>>       >>> We are trying to implement an IOC for the Libera Spark BPM using
>>       >>> stream device, one of the tricky commands for example is TBT_IQ <N>
>>       >>> which returns a variable-length buffer of turn-by-turn data. An
>>      example:
>>       >>>
>>       >>> TBT_IQ 3
>>       >>>
>>       >>> 22 -36  19 -38  16 -31  -3  -31
>>       >>>
>>       >>> 39 -212 29 -212 26 -163 -37 -150
>>       >>>
>>       >>> 39 -175 36 -176 27 -133 -21 -129
>>       >>>
>>       >>> As you can see the output is an array of fixed columns (IQ
>>       >>> parameters) and variable rows (waveforms length). In a single line
>>       >>> you can easily parse and skip any value, but in this format we
>>      have two problems:
>>       >>>
>>       >>> 1.Is it possible to parse variable number of IN commands in the
>>       >>> protocol file?
>>       >>>
>>       >>> 2.Variable "IN" commands mean variable waveform length, is this
>>       >>> possible in EPICS?
>>       >>>
>>       >>> Best Regards,
>>       >>>
>>       >>> Abdalla Ahmad
>>       >>>
>>       >>> Control Engineer
>>       >>>
>>       >>> SESAME
>>       >>>
>>       >>> Allan, Jordan.
>>       >>>
>>       >>> Tel: (+962-5) 3511348 , ext. 265
>>       >>>
>>       >>> Fax: (+962-5) 3511423
>>       >>>
>>       >>> Mob: (+962-7)88183296
>>       >>>
>>       >>> www.sesame.org.jo<http://www.sesame.org.jo> <http://www.sesame.org.jo>
>>      <http://www.sesame.org.jo/>
>>       >>>
>>