Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: Re: querying for sets of pvs in archived data
From: pthomas@caltech.edu
To: "Robert Soliday" <soliday@aps.anl.gov>
Cc: tech-talk@aps.anl.gov
Date: Mon, 1 Jul 2013 16:33:27 -0700 (PDT)
Thank you, that looks interesting. I'll take a closer look.

-Patrick

> The SDDS files that the data is stored in are not database files. Right
> now we don't have a single program to extract the data using queries
> like the one on the web page. We do have a program called
> sddsconvertlogonchange which we can use in combination with a few other
> programs for queries like these.
>
> sddsconvertlogonchange [<input-file>] [<output-file>]
>    [-pipe=[input][,output]]
>    [-binary] [-ascii]
>    [-double] [-float]
>    [-minimumInterval=<seconds>]
>    [-snapshot=<epochtime>]
>    [-time=[start=<epochtime>][,end=<epochtime>]]
>    [-delete={<matching-string>[,...]]
>    [-retain={<matching-string>[,...]]
>
>
> 1. Return the values of a set of variables at a given time:
>
> $ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out
> -snapshot=1372451200 | \
>    sddsconvert -pipe -retain=column,LTP:H1:CurrentAO,LTP:V1:CurrentAO | \
>    sdds2stream -pipe -columns=LTP:H1:CurrentAO,LTP:V1:CurrentAO
>
> 8.768076626281003e-02 -1.253228842690183e-01
>
>
> 2. Compare the values of a set of variables between two given times:
>
> Same as above except for down twice for two different timestamps. The
> script that would be running the program would then be used to do the
> comparison.
> Or you could run:
>
> $ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out
> -time=start=1372451000,end=1372451200 | sddsprocess -pipe
> -clip=1,1,invert -retain=column,LTP:H1:CurrentAO,LTP:V1:CurrentAO
> -process=LTP:H1:CurrentAO,spread,LTP:H1:CurrentAO_SPREAD
> -process=LTP:V1:CurrentAO,spread,LTP:V1:CurrentAO_SPREAD | sdds2stream
> -pipe -parameters=LTP:H1:CurrentAO_SPREAD,LTP:V1:CurrentAO_SPREAD
>
> 0.000000000000000e+00
> 1.060841567856212e-01
>
>
> 3. Return all the values of a set of variables between two given times:
>
> $ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out
> -time=start=1372395600,end=1372476700
> -retain=column,Time,LTP:H1:CurrentAO,LTP:V1:CurrentAO | sdds2stream
> -pipe -column=Time,LTP:H1:CurrentAO,LTP:V1:CurrentAO
>
> 1.372395604763000e+09 8.768076626281003e-02 -1.923872748339710e-02
> 1.372451188445000e+09 8.768076626281003e-02 -1.253228842690183e-01
> 1.372476694892000e+09 2.623497150728903e-01 -1.253228842690183e-01
>
>
>
> I could also extend the sddsconvertlogonchange to accept these queries
> directly.
>
>
> On 07/01/13 15:54, pthomas@caltech.edu wrote:
>> Hi Bob,
>>
>> Does the PV History Tool support queries like these:
>>
>> http://dba.stackexchange.com/questions/45438/choosing-a-database-type-for-recording-infrequent-changes-in-a-large-number-of-v
>>
>> Thank you,
>> Patrick
>>
>>> There is no online documentation for the PV History Tool. The script is
>>> contained in:
>>> http://www.aps.anl.gov/asd/oag/downloads/oag.1.20.tar.gz
>>> at oag/apps/src/tcltkapp/oagapp/PVHistoryTool
>>> If you install the OAGTclTk program first:
>>> http://www.aps.anl.gov/asd/oag/software.shtml#TclTk
>>> you should be able to run the PVHistoryTool script. You will probably
>>> also need the SDDSToolKit which is available on the same web page.
>>>
>>> You will need to set the OAGGlobal(LogOnChangeDirectory) variable to
>>> point to the directory containing the attached
>>> PVHistoryTool.config.1.sdds file that I am attaching. To do this you
>>> should edit the APSSetOAGGlobal.tcl file that comes with the OAGTclTk
>>> installation. Or you can set it directly inside the PVHistoryTool
>>> script.
>>>
>>> After starting the PVHistoryTool script you will select a beamline that
>>> has been predefined in the PVHistoryTool.config.1.sdds file. For
>>> example
>>> we have an LPL beamline. By selecting this it attempts to load
>>> $OAGGlobal(LogOnChangeDirectory)/LPL/LPL.loc which is the input file
>>> for
>>> our LPL sddslogonchange logger. I am attaching our file as an example.
>>>
>>> Our sddslogonchange is launched with the command:
>>>
>>> sddslogonchange LPL.loc LPL -dailyFiles  -loginitial
>>> -connectTimeout=120
>>>
>>> I will be attaching the PVHistoryTool.config.1.sdds and LPL.loc files
>>> in
>>> a separate email.
>>>
>>> On 07/01/13 13:54, pthomas@caltech.edu wrote:
>>>> Hi Bob,
>>>>
>>>> Thank you. I found the page for sddslogonchange. Is there a page for
>>>> the
>>>> PV History Tool as well?
>>>>
>>>> -Patrick
>>>>
>>>>> On 06/28/13 12:25, pthomas@caltech.edu wrote:
>>>>>> Hi Emmanuel,
>>>>>>
>>>>>> Thank you for your reply. I think there may be a difference though.
>>>>>> I
>>>>>> believe that the SCR applications use snapshots taken at periodic
>>>>>> intervals in time? What I had in mind was something that places
>>>>>> monitors
>>>>>> on the process variables and archives the values only when they
>>>>>> change.
>>>>>> The queries I described would then be done on that recorded data.
>>>>>>
>>>>>> MASAR does look like a nice program though.
>>>>>>
>>>>>> Thank you,
>>>>>> Patrick
>>>>>>
>>>>>>>> store and query data as I have described, but I am wondering if I
>>>>>>>> am
>>>>>>>> reinventing the wheel. Does a tool such as this already exist?
>>>>>>> Yes.
>>>>>>> You are describing a feature of Save/Compare/Restore applications.
>>>>>>> SCR are also know as epics-backup application (i.e take a snapshot
>>>>>>> of
>>>>>>> a
>>>>>>> running machine, compare 2 snapshots, compare a snapshot with the
>>>>>>> running
>>>>>>> machine)
>>>>>>>
>>>>>>> You next question is where to get such a piece of software. Well...
>>>>>>> I am using SDDS save/compare/restore (very old)
>>>>>>> SCORE may have the described feature as well (used at SNS and SLAC)
>>>>>>>
>>>>>>> <Shameless plug for MASAR>
>>>>>>> I am currently looking at MASAR part of epics-4, which is straight
>>>>>>> forward
>>>>>>> to install, but still in development.
>>>>>>> For example, comparisons of 2 snapshots using the pyQt GUI is
>>>>>>> missing.
>>>>>>> A CSS GUI is also a work-in-progress.
>>>>>>> At this time, most of the development is at the GUI level
>>>>>>> So if you don't mind a text output (i.e. in a terminal), you can
>>>>>>> use
>>>>>>> MASAR
>>>>>>> with its python API and have something working by tonight.
>>>>>>> </Shameless plug for MASAR>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> That sounds a lot like our sddslogonchange logger paired with our PV
>>>>> History Tool GUI. We have the logger running all the time and
>>>>> recording
>>>>> any changes to individual PVs with an optional tolerance level. We
>>>>> can
>>>>> then use our PV History Tool to plot the changes and to restore the
>>>>> PVs
>>>>> to any given time.
>>>>>
>>>>> --Bob Soliday
>>>>>
>>>>>
>>>
>>
>
>



References:
querying for sets of pvs in archived data pthomas
RE: querying for sets of pvs in archived data Emmanuel Mayssat
RE: querying for sets of pvs in archived data pthomas
Re: querying for sets of pvs in archived data Robert Soliday
Re: querying for sets of pvs in archived data pthomas
Re: querying for sets of pvs in archived data Robert Soliday
Re: querying for sets of pvs in archived data pthomas
Re: querying for sets of pvs in archived data Robert Soliday

Navigate by Date:
Prev: Re: querying for sets of pvs in archived data Robert Soliday
Next: Re: ChanneIRPC never connected White, Greg
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: querying for sets of pvs in archived data Robert Soliday
Next: Sub-arrays nick.rees
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·