Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
<== Date ==> <== Thread ==>

Subject: Re: querying for sets of pvs in archived data
From: Robert Soliday <soliday@aps.anl.gov>
To: pthomas@caltech.edu
Cc: tech-talk@aps.anl.gov
Date: Mon, 01 Jul 2013 16:43:41 -0500
The SDDS files that the data is stored in are not database files. Right now we don't have a single program to extract the data using queries like the one on the web page. We do have a program called sddsconvertlogonchange which we can use in combination with a few other programs for queries like these.

sddsconvertlogonchange [<input-file>] [<output-file>]
  [-pipe=[input][,output]]
  [-binary] [-ascii]
  [-double] [-float]
  [-minimumInterval=<seconds>]
  [-snapshot=<epochtime>]
  [-time=[start=<epochtime>][,end=<epochtime>]]
  [-delete={<matching-string>[,...]]
  [-retain={<matching-string>[,...]]


1. Return the values of a set of variables at a given time:

$ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out -snapshot=1372451200 | \
  sddsconvert -pipe -retain=column,LTP:H1:CurrentAO,LTP:V1:CurrentAO | \
  sdds2stream -pipe -columns=LTP:H1:CurrentAO,LTP:V1:CurrentAO

8.768076626281003e-02 -1.253228842690183e-01


2. Compare the values of a set of variables between two given times:

Same as above except for down twice for two different timestamps. The script that would be running the program would then be used to do the comparison.
Or you could run:

$ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out -time=start=1372451000,end=1372451200 | sddsprocess -pipe -clip=1,1,invert -retain=column,LTP:H1:CurrentAO,LTP:V1:CurrentAO -process=LTP:H1:CurrentAO,spread,LTP:H1:CurrentAO_SPREAD -process=LTP:V1:CurrentAO,spread,LTP:V1:CurrentAO_SPREAD | sdds2stream -pipe -parameters=LTP:H1:CurrentAO_SPREAD,LTP:V1:CurrentAO_SPREAD

0.000000000000000e+00
1.060841567856212e-01


3. Return all the values of a set of variables between two given times:

$ sddsconvertlogonchange LPL-2013-179-0628 -pipe=out -time=start=1372395600,end=1372476700 -retain=column,Time,LTP:H1:CurrentAO,LTP:V1:CurrentAO | sdds2stream -pipe -column=Time,LTP:H1:CurrentAO,LTP:V1:CurrentAO

1.372395604763000e+09 8.768076626281003e-02 -1.923872748339710e-02
1.372451188445000e+09 8.768076626281003e-02 -1.253228842690183e-01
1.372476694892000e+09 2.623497150728903e-01 -1.253228842690183e-01



I could also extend the sddsconvertlogonchange to accept these queries directly.


On 07/01/13 15:54, pthomas@caltech.edu wrote:
Hi Bob,

Does the PV History Tool support queries like these:

http://dba.stackexchange.com/questions/45438/choosing-a-database-type-for-recording-infrequent-changes-in-a-large-number-of-v

Thank you,
Patrick

There is no online documentation for the PV History Tool. The script is
contained in:
http://www.aps.anl.gov/asd/oag/downloads/oag.1.20.tar.gz
at oag/apps/src/tcltkapp/oagapp/PVHistoryTool
If you install the OAGTclTk program first:
http://www.aps.anl.gov/asd/oag/software.shtml#TclTk
you should be able to run the PVHistoryTool script. You will probably
also need the SDDSToolKit which is available on the same web page.

You will need to set the OAGGlobal(LogOnChangeDirectory) variable to
point to the directory containing the attached
PVHistoryTool.config.1.sdds file that I am attaching. To do this you
should edit the APSSetOAGGlobal.tcl file that comes with the OAGTclTk
installation. Or you can set it directly inside the PVHistoryTool script.

After starting the PVHistoryTool script you will select a beamline that
has been predefined in the PVHistoryTool.config.1.sdds file. For example
we have an LPL beamline. By selecting this it attempts to load
$OAGGlobal(LogOnChangeDirectory)/LPL/LPL.loc which is the input file for
our LPL sddslogonchange logger. I am attaching our file as an example.

Our sddslogonchange is launched with the command:

sddslogonchange LPL.loc LPL -dailyFiles  -loginitial  -connectTimeout=120

I will be attaching the PVHistoryTool.config.1.sdds and LPL.loc files in
a separate email.

On 07/01/13 13:54, pthomas@caltech.edu wrote:
Hi Bob,

Thank you. I found the page for sddslogonchange. Is there a page for the
PV History Tool as well?

-Patrick

On 06/28/13 12:25, pthomas@caltech.edu wrote:
Hi Emmanuel,

Thank you for your reply. I think there may be a difference though. I
believe that the SCR applications use snapshots taken at periodic
intervals in time? What I had in mind was something that places
monitors
on the process variables and archives the values only when they
change.
The queries I described would then be done on that recorded data.

MASAR does look like a nice program though.

Thank you,
Patrick

store and query data as I have described, but I am wondering if I am
reinventing the wheel. Does a tool such as this already exist?
Yes.
You are describing a feature of Save/Compare/Restore applications.
SCR are also know as epics-backup application (i.e take a snapshot of
a
running machine, compare 2 snapshots, compare a snapshot with the
running
machine)

You next question is where to get such a piece of software. Well...
I am using SDDS save/compare/restore (very old)
SCORE may have the described feature as well (used at SNS and SLAC)

<Shameless plug for MASAR>
I am currently looking at MASAR part of epics-4, which is straight
forward
to install, but still in development.
For example, comparisons of 2 snapshots using the pyQt GUI is
missing.
A CSS GUI is also a work-in-progress.
At this time, most of the development is at the GUI level
So if you don't mind a text output (i.e. in a terminal), you can use
MASAR
with its python API and have something working by tonight.
</Shameless plug for MASAR>




That sounds a lot like our sddslogonchange logger paired with our PV
History Tool GUI. We have the logger running all the time and recording
any changes to individual PVs with an optional tolerance level. We can
then use our PV History Tool to plot the changes and to restore the PVs
to any given time.

--Bob Soliday






Replies:
Re: querying for sets of pvs in archived data pthomas
References:
querying for sets of pvs in archived data pthomas
RE: querying for sets of pvs in archived data Emmanuel Mayssat
RE: querying for sets of pvs in archived data pthomas
Re: querying for sets of pvs in archived data Robert Soliday
Re: querying for sets of pvs in archived data pthomas
Re: querying for sets of pvs in archived data Robert Soliday
Re: querying for sets of pvs in archived data pthomas

Navigate by Date:
Prev: asyn+streamDevice hangs with synchronousLock:Yes Brian McAllister
Next: Re: querying for sets of pvs in archived data pthomas
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
Navigate by Thread:
Prev: Re: querying for sets of pvs in archived data pthomas
Next: Re: querying for sets of pvs in archived data pthomas
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·