Re: Save/Compare/Restore Application

["Shen, Guobao" <shengb@bnl.gov>]

Hi Emmanuel,

Thanks for you comments. Let me explain some details before I answer your questions.

The MASAR service is implemented to take machine snapshot, save data for future usage,

and retrieve data from database. To simplify the server part, the restore and compare function

is not done in the server since each facility has different restoring procedure, and different algorithm to compare different data set.

Especially for restoring, in many case, user has to be very careful for example some pvs have to be set first, or only under certain condition, we can restore set of pvs. Those should be either embedded into IOC, or implemented as a standalone server.

Fr the client apps, my original goal was to provide a narrow API set, and allow users to manipulate the data in their favorite way. But we try to collect all requirements, and are flexible to expand the client API to make it more user friendly.

We are actively maintaining MASAR server, and improving it here at NSLS II project.

Please see my comments inline.

On Feb 5, 2013, at 8:38 PM, "Emmanuel Mayssat" <emayssat@outlook.com> wrote:

MASAR is awesome !

Thanks.

I have a few questions/comments though.

Q: What is the purpose of the comment tab? (The tab the pyqt application default to)

Although we put the MASAR server in production, but the UI is not unfortunately.

Our plan is to use CSS as our UI, therefore, my focusing for MASAR client is to provide an API.

The PyQt app was originally developed for my development and testing purpose. Since the

development in CSS part is delayed, I distributed the UI with the server for now.

The purpose of the comment tab is to give user a place to add some comments.

It is still a place holder for future improvement.

Q: If files are backed up in sqlite db, how does that db scale with time?

What about saving in several sqlite files? (Let's say one per system? one file per year/month/day? one file per backup?)?

If I use only 1 sqlite file and let the system run for 5 years, am I going to have problems with performance or data management?

Very good point. I did not do very details benchmark myself, but read some articles that with careful tuning, it still has good performance when the file size blows up to a couple GB.

I do have some slides to show our experience we got during our beam commissioning.

Our database grow to 10MB roughly within 3 months. There were over 350 snapshots,

and one configuration has about 600 pvs.

I can send you my slides if you are interesting.

Saving data into few SQLite file might be one approach, also we are considering to merge our database 

to IRMIS database, which is using MySQL currently.

Another alternative approach on my todo list is to save data in hdf5 file, and have an link/index in SQLite database. With this approach, SQLite should be sufficient.

Not decide which direction I should go. But this is on my list.

Q: Are the button labelled correctly?

[fetch events] ... fetches snapshots   ( <-- my operators (and I ) don't know what an event is ...)

[fetch snapshots]... display snapshot, i.e pv values of selected snapshot

No. Definitely not. As I mentioned above, I was and am still hoping CSS can kick in here.

It has much better widgets to do all the jobs provide by my PyQt app. Honestly, I am not a GUI expert.

Anyway, let me explain some term I used in the PyQt app:

Event: each snapshot has real data from IOC, and associated with some head to explain what the event is for. It is like head + data.

An event is the head part, which indicates a snapshot event happened.

Snapshot: this is the real data part. 

The reason I separate them is because we always want to look at the head first before getting the data.

[save to file] ... export snapshot to csv, but do not save a snapshot

Sorry for that. You are right. Here it does mean to export current displayed snapshot to a file.

Q: Is there a way to select the fields that are displayed in the viewer?

SCR applications trend to not be user-friendly. making the UI configurable would be awesome! 

As I mentioned above, CSS has this functionality and widget already. 

For PyQt app, it is doable, but I did not get time to do it.

Q: Some PVs should never be restored (ie input or special output record), is there a way configure a PV to never be restored?

Currently, you can choose those pvs manually in the PyQt app.

If you want this to be done automatically, an entry in database has to be added.

But what to do if you change your mind once you flag that pvs permanently?

Or in some machine mode, operation mode for example, those pvs should never be restored,

but in other mode, physics study mode for example, physicists do want to restore them?

If some pvs should never be restored, will it better to archive those pvs with channel archiever?

Q: When doing the diff between the current configuration and the running machine, is there a way to allow for noise in the reading?

Example: A ai record reports a noisy voltage. I want the software to report that 1.1 Volt is the same as 1 Volt

Yes. Currently I provide a very low level API set. But a user friendly API would be nice to host those function on top of the low level API. Will add this to my todo list.

Q: Can you start the MASAR server without a terminal?

Yes. There are some ways to run it as a daemon. The first approach in my mind is to use procserv.

Q: Is MASAR still actively developed? Is anyone else using it?

Yes. Here at NSLS II, we have been using it for our linac beam commissioning. We will continue to use it for our whole facility.

As far as I know, FRIB/MSU is planning to use it also.

Guobao

Regards,

--

Emmanuel

---

Date: Wed, 30 Jan 2013 13:28:45 -0500
From: shengb@bnl.gov
To: emayssat@outlook.com
Subject: Re: Save/Compare/Restore Application
CC: tech-talk@aps.anl.gov

Hi Emmanuel,
MASAR is developed with all tech you mentioned: Python, PyQt, SQLite.
The server side uses SQLite as back-end data storage engine. All configurations and data are stored in SQLite.
The access to SQLite is thru a python interface, which I called it pymasar. With pymasar, although it is not the
desired way, but if you wish, you are able to play with the MASAR RDB directly.

The client side provides Python interface for scripting users. A GUI is also available, which is based on the client
Python interface and developed with PyQt.

The comparing is inside the Client side.
The restore function is added in the PyQt part, which needs cothread module.

Here is a list of modules dependency:
 - EPICS V3 (most likely, libCom)
 - EPICS V4 (pvData, pvAccess, pvIOC, and normativetype)
 - SQLite
 - Python 2.7.x
 - PyQt

According the requirement you mentioned in your previous, I believe we can do it in the client interface.
I am very interesting to work on that.

Guobao

On 1/30/13 1:00 PM, Emmanuel Mayssat wrote:

Guabao,

I found the mercurial repository (although for some reason I cannot access it, but that maybe on my side)

Is EPICS V4 required? 

I have an EPICS V3.14.11 installation....

--

E

---

From: emayssat@outlook.com
To: shengb@bnl.gov; emmanuel_mayssat@lynceantech.com
Subject: RE: Save/Compare/Restore Application
Date: Wed, 30 Jan 2013 09:51:49 -0800
CC: tech-talk@aps.anl.gov

Guabao,
SCORE is Java based and here we are using .... python (as well as Qt, pyqt, SQLlite, and Fedora 16)!

So our preference go to MASAR, but....

I looked at the documentation and noticed some interesting features.

Assuming I have a signal with white noise that is captured in a ai record,

if I compare its current and previously saved value, there are likely to be different.

Is there a way to say that for that PV, if the value is within 3 standard dev (configurable per PV) then do not report the PV as different?

Are the sources MASAR available?

Is this software packaged and ready to be distributed?

If so, where is a download page?

Regards,

--

Emmanuel

---

From: shengb@bnl.gov
To: emayssat@outlook.com
Subject: Re: Save/Compare/Restore Application
Date: Wed, 30 Jan 2013 03:27:00 +0000
CC: tech-talk@aps.anl.gov

Hi Emmanuel,

Here at NSLS II, we have developed a tool for this purpose, which I named it MASAR, and are using it for our beam commissioning.

It is capable to take a snapshot with any pv combination, for example with scalar pv and waveform pv.

An online user manual can be found at 

http://epics-pvdata.hg.sourceforge.net/hgweb/epics-pvdata/masarService/raw-file/tip/documentation/userManual.html

There are sections to explain how to compile everything from source, and run a demo.

I have all binary packages against Debian 7. If you want them, I can send to you.

Guobao

On Jan 29, 2013, at 8:00 PM, "Emmanuel Mayssat" <emayssat@outlook.com> wrote:

Hello, 

I am exploring save/compare/restore applications out there as I intend to replace ours.

To date, I am only aware of SCORE used at ORNL. 

Is anyone on this mailing list maintaining/developing/administrating SCORE?

Is anybody using something else? If so, what? Is there a project web page?

Regards,

--

Emmanuel