Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
<== Date ==> <== Thread ==>

Subject: Re: Merging store STS/MTS/LTS data from a live system with a backup
From: "Shankar, Murali via Tech-talk" <tech-talk@aps.anl.gov>
To: Rory Clarke - UKRI STFC <rory.clarke@stfc.ac.uk>, "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Thu, 13 Dec 2018 19:51:10 +0000

You should be able to "merge" the backup data into the live data; I don't think I have a merge script but it should be simple to get you a utility to do this. It might take me a day to test this out; what this will do is take two absolute paths as arguments and merge the contents of one file into another. As you do not have overlap, I think we should be able to simply append the "newer" file to the back of the "older" one after making sure that the timestamps etc are ok. 


If you think this will help, let me know and I'll generate and test a script for you. I'll try to get this to you by the end of this week. 



Regards,

Murali






From: Rory Clarke - UKRI STFC <rory.clarke@stfc.ac.uk>
Sent: Thursday, December 13, 2018 11:26:01 AM
To: Shankar, Murali
Subject: Merging store STS/MTS/LTS data from a live system with a backup
 

 

I am trying to merge backup versions of the STS, MTS and LTS data directories to a current Archiver Appliance. The LTS storage was corrupted during a disk upgrade and data was lost. I have a backup of the STS, MTS and LTS directories from that point and would like to merge this data with the live data that is being taken now. There is currently a month’s work of data and a backup of over a year’s worth.

 

I only have one archiver at the moment.

 

I used the pbutils scripts in the mgmt/install folder to scan the .pb file of the STS backup and the LTS files of the current store. The backup is a week older then the oldest live data so there would be no overlap.

 

At first I thought of joining the backup L/M/STS files together and then adding them on to the LTS files in the live archiver. I merged the partitions in the backup STS and MTS folders together OK but saw the MTS dates overlapped when merging the LTS dates. This can be fixed with the repair.sh script to fix this but is too slow for so much data. I could ultimately join the merged backup with the live LTS files however I think I am lacking the tools to do this in a clean way.

 

Next I tried to add the backup data stores to the Archiver Appliance. I tried to defined a new stores in a policies.py file shown below:

 

  # We use the environment variables ARCHAPPL_SHORT_TERM_FOLDER and ARCHAPPL_MEDIUM_TERM_FOLDER to determine the location of the STS and the MTS in the appliance 

  shorttermstore_plugin_url =  'pb://localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

  mediumtermstore_plugin_url = 'pb://localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1'

  longtermstore_plugin_url =   'pb://localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR&etlOutofStoreIf=false'

  #Backup store

  backup_short_plugin_url =    'pb://localhost?name=BACKSTS&rootFolder=${ARCHAPPL_BACKUP_STS_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR'

  backup_med_plugin_url =      'pb://localhost?name=BACKMED&rootFolder=${ARCHAPPL_BACKUP_MTS_TERM_FOLDER}&partitionGranularity=PARTITION_DAY'

  backup_long_plugin_url =     'pb://localhost?name=BACKLTS&rootFolder=${ARCHAPPL_BACKUP_LTS_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR'

 

And added then to the default policy (I think)

 

 

        else:

                pvPolicyDict['samplingPeriod'] = 1.0

                pvPolicyDict['samplingMethod'] = 'MONITOR'

                pvPolicyDict['dataStores'] = [

                        shorttermstore_plugin_url,

                        mediumtermstore_plugin_url,

                        longtermstore_plugin_url,

                        backup_short_plugin_url,

                        backup_med_plugin_url,

                        backup_long_plugin_url

                        ]

                pvPolicyDict['policyName'] = 'Default';

 

However, I still only have the original STS, MTS, LTS and no BACKSTS, BACKMTS, BACKLTS stores in the stores webpage and the backup data is missing. I can see the backup LTS data if I swap it with the live LTS but can’t seem to add stores

 

I’m wondering what is the best way to recover these backup pb files and add them to the current live archiver.

 

Regards,

 

Rory  

 


Navigate by Date:
Prev: RE: FW: Build failure when compiling synApps from source Iain Marcuson via Tech-talk
Next: Re: FW: Build failure when compiling synApps from source Jemian, Pete R. via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
Navigate by Thread:
Prev: Archiver Appliance: merging STS, MTS and LTS pb data from a backup to a live system Rory Clarke - UKRI STFC via Tech-talk
Next: csm module support or alternative Davide Marcato via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
ANJ, 14 Dec 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·