1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 <2018> 2019 2020 2021 2022 2023 2024 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 <2018> 2019 2020 2021 2022 2023 2024 |
<== Date ==> | <== Thread ==> |
---|
Subject: | Archiver Appliance: merging STS, MTS and LTS pb data from a backup to a live system |
From: | Rory Clarke - UKRI STFC via Tech-talk <[email protected]> |
To: | "[email protected]" <[email protected]> |
Date: | Thu, 13 Dec 2018 19:16:01 +0000 |
Hi, I am trying to merge backup versions of the STS, MTS and LTS data directories to a current Archiver Appliance. The LTS storage was corrupted during a disk upgrade and data was lost. I have a backup of the STS, MTS and LTS directories from that point and would like to merge this data with the live data that is being taken now. I used the pbutils scripts in the mgmt/install folder to scan the .pb file of the STS backup and the LTS files of the current store. The backup is a week older then the oldest live data so there would be no overlap. At first I thought of joining the backup L/M/STS files together and then adding them on to the LTS files in the live archiver. I tried several things. I merged the partitions in the backup STS and MTS folders together OK but saw the MTS
dates overlapped the LTS dates. This can be fixed with the repair.sh script to fix this but is too slow for so much data. I could ultimately join the merged backup with the live LTS files however I think I am lacking the tools to do this in a clean way. Next I tried to add the backup data stores to the Archiver Appliance. I tried to defined a new stores in a policies.py file shown below: # We use the environment variables ARCHAPPL_SHORT_TERM_FOLDER and ARCHAPPL_MEDIUM_TERM_FOLDER to determine the location of the STS and the MTS in the appliance
shorttermstore_plugin_url = 'pb://localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true' mediumtermstore_plugin_url = 'pb://localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1' longtermstore_plugin_url = 'pb://localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR&etlOutofStoreIf=false' #Backup store backup_short_plugin_url = 'pb://localhost?name=BACKSTS&rootFolder=${ARCHAPPL_BACKUP_STS_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR' backup_med_plugin_url = 'pb://localhost?name=BACKMED&rootFolder=${ARCHAPPL_BACKUP_MTS_TERM_FOLDER}&partitionGranularity=PARTITION_DAY' backup_long_plugin_url = 'pb://localhost?name=BACKLTS&rootFolder=${ARCHAPPL_BACKUP_LTS_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR' And added then to the default policy (I think) else: pvPolicyDict['samplingPeriod'] = 1.0 pvPolicyDict['samplingMethod'] = 'MONITOR' pvPolicyDict['dataStores'] = [ shorttermstore_plugin_url, mediumtermstore_plugin_url, longtermstore_plugin_url, backup_short_plugin_url, backup_med_plugin_url, backup_long_plugin_url ] pvPolicyDict['policyName'] = 'Default'; However, I still only have the original STS, MTS, LTS and no BACKSTS, BACKMTS, BACKLTS stores in the stores webpage and the backup data is missing. I can see the backup LTS data if I swap it with the live LTS but can’t seem to add stores I’m wondering what is the best way to recover these backup pb files and add them to the current live archiver. Regards, Rory |