RE: Epics Archiver Appliance Storage Question

["Manoussakis, Adamandios via Tech-talk" <tech-talk at aps.anl.gov>]

So I am using the site-specific install from the guide, the policies.py file had been updated with hardcoded paths.

 

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=/arch/sts/ArchiverStore&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

d01_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=8&gather=1'

d03_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=4&gather=1'

d07_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=8&gather=1'

d14_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=15&gather=1'

y01_plugin_url = 'pb://localhost?name=LTS&rootFolder=/arch/lts/ArchiverStore&partitionGranularity=PARTITION_YEAR'

bkh_plugin_url = 'blackhole://localhost?name=LTS'

 

I changed the STS path to where SHORT_TERM is just another directory for testing (~/Documents/sts/ArchiverStore), there is a file called archappl.conf where this is defined (I assume this is similar to your tomcat.conf).

 

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=${ ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

 

also tried just hardcoding it

 

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=/home/user1/Documents/sts/ArchiverStore&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

 

I have tried to stop the archiver and then rerun it but it doesn’t seem to pull in the new policies.py file when it is being run?  You mentioned that its being placed into the MYSQL database which makes sense but I would think when I delete the PV from the archiver and then place it back into the archiver that it would have the new path set for STS.

 

Thanks,

Adam

 

 

From: Wilson, Andy (DLSLtd,RAL,LSCI) <andrew.wilson at diamond.ac.uk>
Sent: Tuesday, October 12, 2021 1:18 AM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>; Manoussakis, Adamandios <manoussakis1 at llnl.gov>; Wilson, Andy (DLSLtd,RAL,LSCI) <andrew.wilson at diamond.ac.uk>
Subject: Re: Epics Archiver Appliance Storage Question

 

A further thought is - where are you setting the environment variables? We are doing it in tomcat.conf.

 

Andy

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Wilson, Andy (DLSLtd, RAL, LSCI) via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, October 12, 2021 09:13
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>; Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Subject: Re: Epics Archiver Appliance Storage Question

 

Hi Adam,

 

It appears that the storage locations are stored in the PVTypeinfo in the MySQL database, which are set in the policies.py file when you first archive a PV.

 

In the case of our install, the storage locations have the environment variables in them, like below, but if they are set to the literal locations in policies.py I guess they will need updating.

 

Snippet from database:

 

INSERT INTO `PVTypeInfo` VALUES ('<some_pv>,'{\"hostName\":\"<some_host>\",\"paused\":\"false\",\"creationTime\":\"2017-10-12T14:47:00.790Z\",\"lowerAlarmLimit\":\"NaN\",\"precision\":\"0.0\",\"lowerCtrlLimit\":\"-11.0\",\"units\":\"log\",\"computedBytesPerEvent\":\"20\",\"computedEventRate\":\"0.033333335\",\"usePVAccess\":\"false\",\"computedStorageRate\":\"0.6666667\",\"modificationTime\":\"2017-10-18T14:24:22.063Z\",\"upperDisplayLimit\":\"3.0\",\"upperWarningLimit\":\"-7.0\",\"DBRType\":\"DBR_SCALAR_DOUBLE\",\"dataStores\":[\"pb:\\/\\/localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_30MIN&consolidateOnShutdown=true\",\"pb:\\/\\/localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1\",\"pb:\\/\\/localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR\"], ...

 

Snippet from policies.py:

# We use the environment variables ARCHAPPL_SHORT_TERM_FOLDER and ARCHAPPL_MEDIUM_TERM_FOLDER to determine the location of the STS and the MTS in the appliance

shorttermstore_plugin_url = 'pb://localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

mediumtermstore_plugin_url = 'pb://localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1'

longtermstore_plugin_url = 'pb://localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR'

Cheers,

Andy

---

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Manoussakis, Adamandios via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, October 12, 2021 08:21
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Epics Archiver Appliance Storage Question

 

Thanks Abdalla, seems like it shouldn’t be too much trouble.  I have tried stopping the tomcat instances, making sure the environment variable is set correctly (I did see in the policies.py file where the paths are set).  I tried to restart the archiver appliance and also re-archived the pv (paused/deleted) but it still seems to be going to the old spot.  It looks like the file I am using is from Hans site-specific install where it was hardcoded the paths actually for sts/mts/lts.  I tried to just change the directory to see if it would work but still no success in getting the archiver to save the files in a different directory for sts.

 

Adam

 

From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Sent: Monday, October 11, 2021 11:43 PM
To: Manoussakis, Adamandios <manoussakis1 at llnl.gov>; EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Epics Archiver Appliance Storage Question

 

Hi Adam

 

I think it should be easy by changing the macro variables, I think we did this before and it was working, just make sure that you have read/write permissions on the new locations otherwise all your PVs will remain in the “Appliance Assigned” state. For your 2^nd question, STORAGE_TOP is optional, the three storage locations definitions are independent, it is not mandatory for the locations or mount points to share the same “STORAGE_TOP”, you can specify each store in the appropriate mount point in your setup.

 

If you have enough RAM on your server, it is best practice to setup the STS using a RAM disk (the tempfs filesystem), this will improve performance on data retrieval for recent periods.

 

Best Regards,

 

Abdalla Al-Dalleh

Control Engineer, SESAME

Phone: +962-788183296

Email: Abdalla.ahmad at sesame.org.jo

 

From: Manoussakis, Adamandios via Tech-talk
Sent: Tuesday, October 12, 2021 1:57 AM
To: EPICS tech-talk
Subject: Epics Archiver Appliance Storage Question

 

Hey All,

 

I am planning on changing where our STS MTS and LTS are stored.  Most likely will use STS MTS on the local SSD I have but wanted to move our LTS over to a NAS or some external storage.  Wanted to know if there are any pitfalls to setting this up or is at as easy as changing up these values in the archappl.conf?  Also was wondering about what STORAGE_TOP does if say my LONG_TERM_FOLDER does not share the same STORAGE_TOP directory for example?

 

ARCHAPPL_STORAGE_TOP="/arch"

ARCHAPPL_SHORT_TERM_FOLDER="/arch/sts/ArchiverStore"

ARCHAPPL_MEDIUM_TERM_FOLDER="/arch/mts/ArchiverStore"

ARCHAPPL_LONG_TERM_FOLDER="/Network/lts/StorageExample "

 

Thanks,

Adam

 

 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom