Experimental Physics and
Industrial Control System

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 <2021> 2022 2023 2024	Index	1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 <2021> 2022 2023 2024
<== Date ==>		<== Thread ==>

Subject:	Re: Epics Archiver Appliance Storage Question
From:	"Wilson, Andy $DLSLtd, RAL, LSCI$ via Tech-talk" <tech-talk at aps.anl.gov>
To:	"Manoussakis, Adamandios" <manoussakis1 at llnl.gov>, Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>, EPICS tech-talk <tech-talk at aps.anl.gov>
Date:	Wed, 13 Oct 2021 14:44:00 +0000

I would think when I delete the PV from the archiver and then place it back into the archiver that it would have the new path set for STS.

Yes this is what I would expect to happen too, if you have updated the policies.py file, then removed the PV and added it again for archiving.

it doesn’t seem to pull in the new policies.py file when it is being run?

I'm afraid I'm not familiar with the site-specific install in the guide, so my advice would just be to double-check that the copy of policies.py you are editing is the one that is being used. It looks from the file below like the install puts it in /opt/epicsarchiverap, so you could check that the file here is being updated as you expect.

https://github.com/jeonghanlee/epicsarchiverap-env/blob/ef43e8d9a4ca5846e2516527f9db1eccc52a1401/configure/CONFIG_SITE

To check the store URLs that have been applied for a PV you can either look directly in the database or request the PVTypeInfo for a PV using the mgmt BPL.

Andy

From: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Sent: Tuesday, October 12, 2021 19:38
To: Wilson, Andy (DLSLtd,RAL,LSCI) <andrew.wilson at diamond.ac.uk>; Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Epics Archiver Appliance Storage Question

So I am using the site-specific install from the guide, the policies.py file had been updated with hardcoded paths.

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=/arch/sts/ArchiverStore&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

d01_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=8&gather=1'

d03_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=4&gather=1'

d07_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=8&gather=1'

d14_plugin_url = 'pb://localhost?name=MTS&rootFolder=/arch/mts/ArchiverStore&partitionGranularity=PARTITION_DAY&hold=15&gather=1'

y01_plugin_url = 'pb://localhost?name=LTS&rootFolder=/arch/lts/ArchiverStore&partitionGranularity=PARTITION_YEAR'

bkh_plugin_url = 'blackhole://localhost?name=LTS'

I changed the STS path to where SHORT_TERM is just another directory for testing (~/Documents/sts/ArchiverStore), there is a file called archappl.conf where this is defined (I assume this is similar to your tomcat.conf).

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=${ ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

also tried just hardcoding it

h01_plugin_url = 'pb://localhost?name=STS&rootFolder=/home/user1/Documents/sts/ArchiverStore&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

I have tried to stop the archiver and then rerun it but it doesn’t seem to pull in the new policies.py file when it is being run? You mentioned that its being placed into the MYSQL database which makes sense but I would think when I delete the PV from the archiver and then place it back into the archiver that it would have the new path set for STS.

Thanks,

Adam

From: Wilson, Andy (DLSLtd,RAL,LSCI) <andrew.wilson at diamond.ac.uk>
Sent: Tuesday, October 12, 2021 1:18 AM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>; Manoussakis, Adamandios <manoussakis1 at llnl.gov>; Wilson, Andy (DLSLtd,RAL,LSCI) <andrew.wilson at diamond.ac.uk>
Subject: Re: Epics Archiver Appliance Storage Question

A further thought is - where are you setting the environment variables? We are doing it in tomcat.conf.

Andy

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Wilson, Andy (DLSLtd, RAL, LSCI) via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, October 12, 2021 09:13
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>; Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Subject: Re: Epics Archiver Appliance Storage Question

Hi Adam,

It appears that the storage locations are stored in the PVTypeinfo in the MySQL database, which are set in the policies.py file when you first archive a PV.

In the case of our install, the storage locations have the environment variables in them, like below, but if they are set to the literal locations in policies.py I guess they will need updating.

Snippet from database:

INSERT INTO `PVTypeInfo` VALUES ('<some_pv>,'{\"hostName\":\"<some_host>\",\"paused\":\"false\",\"creationTime\":\"2017-10-12T14:47:00.790Z\",\"lowerAlarmLimit\":\"NaN\",\"precision\":\"0.0\",\"lowerCtrlLimit\":\"-11.0\",\"units\":\"log\",\"computedBytesPerEvent\":\"20\",\"computedEventRate\":\"0.033333335\",\"usePVAccess\":\"false\",\"computedStorageRate\":\"0.6666667\",\"modificationTime\":\"2017-10-18T14:24:22.063Z\",\"upperDisplayLimit\":\"3.0\",\"upperWarningLimit\":\"-7.0\",\"DBRType\":\"DBR_SCALAR_DOUBLE\",\"dataStores\":[\"pb:\\/\\/localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_30MIN&consolidateOnShutdown=true\",\"pb:\\/\\/localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1\",\"pb:\\/\\/localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR\"], ...

Snippet from policies.py:

# We use the environment variables ARCHAPPL_SHORT_TERM_FOLDER and ARCHAPPL_MEDIUM_TERM_FOLDER to determine the location of the STS and the MTS in the appliance

shorttermstore_plugin_url = 'pb://localhost?name=STS&rootFolder=${ARCHAPPL_SHORT_TERM_FOLDER}&partitionGranularity=PARTITION_HOUR&consolidateOnShutdown=true'

mediumtermstore_plugin_url = 'pb://localhost?name=MTS&rootFolder=${ARCHAPPL_MEDIUM_TERM_FOLDER}&partitionGranularity=PARTITION_DAY&hold=2&gather=1'

longtermstore_plugin_url = 'pb://localhost?name=LTS&rootFolder=${ARCHAPPL_LONG_TERM_FOLDER}&partitionGranularity=PARTITION_YEAR'

Cheers,

Andy

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of Manoussakis, Adamandios via Tech-talk <tech-talk at aps.anl.gov>
Sent: Tuesday, October 12, 2021 08:21
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Epics Archiver Appliance Storage Question

Thanks Abdalla, seems like it shouldn’t be too much trouble. I have tried stopping the tomcat instances, making sure the environment variable is set correctly (I did see in the policies.py file where the paths are set). I tried to restart the archiver appliance and also re-archived the pv (paused/deleted) but it still seems to be going to the old spot. It looks like the file I am using is from Hans site-specific install where it was hardcoded the paths actually for sts/mts/lts. I tried to just change the directory to see if it would work but still no success in getting the archiver to save the files in a different directory for sts.

Adam

From: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Sent: Monday, October 11, 2021 11:43 PM
To: Manoussakis, Adamandios <manoussakis1 at llnl.gov>; EPICS tech-talk <tech-talk at aps.anl.gov>
Subject: RE: Epics Archiver Appliance Storage Question

Hi Adam

I think it should be easy by changing the macro variables, I think we did this before and it was working, just make sure that you have read/write permissions on the new locations otherwise all your PVs will remain in the “Appliance Assigned” state. For your 2^nd question, STORAGE_TOP is optional, the three storage locations definitions are independent, it is not mandatory for the locations or mount points to share the same “STORAGE_TOP”, you can specify each store in the appropriate mount point in your setup.

If you have enough RAM on your server, it is best practice to setup the STS using a RAM disk (the tempfs filesystem), this will improve performance on data retrieval for recent periods.

Best Regards,

Abdalla Al-Dalleh

Control Engineer, SESAME

Phone: +962-788183296

Email: Abdalla.ahmad at sesame.org.jo

From: Manoussakis, Adamandios via Tech-talk
Sent: Tuesday, October 12, 2021 1:57 AM
To: EPICS tech-talk
Subject: Epics Archiver Appliance Storage Question

Hey All,

I am planning on changing where our STS MTS and LTS are stored. Most likely will use STS MTS on the local SSD I have but wanted to move our LTS over to a NAS or some external storage. Wanted to know if there are any pitfalls to setting this up or is at as easy as changing up these values in the archappl.conf? Also was wondering about what STORAGE_TOP does if say my LONG_TERM_FOLDER does not share the same STORAGE_TOP directory for example?

ARCHAPPL_STORAGE_TOP="/arch"

ARCHAPPL_SHORT_TERM_FOLDER="/arch/sts/ArchiverStore"

ARCHAPPL_MEDIUM_TERM_FOLDER="/arch/mts/ArchiverStore"

ARCHAPPL_LONG_TERM_FOLDER="/Network/lts/StorageExample "

Thanks,

Adam

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

References:: Epics Archiver Appliance Storage Question Manoussakis, Adamandios via Tech-talk; RE: Epics Archiver Appliance Storage Question Abdalla Ahmad via Tech-talk; RE: Epics Archiver Appliance Storage Question Manoussakis, Adamandios via Tech-talk; Re: Epics Archiver Appliance Storage Question Wilson, Andy (DLSLtd, RAL, LSCI) via Tech-talk; Re: Epics Archiver Appliance Storage Question Wilson, Andy (DLSLtd, RAL, LSCI) via Tech-talk; RE: Epics Archiver Appliance Storage Question Manoussakis, Adamandios via Tech-talk

Navigate by Date:: Prev: RE: Epics Archiver Appliance Storage Question Shankar, Murali via Tech-talk; Next: JAI problem using ADAravis Steve Kinder - STFC UKRI via Tech-talk; Index: 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 <2021> 2022 2023 2024
Navigate by Thread:: Prev: RE: Epics Archiver Appliance Storage Question Manoussakis, Adamandios via Tech-talk; Next: RE: Epics Archiver Appliance Storage Question Shankar, Murali via Tech-talk; Index: 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 <2021> 2022 2023 2024

ANJ, 13 Oct 2021

· Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System