Experimental Physics and
Industrial Control System

Sky Brewer via Tech-talk <tech-talk at aps.anl.gov> · Thu, 2 May 2024 06:57:34 +0000

I think creation of whole files in the LTS which aren't overlapping with existing data should work fine. 

With the LTS if it doesn't work, there should be low risk of problems. It should be pretty easy to test with a test archiver as well. 

If the PVs exists now I would make sure to do a Pause and consolidate before copying any data to make sure the ETL doesn't conflict with what you're doing.

Hope that helps.
Sky

On 2024-04-30, 16:47, "Michael Davidsaver" <mdavidsaver at gmail.com <mailto:mdavidsaver at gmail.com>> wrote:

On 4/29/24 08:19, Tynan Ford via Tech-talk wrote:
> Hi Sky,
> 
> We have wondered if it's possible to append data to the appliance pb files since we have data from previous archivers that goes back many years. It would be useful to have that data imported into the appliance so all archived data can be retrieved via the same method. I naively thought we could write a script to create pb files like TESTPV:2001.pb, TESTPV:2002.pb, etc but I'm curious what you think?

It is probably suffering from bit-rot at this point,
but my old Channel Archiver to Archiver Appliance
direct file converter is still available:

https: //github.com/mdavidsaver/ca2aa

> Thanks
> Tynan
> 
> On Wed, Apr 24, 2024 at 2:46 AM Sky Brewer via Tech-talk <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov> <mailto:tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>>> wrote:
> 
> Hi Windsor,____
> 
> __ __
> 
> Some examples of python code for handling the pb files can be found at:____
> 
> __ __
> 
> https://urldefense.us/v3/__https://github.com/zhangt58/PyArchAppl/blob/master/main/data/pb/decode.py__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFbN8kGPM$  <https://urldefense.us/v3/__https://github.com/zhangt58/PyArchAppl/blob/master/main/data/pb/decode.py__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFbN8kGPM$ > <https://urldefense.us/v3/__https://github.com/zhangt58/PyArchAppl/blob/master/main/data/pb/decode.py__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omsWogAg71$ <https://urldefense.us/v3/__https://github.com/zhangt58/PyArchAppl/blob/master/main/data/pb/decode.py__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omsWogAg71$>> ____
> 
> https://urldefense.us/v3/__https://github.com/dls-controls/aapy/blob/master/aa/pb.py__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFU1GluYQ$  <https://urldefense.us/v3/__https://github.com/dls-controls/aapy/blob/master/aa/pb.py__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFU1GluYQ$ > <https://urldefense.us/v3/__https://github.com/dls-controls/aapy/blob/master/aa/pb.py__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omsZN2X7I9$ <https://urldefense.us/v3/__https://github.com/dls-controls/aapy/blob/master/aa/pb.py__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omsZN2X7I9$>> ____
> 
> and ____
> 
> https://urldefense.us/v3/__https://gitlab.esss.lu.se/ics-software/py-epicsarchiver/-/blob/master/epicsarchiver/pb.py?ref_type=heads__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFTvdUss5$  <https://urldefense.us/v3/__https://gitlab.esss.lu.se/ics-software/py-epicsarchiver/-/blob/master/epicsarchiver/pb.py?ref_type=heads__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFTvdUss5$ > <https://urldefense.us/v3/__https://gitlab.esss.lu.se/ics-software/py-epicsarchiver/-/blob/master/epicsarchiver/pb.py?ref_type=heads__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omseE1GJdN$ <https://urldefense.us/v3/__https://gitlab.esss.lu.se/ics-software/py-epicsarchiver/-/blob/master/epicsarchiver/pb.py?ref_type=heads__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omseE1GJdN$>> ____
> 
> __ __
> 
> Obviously, I prefer my own code 😊____
> 
> __ __
> 
> In terms of appending data, could you explain your use case? I don’t know why you would want to append data from outside the archiver.____
> 
> The archiver makes many assumptions about these files in terms of what data is in them dependent on the name of the file. I would be ____
> 
> Careful about altering them.____
> 
> __ __
> 
> A simple solution can be to push data into the files using a PV! Note that the archiver doesn’t do any checks that timestamps correspond to the current time. ____
> 
> Only that they are sequential. So as long as you are adding new events you can push extra events in.____
> 
> __ __
> 
> A second solution is to use the merge data from another cluster in the ETL process. You can see some documentation at https://urldefense.us/v3/__https://slacmshankar.github.io/epicsarchiver_docs/redundancy.html__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFXdH8EVX$  <https://urldefense.us/v3/__https://slacmshankar.github.io/epicsarchiver_docs/redundancy.html__;!!G_uCfscf7eWS!aKYfKdrC0qTEYyv7dODxiqpKIbbGwGmXDelYVbsx_jW6RRaz73ACLEXIknh3ia44tvZCZug7FKieMW0SFXdH8EVX$ > <https://urldefense.us/v3/__https://slacmshankar.github.io/epicsarchiver_docs/redundancy.html__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omscva_VhG$ <https://urldefense.us/v3/__https://slacmshankar.github.io/epicsarchiver_docs/redundancy.html__;!!G_uCfscf7eWS!b5YgNXV8LWnk1VGBnMvzi-WzDnvd0WROFnruh9I275nM8J9YqqhG98bXfcER4KCiJ6M4_LjfjjRhn2omscva_VhG$>> ____
> 
> I can probably create an example if you need it.____
> 
> __ __
> 
> You could then set up a dummy PV to push the data you want in, and allow the archiver ETL process to merge the data for you.____
> 
> __ __
> 
> I hope that helps.____
> 
> Sky____
> 
> __ __
> 
> __ __
> 
> __ __
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System