RE: archive appliance some PVs no data

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hello Michael

Thanks for the feedback. Can you please elaborate? I still don't understand how this option won't work. To reiterate, I meant that AA should have the option to archive the timestamp of the OS instead of the timestamp received from CA monitors. For example, in our case with the device that could not sync, both devices are ARM-based embedded Linux, one device were synced successfully by re-compiling the OS timezone, the other was not a success at all and still shows a timestamp from the 90s if I remember. That's where I had to add buffer PVs.

I know this might be an extreme case, but the topic of AA's timestamp is worth discussing. 

Thanks!
Abdalla.

-----Original Message-----
From: Michael Davidsaver <mdavidsaver at gmail.com> 
Sent: Tuesday, September 24, 2024 6:54 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>
Cc: tech-talk at aps.anl.gov
Subject: Re: archive appliance some PVs no data

On 9/23/24 23:12, Abdalla Ahmad via Tech-talk wrote:
> Hello all
> 
> I would like to discuss something similar related to AA, would it make sense if it has an option to take its own timestamp instead of CA’s timestamp? The same thing as caget/camonitor tools do? If the answer is no, why is that?

imo. an emphatic "no".  To do otherwise means adding a per-PV configuration option for this workaround to every CA client application.  This does not scale well.


> We faced the same issue with some device that could not sync with our NTP servers. After many troubleshooting attempts I decided to create “buffer” PVs (calc records) where they just read the value of any PV as it is.

If you want to start a separate thread, I'm happy to help you try to straighten this out, or at least help to point the finger if there is a network/firewall issue.


> Best Regards,
> 
> Abdalla.
> 
> *From:*Tech-talk <tech-talk-bounces at aps.anl.gov> *On Behalf Of *Shen, 
> Guobao via Tech-talk
> *Sent:* Tuesday, September 24, 2024 3:44 AM
> *To:* Huang, Rong <ronghuang at ls-cat.org>; Michael Davidsaver 
> <mdavidsaver at gmail.com>
> *Cc:* tech-talk at aps.anl.gov
> *Subject:* Re: archive appliance some PVs no data
> 
> Rong,
> 
>  From the Archiver Appliance’s website, it has statistic data about the PV connections, drops, and etc.
> 
> Regarding the timestamp, beyond your timestamp, I guess that all PVs inside your IOC got processed?
> 
>  From our deployment at APS accelerator side, we did occasionally see some data point drops.
> 
> But over, it’s pretty healthy.
> 
> Thanks,
> 
> Guobao
> 
> *From: *Tech-talk <tech-talk-bounces at aps.anl.gov 
> <mailto:tech-talk-bounces at aps.anl.gov>> on behalf of Rong Huang via 
> Tech-talk <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>>
> *Date: *Monday, September 23, 2024 at 6:23 PM
> *To: *Michael Davidsaver <mdavidsaver at gmail.com 
> <mailto:mdavidsaver at gmail.com>>
> *Cc: *tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov> 
> <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>>
> *Subject: *Re: archive appliance some PVs no data
> 
> Hi Michael,
> 
> Thank you very much for your reply.
> 
> I searched for keyword "dropped" in all tomcat instance directories where each log directory locates, I find nothing:
> 
> *archiver@strawberry*:*~/epics/archive*$ grep -r "dropped" 
> engine/logs/
> 
> *archiver@strawberry*:*~/epics/archive*$ grep -r "dropped" etl/logs/
> 
> *archiver@strawberry*:*~/epics/archive*$ grep -r "dropped" mgmt/logs/
> 
> *archiver@strawberry*:*~/epics/archive*$ grep -r "dropped" 
> retrieval/logs/
> 
>   You are right that on the detail page, the problemed PV's "how many events so far" is 0.
> 
> Paul Sichta from PPPL emailed me and mentioned the possibility of a wrong PV timestamp. I find out our troubled PVs have the timestamp of year 1990, and I will discuss with our IT staff to find out what changed to our IOC booting host PC.
> 
> Our IOC cmd file uses `putenv ("EPICS_TS_NTP_INET=10.1.0.3")` to sync time with the PC.
> 
> Thanks a lot,
> 
> Rong
> 
> On Mon, Sep 23, 2024 at 12:50 PM Michael Davidsaver <mdavidsaver at gmail.com <mailto:mdavidsaver at gmail.com>> wrote:
> 
>     On 9/23/24 12:20, Rong Huang via Tech-talk wrote:
>     > Hello,
>     > 
>     > I installed Archiver Appliance last year while APS was down. To learn how to use it, I started archiving some PVs, and paused them. I "resumed" the archiving now while APS is back, but some PVs can be archived, some PVs cannot.
> 
>     Some general AA troubleshooting advice...
> 
>     I would suggest looking at the "error" reports pages,
>     paying special attention to those with names beginning
>     with "PVs by dropped ...".
> 
>     Also, look on the details page for one of the PVs in question.
> 
>     eg. look for the "How many events so far?" line.  Refresh a
>     few times to see that it increments when that PV should update.
>     Also look for "How many events lost totally so far?" and see
>     that it is zero, or at least not incrementing.
> 
> 
>     > I use https://urldefense.us/v3/__http://strawberry.ls-cat.org:17665/mgmt/ui/index.html__;!!G_uCfscf7eWS!eKLRsTGJw6I9EullBTnzMSlvMsZMdp8d0WxUuaAf_tStehNkVkD2DOY8rfZ3zPkwuo8w5uKpklAxMGWnMEwnRS1An_Xf1yw$  <https://urldefense.us/v3/__http:/strawberry.ls-cat.org:17665/mgmt/ui/index.html__;!!G_uCfscf7eWS!dHYEzcrH9P3u_wgcUb1ZpJU4vt2SkpDtGKhCxVSf471VA82Xk55KW0uG_5n0cJsEY49qyusJDJZB95f3ymFSkbuK$> <https://urldefense.us/v3/__http://strawberry.ls-cat.org:17665/mgmt/ui/index.html__;!!G_uCfscf7eWS!dGt06d4Y35r21tzRJQ1P4MfKQL9FjIyz9ZfpC9EqT_x-zgfhGmPeWFfshx886REUI6bsf-LGb559zQ0dOJG_bOkZ$ <https://urldefense.us/v3/__http:/strawberry.ls-cat.org:17665/mgmt/ui/index.html__;!!G_uCfscf7eWS!dGt06d4Y35r21tzRJQ1P4MfKQL9FjIyz9ZfpC9EqT_x-zgfhGmPeWFfshx886REUI6bsf-LGb559zQ0dOJG_bOkZ$>> to Achive/Pause/Resume archiving.
>     > 
>     > I use https://urldefense.us/v3/__http://strawberry.ls-cat.org:17668/retrieval/ui/viewer/archViewer.html__;!!G_uCfscf7eWS!eKLRsTGJw6I9EullBTnzMSlvMsZMdp8d0WxUuaAf_tStehNkVkD2DOY8rfZ3zPkwuo8w5uKpklAxMGWnMEwnRS1A-H0bIDo$  <https://urldefense.us/v3/__http:/strawberry.ls-cat.org:17668/retrieval/ui/viewer/archViewer.html__;!!G_uCfscf7eWS!dHYEzcrH9P3u_wgcUb1ZpJU4vt2SkpDtGKhCxVSf471VA82Xk55KW0uG_5n0cJsEY49qyusJDJZB95f3yrttuEsh$> <https://urldefense.us/v3/__http://strawberry.ls-cat.org:17668/retrieval/ui/viewer/archViewer.html__;!!G_uCfscf7eWS!dGt06d4Y35r21tzRJQ1P4MfKQL9FjIyz9ZfpC9EqT_x-zgfhGmPeWFfshx886REUI6bsf-LGb559zQ0dODFD8Xz_$ <https://urldefense.us/v3/__http:/strawberry.ls-cat.org:17668/retrieval/ui/viewer/archViewer.html__;!!G_uCfscf7eWS!dGt06d4Y35r21tzRJQ1P4MfKQL9FjIyz9ZfpC9EqT_x-zgfhGmPeWFfshx886REUI6bsf-LGb559zQ0dODFD8Xz_$>> to view/retrive data.
>     > 
>     > The problem, using "21:C1:MO:E:ActPos" as an example, the archViewer website showing one value straight line, "113" Kev, for any display time ranges. "113" comes from the mono E reading when I started archiver appliance last year but not the current E reading.
>     > 
>     > I deleted the PV "21:C1:MO:E:ActPos" and its data from archiver appliance, re-entered the PV and started its archiving, the mgmt page showed it "Being archived", "appliance0", "Connected? true", "Sampling period: 10.0".
>     > 
>     > archViewer web page shows a flat line of "0" for any time range.
>     > 
>     > There is no data file of this PV saved in /arch/sts/ArchiverStore/21. (Even there is no subdirectory of "C1" in /arch/sts/ArchiverStore/21 after I manually removed it and deleted all 21:C1:* PVs)
>     > 
>     > Restarting archiving appliance (sampleStart.sh stop & sampleStart.sh start) does not help.
>     > 
>     > But some PVs can be archived after "resume" (some of them might have noise, therefore the values were varying all the time in the past).
>     > 
>     > I am using apache-tomcat-9.0.86. mysql 8.0.37-0ubuntu0.22.04.3, I installed Archinver Appliance using single_mechine_install.sh.
>     > 
>     > Thanks a lot!
>     > Rong
>     >
>