RE: Investigating Archiver Appliance lost events

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hi Yong

 

Your observation is valid, and we faced it before but it is not the case this time as the fastest PV we have is archived at 10 Hz, and the issue happens with a large number of PVs, we are archiving 18K+ PVs and I think there was less than 2K PVs in the STS. I am not sure how valid is this, but I remember the issue happened when I executed a script that fetches PV names list and consolidates data for all PVs to the LTS (The management BPL consolidateDataForPV). Now the server is working, I will execute the script again and see how it goes.

 

Best Regards,

Abdalla Al-Dalleh

Control Engineer

SESAME

 

From: Hu, Yong <yhu at bnl.gov>
Sent: Sunday, January 25, 2026 4:41 PM
To: Abdalla Ahmad <Abdalla.Ahmad at sesame.org.jo>; tech-talk at aps.anl.gov
Subject: Re: Investigating Archiver Appliance lost events

 

Hi Abdalla,

 

I recall that we had similar issues at NSLS-2. The buffer refers to the troublesome PV on the AA, not the OS. For us, 'buffer full' means the PV updates too fast (maybe someone has made the changes on the PV). I remember the default monitor-based archiving rate is limited to 1Hz for scaler PVs. I am guessing your problematic PVs update faster the default rate.  You can use camonitor (or caEventRate) to verify the update rate.

 

Cheers,

Yong

 

From:  Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>
Date: Sunday, January 25, 2026 at 7:47 AM

 

Hi

 

This morning we noticed strange behavior on the archiver, it unable to write PB files for a large number of PVs, we noticed this behavior when doing live plots where the plotter plots only the first few 3 or 4 sample and no archived data at all. During troubleshooting, I noticed on one of the problematic PVs it had on its Details page the following parameters increasing every second (i.e. every sample received):

 

• How many events lost because the sample buffer is full so far?
• How many events lost totally so far?

 

Both AA and OS logs were not helpful, I ended up rebooting the server itself. My question is which buffer is the details page referring to? Is it buffer within AA itself or the OS?

 

Best Regards,

Abdalla Al-Dalleh

Control Engineer

SESAME

P.O. Box 7, Allan 19252, Jordan
Tel: +96253511348 , ext. 265

Fax: +96253511423

Email : abdalla.ahmad at sesame.org.jo
Website: www.sesame.org.jo