RE: Archiver Appliance stuck on initial sampling

[Dennis Hilhorst via Tech-talk <tech-talk at aps.anl.gov>]

Hi Sky,

 

Thanks for you quick reply! Most PVs are PVA and CA. They are not submitted with a pva:// prefix, though the archiver does seem to detect they are PVA. I restarted the archiver after clearing out all data again and submitted all PVs which are not from a python softioc. They are all from a single ioc running on the same host as the archiver itself, so the subnet should be okay. I have let it run for a few hours, but it still seems to get stuck. All PVs that “may not exist” and are not in the START state are in the METAINFO_GATHERING state, though the PV’s that are currently in the METAINFO_REQUESTED state report does show 1000 PVs, but I can’t abort from there. They seem to be a rather random assortment of PVs, and I don’t see a very common theme between them besides the values that are in the report: all of them say “false” for Monitoring started, and the other values (When, Remaining, Timer Done, Events so far, storage so far and Internal State and all the MainMeta values) are all N/A, 0 or null (the initial values?).

 

The repo you posted seems to unreachable from outside your facility maybe? I will try to tune these settings after the weekend to see if they have an effect.

 

Sincerely,

 

Dennis Hilhorst

 

From: Sky Brewer <jaco.brewer at gmail.com>
Sent: vrijdag 9 januari 2026 11:49
To: tech-talk at aps.anl.gov; Dennis Hilhorst <dhilhorst at highvolteng.com>
Subject: RE: Archiver Appliance stuck on initial sampling

 

Hi Dennis,

 

I'm guessing most of the PVs are pvAccess?

 

You can clear the queue of stuck PVs (I would do that for all the python soft IOC pvs that aren't working) by going to 

 

mgmt interface -> reports -> PV's that may not exist 

 

then clicking abort.

 

I had a similar problem due to having thousands of PVs which were not on the correct subnet blocking the queue and I ended up adding a feature to https://gitlab.esss.lu.se/ics-software/archiver-appliance/epicsarchiver-mgmt to clear them out. 

 

The report PV's that may not exist  and PV's that are currently in METAINFO_REQUESTED are good for debugging this kind of issue.

 

You can also tune the queue with the section in the arch.properties:

"""

# We impose a throttle on archive requests that are pending. This is an attempt to conserve resources on the engine and also to control CA search broadcast storms
# What this means is that if you have more that this many invalid PV's (PVs that will never connect) in the archive workflow; the ones later in the queue will never get fulfilled unless the archive request queue is cleaned up.
# See abortNeverConnectedPV for a example of how to do this.
# Use this property to increase this batch size.
# Note that there is a limit on how high this can go. We sample the PV for a minute to determine storage rates etc; if this is set very high (> 10000), there may not be enough time to perform the sampling for the archive workflow.
# org.epics.archiverappliance.mgmt.MgmtRuntimeState.archivePVWorkflowBatchSize = 1000
# For larger archivePVWorkflowBatchSize, you may need a longer time between the workflow state machine ticks.
# Set this to 60 seconds or more if you are using archivePVWorkflowBatchSize's of 5000 or greater.
# The workflow may take a little longer per PV but overall may be much faster.
# org.epics.archiverappliance.mgmt.MgmtRuntimeState.archivePVWorkflowTickSeconds = 10
# Abort PV's in the archive PV workflow after this many minutes if the archiver is not able to connect to the PV.
# The workflow can take a few minutes; so this should be set to a reasonable value (for example, 1 minute would mean that no PV would complete the workflow) 
# By default, this is set to a day. So, if the archiver cannot connect to the PV in a day, it will give up and abort.
# To turn this off, set this to -1.
# org.epics.archiverappliance.mgmt.MgmtRuntimeState.abortArchiveRequestInMins = 1440
"""

 

Hope that helps!

 

--

Sky Brewer