Experimental Physics and
Industrial Control System

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024	Index	1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024
<== Date ==>		<== Thread ==>

Subject:	Re: Epics archiver appliance freezing
From:	"Shankar, Murali via Tech-talk" <tech-talk at aps.anl.gov>
To:	"Manoussakis, Adamandios" <manoussakis1 at llnl.gov>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date:	Thu, 1 Sep 2022 19:06:10 +0000

Yeah; then something in the engine is preventing the call from completing. You could do a kill -9 SIGQUIT on the engine process to check the threads and see where they are hung.. Also look in the engine's logs to see if you can spot anything.

Regards,

Murali

From: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Sent: Thursday, September 1, 2022 11:41 AM
To: Shankar, Murali <mshankar at slac.stanford.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: Epics archiver appliance freezing

I only have the one appliance running and its on the computer where the AA lives (I replaced the ip with 192.168.0.31 but I just have our ip there). The other appliance below is commented out.

<?xml version='1.0' encoding='utf-8'?>

<!--

Took the contents from single\_machine\_install.sh, and modified

them according to our configuration.

-->

<identity>appliance0</identity>

<cluster_inetport>192.168.0.31:16670</cluster_inetport>

<mgmt_url>http://192.168.0.31:17665/mgmt/bpl</mgmt_url>

<engine_url>http://192.168.0.31:17666/engine/bpl</engine_url>

<etl_url>http://192.168.0.31:17667/etl/bpl</etl_url>

<retrieval_url>http://192.168.0.31:17668/retrieval/bpl</retrieval_url>

<data_retrieval_url>http://192.168.0.31:17668/retrieval</data_retrieval_url>

</appliance>

</appliances>

<!--

<identity>appliance0</identity>

<cluster_inetport>192.168.40.5:16670</cluster_inetport>

<mgmt_url>http://192.168.40.5:17665/mgmt/bpl</mgmt_url>

<engine_url>http://192.168.40.5:17666/engine/bpl</engine_url>

<etl_url>http://192.168.40.5:17667/etl/bpl</etl_url>

<retrieval_url>http://192.168.40.5:17668/retrieval/bpl</retrieval_url>

<data_retrieval_url>http://192.168.40.5:17668/retrieval</data_retrieval_url>

</appliance>

</appliances>

-->

From: Shankar, Murali <mshankar at slac.stanford.edu>
Sent: Thursday, September 1, 2022 11:21 AM
To: Manoussakis, Adamandios <manoussakis1 at llnl.gov>; tech-talk at aps.anl.gov
Subject: Re: Epics archiver appliance freezing

Do you have unreachable appliances in your appliances.xml? That might explain the "The web application [mgmt] is still processing a request that has yet to finish." and your other issues. The call at org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90) is a call to the engine to get the status of the PV and for some reason this is not completing and gets stuck. And eventually you run out of HTTP threads and the UI freezes.

Regards,

Murali

From: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Sent: Thursday, September 1, 2022 10:10 AM
To: Shankar, Murali <mshankar at slac.stanford.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: Epics archiver appliance freezing

Hi Murali,

So far been looking through mgmt logs again but todays logs 9-1-2022 after a crash did not show this null pointer exception below.

from localhost.2022-08-31.log looks like this may be the culprit for crashing but I am not sure where to find or get rid of this issue. I have looked at other setups I have of the AA and nothing similar has popped up in the logs. The interesting part is a reboot will seem to get the AA back to normal and you can use it up until I run our CSS and cause PVs to change -> causing the AA to start archiving. After say 25 runs then the AA freezes.

31-Aug-2022 19:02:35.663 SEVERE [http-nio-17665-exec-1] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null

java.lang.NullPointerException

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:242)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.base/java.lang.Thread.run(Thread.java:829)

from catalina.2022-08-31.log I am guessing this is due to this null pointer issue, is it possibly a bad PV name or a bug in the web viewer/archiver that is allowing some malformed PV name to get stored.

01-Sep-2022 09:12:41.526 WARNING [Thread-5] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [mgmt] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[

[email protected]/java.net.SocketInputStream.socketRead0(Native Method)

[email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)

[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:168)

[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:140)

org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)

org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)

org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)

org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)

org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)

org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)

org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)

org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)

org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)

org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)

org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)

org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)

org.epics.archiverappliance.utils.ui.GetUrlContent.postStringListAndGetJSON(GetUrlContent.java:350)

org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90)

org.epics.archiverappliance.common.BasicDispatcher.dispatch(BasicDispatcher.java:99)

org.epics.archiverappliance.mgmt.BPLServlet.doGet(BPLServlet.java:198)

javax.servlet.http.HttpServlet.service(HttpServlet.java:634)

javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)

org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

[email protected]/java.lang.Thread.run(Thread.java:829)]

from localhost_access_log.2022-08-31.txt, I assume the 1 second intervals for this request is some sort of polling to grab them? And the other endpoint seems to poll on a 5 min interval.

27.0.0.1 - - [31/Aug/2022:19:39:53 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:02 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:03 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:04 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:08 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:09 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:10 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:12 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:16 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:18 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:19 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:20 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:23 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

IP ADDR - - [01/Sep/2022:00:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:09:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:09:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

I am getting close to just reinstall the AA, but wanted to check if there is a way to clear out the list of PVs the AA SQL database has? The Website doesn’t seem to have an option to mass delete PVs, although not sure this would solve the issue.

Thanks,

Adam

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Shankar, Murali via Tech-talk
Sent: Thursday, September 1, 2022 7:11 AM
To: tech-talk at aps.anl.gov
Subject: Re: Epics archiver appliance freezing

>> Is there a specific log to look into to see the freeze up, most of the logs I’ve checked out have tons of information but not sure what I am searching for as the culprit

I would first look in the mgmt's ( and subsequently the other component's) localhost logs to determine which request is causing the issue. Once you narrow down the request; you can look in the code/try adding additional log statements etc. Check for anything related to out of memory etc. Does restarting fix this issue temporarily? Also, did you check the browser's console log for any client side exception?

>> but it just will not return anything status wise about the PV search.

I think most of this is done in memory on the mgmt instance. However; the cluster needs to be active and alive for this to work correctly. So any info on actions that takes the cluster down ( out of memory, for example) would help.

Regards,

Murali

Replies:: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk

References:: Re: Epics archiver appliance freezing Shankar, Murali via Tech-talk; RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk; Re: Epics archiver appliance freezing Shankar, Murali via Tech-talk; RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk

Navigate by Date:: Prev: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk; Next: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk; Index: 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024
Navigate by Thread:: Prev: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk; Next: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk; Index: 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024

ANJ, 14 Sep 2022

· Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System