| 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024 2025 | Index | 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 <2022> 2023 2024 2025 |
| <== Date ==> | <== Thread ==> |
|---|
| Subject: | RE: Epics archiver appliance freezing |
| From: | "Manoussakis, Adamandios via Tech-talk" <tech-talk at aps.anl.gov> |
| To: | "Shankar, Murali" <mshankar at slac.stanford.edu>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov> |
| Date: | Thu, 1 Sep 2022 21:44:49 +0000 |
|
Still looking around, not seeing any huge red flags. I was able to pause all of the PVs and did get the freezing to stop. I am slowly reintroducing PVs to see if I can figure out which one might be causing it. I still am not sure why
a PV would cause the whole cluster to freeze though. Adam From: Shankar, Murali <mshankar at slac.stanford.edu> Yeah; then something in the engine is preventing the call from completing. You could do a kill -9 SIGQUIT on the engine process to check the threads and see where they
are hung.. Also look in the engine's logs to see if you can spot anything. Regards, Murali From: Manoussakis, Adamandios <manoussakis1 at llnl.gov> I only have the one appliance running and its on the computer where the AA lives (I replaced the ip with 192.168.0.31 but I just have our ip there). The other appliance below is commented out. <?xml version='1.0' encoding='utf-8'?> <!-- Took the contents from single\_machine\_install.sh, and modified
them according to our configuration. --> <appliances> <appliance> <identity>appliance0</identity> <cluster_inetport>192.168.0.31:16670</cluster_inetport> <mgmt_url>http://192.168.0.31:17665/mgmt/bpl</mgmt_url> <engine_url>http://192.168.0.31:17666/engine/bpl</engine_url> <etl_url>http://192.168.0.31:17667/etl/bpl</etl_url> <retrieval_url>http://192.168.0.31:17668/retrieval/bpl</retrieval_url> <data_retrieval_url>http://192.168.0.31:17668/retrieval</data_retrieval_url> </appliance> </appliances> <!-- <appliances> <appliance> <identity>appliance0</identity> <cluster_inetport>192.168.40.5:16670</cluster_inetport> <mgmt_url>http://192.168.40.5:17665/mgmt/bpl</mgmt_url> <engine_url>http://192.168.40.5:17666/engine/bpl</engine_url> <etl_url>http://192.168.40.5:17667/etl/bpl</etl_url> <retrieval_url>http://192.168.40.5:17668/retrieval/bpl</retrieval_url> <data_retrieval_url>http://192.168.40.5:17668/retrieval</data_retrieval_url> </appliance> </appliances> --> From: Shankar, Murali <mshankar at slac.stanford.edu>
Do you have unreachable appliances in your appliances.xml? That might explain the "The web application [mgmt] is still processing a request that has yet to finish." and
your other issues. The call at org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90) is a call to the engine to get the status of the PV and for some reason this is not completing and gets stuck. And eventually you run out
of HTTP threads and the UI freezes. Regards, Murali From: Manoussakis, Adamandios <manoussakis1 at llnl.gov> Hi Murali, So far been looking through mgmt logs again but todays logs 9-1-2022 after a crash did not show this null pointer exception below.
from localhost.2022-08-31.log looks like this may be the culprit for crashing but I am not sure where to find or get rid of this issue. I have looked at other setups I have of the AA and nothing similar has popped up in the logs. The
interesting part is a reboot will seem to get the AA back to normal and you can use it up until I run our CSS and cause PVs to change -> causing the AA to start archiving. After say 25 runs then the AA freezes. 31-Aug-2022 19:02:35.663 SEVERE [http-nio-17665-exec-1] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null java.lang.NullPointerException at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:242) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) from catalina.2022-08-31.log I am guessing this is due to this null pointer issue, is it possibly a bad PV name or a bug in the web viewer/archiver that is allowing some malformed PV name to get stored.
01-Sep-2022 09:12:41.526 WARNING [Thread-5] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [mgmt] is still processing a request that has yet to finish. This is very likely to create a memory
leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[ [email protected]/java.net.SocketInputStream.socketRead0(Native Method) [email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:168) [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:140) org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152) org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270) org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153) org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254) org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) org.epics.archiverappliance.utils.ui.GetUrlContent.postStringListAndGetJSON(GetUrlContent.java:350) org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90) org.epics.archiverappliance.common.BasicDispatcher.dispatch(BasicDispatcher.java:99) org.epics.archiverappliance.mgmt.BPLServlet.doGet(BPLServlet.java:198) javax.servlet.http.HttpServlet.service(HttpServlet.java:634) javax.servlet.http.HttpServlet.service(HttpServlet.java:741) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [email protected]/java.lang.Thread.run(Thread.java:829)] from localhost_access_log.2022-08-31.txt, I assume the 1 second intervals for this request is some sort of polling to grab them? And the other endpoint seems to poll on a 5 min interval. 27.0.0.1 - - [31/Aug/2022:19:39:53 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:02 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:03 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:03 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:04 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:08 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:09 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:10 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:10 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:12 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:12 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:16 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:16 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:18 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:18 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:19 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:19 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:20 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 127.0.0.1 - - [31/Aug/2022:19:40:23 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202 IP ADDR - - [01/Sep/2022:00:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:00:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:01:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:02:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:03:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:04:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:05:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:06:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:07:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:08:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:09:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 IP ADDR - - [01/Sep/2022:09:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961 I am getting close to just reinstall the AA, but wanted to check if there is a way to clear out the list of PVs the AA SQL database has? The Website doesn’t seem to have an option to mass delete PVs, although not sure this would solve
the issue. Thanks, Adam From: Tech-talk <tech-talk-bounces at aps.anl.gov>
On Behalf Of Shankar, Murali via Tech-talk >> Is there a specific log to look into to see the freeze up, most of the logs I’ve checked out have tons of information but not sure what I am searching for as the culprit I would first look in the mgmt's ( and subsequently the other component's) localhost logs to determine which request is causing the issue. Once you narrow down the request;
you can look in the code/try adding additional log statements etc. Check for anything related to out of memory etc. Does restarting fix this issue temporarily? Also, did you check the browser's console log for any client side exception? >> but it just will not return anything status wise about the PV search. I think most of this is done in memory on the mgmt instance. However; the cluster needs to be active and alive for this to work correctly. So any info on actions that
takes the cluster down ( out of memory, for example) would help. Regards, Murali |