EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Epics archiver appliance freezing
From: "Shankar, Murali via Tech-talk" <tech-talk at aps.anl.gov>
To: "Manoussakis, Adamandios" <manoussakis1 at llnl.gov>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date: Thu, 1 Sep 2022 19:06:10 +0000
Yeah; then something in the engine is preventing the call from completing. You could do a kill -9 SIGQUIT on the engine process to check the threads and see where they are hung.. Also look in the engine's logs to see if you can spot anything. 

Regards,
Murali


From: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Sent: Thursday, September 1, 2022 11:41 AM
To: Shankar, Murali <mshankar at slac.stanford.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: Epics archiver appliance freezing
 

I only have the one appliance running and its on the computer where the AA lives (I replaced the ip with 192.168.0.31 but I just have our ip there).  The other appliance below is commented out.

 

<?xml version='1.0' encoding='utf-8'?>

<!--

  Took the contents from single\_machine\_install.sh, and modified

  them according to our configuration.

-->

<appliances>

   <appliance>

     <identity>appliance0</identity>

     <cluster_inetport>192.168.0.31:16670</cluster_inetport>

     <mgmt_url>http://192.168.0.31:17665/mgmt/bpl</mgmt_url>

     <engine_url>http://192.168.0.31:17666/engine/bpl</engine_url>

     <etl_url>http://192.168.0.31:17667/etl/bpl</etl_url>

     <retrieval_url>http://192.168.0.31:17668/retrieval/bpl</retrieval_url>

     <data_retrieval_url>http://192.168.0.31:17668/retrieval</data_retrieval_url>

   </appliance>

</appliances>

 

<!--

<appliances>

   <appliance>

     <identity>appliance0</identity>

     <cluster_inetport>192.168.40.5:16670</cluster_inetport>

     <mgmt_url>http://192.168.40.5:17665/mgmt/bpl</mgmt_url>

     <engine_url>http://192.168.40.5:17666/engine/bpl</engine_url>

     <etl_url>http://192.168.40.5:17667/etl/bpl</etl_url>

     <retrieval_url>http://192.168.40.5:17668/retrieval/bpl</retrieval_url>

     <data_retrieval_url>http://192.168.40.5:17668/retrieval</data_retrieval_url>

   </appliance>

</appliances>

-->

 

From: Shankar, Murali <mshankar at slac.stanford.edu>
Sent: Thursday, September 1, 2022 11:21 AM
To: Manoussakis, Adamandios <manoussakis1 at llnl.gov>; tech-talk at aps.anl.gov
Subject: Re: Epics archiver appliance freezing

 

Do you have unreachable appliances in your appliances.xml? That might explain the "The web application [mgmt] is still processing a request that has yet to finish." and your other issues. The call at org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90) is a call to the engine to get the status of the PV and for some reason this is not completing and gets stuck. And eventually you run out of HTTP threads and the UI freezes.

 

Regards,

Murali

 

 

 

 

 


From: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Sent: Thursday, September 1, 2022 10:10 AM
To: Shankar, Murali <mshankar at slac.stanford.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: Epics archiver appliance freezing

 

Hi Murali,

 

So far been looking through mgmt logs again but todays logs 9-1-2022 after a crash did not show this null pointer exception below. 

 

from localhost.2022-08-31.log looks like this may be the culprit for crashing but I am not sure where to find or get rid of this issue.  I have looked at other setups I have of the AA and nothing similar has popped up in the logs.  The interesting part is a reboot will seem to get the AA back to normal and you can use it up until I run our CSS and cause PVs to change -> causing the AA to start archiving.  After say 25 runs then the AA freezes.

 

31-Aug-2022 19:02:35.663 SEVERE [http-nio-17665-exec-1] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null

java.lang.NullPointerException

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:242)

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

                at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)

                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                at java.base/java.lang.Thread.run(Thread.java:829)

 

 

from catalina.2022-08-31.log I am guessing this is due to this null pointer issue, is it possibly a bad PV name or a bug in the web viewer/archiver that is allowing some malformed PV name to get stored. 

 

01-Sep-2022 09:12:41.526 WARNING [Thread-5] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [mgmt] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[

[email protected]/java.net.SocketInputStream.socketRead0(Native Method)

[email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)

[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:168)

[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:140)

org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)

org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)

org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)

org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)

org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)

org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)

org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)

org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)

org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)

org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)

org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254)

org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)

org.epics.archiverappliance.utils.ui.GetUrlContent.postStringListAndGetJSON(GetUrlContent.java:350)

org.epics.archiverappliance.mgmt.bpl.GetPVStatusAction.execute(GetPVStatusAction.java:90)

org.epics.archiverappliance.common.BasicDispatcher.dispatch(BasicDispatcher.java:99)

org.epics.archiverappliance.mgmt.BPLServlet.doGet(BPLServlet.java:198)

javax.servlet.http.HttpServlet.service(HttpServlet.java:634)

javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)

org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

[email protected]/java.lang.Thread.run(Thread.java:829)]

 

 

 

 

from localhost_access_log.2022-08-31.txt, I assume the 1 second intervals for this request is some sort of polling to grab them? And the other endpoint seems to poll on a 5 min interval.

 

 

27.0.0.1 - - [31/Aug/2022:19:39:53 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:02 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:03 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:03 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:04 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:08 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:09 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:10 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:10 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:11 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:12 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:12 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:13 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:14 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:15 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:16 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:16 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:17 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:18 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:18 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:19 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:19 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:20 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

127.0.0.1 - - [31/Aug/2022:19:40:23 -0700] "GET /mgmt/bpl/getPVStatus?pv=PV1%3AST%3AID%3A*&reporttype=short HTTP/1.1" 200 1202

 

IP ADDR - - [01/Sep/2022:00:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:00:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:01:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:04:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:09:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:14:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:19:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:24:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:29:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:34:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:39:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:44:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:49:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:54:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:02:59:03 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:03:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:04:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:04:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:09:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:14:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:19:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:24:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:29:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:34:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:39:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:44:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:49:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:54:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:05:59:04 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:06:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:07:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:14:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:19:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:24:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:29:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:34:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:39:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:44:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:49:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:54:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:08:59:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:09:04:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

IP ADDR - - [01/Sep/2022:09:09:05 -0700] "GET /mgmt/bpl/getPVsForThisAppliance HTTP/1.1" 200 183961

 

I am getting close to just reinstall the AA, but wanted to check if there is a way to clear out the list of PVs the AA SQL database has?  The Website doesn’t seem to have an option to mass delete PVs, although not sure this would solve the issue.

 

Thanks,

Adam

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Shankar, Murali via Tech-talk
Sent: Thursday, September 1, 2022 7:11 AM
To: tech-talk at aps.anl.gov
Subject: Re: Epics archiver appliance freezing

 

>> Is there a specific log to look into to see the freeze up, most of the logs I’ve checked out have tons of information but not sure what I am searching for as the culprit

 

I would first look in the mgmt's ( and subsequently the other component's) localhost logs to determine which request is causing the issue. Once you narrow down the request; you can look in the code/try adding additional log statements etc. Check for anything related to out of memory etc. Does restarting fix this issue temporarily? Also, did you check the browser's console log for any client side exception?

 

>> but it just will not return anything status wise about the PV search.

I think most of this is done in memory on the mgmt instance. However; the cluster needs to be active and alive for this to work correctly. So any info on actions that takes the cluster down ( out of memory, for example) would help.

 

Regards,

Murali

 

 

 


Replies:
RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
References:
Re: Epics archiver appliance freezing Shankar, Murali via Tech-talk
RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
Re: Epics archiver appliance freezing Shankar, Murali via Tech-talk
RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk

Navigate by Date:
Prev: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
Next: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
Navigate by Thread:
Prev: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
Next: RE: Epics archiver appliance freezing Manoussakis, Adamandios via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <20222023  2024 
ANJ, 14 Sep 2022 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·