EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: connection timeout between ioc and logstash
From: 吴煊 via Tech-talk <[email protected]>
To: "johnson, andrew n." <[email protected]>, "j. lewis muir" <[email protected]>, "kivel niko (psi)" <[email protected]>
Cc: tech-talk <[email protected]>
Date: Sat, 17 Aug 2019 22:13:09 +0800 (GMT+08:00)


Hi all,

  I've test the tcp_keep_alive option, unfortunately, it didn't work either. The difference is the IOC doesn't prompt error info when timeout. I can tell that the time of timeout is half an hour(28 minutes later is ok, 31 minutes later is abnormal). Then, logstash can never receive new message except I restarting the IOC. It's no use to restart logstash. The logstash configuration I use as follow: 

input {

        tcp {

                host => "${EPICS_IOC_LOG_INET:10.1.236.30}"

                port => "${EPICS_IOC_LOG_PORT:7011}"

                id => "ioclog-${EPICS_IOC_LOG_PORT:7011}"

                codec => "line"

                tcp_keep_alive => true

        }

}

 

filter {

        grok {

                pattern_definitions => {

                        MONTHNAM => "(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)"

                        TIMESTAMP => "%{MONTHDAY}-%{MONTHNAM}-%{YEAR} %{TIME}"

                        PV => "\S+"

                }

#               named_captures_only => true

                # The following string must be all one line, one space between parts

                match => { "message" => "%{TIMESTAMP:timestamp} %{HOSTNAME:client} %{USER:user} %{PV:pv} new=%{NUMBER:new:float} old=%{NUMBER:old:float}" }

        }


#       date {

#               # Extract the timestamp from the message as a proper logstash

#               # timestamp field type, but since the IOC's clock might not be

#               # accurate we don't replace the locally generated @timestamp.

#               match => ["timestamp", "dd-MMM-yy HH:mm:ss"]

#               locale => "en"

#               target => "@ioctimestamp"

#       }

}

 

output {

#        elasticsearch {

#                hosts => "10.1.236.30:9200"

#                index => "caputlog-%{+YYYY-MM-dd}"

#        }

       stdout { codec => rubydebug }

        } 

It can't parse correctly if I added "(?: min=%{NUMBER:min} max=%{NUMBER:max})?" to the end of grok match part. It would work only if there was no burst. I haven't found a solution for this.


-----原始邮件-----
发件人:"Johnson, Andrew N." <[email protected]>
发送时间:2019-08-17 00:43:01 (星期六)
收件人: "吴煊" <[email protected]>, "j. lewis muir" <[email protected]>, "kivel niko (psi)" <[email protected]>
抄送: tech-talk <[email protected]>
主题: Re: connection timeout between ioc and logstash

Hi,

On 8/16/19 10:22 AM, 吴煊 via Tech-talk wrote:
I simplly followed the instruction written by Andrew Johnson https://events01.synchrotron.org.au/event/85/contributions/2601/attachments/689/997/5_06_Logstash_for_IOCs.pdf A thread named "IOClog" in May 2th 2019 had mentioned this.
I've installed logstash 7.2 in a linux box. I configured 'caPutLogInit "10.1.236.30:7011", 1' in st.cmd. The configuration file for logstash is fellowing:
input {
 tcp {
 host => "${EPICS_IOC_LOG_INET:10.1.236.30}"
 port => "${EPICS_IOC_LOG_PORT:7011}"
 codec => "line"
 }
output {
 file {
 path => "/log/ioc/putlog-%{+YYYY-MM-dd}.json"
 }
}
}
That's all the configuration. I can not figure out who closed the connection, but I think logstash may closed it, because iocLogServer will not. I'll try the tcp_keep_alive option next and to see it would work or not.
Apparently the default setting for the TCP SO_KEEPALIVE option can be different on different OSs, and the logstash configuration file I presented had not been widely tested (I've been running logstash on MacOS here). The iocLogServer code does explicitly set SO_KEEPALIVE in its acceptNewClient() routine so I think that logstash option will be the correct solution to this problem — please let tech-talk know your results.

By the way, I found that the grok filter in the example file is not correct for logstash 7.2
I developed those regular expressions using http://grokdebug.herokuapp.com/ and logstash 6.3.2 and am currently running 7.0.0, but we're not doing put-logging here yet. If you have a fix for logstash 7.2 I would be interested in seeing it.

Thanks,

- Andrew
-- 
Complexity comes for free, Simplicity you have to work for.

Replies:
RE: connection timeout between ioc and logstash Kivel Niko (PSI) via Tech-talk
References:
connection timeout between ioc and logstash 吴煊 via Tech-talk
Re: connection timeout between ioc and logstash J. Lewis Muir via Tech-talk
RE: connection timeout between ioc and logstash Kivel Niko (PSI) via Tech-talk
Re: connection timeout between ioc and logstash J. Lewis Muir via Tech-talk
Re: Re: connection timeout between ioc and logstash 吴煊 via Tech-talk
Re: connection timeout between ioc and logstash Johnson, Andrew N. via Tech-talk

Navigate by Date:
Prev: RE: Undefined Timestamp on Motor Record Pearson, Matthew R. via Tech-talk
Next: RE: connection timeout between ioc and logstash Kivel Niko (PSI) via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: connection timeout between ioc and logstash Johnson, Andrew N. via Tech-talk
Next: RE: connection timeout between ioc and logstash Kivel Niko (PSI) via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  <20192020  2021  2022  2023  2024 
ANJ, 19 Aug 2019 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·