Re: IOClog

[Dale Cox via Tech-talk <tech-talk at aps.anl.gov>]

At NEXTLab, we use rsyslog to ship the logs out to a remote server.  I was looking for more than just the IOC errors that come across the error log.  We log all of the console output to a log file, then use rsyslog to send that to a central log storage server for review later.

For reviewing/analytics, we use Grafana.  To get the data into a datasource so grafana can show it, we use a pipeline of Promtail > Loki running on the log server.  It works well, and if you set your dashboard up right, you can do searches and analytics pretty easily.  

It also pairs with other grafana-focused tools like Prometheus.  We can run the node-exporter on our IOC machines and view the logs alongside hardware stats like CPU / RAM usage, etc.

This workflow has made me want to start logging other stuff to that same location.

It's great that there are so many IT-centric tools out there that have solved a lot of this.

Dale

On Thu, Nov 7, 2024 at 2:18 PM Miroslaw Dach via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hi All,

We have experimented with logstash and we have the environment setup for caput log and error messages setup.
All seems to work fine when saving datas into the logfiles.
The configuration for logstash which Andrew has summarised:

https://events01.synchrotron.org.au/event/85/contributions/2601/attachments/689/997/5_06_Logstash_for_IOCs.pdf

was very helpful.

Do you have a similar summary for elastic search and Kibana configuration?

Best Regards
Mirek

Subject: Re: IOClog
From: "Konrad, Martin via Tech-talk" <tech-talk at aps.anl.gov>
To: Neil Hamilton Jr <neil.hamilton.jr at gmail.com>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date: Thu, 2 May 2019 13:36:10 +0000
Hi Neil,
> Is the recommended logging facility IOClog?
20 years ago it was a good choice. If you are looking for a simple
solution for testing it might still get the job done.

For production use there are more powerful tools around, though. Some
labs including ANL and FRIB are successfully using Logstash now which is
very popular in the IT world (there is a good chance your sysadmin is
already using it for system log files etc.). You might want to consider
pairing Logstash with the Elastic Stack to allow fast search and Kibana
for visualization. Andrews had a few slides about how to configure it to
receive EPICS log messages:

https://events01.synchrotron.org.au/event/85/contributions/2601/attachments/689/997/5_06_Logstash_for_IOCs.pdf

Unfortunately, we didn't find time to document this properly. We
appreciate your help turning Andrews slides into a description that can
go into our official documentation.

Thanks,

Martin

--

Dale Cox

Instrumentation and Electrical Engineer

ACU NEXT Lab