Re: IOClog

[Saisrikiran Mudigonda via Tech-talk <tech-talk at aps.anl.gov>]

At EPIC, we log different types of errors to log files using procServ. Error logs, Console logs, CaPut logs and Server logs are the types of logs we capture. We push the logs to a central server using NFS share. 

For analytics, we use Grafana dashboards. We use Elastic search to process the logs ingested from Logstash. We also use "tags" to differentiate between errors (db_error, connection_error, etc.) in the logstash pipeline.

For capturing the status of the servers, we use Prometheus with Node_exporter and Blackbox_exporter for the health (UP/DOWN status) of the servers. 

We also show the number of IOCs up/down using "node_systemd_unit_state" of Node_exporter (as each IOC runs as a systemd service).

Our dashboard is distributed in 3 panels - first panel contains IOC count (server1, server2, etc), second panel contains Hardware errors and the third panel has IOC errors.

The IOC errors are captured using keywords like ERROR, etc. and using "tags".

Best Regards,

Sai

--

Saisrikiran Mudigonda

Project Software Engineer

Extreme Photonics Innovation Centre

TIFR, Hyderabad, India

On Fri, Nov 8, 2024 at 2:33 AM Dale Cox via Tech-talk <tech-talk at aps.anl.gov> wrote:

At NEXTLab, we use rsyslog to ship the logs out to a remote server.  I was looking for more than just the IOC errors that come across the error log.  We log all of the console output to a log file, then use rsyslog to send that to a central log storage server for review later.

For reviewing/analytics, we use Grafana.  To get the data into a datasource so grafana can show it, we use a pipeline of Promtail > Loki running on the log server.  It works well, and if you set your dashboard up right, you can do searches and analytics pretty easily.  

It also pairs with other grafana-focused tools like Prometheus.  We can run the node-exporter on our IOC machines and view the logs alongside hardware stats like CPU / RAM usage, etc.

This workflow has made me want to start logging other stuff to that same location.

It's great that there are so many IT-centric tools out there that have solved a lot of this.

Dale

On Thu, Nov 7, 2024 at 2:18 PM Miroslaw Dach via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hi All,

We have experimented with logstash and we have the environment setup for caput log and error messages setup.
All seems to work fine when saving datas into the logfiles.
The configuration for logstash which Andrew has summarised:

https://events01.synchrotron.org.au/event/85/contributions/2601/attachments/689/997/5_06_Logstash_for_IOCs.pdf

was very helpful.

Do you have a similar summary for elastic search and Kibana configuration?

Best Regards
Mirek

Subject: Re: IOClog
From: "Konrad, Martin via Tech-talk" <tech-talk at aps.anl.gov>
To: Neil Hamilton Jr <neil.hamilton.jr at gmail.com>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Date: Thu, 2 May 2019 13:36:10 +0000
Hi Neil,
> Is the recommended logging facility IOClog?
20 years ago it was a good choice. If you are looking for a simple
solution for testing it might still get the job done.

For production use there are more powerful tools around, though. Some
labs including ANL and FRIB are successfully using Logstash now which is
very popular in the IT world (there is a good chance your sysadmin is
already using it for system log files etc.). You might want to consider
pairing Logstash with the Elastic Stack to allow fast search and Kibana
for visualization. Andrews had a few slides about how to configure it to
receive EPICS log messages:

https://events01.synchrotron.org.au/event/85/contributions/2601/attachments/689/997/5_06_Logstash_for_IOCs.pdf

Unfortunately, we didn't find time to document this properly. We
appreciate your help turning Andrews slides into a description that can
go into our official documentation.

Thanks,

Martin

--

Dale Cox

Instrumentation and Electrical Engineer

ACU NEXT Lab