Chapter 10
IOC Error Logging

 10.1 Overview
 10.2 Error Message Routines
  10.2.1 Basic Routines
  10.2.2 Log with Severity
  10.2.3 Status Routines
  10.2.4 Obsolete Routines
 10.3 errlog Listeners
 10.4 errlogThread
 10.5 console output and message queue size
 10.6 Status Codes
 10.7 iocLog
  10.7.1 iocLogServer
  10.7.2 iocLogClient
  10.7.3 Configuring a Private Log Server
  10.7.4 iocLogPrefix

10.1 Overview

Errors detected by an IOC can be divided into classes: Errors related to a particular client and errors not attributable to a particular client. An example of the first type of error is an illegal Channel Access request. For this type of error, a status value should be passed back to the client. An example of the second type of error is a device driver detecting a hardware error. This type of error should be reported to a system wide error handler.

Dividing errors into these two classes is complicated by a number of factors.

If used properly, the error handling facilities described in this chapter can process both types of errors.

This chapter describes the following:

NOTE: recGbl error routines are also provided. They in turn call one of the error message routines.

10.2 Error Message Routines

10.2.1 Basic Routines

   int errlogPrintf(const char pformat, ...); 
   int errlogVprintf(const char pformat,va_list pvar); 
   int errlogMessage(const char message); 
   void errlogFlush(void);

errlogPrintf and errlogVprintf are like printf and vprintf provided by the standard C library, except that their output is sent to the errlog task; unless configured not to, the output will appear on the console as well. Consult any book that describes the standard C library such as “The C Programming Language ANSI C Edition” by Kernighan and Ritchie.

errlogMessage sends message to the errlog task.

errlogFlush wakes up the errlog task and then waits until all messages are flushed from the queue.

10.2.2 Log with Severity

typedef enum { 
    errlogInfo,errlogMinor,errlogMajor,errlogFatal 
}errlogSevEnum; 
 
int errlogSevPrintf(const errlogSevEnum severity, 
    const char pformat, ...); 
int errlogSevVprintf(const errlogSevEnum severity, 
    const char pformat,va_list pvar); 
 
char errlogGetSevEnumString(const errlogSevEnum severity); 
 
void  errlogSetSevToLog(const errlogSevEnum severity ); 
errlogSevEnum errlogGetSevToLog(void);

errlogSevPrintf and errlogSevVprintf are like errlogPrintf and errlogVprintf except that they add the severity to the beginning of the message in the form “sevr=<value>” where value is one of “info, minor, major, fatal”. Also the message is suppressed if severity is less than the current severity to suppress. If epicsThreadIsOkToBlock is true, which is true during iocInit, errlogSevVprintf does NOT send output to the errlog task.

errlogGetSevEnumString gets the string value of severity.

errlogSetSevToLog sets the severity to log. errlogGetSevToLog gets the current severity to log.

10.2.3 Status Routines

void errMessage(long status, char message); 
 
void errPrintf(long status, const char pFileName, 
    int lineno, const char pformat, ...);

Routine errMessage (actually a macro that calls errPrintf) has the following format:

void errMessage(long  status, char  message);

Where status is defined as:

errMessage, via a call to errPrintf, prints the message, the status symbol and string values, and the name of the task which invoked errMessage. It also prints the name of the source file and the line number from which the call was issued.

The calling routine is expected to pass a descriptive message to this routine. Many subsystems provide routines built on top of errMessage which generate descriptive messages.

An IOC global variable errVerbose, defined as an external in errMdef.h, specifies verbose messages. If errVerbose is TRUE then errMessage should be called whenever an error is detected even if it is known that the error belongs to a specific client. If errVerbose is FALSE then errMessage should be called only for errors that are not caused by a specific client.

Routine errPrintf is normally called as follows:

errPrintf(status, __FILE__, __LINE__,"<fmt>",...);

Where status is defined as:

FILE and LINE are defined as:

The remaining arguments are just like the arguments to the C printf routine. errVerbose determines if the filename and line number are shown.

An EPICS status code can also be converted to a string. If the supplied status code isn’t registered in the status code database then the raw status code number is converted into a string in the destination buffer.

#include "errMdef.h" 
void errSymLookup(long status, char pBuf, unsigned bufLength);

10.2.4 Obsolete Routines

int epicsPrintf(const char pformat, ...); 
int epicsVprintf(const char pformat,va_list pvar);

These are macros that call errlogPrintf and errlogVprintf. They are provided for compatibility.

10.3 errlog Listeners

Any code can receive errlog message. The following are the calls to add and remove a listener.

typedef void(⋆errlogListener) (void pvt,const char message); 
void errlogAddListener(errlogListener listener,void pPrivate); 
void errlogRemoveListener(errlogListener listener);

These routines add/remove a callback that receives each error message. These routines are the interface to the actual system wide error handlers.

10.4 errlogThread

The error message routines can be called by any non-interrupt level code. These routines pass the message to the errlog Thread. If any of the error message routines are called at interrupt level, epicsInterruptContextMessage is called with the message “errlogPrintf called from interrupt level”.

errlogThread manages the messages. Messages are placed in a message queue, which is read by errlogThread. The message queue uses a fixed block of memory to hold all messages. When the message queue is full additional messages are rejected but a count of missed messages is kept. The next time the message queue empties an extra message about the missed messages is generated.

The maximum message size is by default 256 characters. If a message is longer, the message is truncated and a message explaining that it was truncated is appended. There is a chance that long messages corrupt memory. This only happens if client code is defective. Long messages most likely result from “%s” formats with a bad string argument.

errlogThread passes each message to any registered listener.

10.5 console output and message queue size

The errlog system can also display messages on the ioc console. It calls epicsThreadIsOkToBlock to decide when to display the message. If it is OK to block, the message is displayed by the same thread that calls one of the errlog print routines. If it is not OK to block, errlogThread displays the messages.

Normally the errlog system displays all messages on the console. eltc can be used to suppress these messages.

int eltc(int yesno); /⋆ error log to console (0 or 1) ⋆/ 
int errlogInit(int bufsize); 
int errlogInit2(int bufsize, int maxMsgSize);

eltc determines if errlog task writes message to the console. During error message storms this command can be used to suppress console messages. A argument of 0 suppresses the messages, any other value lets messages go to the console.

errlogInit or errlogInit2 can be used to initialize the error logging system with a larger buffer and maximum message size. The default buffer size is 1280 bytes, and the default maximum message size is 256.

10.6 Status Codes

EPICS defined status values provide the following features:

IOC routines often return a long integer status value, encoded similar to the vxWorks error status encoding. On some modern architectures a long integer is more than 32 bits wide, but in order to keep the API compatible the status values are still passed as long integers, even though only 32 bits are ever used. The most significant 16 bits indicate the subsystem or module where the error occurred. The least significant 16 bits contain a subsystem-specific status value. In order that status values do not conflict with the vxWorks error status values, all subsystem numbers are greater than 500.

A header file errMdef.h defines macros for all the subsystem numbers. For example the database access routines use this module number:

#define M_dbAccess  (501 << 16)   /⋆Database Access Routines⋆/

There are header files for every IOC subsystem that returns standard status values. The status values are encoded with lines of the following format:

#define S_xxxxxxx value /⋆string value⋆/

For example:

#define S_dbAccessBadDBR (M_dbAccess|3) /⋆Invalid Database Request⋆/

For example, when dbGetField detects a bad database request type, it executes the statement:

return(S_dbAccessBadDBR);

The calling routine checks the return status as follows:

status = dbGetField(...); 
if(status) {/⋆ Call was not successful ⋆/ }

10.7 iocLog

NOTE: Many sites use CMLOG instead of iocLog. See the CMLOG documentation for details.

This consists of two modules: iocLogServer and iocLogClient. The client code runs on each ioc and listens for the messages generated locally by the errlog system. It also reports the messages from the vxWorks logMsg facility.

10.7.1 iocLogServer

This runs on a host. It receives messages for all enabled iocLogClients in the local area network. The messages are written to a file. Epics base provides a startup file “base/src/libCom/log/rc2.logServer”, which is a SystemV init script to start the server. Consult this script for details.

To start a log server on a UNIX or PC workstation you must first set the following environment variables and then run the executable “iocLogServer” on your PC or UNIX workstation.

EPICS_IOC_LOG_FILE_NAME

The name and path to the log file.

EPICS_IOC_LOG_FILE_LIMIT

The maximum size in characters for the log file. If the file grows larger than this limit the server will seek back to the beginning of the file and write new messages over the old messages starting from the beginning. If the value is zero then there is no limit on the size of the log file.

EPICS_IOC_LOG_FILE_COMMAND

A shell command string used to obtain the log file path name during initialization and in response to SIGHUP. The new path name will replace any path name supplied in EPICS_IOC_LOG_FILE_NAME.

Thus, if EPICS_IOC_LOG_FILE_NAME is “a/b/c.log” and EPICS_IOC_LOG_FILE_COMMAND returns “A/B” or “A/B/” the log server will be stored at “A/B/c.log”

If EPICS_IOC_LOG_FILE_COMMAND is empty then this behavior is disabled. This feature is used at some sites for switching the server to a new directory at a fixed time each day. This variable is currently used only by the UNIX version of the log server.

EPICS_IOC_LOG_PORT

THE TCP/IP port used by the log server.

To configure an IOC to log its messages it must have an environment variable EPICS_IOC_LOG_INET set to the IP address of the host that is running the log server, and EPICS_IOC_LOG_PORT to the TCP/IP port used by the log server.

Defaults for all of the above parameters are specified in the files $(EPICS_BASE)/config/CONFIG_SITE_ENV and $(EPICS_BASE)/config/CONFIG_ENV.

10.7.2 iocLogClient

This runs on each ioc. It is started by calling:

iocLogInit();

The global variable iocLogDisable can be used to enable/disable the messages from being sent to the server. Setting this variable to (0,1) (enables, disables) the messages generation. If iocLogDisable is set to 1 before calling iocLogInit then iocLogClient will not even initialize itself. iocLogDisable can also be changed to turn logging on or off.

iocLogClient calls errlogAddListener and sends each message to the iocLogServer.

10.7.3 Configuring a Private Log Server

In a testing environment it is desirable to use a private log server. This can be done as follows:

10.7.4 iocLogPrefix

Many sites run multiple soft IOCs on the same machine. With some log viewers like cmlogviewer it is not possible to distinguish between the log messages from these IOCs since their hostnames are all the same. One solution to this is to add a unique prefix to every log message.

The iocLogPrefix command can be run from the startup file during IOC initialization to establish such a prefix that will be prepended to every log message when it is sent to the iocLogServer.

For example, adding the following lines to your st.cmd file

  epicsEnvSet("IOC","sioc-b34-mc10");
  iocLogPrefix("fac=LI21 proc=${IOC} ");

will categorize all log messages from this IOC as belonging to the facility LI21 and to the process sioc-b34-mc10.

Note that log messages echoed to the IOC’s standard output will not show the prefix, it only appears in the version sent to the log server. iocLogPrefix should appear fairly early in the startup script so the IOC doesn’t try to send any log messages without the prefix. Once the prefix has been set, it cannot be changed without rebooting the IOC. One can determine if a log prefix has been set using iocLogShow.