Log Monitoring

Introduction

With NetXMS you can monitor changes in text log files, Windows Event Log, and built-in syslog server. All log monitoring done by agents, except for built-in syslog server. In general, log processing goes as following:

1. When new line added to log file, it is passed to appropriate log parser;
2. If line matched one of the patterns, event associated with this pattern sent to server;
3. Server receives event and passess to event processing policy as usual, with event source set to node from which event was received.

Agent Configuration for Log Monitoring

To be able to monitor logs with NetXMS agent, you should load LOGWATCH subagent and define parser configuration for each log file you wish to monitor. Example of agent configuration file:

SubAgent = logwatch.nsm

# Below is log parsers definitions
*LOGWATCH
Parser = C:\NetXMS\parser1.xml
Parser = C:\NetXMS\parser2.xml

Syslog Monitoring

NetXMS has built-in syslog server, which can be used to receive logs from network devices and servers. It is also possible to parse incoming syslog messages in a way similar to Windows Event Log monitoring. To parse syslog messages, LOGWATCH subagent is not needed – parsing is done by the server itself. You should only create parser configuration file for syslog in console via Control Panel – Syslog Parser.

Parser Definition File

Overview

Parser definition file is an XML document with the following structure:

<parser>
	<file>file name</file>
	<macros>
		<macro name="name">macro body</macro>
		<!-- more <macro> tags can follow --> 
	</macros>
	<rules>
		<rule>
			<match>regexp</match>
			<id>event id</id>
			<level>severity level</level>
			<source>event source</source>
			<event>event</event>
			<context>context</context>
		</rule>
		<!-- more <rule> tags can follow -->
	</rules>
</parser>

Entire <macros> section can be omited, and inside <rule> tag only <match> is mandatory.

Global Parser Options

In the <parser> tag you can specify the following options:

Option	Description	Default value
processAll	If this option set to 1, parser will always pass log record through all rules. If this option set to 0, processing will stop after first match.	0
trace	Trace level.	0

<file> Tag

In the <file> tag you should specify log file to apply this parser to. To specify Windows Event Log, prepend it's name with asterisk (*), for example *System.

Macros

In the <macros> section you can define macros for use in matching rules. For example, it can be useful to define macro for a timestamp preceding each log record and use it in matching rules instead of actual regexp. You can define as many macros as you wish, each within it's own <macro> tag. Each macro should have unique name, defined in name attribute, and can be used in matching rules in form @{name}.

Example: you need to parse log file where each line starts with timestamp in format dd/mm/yy HH:MM:SS. You can define the following macro:

<macro name="timestamp">[0-9]{2}/[0-9]{2}/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}</macro>

and then use it in matching rules:

<rules>
	<rule>
		<match>@{timestamp}.*([A-Za-z]+) failed.*</match>
		<event>12345</event>
	</rule>
	<rule>
		<match>@{timestamp}.*error.*</match>
		<event>45678</event>
	</rule>
</rules>

Please note that <macros> section always should be located before <rules> section in parser definition file.

Matching rules

In the <rules> section you define matching rules for log records. Each rule placed inside it's own <rule> tag. Each rule can have aditional options:

Option	Description	Default value
break	If this option set to 1 and curent line match to regular expression in the rule, parser will stop processing of current line, even if global parser option processAll was set to 1. If this option set to 0 (which is by default), processing will stop according to processAll option settings.	0
context	Name of the context this rule belongs to. If this option is set, rule will be processed only if given context was already activated with <context> tag in one of the rules processed earlier (it can be either same line or one of the previous lines).	empty

Inside the <rule> section are additional tags: <match>, <description>, <event>, and <context>. Only <match> section are mandatory – it specifies regular expression against which log record should be matched. All other tags are optional and defines parser behavior if record match regular expression.

<match> Tag

Tag <match> contains POSIX regular expression used to match log records. Parts enclosed in parenthesis can be extracted from log record and passed as arguments of generated event. You can use macros defined in <macros> section. Also, is is possible to define inverted match rules (rules when log record considered matching if it does not match regular expression). Inverted match can be set by setting attribute invert to 1.

Some examples:

<match>^Error: (.*)</match>

This regular expression will match any line started with word Error: , and everything after that word will be extracted from log record for use with event.

<match>[0-9]{3}</match>

This regular expression will match any line containing at least 3 consecutive digits.

<match invert="1">abc</match>

This regular expression will match any line not containing character sequence abc.

<id> Tag

Tag <id> can be used to filter records from Windows Event Log by event ID. You can specify either single event ID or ID range (by using two nubers separated with minus sign). For example:

<id>7</id>

will match records with event ID equal 7, and

<id>10-20</id>

will match records with ID in range from 10 to 20 (inclusive).

This tag has no effect for text log files, and can be used as a synonim for <facility> tag for syslog monitoring.

<source> Tag

Tag <source> can be used to filter records from Windows Event Log by event source. You can specify exact event source name or pattern with * and ? metacharacters.

Some examples:

<source>Tcpip</source>

will match records with event source "Tcpip" (case-insensetive), and

<source>X*</source>

will match records with event source started from letter "X".

This tag has no effect for text log files, and can be used as a synonim for <tag> tag for syslog monitoring.

<level> Tag

Tag <level> can be used to filter records from Windows Event log by event severity level (also called event type in older Windows versions). Each severity level has it's own code, and to filter by multiple severity levels you should specify sum of appropriate codes. Severity level codes are following:

1	Error
2	Warning
4	Information
8	Audit Success
16	Audit Failure

Some examples:

<level>1</level>

will match all records with severity level "Error", and

<level>6</level>

will match all records with severity level "Warning" or "Information".

This tag has no effect for text log files, and can be used as a synonim for <severity> tag for syslog monitoring.

<facility> Tag

Tag <facility> can be used to filter syslog records (received by NetXMS built-in syslog server) by facility code. The following facility codes can be used:

0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16	local use 0 (local0)
17	local use 1 (local1)
18	local use 2 (local2)
19	local use 3 (local3)
20	local use 4 (local4)
21	local use 5 (local5)
22	local use 6 (local6)
23	local use 7 (local7)

You can specify either single facility code or facility code range (by using two nubers separated with minus sign). For example:

<facility>7</facility>

will match records with facility code equal 7, and

<facility>10-20</facility>

will match records with facility code in range from 10 to 20 (inclusive).

This tag has no effect for text log files, and can be used as a synonim for <id> tag for Windows Event Log monitoring.

<tag> Tag

Tag <tag> can be used to filter syslog records (received by NetXMS built-in syslog server) by content of "tag" field. You can specify exact value or pattern with * and ? Metacharacters.

Some examples:

<tag>httpd</tag>

will match records with tag "httpd" (case-insensetive), and

<tag>X*</tag>

will match records with tag started from letter "X".

This tag has no effect for text log files, and can be used as a synonim for <source> tag for Windows Event Log monitoring.

<severity> Tag

Tag <severity> can be used to filter syslog records (received by NetXMS built-in syslog server) by severity level. Each severity level has it's own code, and to filter by multiple severity levels you should specify sum of appropriate codes. Severity level codes are following:

1	Emergency
2	Alert
4	Critical
8	Error
16	Warning
32	Notice
64	Informational
128	Debug

Some examples:

<severity>1</severity>

will match all records with severity level "Emergency", and

<severity>6</severity>

will match all records with severity level "Alert" or "Critical".

This tag has no effect for text log files, and can be used as a synonim for <level> tag for Windows Event Log monitoring.

<description> Tag

Tag <description> contains textual description of the rule, which will be shown in parser trace.

<event> Tag

Tag <event> defines event to be generated if current log record match to regular expression defined in <match> tag. Inside <event> tag you should specify event code to be generated (or event name if you configure server-side syslog parsing). If you wish to pass parts of log record text extracted with regular expression as event's parameters, you should specify correct number of parameters in params attribute.

<context> Tag

Tag <context> defines activation or deactivation of contexts. It has the following format:

<context action="action" reset="reset mode">context name</context>

Possible actions are:

set	Activate (set "active" flag of) given context.
clear	Deactivate (clear "active" flag of) given context.

Reset mode determines how context will be deactivated (reset). Possible values for reset mode are:

auto	Deactivate context automatically after first match in context (match rule with context attribute set to given context).
manual	Context can be deactivated only by explicit <context action="clear"> statement.

Both action and reset attributes can be omited; default value for action is set, and default value for reset is auto.

Examples of Parser Definition File

1. Generate event with code 100000 if line in the log file /var/log/messages contains word error:

<parser>
	<file>/var/log/messages</file>
	<rules>
		<rule>
			<match>error</match>
			<event>100000</event>
		</rule>
	</rules>
</parser>

2. Generate event with code 200000 if line in the log file C:\demo.log contains word process: and is immediatelly following line containing text process startup failed; everything after word process: will be sent as event's parameter.

<parser>
	<file>C:\demo.log</file>
	<rules>
		<rule>
			<match>process startup failed</match>
			<context action="set" reset="auto">STARTUP_FAILED</context>
		</rule>
		<rule context="STARTUP_FAILED">
			<match>process:(.*)</match>
			<event params="1">200000</event>
		</rule>
	</rules>
</parser>