Juniper NETWORKS STRM - TECHNICAL NOTE REV 6-2008, STRM User Manual

JUNIPER NETWORKS STRM
T

ECHNICAL NOTE

USING EXTENSION DOCUMENTS

JUNE 2008

Device extensions allow you to modify how a DSM parses logs, which is useful for
resolving parsing issues. However, before you define a device extension, you must
build an extension document. This document provides information on defining an
extension document including:

• About Extension Document s

• Understanding Extension Document Elements

• Creating Extension Documents

• Device Type IDs

This document assumes an advanced knowledge of XML coding.

About Extension
Documents

The extension document is specified in Extensible Markup Language (XML)
format. You can create and edit the document using any common word processing
application. You can create multiple extension documents and associate an
extension document to various device types. Using an extension document, you
can resolve parsing issues, such as:

• Fixing an event that has missing or incorrect fields (for example, if the

username is not being parsed).

• Completing the parsing of an event when the DSM to which it is attached fails to

produce a result. Any new events produced by the device extension are
associated to the device that failed to parse the original payload. This action
prevents these events from appearing as un-parsed in the STRM interface.

XML format allows for a simplification of the extension document’s organization,
and verifies the extension document’s contents. Using this format requires that all
regular expressions be contained in character data (CDATA) sections to prevent
the special characters that are needed by regular expressions from interfering with
the markup format. For example:

<pattern id="Protocol" case-insensitive="true" xmlns="">
<![CDATA[(tcp|udp|icmp|gre)]]></pattern>

Where (tcp|udp|icmp|gre) is the actual regular expression pattern.

Release 2008.2

2

The extension document allows you to parse a device’s payload. Within the
extension document you can include statements of varying degrees of complexity,
as required to parse the desired information.

Understanding
Extension
Document
Elements

Patterns Rather than associating a regular expression directly with a particular field name,

This section explains the two main divisions of the extension document:

• Patterns

• Match Groups

patterns (

patterns) are declared separately at the top of the extension document

and can be subsequently referenced multiple times within the file.

Table 1 Pattern Parameters

Parameter Description

id

(Required)

case-insensitive

(Optional)

trim-whitespace

(Optional)

Specify a regular string that is unique within the extension
document.

Specify if you wish the pattern to ignore character case when
doing a match, for example abc is the same as ABC.

If not specified, this parameter defaults to false.
Specify if you wish the pattern to ignore white space and

carriage returns. If the CDATA sections are split onto different
lines, this parameter ensures that any extra spaces and
carriage returns are not interpreted as part of the pattern.

If not specified, this parameter defaults to false.

Match Groups A match group (match-group) is a set of patterns used for parsing or modifying

Release 2008.2

one or more types of events. A matcher is an entity within a match group that is
parsed (for example, EventName) and is paired with the appropriate pattern and
group for parsing. Any number of match groups may appear in the extension
document.

Table 2 Match Group Parameters

Parameter Description

order

(Required)

description
(Optional)

Specify an integer greater than zero to specify the order in which
the match groups are executed. It must be unique within the
extension document.

Specify a description for the match group, which can be any str ing.
This information may appear in the logs.

If not specified, this parameter defaults to empty.

Understanding Extension Document Elements 3

Table 2 Match Group Parameters (continued)

Parameter Description

device-type­id-override

(Optional)

Specify a different device’s QID. Allows the particular match group
to search in the specified device for the event type. It must be a
valid device type ID, represented as an integer. A list of device type
IDs is presented in

If not specified, this parameter defaults to the device type of the
device to which the extension is attached.

Table 6.

Match groups can have up to three different types of entities:

• Matcher (matcher)

• Single-Event Modifier (event-match-single)

• Multi-Event Modifier (event-match-multiple)

Matcher (matcher)

A matcher entity is a field that is parsed (for example, EventName) and is paired
with the appropriate pattern and group for parsing. Matchers have an associated
order, so if multiple matchers are specified for the same field name, the matchers
are executed in the order presented until a successful parse is found or a failure
occurs.

Table 3 Matcher Entity Parameters

Parameter Description

field

(Required)

pattern-id

(Required)

order

(Required)

Specify the field to which you wish the pattern to apply,
for example, EventName, or SourceIp. See
list of valid field names.

Specify the pattern you wish to use when parsing the
field out of the payload. This value must match (including
case) the ID parameter of the pattern previously defined
in a pattern ID parameter (

Specify the order that you wish this pattern to attempt
among matchers assigned to the same field. If there are
two matchers assigned to the EventName field, the one
with the lowest order is attempted first.

Table 4 for a

Table 1).

Release 2008.2

4

Table 3 Matcher Entity Parameters (continued)

Parameter Description

capture-group

(Optional)

Specify capture group(s), as denoted in the regular
expression inside parenthesis ( ). These captures are
indexed starting at one and processed from left to right
in the pattern. The capture-group field must be a
positive integer less than or equal to the number of
capture groups contained in the pattern. The default
value is zero, which will be the entire match.

For example, you can define a single pattern for a
source IP address and port; where the SourceIp matcher
may use a capture group of 1, and the SourcePort
matcher may use a capture group of 2, but only one
pattern needs to be defined.

This field has a dual purpose when combined with the

enable-substitutions parameter.

enable-substitutions

(Optional)

Specify this Boolean parameter as true when a field
cannot be adequately represented with a straight group
capture.

You can combine multiple groups together with extra
text to form a value. This parameter enables that
behavior.

This parameter changes the meaning of the
capture-group parameter. The capture-group
parameter creates the new value, and group
substitutions are specified using \x where x is a group
number from 1 to 9. You may use groups multiple times,
and any free-form text can also be inserted into the
value. For example, if you need to form a value out of
group 1, followed by an underscore, followed by group 2,
an @, and then group 1 again, the appropriate
capture-group syntax is:

capture-group=”\1_\2@\1”

In another example, a MAC address is separated by
colons, but STRM assumes that MAC addresses are
hyphen separated. The syntax to parse and capture the
individual portions is:

capture-group=”\1:\2:\3:\4:\5:\6”

If no groups are specified in the capture-group when
substitutions are enabled, a direct text replacement
occurs.

Default is false.

Release 2008.2

Understanding Extension Document Elements 5

Table 4 provides a list of valid field names for use in the matcher field parameter (see
Table 3).

Table 4 Matcher Field Names

Field Name Description

EventName
(Required)

Specify the event name to be retrieved from the QID to
identify the event.

EventCategory Specify an event category for any event with a category

not handled by an event-match-single entity or an
event-match-multiple entity.

Combined with EventName, EventCategory is used to

search for the event in the QID.
SourceIp Specify the source IP address for the message.
SourcePort Specify the source port for the message.
SourceIpPreNAT Specify the source IP address for the message before

Network Address Translation (NAT) occurs.
SourceIpPostNAT Specify the source IP address for the message after NAT

occurs.
SourceMAC Specify the source MAC address fo r the message.
SourcePortPreNAT Specify the source port for the message before NAT

occurs.
SourcePortPostNAT Specify the source port for the message after NAT occurs.
DestinationIp Specify the destination IP address for the message.
DestinationPort Specify the destination port for the message.
DestinationIpPreNAT Specify the destination IP address for the message before

NAT occurs.
DestinationIpPostNAT Specify the destination IP address for the message after

NAT occurs.
DestinationPortPreNAT Specify the destination port for the message before NAT

occurs.
DestinationPortPostNAT Specify the destination port for the message after NAT

occurs.
DestinationMAC Specify the destination MAC address for the message.
DeviceTime Specify the time that the event was sent, according to the

device (this is NOT the time that the event arrived).

STRM detects timestamps in the following formats:

• Valid syslog timestamp in the form of mm dd hh:mm:ss,

for example:

Jan 13 12:33:10

• Current locale timestamp

Any other formats will not properly convert.

Release 2008.2