5. Event and Accounting Logs

This chapter provides information about configuring event and accounting logs on the 7210 SAS.

5.1. Logging Overview

The two primary types of logging supported in the 7210 SAS OS are event logging and accounting logs.

Event logging controls the generation, dissemination and recording of system events for monitoring status and troubleshooting faults within the system. The 7210 SAS groups events into three major categories or event sources:

  1. Security events
    Events that pertain to attempts to breach system security.
  2. Change events
    Events that pertain to the configuration and operation of the node.
  3. Main events
    Events that pertain to applications that are not assigned to other event categories/sources.
  4. Debug events
    Events that pertain to trace or other debugging information.

The following are events within the 7210 SAS and have the following characteristics:

  1. a time stamp in UTC or local time
  2. the generating application
  3. a unique event ID within the application
  4. the VRF-ID
  5. a subject identifying the affected object
  6. a short text description

Event control assigns the severity for each application event and whether the event should be generated or suppressed. The severity numbers and severity names supported in the 7210 SAS OS conform to ITU standards M.3100 X.733 and X.21 and are listed in Table 41.

Table 41:  Event Severity Levels  

Severity Number

Severity Name

1

cleared

2

indeterminate (info)

3

critical

4

major

5

minor

6

warning

Events that are suppressed by event control will not generate any event log entries. Event control maintains a count of the number of events generated (logged) and dropped (suppressed) for each application event. The severity of an application event can be configured in event control.

An event log within the 7210 SAS OS associates the event sources with logging destinations. Examples of logging destinations include, the console session, a specific telnet or SSH session, memory logs, file destinations, SNMP trap groups and syslog destinations. A log filter policy can be associated with the event log to control which events will be logged in the event log based on combinations of application, severity, event ID range, VRF ID, and the subject of the event.

The 7210 SAS accounting logs collect comprehensive accounting statistics to support a variety of billing models. The routers collect accounting data on services and network ports on a per-service class basis. In addition to gathering information critical for service billing, accounting records can be analyzed to provide insight about customer service trends for potential service revenue opportunities. Accounting statistics on network ports can be used to track link utilization and network traffic pattern trends. This information is valuable for traffic engineering and capacity planning within the network core.

Accounting statistics are collected according to the parameters defined within the context of an accounting policy. Accounting policies are applied to access objects (such as access ports and SAPs or network objects (such as SDPs, network ports, network IP interface). Accounting statistics are collected by counters for individual service meters defined on the customer SAP or by the counters within forwarding class (FC) queues defined on the network ports.

The type of record defined within the accounting policy determines where a policy is applied, what statistics are collected and time interval at which to collect statistics.

The “location” field of the file-id lets the user configure the device and store it in any directory. The default value is cf1:, but it can also be uf1: (for devices supporting USB) and uf1: and cf2: for 7210 SAS-T

5.2. Log Destinations

Both event logs and accounting logs use a common mechanism for referencing a log destination.

Only a single log destination can be associated with an event log or with an accounting log. An event log can be associated with multiple event sources, but it can only have a single log destination.

A file destination is the only type of log destination that can be configured for an accounting log.

5.2.1. Console

Sending events to a console destination means the message will be sent to the system console The console device can be used as an event log destination.

5.2.2. Session

A session destination is a temporary log destination which directs entries to the active telnet or SSH session for the duration of the session. When the session is terminated, for example, when the user logs out, the event log is removed. Event logs configured with a session destination are not stored in the configuration file. Event logs can direct log entries to the session destination.

5.2.3. Memory Logs

A memory log is a circular buffer. When the log is full, the oldest entry in the log is replaced with the new entry. When a memory log is created, the specific number of entries it can hold can be specified, otherwise it will assume a default size. An event log can send entries to a memory log destination.

5.2.4. Log Files

Log files can be used by both event logs and accounting logs and are stored on the compact flash devices (specifically cf1:) in the file system.

A log file is identified with a single log file ID, but a log file will generally be composed of a number individual files in the file system. A log file is configured with a rollover parameter, expressed in minutes, which represents the length of time an individual log file should be written to before a new file is created for the relevant log file ID. The rollover time is checked only when an update to the log is performed. Therefore, complying to this rule is subject to the incoming rate of the data being logged. For example, if the rate is very low, the actual rollover time may be longer than the configured value.

The retention time for a log file specifies the amount of time the file should be retained on the system based on the creation date and time of the file. The system continuously checks for log files with expired retention periods once every hour and deletes as many files as possible during a ten second interval.

When a log file is created, only the compact flash device for the log file is specified. Log files are created in specific subdirectories with standardized names depending on the type of information stored in the log file.

Event log files are always created in the \log directory on the specified compact flash device. The naming convention for event log files is:

log eeff-timestamp

where:

  1. ee is the event log ID
  2. ff is the log file destination ID
  3. timestamp is the timestamp when the file is created in the form of yyyymmdd-hhmmss where:
    1. yyyy is the four-digit year (for example, 2017)
    2. mm is the two digit number representing the month (for example, 12 for December)
    3. dd is the two digit number representing the day of the month (for example, 03 for the 3rd of the month)
    4. hh is the two digit hour in a 24-hour clock (for example, 04 for 4 a.m.)
    5. mm is the two digit minute (for example, 30 for 30 minutes past the hour)
    6. ss is the two digit second (for example, 14 for 14 seconds)

Accounting log files are created in the \act-collect directory on a compact flash device (cf1). The naming convention for accounting log files is nearly the same as for log files except the prefix act is used instead of the prefix log. The naming convention for accounting logs is:

act aaff-timestamp.xml.gz

where:

  1. aa is the accounting policy ID
  2. ff is the log file destination ID
  3. timestamp is the timestamp when the file is created in the form of yyyymmdd-hhmmss where:
    1. yyyy is the four-digit year (for example, 2007)
    2. mm is the two digit number representing the month (for example, 12 for December)
    3. dd is the two digit number representing the day of the month (for example, 03 for the 3rd of the month)
    4. hh is the two digit hour in a 24-hour clock (for example, 04 for 4 a.m.)
    5. mm is the two digit minute (for example, 30 for 30 minutes past the hour)
    ss is the two digit second (for example, 14 for 14 seconds)

Accounting logs are .xml files created in a compressed format and have a .gz extension.

The \act-collect directory is where active accounting logs are written. When an accounting log is rolled over, the active file is closed and archived in the \act directory before a new active accounting log file created in \act-collect.

5.2.5. SNMP Trap Group

An event log can be configured to send events to SNMP trap receivers by specifying an SNMP trap group destination.

An SNMP trap group can have multiple trap targets. Each trap target can have different operational parameters.

A trap destination has the following properties:

  1. The IP address of the trap receiver.
  2. The UDP port used to send the SNMP trap.
  3. SNMP version (v1, v2c, or v3) used to format the SNMP notification.
  4. SNMP community name for SNMPv1 and SNMPv2c receivers.
  5. Security name and level for SNMPv3 trap receivers.

For SNMP traps that will be sent in-band, the source IP address of the trap is the system IP address of the 7210 SAS.

Each trap target destination of a trap group receives the identical sequence of events as defined by the log ID and the associated sources and log filter applied.

5.2.6. Syslog

An event log can be configured to send events to one syslog destination. Syslog destinations have the following properties:

  1. Syslog server IP address.
  2. The UDP port used to send the syslog message.
  3. The Syslog Facility Code (0 - 23) (default 23 - local 7).
  4. The Syslog Severity Threshold (0 - 7) - events exceeding the configured level will be sent.

Because syslog uses eight severity levels whereas the 7210 SAS-Series uses six internal severity levels, the severity levels are mapped to syslog severities. Table 42 lists the severity level mappings to syslog severities.

Table 42:  7210 SAS to Syslog Severity Level Mappings 

Severity Level

Numerical Severity (highest to lowest)

Syslog Configured Severity

Definition

0

emergency

System is unusable

critical (3)

1

alert

Action must be taken immediately

major (4)

2

critical

Critical conditions

minor (5)

3

error

Error conditions

warning (6)

4

warning

Warning conditions

5

notice

Normal but significant condition

cleared (1) indeterminate (2)

6

info

Informational messages

7

debug

Debug-level messages

5.3. Event Logs

Event logs are the means of recording system generated events for later analysis. Events are messages generated by the system by applications or processes within the 7210 SAS.

Figure 8 shows a function block diagram of event logging.

Figure 8:  Event Logging Block Diagram 

5.3.1. Event Sources

In Figure 8, the event sources are the main categories of events that feed the log manager.

  1. Security — The security event source is all events that affect attempts to breach system security such as failed login attempts, attempts to access MIB tables to which the user is not granted access or attempts to enter a branch of the CLI to which access has not been granted. Security events are generated by the SECURITY application and the authenticationFailure event in the SNMP application.
  2. Change — The change activity event source is all events that directly affect the configuration or operation of the node. Change events are generated by the USER application. The Change event stream also includes the tmnxConfigModify(#2006), tmnxConfigCreate (#2007), tmnxConfigDelete (#2008) and tmnxStateChange (#2009) change events from the SYSTEM application.
  3. Debug — The debug event source is the debugging configuration that has been enabled on the system. Debug events are generated by the DEBUG application.
  4. Main — The main event source receives events from all other applications within the 7210 SAS.

Examples of applications within 7210 SAS include IP, MPLS, OSPF, CLI, services, and so on. The following output is an example of the show log applications command output, which displays all applications.

*A:ALU-7210# show log applications
==================================
Log Event Application Names
==================================
Application Name
----------------------------------
CHASSIS
DEBUG
DOT1AG
DOT1X
EFM_OAM
FILTER
IGMP
IP
LAG
LOGGER
MIRROR
NTP
OAM
PORT
QOS
SECURITY
SNMP
STP
SVCMGR
SYSTEM
TIP
TOD
USER
VRTR
==================================
*A:ALU-7210#
 

5.3.2. Event Control

Event control preprocesses the events generated by applications before the event is passed into the main event stream. Event control assigns a severity to application events and can either forward the event to the main event source or suppress the event. Suppressed events are counted in event control, but these events will not generate log entries as it never reaches the log manager.

Simple event throttling is another method of event control and is configured similarly to the generation and suppression options. See Simple Logger Event Throttling.

Events are assigned a default severity level in the system, but the application event severities can be changed by the user.

Application events contain an event number and description that describes why the event is generated. The event number is unique within an application, but the number can be duplicated in other applications.

The following example, generated by querying event control for application generated events, displays a partial list of event numbers and names.

router# show log event-control
=======================================================================
Log Events
=======================================================================
Application
 ID#    Event Name                       P   g/s     Logged     Dropped
-----------------------------------------------------------------------
CHASSIS:
   2001 cardFailure                      MA  gen          0           0
   2002 cardInserted                     MI  gen          2           0
   2003 cardRemoved                      MI  gen          0           0
   2004 cardWrong                        MI  gen          0           0
   2005 EnvTemperatureTooHigh            MA  gen          0           0
   2006 fanFailure                       CR  gen          0           0
...
EFM_OAM:
   2001 tmnxDot3OamPeerChanged           MI  gen          0           0
   2002 tmnxDot3OamLoopDetected          MI  gen          0           0
   2003 tmnxDot3OamLoopCleared           MI  gen          0           0
FILTER:
   2001 tIPFilterPBRPacketsDrop          WA  gen          0           0
   2002 tFilterEntryActivationFailed     WA  gen          0           0
   2003 tFilterEntryActivationRestored   WA  gen          0           0
IGMP:
   2001 vRtrIgmpIfRxQueryVerMismatch     WA  gen          0           0
   2002 vRtrIgmpIfCModeRxQueryMismatch   WA  gen          0           0
   2003 vRtrIgmpMaxGrpsLimitExceeded     WA  gen          0           0
   2004 vRtrIgmpMcacPlcyDropped          WA  gen          0           0
IP:
L  2001 clearRTMError                    MI  gen          0           0
L  2002 ipEtherBroadcast                 MI  gen          0           0
L  2003 ipDuplicateAddress               MI  gen          0           0
L  2004 ipArpInfoOverwritten             MI  gen          0           0
L  2005 fibAddFailed                     MA  gen          0           0
...
SYSTEM:
   2001 stiDateAndTimeChanged            WA  gen          0           0
   2002 ssiSaveConfigSucceeded           MA  gen          1           0
   2003 ssiSaveConfigFailed              CR  gen          1           0
   2004 sbiBootConfig                    MA  gen          1           0
   2005 sbiBootSnmpd                     MA  gen          1           0
...
VRTR:
   2001 tmnxVRtrMidRouteTCA              MI  gen          0           0
   2002 tmnxVRtrHighRouteTCA             MI  gen          0           0
   2003 tmnxVRtrHighRouteCleared         MI  gen          0           0
...
=======================================================================
router# 

5.3.3. Log Manager and Event Logs

Events that are forwarded by event control are sent to the log manager. The log manager manages the event logs in the system and the relationships between the log sources, event logs and log destinations, and log filter policies.

An event log has the following properties:

  1. A unique log ID
    The log ID is a short, numeric identifier for the event log. A maximum of ten logs can be configured at a time.
  2. One or more log sources
    The source stream or streams to be sent to log destinations can be specified. The source must be identified before the destination can be specified. The events can be from the main event stream, events in the security event stream, or events in the user activity stream.
  3. One event log destination
    A log can only have a single destination. The destination for the log ID destination can be one of console, session, syslog, snmp-trap-group, memory, or a file on the local file system.
  4. An optional event filter policy
    An event filter policy defines whether to forward or drop an event or trap-based on match criteria.

5.3.4. Event Filter Policies

The log manager uses event filter policies to allow fine control over which events are forwarded or dropped based on various criteria. Like other policies with the 7210 SAS, filter policies have a default action. The default actions are either:

  1. Forward
  2. Drop

Filter policies also include a number of filter policy entries that are identified with an entry ID and define specific match criteria and a forward or drop action for the match criteria.

Each entry contains a combination of matching criteria that define the application, event number, router, severity, and subject conditions. The entry action determines how the packets should be treated if they have met the match criteria.

Entries are evaluated in order from the lowest to the highest entry ID. The first matching event is subject to the forward or drop action for that entry.

Valid operators are described in Table 43.

Table 43:  Valid Operators 

Operator

Description

eq

equal to

neq

not equal to

lt

less than

lte

less than or equal to

gt

greater than

gte

greater than or equal to

A match criteria entry can include combinations of:

  1. Equal to or not equal to a specific system application.
  2. Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal to an event number within the application.
  3. Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal to a severity level.
  4. Equal to or not equal to a router name string or regular expression match.
  5. Equal to or not equal to an event subject string or regular expression match.

5.3.5. Event Log Entries

Log entries that are forwarded to a destination are formatted in a way appropriate for the specific destination whether it be recorded to a file or sent as an SNMP trap, but log event entries have common elements or properties. All application generated events have the following properties:

  1. A time stamp in UTC or local time.
  2. The generating application.
  3. A unique event ID within the application.
  4. A router name identifying the VRF-ID that generated the event.
  5. A subject identifying the affected object.
  6. A short text description.

The general format for an event in an event log with either a memory, console or file destination is as follows.

nnnn YYYY/MM/DD HH:MM:SS.SS <severity>:<application> # <event_id> <router-
name> <subject> description

The following is an event log example:

475 2006/11/27 00:19:40.38 WARNING: SNMP #2007 Base 1/1/1 
"interface 1/1/1 came up" 

The specific elements that compose the general format are described in Table 44.

Table 44:  Log Entry Field Descriptions 

Label

Description

nnnn

The log entry sequence number.

YYYY/MM/DD

The UTC date stamp for the log entry.

YYYY — Year

MM — Month

DD — Date

HH:MM:SS.SS

The UTC time stamp for the event.

HH — Hours (24 hour format)

MM — Minutes

SS.SS — Seconds

<severity>

The severity level name of the event.

CLEARED A cleared event (severity number 1).

INFO An indeterminate/informational severity event (severity level 2).

CRITICAL A critical severity event (severity level 3).

MAJOR A major severity event (severity level 4).

MINOR A minor severity event (severity level 5).

WARNING A warning severity event (severity 6).

<application>

The application generating the log message.

<event_id>

The application event ID number for the event.

<router>

The router name representing the VRF-ID that generated the event.

<subject>

The subject/affected object for the event.

<description>

A text description of the event.

5.3.6. Simple Logger Event Throttling

Simple event throttling provides a mechanism to protect event receivers from being overloaded when a scenario causes many events to be generated in a very short period of time. A throttling rate, # events/# seconds, can be configured. Specific event types can be configured to be throttled. When the throttling event limit is exceeded in a throttling interval, any further events of that type cause the dropped events counter to be incremented. Dropped events counts are displayed by the show>log>event-control context. Events are dropped before being sent to one of the logger event collector tasks. There is no record of the details of the dropped events and therefore no way to retrieve event history data lost by this throttling method.

A particular event type can be generated by multiple managed objects within the system. At the point this throttling method is applied the logger application has no information about the managed object that generated the event and cannot distinguish between events generated by object “A” from events generated by object “B”. If the events have the same event-id, they are throttled regardless of the managed object that generated them. It also does not know which events may eventually be logged to destination log-id <n> from events that will be logged to destination log-id <m>.

Throttle rate applies commonly to all event types. It is not configurable for a specific event-type.

A timer task checks for events dropped by throttling when the throttle interval expires. If any events have been dropped, a TIMETRA-SYSTEM-MIB::tmnxTrapDropped notification is sent.

5.3.7. Default System Log

Log 99 is a preconfigured memory-based log which logs events from the main event source (not security, debug, etc.). Log 99 exists by default.

The following is a sample log 99 configuration output.

ALA-1>config>log# info detail
#------------------------------------------
echo "Log Configuration "
#------------------------------------------
...
        snmp-trap-group 7
        exit
...
        log-id 99
            description "Default system log"
            no filter
            from main
            to memory 500
            no shutdown
        exit
----------------------------------------------
ALA-1>config>log#

5.4. Accounting Logs

Before an accounting policy can be created, a target log file must be created to collect the accounting records. The files are stored in system memory on compact flash (cf1:) in compressed (tar) XML format and can be retrieved using FTP or SCP.

A file ID can only be assigned to either one event log ID or one accounting log.

5.4.1. Accounting Records

An accounting policy must define a record name and collection interval. Only one record name can be configured per accounting policy. Also, a record name can only be used in one accounting policy.

The record name, sub-record types, and default collection period for access and network accounting policies are shown as follows.

The 7210 SAS provides 21 accounting records for the following accounting policies:

  1. access
  2. accessport
  3. network
  4. networkIf
  5. sdp

When creating accounting policies, only one (of each) access, accessport, network, networkIf and sdp accounting-policy can be defined as default. If statistics collection is enabled on an accounting object and no accounting policy is applied, the default accounting policy is used. If a default policy is not defined, no statistics are collected unless a specifically defined accounting policy is applied.

Each accounting record name is composed of one or more sub-records composed of multiple fields.

For VLL and VPLS services on the 7210 SAS, the user can run the config>service>epipe/vpls>sap>statistics>ingress>counter-mode {in-out-profile-count | forward-drop-count} to change the counter-mode of counters associated with SAP ingress meters or policers. Refer to the 7210 SAS-M, T, Mxp, Sx, S Services Guide and 7210 SAS-R6, R12 Services Guide for more information about the counter-mode command.

The statistics collected for the following accounting records vary based on the counter-mode selected:

  1. Service-ingress-octets
  2. Service-ingress-packets
  3. Combined-service-ingress
  4. Complete-service-ingress-egress

See Appendix: Accounting Record Name Details for 7210 SAS Platforms for more information about accounting records and counters for the 7210 SAS platforms.

5.4.2. Configuration Guidelines

Before modifying the counter, disable account log generation. Execute the no collect-stats command. Changing the mode of the counter results in loss of previously collected counts and resets the counter.

5.4.3. Accounting Files

When a policy has been created and applied to a service or network port, the accounting file is stored on the compact flash in a compressed XML file format. The device creates two directories on the compact flash to store the files. The following output displays a directory named act-collect that holds accounting files that are open and actively collecting statistics. The directory named act stores the files that have been closed and are awaiting retrieval.

ALA-1>file cf1:\# dir act*
12/19/2006 06:08a <DIR> act-collect
12/19/2006 06:08a <DIR> act
 
ALA-1>file cf1:\act-collect\ # dir
Directory of cf1:\act-collect#
12/23/2006 01:46a <DIR> .
12/23/2006 12:47a <DIR>  ..
12/23/2006 01:46a 112 act1111-20031223-014658.xml.gz
12/23/2006 01:38a 197 act1212-20031223-013800.xml.gz

Accounting files always have the prefix act followed by the accounting policy ID, log ID and timestamp. The accounting log file naming and log file destination properties like rollover and retention are described in more detail in Log Files.

5.4.4. Design Considerations

When preparing for an accounting policy deployment, verify that data collection, file rollover, and file retention intervals are properly tuned for the amount of statistics to be collected.

If the accounting policy collection interval is too brief, there may be insufficient time to store the data from all the services within the specified interval. If that is the case, some records may be lost or incomplete. Interval time, record types, and number of services using an accounting policy are all factors that should be considered when implementing accounting policies.

The rollover and retention intervals on the log files and the frequency of file retrieval must also be considered when designing accounting policy deployments. The amount of data stored depends on the type of record collected, the number of services that are collecting statistics, and the collection interval that is used.

5.5. Configuration Notes

This section describes logging configuration restrictions.

  1. A file or filter cannot be deleted if it has been applied to a log.
  2. File IDs, syslog IDs, or SNMP trap groups must be configured before they can be applied to a log ID.
  3. A file ID can only be assigned to either one log ID or one accounting policy.
  4. Accounting policies must be configured in the config>log context before they can be applied to a service SAP or service interface, or applied to a network port.
  5. The snmp-trap-id must be the same as the log-id.