5.1.1. Event Logging

Event logging controls the generation, dissemination and recording of system events for monitoring status and troubleshooting faults within the system. Events are messages generated by the system by applications or processes within the 7705 SAR. The 7705 SAR groups events into four major categories or event sources:

The applications listed above have the following properties:

Event control assigns the severity for each application event and determines whether the event should be generated or suppressed. The severity numbers and severity names supported in the 7705 SAR conform to ITU standards M.3100 X.733 and X.21 and are listed in Table 32.

Event control maintains a count of the number of events generated (logged) and dropped (suppressed) for each application event. The severity of an application event can be configured in event control.

An event log within the 7705 SAR associates the event sources with logging destinations. Examples of logging destinations include the console session, memory logs, file destinations, SNMP trap groups, and syslog destinations. A log filter policy can be associated with the event log to control which events will be logged in the event log based on combinations of application, severity, event ID range, and the subject of the event.

5.1.2. Accounting Logs

The 7705 SAR accounting logs collect comprehensive statistics to support several billing models. The 7705 SAR collects accounting data on services and on network interfaces on a per-forwarding class basis.

In addition to gathering information critical for service billing, accounting records can be analyzed to provide insight about customer service trends for potential service revenue opportunities. Accounting statistics on network ports can be used to track link utilization and network capacity planning. This information is valuable for traffic engineering and capacity planning within the network core.

The 7705 SAR also supports SAA accounting policies.

Accounting statistics are collected according to the parameters defined within the context of an accounting policy. Accounting policies are applied to customer Service Access Points (SAPs) and network interfaces. Accounting statistics are collected by counters for individual service queues defined on the customer’s SAPs or by the counters within forwarding class (FC) queues defined on the network ports.

The type of record defined within the accounting policy determines where a policy is applied, which statistics are collected, and the time interval at which to collect statistics.

The only supported destination for an accounting log is a compact flash system device (cf3: on all platforms; also cf1: or cf2: on the 7705 SAR-18). Accounting data is stored within a standard directory structure on the device in compressed XML format.

5.2.1. Console

Sending events to a console destination means the message will be sent to the system console. The console device can be used as an event log destination.

5.2.2. Session

A session destination is a temporary log destination that directs entries to the active Telnet or SSH session for the duration of the session. When the session is terminated, for example, when the user logs out, the to session configuration is removed. Event logs configured with a session destination are stored in the configuration file but the to session part of the configuration is not stored. Event logs can direct log entries to the session destination.

5.2.3. Memory Logs

A memory log is a circular buffer. When the log is full, the oldest entry in the log is replaced with the new entry. When a memory log is created, the specific number of entries it can hold can be specified; otherwise, it will assume a default size. An event log can send entries to a memory log destination.

5.2.4. Log Files

Log files can be used by both event logs and accounting logs and are stored on the compact flash device (cf3: on all platforms; also cf1: or cf2: on the 7705 SAR-18) in the file system. A log file destination is configured using the config>log>file-id log-file-id command. A log file destination is applied to an event log using the config>log>log-id>to file command and to an accounting file using the config>log>accounting-policy>to file command.

A log file is identified by a single log file ID, but a log file will generally be composed of a number of individual files in the file system. A log file is configured with the following parameters:

When a log file is created, only the compact flash device for the log file is specified. Log files are created in specific subdirectories with standardized names depending on the type of information stored in the log file.

5.2.5. SNMP Trap Group

An event log can be configured to send events to SNMP trap receivers by specifying an SNMP trap group destination.

An SNMP trap group can have multiple trap targets. Each trap target can have different operational parameters.

A trap destination has the following properties:

For SNMP traps that will be sent out-of-band through the Management Ethernet port on the CSM, the source IP address of the trap is the IP interface address defined on the Management Ethernet port. For SNMP traps that will be sent in-band, the source IP address of the trap is the system IP address of the 7705 SAR.

Each trap target destination of a trap group receives the identical sequence of events as defined by the log ID and the associated sources and log filter applied.

5.2.6. Syslog

An event log can be configured to send events to one syslog destination. Syslog destinations have the following properties:

Because syslog uses eight severity levels, whereas the 7705 SAR uses six internal severity levels, the severity levels are mapped to syslog severities. Table 33 displays the severity level mappings to syslog severities.

5.3.1. Event Sources

In Figure 3, the event sources are the main categories of events that feed the log manager.

The show log applications command displays all applications:

5.3.2. Event Control

Event control preprocesses the events generated by applications before the event is passed into the main event stream. Event control assigns a severity to application events and can either forward the event to the main event source or suppress the event. Suppressed events are counted in event control, but these events will not generate log entries as they never reach the log manager.

Simple event throttling is another method of event control and is configured in the same way as the generation and suppression options. See Simple Logger Event Throttling.

Events are assigned a default severity level in the system, but the application event severities can be changed by the user.

Application events contain an event number and description that explains why the event is generated. The event number is unique within an application, but the number can be duplicated in other applications.

The following example, generated by querying event control for application-generated events, displays a partial list of event numbers and names.

5.3.3. Log Manager and Event Logs

Events that are forwarded by event control are sent to the log manager. The log manager manages the event logs in the system and the relationships between the log sources, event logs and log destinations, and log filter policies.

An event log has the following properties:

5.3.4. Event Filter Policies

The log manager uses event filter policies to control which events are forwarded or dropped based on various criteria. Like other policies with the 7705 SAR, filter policies have a default action. The default actions are either:

Filter policies also include a number of filter policy entries that are identified with an entry ID and define specific match criteria and a forward or drop action for the match criteria.

Each entry contains a combination of matching criteria that define the application, event number, router, severity, and subject conditions. The entry’s action determines how the packets should be treated if they have met the match criteria.

Entries are evaluated in order from the lowest to the highest entry ID. The first matching event is subject to the forward or drop action for that entry.

Filter policy 1001 exists by default and collects events for the Serious Error Log (log ID 100). Filter policy 1001 is preconfigured with one entry that is configured to collect events of major severity or higher. Filter policy 1001 can be reconfigured by the user.

Valid operators are displayed in Table 34.

A match criteria entry can include combinations of:

5.3.5. Event Log Entries

Log entries that are forwarded to a destination are formatted in a way that is appropriate for the specific destination; for example, whether it is to be recorded to a file or sent as an SNMP trap, but log event entries also have common elements or properties. All application-generated events have the following properties:

The general format for an event in an event log with either a memory, console or file destination is as follows:

The following is an event log example:

The specific elements that make up the general format are described in Table 35.

5.3.6. Simple Logger Event Throttling

Simple event throttling provides a mechanism to protect event receivers from being overloaded when a scenario causes many events to be generated in a very short period of time. A throttling rate (events/seconds), can be configured. Specific application events can be configured to be throttled. Once the throttling event limit is exceeded in a throttling interval, any further events of that type are dropped and the dropped events counter is incremented. Dropped events counts are displayed with the show>log>event-control command. Events are dropped before being sent to one of the logger event collector tasks. There is no record of the details of the dropped events and therefore no way to retrieve event history data lost by this throttling method.

A particular event type can be generated by multiple managed objects within the system. At the point that this throttling method is applied, the logger application has no information about the managed object that generated the event and cannot distinguish between events generated by object “A” from events generated by object “B”. If the events have the same event-id, they are throttled regardless of the managed object that generated them. The logger application also cannot distinguish between events that will be logged to destination log-id <n> from events that will be logged to destination log-id <m>.

Throttle rate applies commonly to all event types. It is not configurable for a specific event type.

A timer task checks for events dropped by throttling when the throttle interval expires. If any events have been dropped, a TIMETRA-SYSTEM-MIB::tmnxTrapDropped notification is sent.

By default, event throttling is set to off for each specific event type. It must be explicitly enabled for each event type where throttling is desired. This makes backwards compatibility of configuration files easier to manage.

5.3.7. Default System Logs

Log 99 is a preconfigured memory-based log that collects events from the main event source (that is, not the security, debug, or change source). Log 100 is preconfigured to be associated with filter policy 1001, which is preconfigured to collect events of major severity or higher. Log 100 can be reconfigured by the user.

Log 99 and log 100 exist by default.

The following example displays the log 99 and log 100 configurations.

5.3.8. Event Handling System

The Event Handling System (EHS) is a tool that enables operator-defined behavior to be configured on the 7705 SAR. The operator can define a CLI script that the router executes in response to a log event. The event is referred to as the trigger, where the trigger can be all or part of any event message. Regular expression (regexp) matching can be done on various fields in the log event to give flexibility in the trigger definition.

EHS gives operators the flexibility to configure the 7705 SAR to take actions based on certain events that cannot be done by protocols or services. For example, event-triggered actions can:

EHS objects are used to tie together trigger events (typically log events that match some configurable criteria) and a set of actions to perform (typically one or more CLI scripts).

EHS, along with CRON, makes use of the script-control functions for scripts. Any command available in the CLI can be executed in a script as the result of an event handler being triggered, except for commands that require interaction (for example, a y/n prompt for admin reboot without the now keyword, or commands that require a password). A script will error out if it encounters a command that requests input.

Figure 4 shows the relationships between the different configurable objects used by EHS (and CRON).

Figure 4:  EHS Object Relationships 

5.3.8.1. Configuring Event Handling

As shown in Figure 4, the steps involved in configuring EHS are:

1. configure a script and script policy under the config>system>script-control context; the script policy references the configured script
2. configure an event handler under the config>log>event-handling context and assign actions that reference the previously configured script policy
3. configure the event trigger under the config>log>event-trigger context that defines the event that triggers the running of the script

Refer to the 7705 SAR Basic System Configuration Guide, “CLI Script Control” for information on configuring scripts and script policies.

5.3.8.1.1. Event Handlers

Event handlers are created under the config>log>event-handling context. Each event handler is assigned an event handler name and an action list that consists of one or more entries. Each entry in the list references a configured script policy, which in turn references a configured script.

5.3.8.1.2. Event Triggers

Event triggers are created under the config>log>event-trigger context. Each event trigger is associated with an application and event ID. One or more trigger entries can be configured for the event.

Each trigger entry references a previously configured event handler (which references a configured script policy, which in turn references the script that should be run). A trigger entry can be configured with a previously configured log filter. If a filter is configured, the event trigger calls the filter to determine whether the event should be dropped or forwarded. If the event is to be forwarded, the event trigger invokes the event handler.

All existing log filter matching options are supported, as well as the option introduced in Release 9.0 to add system messages as a match criterion. Regexp matching is supported. Complex rules can be configured to match on log events as a trigger for an EHS event handler.

EHS will trigger on log events that are dropped by user-configured log filters that are assigned to individual logs (with the config>log>log-id>filter command). The EHS event trigger occurs before the distribution of log event streams into individual logs.

If there is no filter configured for the trigger entry, the event trigger invokes the event handler as soon as the event occurs.

Log events can be configured to be suppressed or throttled (with the config>log>event-control command). EHS will not trigger on these events.

5.3.8.1.2.1. Debounce

EHS debounce is the ability to trigger an action (for example, an EHS script), if an event happens (N) times within a specific time period (window) in seconds (S):

where:

N = 2 to 15 occurrences

S = 1 to 604800 seconds

For example, if linkDown occurs N times in S seconds, an EHS script is triggered to shut down the port.

Note: Triggering happens with the Nth event, not at the end of the time window (S). There is no sliding time window (for example, a trigger at the Nth event, N+1 event, and N+2 event) because N is reset after a trigger and the count is restarted. When EHS debouncing is used, the varbinds passed in to an EHS script at script triggering time are from the Nth event occurrence (the Nth triggering event); see Variable Passing. If S is not specified, the 7705 SAR will continue to trigger every Nth event.

5.3.8.1.2.2. Variable Passing

The common parameters and variable bindings (varbinds) of a triggering log event are passed in to the triggered EHS script and can be used in the script as passed-in (dynamic) variables. These variables are:

1. the common event parameters: appid, name, eventid, severity, subject, and gentime
2. the predefined varbinds in a log event message; a varbind is a list of values or attributes included in a log event

Passed-in variables are read-only.

Note: To view event parameters and varbinds, use the show log event-parameters command. The passed-in event gentime is always UTC. The event sequence number is not passed in to the script.

5.3.8.1.3. EHS Scripting

An EHS script can contain local (static) variables and use some basic .if and .set commands. The use of variables with .if and .set commands in an EHS script adds more logic to EHS scripting and allows the reuse of a single EHS script for more than one trigger or action.

Both the passed-in and local variables can be used in the EHS script either as part of the CLI commands or as part of the .if or .set commands.

The following applies to both CLI commands and .if or .set commands.

1. Using $X (without using single or double quotes) replaces the variable X with its string or integer value.

1. Using “X” (with double quotes) means the literal string X.

1. Using “$X” (with double quotes) replaces the variable X with its string or integer value.

1. Using 'X' (with single quotes) means the literal string X.

1. Using ‘$X’ (with single quotes) does not replace the variable X with its value but means the literal string $X.

In summary:

1. All characters within single quotes are interpreted as string characters.

1. All characters within double quotes are interpreted as string characters except for $, which replaces the variable with its value (for example, shell expansion inside a string).

Some supported shell command scenarios are as follows (the commands are pseudo commands):

1. .if $string_variable==string_value_or_string_variable {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
2. .if ($string_variable==string_value_or_string_variable) {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
3. .if $integer_variable==integer_value_or_integer_variable {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
4. .if ($integer_variable==integer_value_or_integer_variable) {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
5. .if $string_variable!=string_value_or_string_variable {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
6. .if ($string_variable!=string_value_or_string_variable) {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
7. .if $integer_variable!=integer_value_or_integer_variable {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
8. .if ($integer_variable!=integer_value_or_integer_variable) {
             CLI_commands_set1
   .} else {
             CLI_commands_set2
   .} endif
9. .set $string_variable = string_value_or_string_variable
10. .set ($string_variable = string_value_or_string_variable)
11. .set $integer_variable = integer_value_or_integer_variable
12. .set ($integer_variable = integer_value_or_integer_variable)

where:

1. CLI_commands_set1 is a set of one or more CLI commands
2. CLI_commands_set2 is a set of one or more CLI commands
3. string_variable is a local string variable
4. string_value_or_string_variable is a string value/variable
5. integer_variable is a local integer variable
6. integer_value_or_integer_variable is an integer value/variable

Note: A maximum of 100 local variables per EHS script is imposed. Exceeding this limit may result in an error and only partial execution of the script. When a set statement is used to set a string_variable to a string_value, the string_value can be any non-integer value with optional single or double quotes. A "." preceding a directive (for example, if, and set) is always expected to start a new line. An end of line is always expected after {. A CLI command is always expected to start a new line. Passed-in (dynamic) variables are always read-only inside an EHS script and cannot be overwritten using a set statement. .if commands support == and != operators only. .if and .set commands support addition, subtraction, multiplication, and division of integers. .if and .set commands support concatenation of strings.

Valid Examples:

1. configure service epipe $serviceID
   where $serviceID is either a local integer variable or passed-in integer variable
2. echo srcAddr is $srcAddr
   where $srcAddr is a passed-in string variable
3. .set $ipAddr = "10.0.0.1"
   where $ipAddr is a local string variable
4. .set $ipAddr = $srcAddr
   where $srcAddr is a passed-in string variable
              $ipAddr is a local string variable
5. .set ($customerID = 50)
   where $customerID is a local integer variable
6. .set ($totalPackets = $numIngrPackets + $numEgrPackets)
   where $totalPackets, $numIngrPackets, $numEgrPackets are local integer variables
7. .set ($portDescription = $portName + $portLocation)
   where $portDescription, $portName, $portLocation are local string variables
8. if ($srcAddr == "CONSOLE") {
             CLI_commands_set1
   .else {
             CLI_commands_set2
   .} endif
   where $srcAddr is a passed-in string variable
              CLI_commands_set1 is a set of one or more CLI commands
              CLI_commands_set2 is a set of one or more CLI commands
9. .if ($customerId == 10) {
             CLI_commands_set1
   .else {
             CLI_commands_set2
   .} endif
   where $customerID is a passed-in integer variable
              CLI_commands_set1 is a set of one or more CLI commands
              CLI_commands_set2 is a set of one or more CLI commands
10. .if ($numIngrPackets == $numEgrPackets) {
              CLI_commands_set1
    .else {
              CLI_commands_set2
    .} endif
    where $numIngrPackets and $numEgrPackets are local integer variables
               CLI_commands_set1 is a set of one or more CLI commands
               CLI_commands_set2 is a set of one or more CLI commands

Invalid Examples:

1. .set $srcAddr = "10.0.0.1"
   where $srcAddr is a passed-in string variable
   Reason: passed-in variables are read-only in an EHS script
2. .set ($ipAddr = ‘$numIngrPackets' + $numEgrPackets)
   where $ipAddr is a local string variable  
             $numIngrPackets and $numEgrPackets are local integer variables
   Reason: variable types do not match; cannot assign a string to an integer
3. .set ($numIngrPackets = $ipAddr + $numEgrPackets)
   where $ipAddr is a local string variable
               $numIngrPackets and $numEgrPackets are local integer variables
   Reason: variable types do not match; cannot concatenate a string to an integer
4. .set $ipAddr = "10.0.0.1"100
   where $ipAddr is a local string variable
   Reason: when double quotes are used, they must enclose the entire string
5. .if ($totalPackets == "10.1.1.1") {
   .} endif
   where $totalPackets is a local integer variable
   Reason: cannot compare an integer variable to a string value
6. .if ($ipAddr == 10) {
   .} endif
   where $ipAddr is a local string variable
   Reason: cannot compare a string variable to an integer value
7. .if ($totalPackets == $ipAddr) {
   where $totalPackets is a local integer variable
              $ipAddr is a local string variable
   Reason: cannot compare an integer variable to a string variable

5.3.8.1.4. Hardware Support

EHS is supported on all 7705 SAR cards, modules, and fixed platforms.

5.4.1. Accounting Records

An accounting policy must define a record name and collection interval. Only one record name can be configured per accounting policy. Also, a record name can only be used in one accounting policy.

Table 36 lists the record name, sub-record types, and default collection period for service and network accounting policies.

The 7705 SAR supports simultaneous collection for some records. For example, “complete-network-ingr-egr” (cpNipo and cpNepo) simultaneously collects statistics on network-ingress octets, network-ingress packets, network-egress octets, and network-egress packets for the same network port.

Similarly, on the service side, “complete-service-ingr-egr” (cpSipo and cpSepo) simultaneously collects statistics on service-ingress octets, service-ingress packets, service-egress octets, and service-egress packets from a single SAP.

When creating accounting policies, one service accounting policy and one network accounting policy can be defined as the default. If statistics collection is enabled on a SAP or network port and no accounting policy is applied, the respective default policy is used. If no default policy is defined, no statistics are collected unless a specifically defined accounting policy is applied.

Each accounting record name is composed of one or more sub-records, which are in turn composed of multiple fields. Table 37 lists the accounting policy record names and the statistics that are collected with each.

5.4.2. Accounting Files

When a policy has been created and applied to a service or network port, the accounting file is stored on the compact flash in a compressed XML file format. The 7705 SAR creates two directories on the compact flash to store the files. The following output displays a directory named act-collect that holds accounting files that are open and actively collecting statistics, and a directory named act that stores the files that have been closed and are awaiting retrieval.

Accounting files always have the prefix act followed by the accounting policy ID, log ID and timestamp. The accounting log file naming and log file destination properties (such as rollover and retention) are discussed in more detail in Log Files.

A file ID can only be assigned to either one event log ID or one accounting log.

5.4.3. Design Considerations

The 7705 SAR has ample resources to support large-scale accounting policy deployments. When preparing for an accounting policy deployment, verify that data collection, file rollover, and file retention intervals are properly tuned for the amount of statistics to be collected.

If the accounting policy collection interval is too brief, there may be insufficient time to store the data from all the services and network interfaces within the specified interval. If that is the case, some records may be lost or incomplete. Interval time, record types, and number of services using an accounting policy are all factors that should be considered when implementing accounting policies.

The rollover and retention intervals on the log files and the frequency of file retrieval must also be considered when designing accounting policy deployments. The amount of data stored depends on the type of record collected, the number of services that are collecting statistics, and the collection interval that is used.

5.5.1. Reference Sources

For information on supported IETF drafts and standards as well as standard and proprietary MIBS, refer to Standards and Protocol Support.