2.1.1. Service Types

The Nokia routers offer the following types of subscriber services which are described in more detail in the referenced chapters:

2.1.2. Service Policies

Common to all Nokia service router connectivity services are policies that are assigned to the service. Policies are defined at a global level and then applied to a service on the router. Policies are used to define Nokia service router service enhancements. The types of policies that are common to the router’s connectivity services are:

2.1.2.1. Multipoint Shared Queuing

Multipoint shared queuing is supported only on Nokia service router routers.

Multipoint shared queuing is supported to minimize the number of multipoint queues created for ingress VPLS, IES or VPRN SAPs or ingress subscriber SLA profiles. Normally, ingress multipoint packets are handled by multipoint queues created for each SAP or subscriber SLA profile instance. In some instances, the number of SAPs or SLA profile instances are sufficient for the in use multipoint queues to represent many thousands of queues on an ingress forwarding plane. If multipoint shared queuing is enabled for the SAPs or SLA profile instances on the forwarding plane, the multipoint queues are not created. Instead, the ingress multipoint packets are handled by the unicast queue mapped to the forwarding class of the multipoint packet.

Functionally, multipoint shared queuing is a superset of shared queuing. With shared queuing on a SAP or SLA profile instance, only unicast packets are processed twice, once for the initial service level queuing and a second time for switch fabric destination queuing. Shared queuing does not affect multipoint packet handling. Multipoint packet handling in normal (service queuing) is the same as shared queuing. When multipoint shared queuing is enabled, shared queuing for unicast packets is automatically enabled.

2.1.2.1.1. Ingress Queuing Modes of Operation

Three modes of ingress SAP queuing are supported for multipoint services (IES, VPLS and VPRN); service, shared, and multipoint shared. The same ingress queuing options are available for IES and VPLS subscriber SLA profile instance queuing.

2.1.2.1.2. Ingress Service Queuing

Normal or service queuing is the default mode of operation for SAP ingress queuing. Service queuing preserves ingress forwarding bandwidth by allowing a service queue defined in an ingress SAP QoS policy to be represented by a group of hardware queues. A hardware queue is created for each switch fabric destination to which the logical service queue must forward packets. For a VPLS SAP with two ingress unicast service queues, two hardware queues are used for each destination forwarding engine the VPLS SAP is forwarding to. If three switch fabric destinations are involved, six queues are allocated (two unicast service queues multiplied by three destination forwarding complexes equals six hardware queues). Figure 1 demonstrates unicast hardware queue expansion. Service multipoint queues in the ingress SAP QoS policy are not expanded to multiple hardware queues, each service multipoint queue defined on the SAP equates to a single hardware queue to the switch fabric.

When multiple hardware queues represent a single logical service queue, the system automatically monitors the offered load and forwarding rate of each hardware queue. Based on the monitored state of each hardware queue, the system imposes an individual CIR and PIR rate for each queue that provides an overall aggregate CIR and PIR reflective of what is provisioned on the service queue.

Figure 1:  Unicast Service Queue Mapping to Multiple Destination Based Hardware Queues 

2.1.2.1.3. Ingress Shared Queuing

To avoid the hardware queue expansion issues associated with normal service based queuing, the system allows an ingress logical service queue to map to a single hardware queue when shared queuing is enabled. Shared queuing uses two passes through the ingress forwarding plane to separate ingress per service queuing from the destination switch fabric queuing. In the case of shared queuing, ingress unicast service queues are created one-for-one relative to hardware queues. Each hardware queue representing a service queue is mapped to a special destination in the traffic manager that ‘forwards’ the packet back to the ingress forwarding plane allowing a second pass through the traffic manager. In the second pass, the packet is placed into a ‘shared’ queue for the destination forwarding plane. The shared queues are used by all services configured for shared queuing.

When the first SAP or SLA profile instance is configured for shared queuing on an ingress forwarding plane, the system allocates eight hardware queues per available destination forwarding plane, one queue per forwarding class. Twenty four hardware queues are also allocated for multipoint shared traffic. The shared queue parameters that define the relative operation of the forwarding class queues are derived from the Shared Queue policy defined in the QoS CLI node. Figure 2 demonstrates shared unicast queuing. SAP or SLA profile instance multipoint queuing is not affected by enabling shared queuing. Multipoint queues are still created as defined in the ingress SAP QoS policy and ingress multipoint packets only traverse the ingress forwarding plane a single time, as demonstrated in Figure 3.

Enabling shared queuing may affect ingress performance due to double packet processing through the service and shared queues.

Figure 2:  Unicast Service Queuing With Shared Queuing Enabled 

Figure 3:  Multipoint Queue Behavior with Shared Queuing Enabled 

2.1.2.1.4. Ingress Multipoint Shared Queuing

Ingress multipoint shared queuing is a variation to the unicast shared queuing defined in Ingress Shared Queuing. Ingress unicast service queues are mapped one-for-one with hardware queues and unicast packets traverse the ingress forwarding plane twice. In addition to the above, the multipoint queues defined in the ingress SAP QoS policy are not created. Instead, multipoint packets (broadcast, multicast and unknown unicast destined) are treated to the same dual pass ingress forwarding plane processing as unicast packets. In the first pass, the forwarding plane uses the unicast queue mappings for each forwarding plane. The second pass uses the multipoint shared queues to forward the packet to the switch fabric for special replication to all egress forwarding planes that need to process the packet.

The benefit of defining multipoint shared queuing is the savings of the multipoint queues per service. By using the unicast queues in the first pass and then the aggregate shared queues in the second pass, per service multipoint queues are not required. The predominate scenario where multipoint shared queuing may be required is with subscriber managed QoS environments using a subscriber per SAP model. Usually, ingress multipoint traffic is minimal per subscriber and the extra multipoint queues for each subscriber reduces the overall subscriber density on the ingress forwarding plane. Multipoint shared queuing eliminates the multipoint queues sparing hardware queues for better subscriber density. Figure 4 demonstrates multipoint shared queuing.

One disadvantage of enabling multipoint shared queuing is that multipoint packets are no longer managed per service (although the unicast forwarding queues may provide limited benefit in this area). Multipoint packets in a multipoint service (VPLS, IES and VPRN) use significant resources in the system, consuming ingress forwarding plane multicast bandwidth and egress replication bandwidth. Usually, the per service unicast forwarding queues are not rate limited to a degree that allows adequate management of multipoint packets traversing them when multipoint shared queuing is enabled. It is possible to minimize the amount of aggregate multipoint bandwidth by setting restrictions on the multipoint queue parameters in the QoS node’s shared queue policy. Aggregate multipoint traffic can be managed per forwarding class for each of the three forwarding types (broadcast, multicast or unknown unicast – broadcast and unknown unicast are only used by VPLS).

A second disadvantage to multipoint shared queuing is the fact that multipoint traffic now consumes double the ingress forwarding plane bandwidth due to dual pass ingress processing.

Figure 4:  Multipoint Shared Queuing Using First Pass Unicast Queues 

2.3.1. Customers

In this section, the terms customers and subscribers are used synonymously. The most basic required entity is the customer ID value which is assigned when the customer account is created. To provision a service, a customer ID must be associated with the service at the time of service creation.

2.3.2. Service Access Points (SAPs)

Each subscriber service type is configured with at least one service access point (SAP). A SAP identifies the customer interface point for a service on a router (for example Figure 6). The SAP configuration requires that slot, XMA or MDA, and port/channel information be specified. The slot, XMA or MDA, and port/channel parameters must be configured prior to provisioning a service (Refer to the XMAs, Cards, MDAs, and Ports sections of the SR OS Interface Configuration Guide).

A SAP is a local entity to the router and is uniquely identified by:

Depending on the encapsulation, a physical port or channel can have more than one SAP associated with it. SAPs can only be created on ports or channels designated as “access” in the physical port configuration. SAPs cannot be created on ports designated as core-facing “network” ports as these ports have a different set of features enabled in software.

Figure 6:  7750 SR/7950 XRS Service Access Point (SAP) 

A SAP can also be associated with a pseudowire port rather than an access port. Such SAPs are called pseudowire SAPs. This is only applicable to IES, VPRN, and Epipe services. Pseudowire ports represent pseudowires in enhanced subscriber management (ESM). For a description of pseudowire ports, see the 7450 ESS, 7750 SR, and VSR Triple Play Service Delivery Architecture Guide.

2.3.2.4. QinQ SAPs

A QinQ SAP has the following format:

qinq <port-id | lag-id>:qtag1.qtag2

Where:

1. qtag1 is the outer qtag value - [*, 0 to 4094]
2. qtag2 is the inner qtag value - [*, null, 0 to 4094]

Regular QinQ SAPs have qtag1 and qtag2 values between 1 and 4094. In addition, QinQ Ethernet and LAG ports support the following “default” SAPs that can be enabled by the new-qinq-untagged-sap command:

1. ‘*.null’ is defined as a default sap for singly-tagged frames in a QinQ port. This SAP accepts single tags in the range 0 to 4095 as well as untagged traffic. This SAP never pushes any tags on egress.
2. ‘*.*’ is defined as a default sap for doubly-tagged frames in a QinQ port. This SAP accepts untagged, singly tagged, and doubly tagged frames with tags in the range 0..4095. This SAP never pushes any tags on egress.
3. In addition to the above-mentioned SAPs, qtag2 can also be '0' or '*' when qtag1 is an explicit value in the 1 to 4094 range, for instance: 1/1/1:10.0 or 1/1/1:10.*. Assuming qtag1 is the same value, qtag1.* and qtag1.0 are supported in the same QinQ port. The system never pushes any qtag2 on egress for 1/1/110.0 and 1/1/1:10.*, only qtag1 is pushed. The x.0 accepts only 0 as second tag or not tag (and nothing else), while x.* accepts anything as second tag or no tag.

A SAP lookup is performed when a new frame arrives to a QinQ port. This 'lookup' is based on the <outer-tag, inner-tag> values of the frame.

Table 3 shows the SAP lookup precedence order for incoming frames with <qtag1.qtag2> qtag values.

Table 3:  SAP Lookup Precedence Order for Incoming Frames 

Incoming Frame qtag1.qtag2	System/Port settings [new-qinq-untagged-sap=YES]
	SAP Lookup Precedence Order
	:X.Y	:X.0	:X.*	:0.*	:*.null	:*.*
x.y	1st		2nd			3rd
x.0		1st	2nd			3rd
0.y				1st		2nd
0.0				1st		2nd
x		1st	2nd		3rd	4th
0				1st	2nd	3rd
<untagged>				1st	2nd	3rd

The following considerations apply to the information described in Table 3:

1. All six SAP types (:X.Y, :X.0, :X.*, :0.*, :*.null and :*.*) are supported in the same QinQ port and, in the table, they are ordered from the most specific (left-hand side) to the least specific with the following VID matching ranges:
   1. X or Y means <1 to 4094>
   2. * means <0 to 4095> or untagged
   3. null means 'no tag'
   4. 0 means VID 0 or untagged
2. On egress, the system will push a tag with a VID value for X and Y, whereas no tag is pushed on egress for values 0, * or null. For example:
   1. On SAP 1/1/1:10.20, the system pushes tags with VIDs 10 and 20 (outer and inner respectively) on egress.
   2. On SAP 1/1/1:10.0 or 1/1/1:10.* the system pushes only one tag with VID 10 on egress.
   3. On SAPs 1/1/1:0.*, 1/1/1:*.null or 1/1/1:*.* the system never pushes any tags on egress.
3. The user can decide the SAP types that are configured in a specific port. Not all SAP types must be configured in a port.
4. Table 3 shows the lookup behavior for ingress frames and priority across SAPs in case more than one can match a given ingress frame. The SAP lookup result for a given frame does not depend on the operational status of the SAP. For instance:
   1. In a port with SAPs 1/1/1:0.* and 1/1/1:*.* defined, the SAP lookup for a given frame with VIDs (0, 300) will yield SAP 1/1/1:0.* regardless of its operational status.
   2. The frame will only match SAP 1/1/1:*.* when the 0.* SAP is removed from the configuration.
5. The following apply to VLAN tag handling:
   1. The system will not strip-off any tags for frames entering the default SAPs (:0.*, :*.null or :*.*).
   2. No extra tags are added when the system transmits frames on the default SAPs (:0.*, :*.null or :*.*).

The following examples illustrate the SAP classification QinQ ports. The examples assume that the new-qinq-untagged-sap command is enabled.

Example - 1

As shown in Figure 8, assuming that the new-qinq-untagged-sap command is enabled, the following SAPs are defined on the same port:

1. 1/1/1:3000.1001 - business customer - vpls-1
2. 1/1/1:2000.1002 - business customer - vpls-2
3. 1/1/1:20.0 - BNG traffic - vpls-3
4. 1/1/1:20.* - business customer - epipe-4
5. 1/1/1:0.* - business customer - epipe-5
6. 1/1/1:*.null - wholesaling single tag - epipe-6
7. 1/1/1:*.* - wholesaling double tag - epipe-7

Figure 8:  Example 1 SAP Classification QinQ Ports 

Based on the SAPs configuration described above, the incoming traffic is classified in the following way - notation (outer-VID, inner-VID):

1. (3000, 1001) goes to vpls-1
2. (20) goes to BNG (vpls-3)
3. (20, 0) goes to BNG (vpls-3)
4. (20, 10) goes to epipe-4
5. untagged, (0), (0, 0), and (0, 10) go to epipe-5
6. (500) goes to wholesaling single tag (epipe-6)
7. (500, 300) and (500, 0) go to wholesaling double tag (epipe-7)

Example - 2

Figure 9 highlights how untagged, VID=0 tagged frames and 20.X frames are classified in the absence of the 0.* and 20.* SAPs.

Figure 9:  Example 2 SAP Classification QinQ Ports 

As outlined in Figure 9, assuming the new-qinq-untagged-sap command is enabled, the following SAPs are defined on the same port:

1. 1/1/1:3000.1001 - business customer - vpls-1
2. 1/1/1:2000.1002 - business customer - vpls-2
3. 1/1/1:20.0 - BNG traffic - vpls-3
4. 1/1/1:*.null - wholesaling single tag - epipe-6
5. 1/1/1:*.* - wholesaling double tag - epipe-7

Incoming traffic - notation (outer-VID, inner-VID)

1. (3000, 1001) goes to vpls-1
2. (20) goes to BNG (vpls-3)
3. (20, 0) goes to BNG (vpls-3)
4. (20, 10) goes to wholesaling double tag (epipe-7)
5. untagged and (0) go to wholesaling single tag (epipe-6)
6. (500) goes to wholesaling single tag (epipe-6)
7. (500, 300) and (500, 0) go to wholesaling double tag (epipe-7)
8. (0,0), and (0,10) goes to wholesaling double tag (epipe-7)

Note: The system will not add service-delimiting tags with VID=0; however, tags with VID=0 are accepted and classified appropriately.

The following constraints must be considered when configuring default QinQ SAPs (:0.*, :*.null, :*.*):

1. Only supported in Ethernet ports or LAG.
2. Only supported on Epipe, PBB-Epipe, VPLS and I-VPLS services. They are not supported on VPRN, IES, R-VPLS or B-VPLS services.
3. Capture SAPs with encapsulation :*.* cannot coexist with a default :*.* SAP on the same port.
4. Inverse-capture SAPs (*.x) are mutually-exclusive with :*.null SAPs.
5. *.null SAPs are not supported for Open Flow matching and forwarding.
6. The following applies to Eth-CFM:
   1. Primary VLAN is not supported.
   2. Eth-CFM extractions occur within the service after the packet lookup has determined which service the inbound packet belongs to.
   3. All three SAPs (*.null, *.* and 0.*) are treated equally by ETH-CFM. Only untagged CFM PDUs are extracted by a local MEP or MIP. Additional tags in the header may match the service context but are not extracted by ETH-CFM for processing.
   4. ETH-CFM PDU transmission encapsulation is based on the SAP configuration. This means that the ETH-CFM PDUs will be transmitted out all three of these SAPs untagged. Care must be taken to ensure that there is no downstream service that may intercept the ETH-CFM PDUs that are not intended for that service. See Table 3 for a description of the SAP lookup precedence order for incoming frames and to understand the potential consequences.
7. Default QinQ SAPs do not support the following features:
   1. PW-SAPs
   2. Eth-tunnel or eth-ring SAPs
   3. VLAN-translation copy-outer
   4. E-Tree root-leaf-tag SAPs
   5. Subscriber-management features
   6. BPDU-translation
   7. Eth-tunnels
   8. IGMP-snooping
   9. MLD-snooping

2.3.3. Connection Profile VLAN SAPs

The connection-profile-vlan SAPs (CP SAPs) allow the association of a range of customer VIDs to a given SAP. CP SAPs can be used to build Layer 2 Services that are fully compatible with MEF 10.3 Bundling Service Attributes and RFC 7432 EVPN VLAN Bundle Service interfaces.

The config>connection-profile-vlan>vlan-range command defines the range of customer VIDs to be matched when the connection-profile-vlan is associated with a dot1q or QinQ SAP. The following CLI output example shows the use of connection-profile-vlan in dot1q and QinQ SAPs:

As far as VLAN manipulation is concerned, the CP SAP behavior is equivalent to the default SAP’s (when the ingress VID falls into the range configured in the CP), where the range of VIDs included is not service-delimiting and therefore, the VIDs are not pushed/popped. The main differences between the CP SAPs and the default SAPs are:

Figure 10 shows customer VID processing by SAPs with service-delimiting VIDs, and by CP SAPs. SAP 1/1/1:cp-1 does not strip off or push VID 10, whereas SAP 1/1/1:100 and SAP 1/1/1:200 do strip off and push the corresponding VID.

Figure 10:  VLAN Tag Handling 

A connection-profile-vlan allows the configuration of VLAN ranges with the following characteristics.

2.3.4. Service Distribution Points

A Service Distribution Point (SDP) acts as a logical way to direct traffic from one router to another through a uni-directional (one-way) service tunnel. The SDP terminates at the far-end device which directs packets to the correct service egress SAPs on that device. A distributed service consists of a configuration with at least one SAP on a local node, one SAP on a remote node, and an SDP binding the service to the service tunnel.

An SDP has the following characteristics:

An SDP from the local device to a far-end router requires a return path SDP from the far-end router back to the local router. Each device must have an SDP defined for every remote router to which it wants to provide service. SDPs must be created first, before a distributed service can be configured.

2.3.4.1. SDP Binding

To configure a distributed service from ALA-A to ALA-B, the SDP ID (1) (shown in Figure 11) must be specified in the service creation process in order to “bind” the service to the tunnel (the SDP). Otherwise, service traffic is not directed to a far-end point and the far-end device(s) cannot participate in the service (there is no service). To configure a distributed service from ALA-B to ALA-A, the SDP ID (5) must be specified.

Figure 11:  GRE Service Distribution Point (SDP) Pointing From ALA-A to ALA-B 

2.3.4.5. SDP Administrative Groups

This feature introduces the support of SDP administrative groups, referred to as SDP admin groups. SDP admin groups provide a way for services using a pseudowire template to automatically include or exclude specific provisioned SDPs. SDPs sharing a specific characteristic or attribute can be made members of the same admin group.

The user first creates the admin groups that are to be used by SDPs on this node:

config>service>sdp-group>group-name group-name value group-value create

A maximum of 32 admin groups can be created. The no option is only allowed if the group-name is not referenced in a PW template or SDP.

The group value ranges from zero (0) to 31. It is uniquely associated with the group name at creation time. If the user attempts to configure another group name for a group value that is already assigned to an existing group name, the SDP admin group creation is failed. The same happens if the user attempts to configure an SDP admin group with a new name but associates it to a group value already assigned to an existing group name.

Next, the user configures the SDP membership in admin groups:

config>service>sdp>sdp-group group-name

The user can enter a maximum of one (1) admin group name at once. The user can execute the command multiple times to add membership to more than one admin group. The admin group name must have been configured or the command is failed. Admin groups are supported on an SDP of type GRE and of type MPLS (BGP/RSVP/LDP). They are also supported on an SDP with the mixed-lsp-mode option enabled.

The user then selects which admin groups to include or exclude in a given pseudowire template:

config>service>pw-template>sdp-include group-name

config>service>pw-template>sdp-exclude group-name

The admin group name must have been configured or the command is failed. The user can execute the command multiple times to include or exclude more than one admin group. The sdp-include and sdp-exclude commands can only be used with the use-provisioned-sdp or prefer-provisioned-sdp options. If the same group name is included and excluded within the same PW template, only the exclude option will be enforced.

Any changes made to the admin group sdp-include and sdp-exclude constraints will only be reflected in existing spoke SDPs after the following command has been executed:

tools>perform>service>eval-pw-template>allow-service-impact

When the service is bound to the PW template, the SDP selection rules will enforce the admin group constraints specified in the sdp-include and sdp-exclude commands.

config>service>vpls>bgp>pw-template-binding policy-id

config>service>epipe>spoke-sdp-fec>pw-template-bind policy-id

Note: The group value is used to uniquely identify an SDP admin group throughout the network in NSP NFM-P. The node will send both the group name and value to NSP NFM-P (or other SNMP device) at the creation of the SDP admin group. In all other operations in the node, such as adding an SDP to an admin group or including or excluding an SDP admin group in a service context, only the group name is sent to NSP NFM-P or the SNMP device.

SDP admin groups can be enabled on all router services that make use of the pseudowire template (BGP-AD VPLS service, BGP-VPLS service, BGP-VPWS and FEC129 VLL service). In the latter case, SR OS provides support at the T-PE nodes only.

2.3.4.7. Class-Based Forwarding

2.3.4.7.1. Application of Class-Based Forwarding over RSVP LSPs

Class-based forwarding over RSVP LSPs allows a service packet to be forwarded over a specific RSVP LSP, part of an SDP, based on its ingress determined forwarding class. The LSP selected depends on the operational status and load-balancing algorithms used for ECMP and LAG spraying.

Figure 12:  Class-Based Forwarding over SDP LSPs 

Figure 12 illustrates the use of class-based forwarding to direct packets of a service to specific RSVP or static LSPs that are part of the same SDP based on the packets' forwarding class. The forwarding class of the packet is the one assigned to the packet as a result of applying the ingress QoS policy to the service SAP. The VLL service packets are all classified into the ef forwarding class and those that are destined to PE2 are forwarded over LSP1. Multicast and broadcast are classified into the be class and are forwarded over LSP2.

This feature allows service providers to dedicate specific LSPs with a determined level of traffic engineering and protection to select service packets. For example, packets of a VoIP service are assigned the ef class to expedite their forwarding but are also sent over carefully traffic-engineered and FRR-protected LSP paths across the service provider network.

2.3.4.7.2. Operation of Class-Based Forwarding over RSVP LSPs

The Nokia router’s class-based forwarding feature applies to a set of LSPs that are part of the same SDP. Each LSP must be configured as part of an SDP specifying the forwarding classes it will support. A forwarding class can only be assigned to one LSP in a given SDP, meaning that only one LSP within an SDP will support a given class of service. However, multiple classes of services can be assigned to an LSP. Both RSVP and static LSPs are allowed. All subclasses will be assigned to the same LSP as the parent forwarding class.

When a service packet is received at an ingress SAP, it is classified into one of the eight forwarding classes. If the packet will leave the SR on an SDP that is configured for class-based forwarding, the outgoing LSP will be selected based on the packet's forwarding class. Each SDP has a default LSP. The default LSP is used to forward a received packet that was classified at the ingress SAP into a forwarding class for which the SDP does not have an explicitly-configured LSP association. It is also used to forward a received packet if the LSP supporting its forwarding class is down.

Note: The SDP goes down if the default LSP is down.

Class-based forwarding can be applied to all services supported by the Nokia routers. For VPLS services, explicit FC-to-LSP mappings are used for known unicast packets. Multicast and broadcast packets use the default LSP. There is a per-SDP user configuration that optionally overrides this behavior to specify an LSP to be used for multicast/broadcast packets.

VLL service packets are forwarded based on their forwarding class only if shared queuing is enabled on the ingress SAP. Shared queuing must be enabled on the VLL ingress SAP if class-forwarding is enabled on the SDP the service is bound to. Otherwise, the VLL packets will be forwarded to the LSP which is the result of hashing the VLL service ID. Since there are eight entries in the ECMP table for an SDP, one LSP ID for each forwarding class, the resulting load balancing of VLL service ID is weighted by the number of times an LSP appears on that table. For instance, if there are eight LSPs, the result of the hashing will be similar to when class based forwarding is disabled on the SDP. If there are fewer LSPs, then the LSPs which were mapped to more than one forwarding class, including the default LSP, will have proportionally more VLL services forwarding to them.

Only user packets are forwarded based on their forwarding class. OAM packets are forwarded in the same way as an SDP with class-based forwarding disabled. In other words, LSP ping and LSP trace messages are queued in the queue corresponding to the forwarding class specified by the user and are forwarded over the LSP being tested. Service and SDP OAM packets, such as service ping, VCCV ping, and SDP ping are queued in the queue corresponding to the forwarding class specified by the user and forwarded over the first available LSP.

Class-based forwarding is not supported for protocol packets tunneled through an SDP. All packets are forwarded over the default LSP.

Class-based forwarding is not supported on a spoke SDP used for termination on an IES or VPRN service. All packets are forwarded over the default LSP.

2.3.4.8. Source IPv4 Address Configuration in GRE SDP and GRE Tunnel

2.3.4.8.1. Introduction and Feature Configuration

When the GRE tunnel is used as part of a provisioned SDP, the following command is relaxed to allow the user to configure a source address for an GRE SDP:

configure>service>sdp>local-end ip-address

The default value of the local-end parameter is the primary IPv4 address of the system interface. To change the local-end address, the SDP must be shut down.

The primary IPv4 address of any local network IP interface, loopback or otherwise, may be used as the source address. The address does not need to match the primary address of an interface which has the MPLS-over-GRE termination subnet configured, unless a GRE SDP or tunnel from the far-end router terminates on this address.

The address of the following interfaces are not supported:

1. unnumbered network IP interface
2. IES interface
3. VPRN interface
4. CSC VPRN interface

The following rules apply to the local-end command:

1. A maximum of 15 distinct address values can be configured for all GRE SDPs under the configure>service>sdp>local-end context, and all L2oGRE SDPs under the config>service>system>gre-eth-bridged>tunnel-termination context.
   The same source address cannot be used in both contexts since an address configured for a L2oGRE SDP matches an internally created interface which is not available to other applications.
2. The local-end address of a GRE SDP, when different from system, need not match the primary address of an interface which has the MPLS-over-GRE termination subnet configured, unless a GRE SDP or tunnel from the far-end router terminates on this address.

The user must ensure that the local-end address is reachable from the far-end router that terminates the GRE SDP. To help ensure reachability, the interface for this address may be added to IGP or BGP, or a static route may be configured on the far-end router.

The following services can be bound to a GRE SDP when the local-end address is modified:

1. VPRN or IES with a spoke-sdp interface
   (configure>service>vprn>interface>spoke-sdp)
2. VPLS with provisioned spoke-sdp
3. BGP-AD VPLS and use-provisioned-sdp or prefer-provisioned-sdp option enabled
4. BGP-VPLS and use-provisioned-sdp or prefer-provisioned-sdp or prefer-provisioned option enabled
5. Epipe with provisioned spoke-sdp
6. Epipe with BGP-VPWS and use-provisioned-sdp or prefer-provisioned-sdp or prefer-provisioned option enabled

For services that support auto-binding to a GRE tunnel, a new CLI command is introduced to configure a single alternate source address per system:

configure>service>system>vpn-gre-source-ip ip-address

The default value is the primary IPv4 address of the system interface.

A change to the value of the vpn-gre-source-ip parameter can be performed without shutting down the service. Once the new value is configured, the system address will not be used in services that bind to the GRE tunnel.

The primary IPv4 address of any local network IP interface, loopback or otherwise, may be used.

The address of the following interfaces are not supported:

1. unnumbered network IP interface
2. IES interface
3. VPRN interface
4. CSC VPRN interface

The following rules apply to the vpn-gre-source-ip parameter value:

1. This single source address counts towards the maximum of 15 distinct address values per system that are used by all GRE SDPs under the configure>service>sdp>local-end context and all L2oGRE SDPs under the config>service>system>gre-eth-bridged>tunnel-termination context.
2. The same source address can be used in both vpn-gre-source-ip and configure>service>sdp>local-end contexts.
3. The same source address cannot be used in both vpn-gre-source-ip and config>service>system>gre-eth-bridged>tunnel-termination contexts because an address configured for a L2oGRE SDP matches an internally created interface that is not available to other applications.
4. The vpn-gre-source-ip address, when different from system, need not match the primary address of an interface which has the MPLS-over-GRE termination subnet configured, unless a GRE SDP or tunnel from the far-end router terminates on this address.

The following contexts can use a GRE tunnel when the source IP address is modified:

1. VPRN service with a SDP (config>service>vprn>spoke-sdp)
2. VPRN auto-bind-tunnel

The source address cannot be configured for the following services with auto-created GRE-SDP:

1. BGP-AD VPLS
2. BGP-VPLS
3. VGP-VPWS

An alternative solution to bind any one of these services to its own specific GRE SDP with its own source IP address, is to tag a pre-provisioned GRE SDP with a SDP admin-group (sdp-group command) and include the admin-group with the PW template binding of this service (config>service>pw-template policy-id [use-provisioned-sdp]> sdp-include group-name). The command prefer-provisioned-sdp can also be used.

2.3.4.8.2. Feature Operation with T-LDP and BGP Service Label Signaling

The origination function continues to operate as in previous releases. The only change is the ability to insert the user configured address in the source address field of the GRE/IPv4 header as explained in Introduction and Feature Configuration.

Note: The service manager does not explicitly request from the LDP module that an SDP auto-generated T-LDP session for the MPLS-over-GRE SDP uses the source address configured with the local-end CLI command. LDP ensures that either a user-configured T-LDP session, or a peer template based auto-created T-LDP session, exists and is connected to the far-end address of the SDP. LDP will use one of these sessions, or will auto-create one using the default local transport address of system.

Consequently, if the source transport address used by the T-LDP control plane session does not match the destination transport address set by the remote PE in the targeted LDP Hello messages, the T-LDP session does not come up.

For example, the setup in Figure 13 will result in both GRE SDP1 and SDP2 to remain down because the targeted Hello adjacency and LDP session will not come up between the two LDP LSRs.

Figure 13:  Mismatched T-LDP Control Plane Parameters 

The user must match the local transport address of the T-LDP session to the local-end address of the GRE SDP in both the local and remote PE routers. This can be achieved by manually configuring a T-LDP session to the peer, or by auto-creating a T-LDP session with the targeted peer template feature, and setting the local-lsr-id command to the address configured in the local-end command of the GRE SDP. In addition, the far-end address must be in a GRE termination subnet at the remote PE and be the primary address of an interface in order for T-LDP to use it as its local LSR ID at the remote PE. Figure 14 shows an example of a correct configuration.

Figure 14:  Proper Setting of T-LDP Control Plane Parameters 

The source address used by the GRE tunnel in the data plane can be different than the local transport address used by T-LDP in the control plane and the GRE SDPs will still come up. For example, the setup in Figure 15 uses at each end the system address for the T-LDP session but uses a loopback interface address as the source address of the GRE SDP.

Figure 15:  Source Address Mismatch between Control and Data Planes 

Note: The LDP uses a priority mechanism to select which parameters to use to instantiate a T-LDP session to the same far-end transport address. A manually provisioned T-LDP session overrides one that is signaled using the targeted peer template which overrides one that is auto-created by an SDP. This is done automatically by LDP by issuing, an ad-hoc update to the Hello message to the far-end with the new parameters. As long as the corresponding change is performed at the far-end router to match the local-end parameter change (for example, changing the local transport address requires a change of the far-end transport address in the remote LSR to the same value) the T-LDP session remains up while the Hello adjacency is being synchronized by both LSRs.

The same recommendation applies when the SDP uses BGP for signaling the VC labels of the services. The user must configure the BGP session to the peer and set the local-address CLI command under the BGP group context or under the neighbor context to the address configured in the local-end command of the GRE SDP.

Replies to OAM messages such as an SDP keep-alive and sdp-ping are sent by the far-end PE using the MPLS-over-GRE encapsulation to the source address of the received OAM message. This means, the source transport address of the T-LDP control plane session or the BGP control plane session is used for the signaling of the VC-label in the local PE. Replies to OAM messages when the VC label is static are sent to the source address of the local PE. In all cases however, the system can properly extract them to the CPM as long as the subnet of that local interface is reachable.

2.3.5. SAP and MPLS Binding Loopback with MAC Swap

SAPs and MPLS SDP bindings within Ethernet services, Epipe and VPLS, may be placed into a loopback mode that allows all packets that arrive on the looped entity to be reflected back into the service. The function is specific to the entity on which the loopback is configured and is non-disruptive to other SAPs and SDP bindings on the same port or LAG.

Epipe and PBB Epipe service constructs support both ingress and egress loopbacks on Ethernet SAPs or MPLS SDP bindings.

VPLS and I-VPLS service constructs support both in ingress and egress loopback on Ethernet SAPs or MPLS SDP bindings.

Do not enable this functionality in the core PBB context because there is no ISID awareness. If this feature is enabled within the core PBB context all traffic that arrives on the B-SAP or B-MPLS binding will be looped back into the PBB context without regard for ISID or customer specific MAC headers.

An ingress loopback configured on the entity will have the following effects on forwarding for the entity:

Essentially an ingress loopback function will isolate the SAP or MPLS SDP binding from the rest of the service. Figure 16 uses a simple Epipe service to illustrate the various touch points and processing that occurs on a packet that is processed by an ingress loopback as it moves through the network element.

Figure 16:  Ingress Loopback 

An egress loopback configured on the entity will have the following effects on the forwarding for the entity.

In the case of the egress loopback, the SAP or MPLS SDP binding is not isolated from the rest of the service it remains part of the service and reflects traffic back into the service. Extreme care must be used when considering the application of an egress loopback in a VPLS or I-VPLS service. Since a VPLS service rely on MAC based forwarding any packet that arrives at an egress loopback will be reflected back into the service and use MAC based forwarding to apply the proper forwarding decision. If this is a live multipoint service with active endpoints this could have very negative effects on the service and the clients connected to this service. Even if the forwarding database is primed any broadcast, unknown or multicast that arrives in the service will arrive on the egress loopback and will be reflected back into the service causing at the very least duplication of all of this type of traffic.

Figure 17 uses a simple Epipe service to illustrate the various touch points and processing that occurs on a packet that is processed by an egress loopback as it moves through the network element. Egress processing will not perform queuing functions on the egress it will only perform the functions of the forwarding plane like remarking.

Figure 17:  Egress Loopback 

The operational state of the SAP or MPLS SDP binding will not change as a result of the loopback function. This means a SAP or MPLS SDP binding that is operationally up will not change state strictly because of the loopback be started or stopped. Of course control protocols that are attempting to gain access via the entity that is not allowing packets to enter the service will eventually time out.

Care must be taken when considering the use of control protocols in a service with enabled loopbacks. The operator must be very aware of the impact that interrupting control protocols can have on the state of the SAP. When SAPs are dynamically created using a protocol or a protocol is required to maintain the operational state of the SAP, interruption of this control protocol will cause the SAP to fail. Other SAPs linking their state to a failed SAP will react to that failure as well. This loopback function is per Ethernet SAP or MPLS SDP binding. This means that all traffic that is extracted and sent to the CPM prior to the loopback process will all be looped back to in the direction it was received, or in the case of VPLS, back into the service. All service based control protocols that are included with this service should be removed to ensure the loopback process is handling the packets and not some other function on the node that can extract the control protocol but never respond because the service is block. However, there may be instances where an operator would want to continue to run control protocols for the service during a loopback. For example, Down MEPs on an Ethernet SAP could continue to process ETH-CFM packets if the loopback is on the mate Ethernet SAP and was configured as an egress loopback.

By default no MAC swap functions are performed. Options are available to allow for various MAC swap functions. Table 7 lists the various options and functions based on the configured mac-swap and associated options.

Only the outer Layer 2 header can be manipulated.

In order for the loopback function to operate, the service must be operationally up, and the SAP, port, or LAG must be administratively up. In the case of a LAG, the LAG must have members port that are administratively up. If any of these conditions are not met, the loopback function will fail.

A SAP that is configured for egress loopback is not required to be operationally up, and the cabling does not need to be connected to the port. However, all necessary hardware must be installed in the network element for the ingress packets to be routed to the egress. Ghost ports do not support loopback operations.

An Epipe service will enter an operationally Down state when one of the SAPs is non-operational. The service state will remain or be returned to an operational state if the ignore-oper-down command is configured under the non-operational SAP. A VPLS service will remain operational as long as one SAP in the service is operational. However, if the SAP is a VPLS is configured over a LAG, the SAP is removed from the forwarding table if it has a non-operational state, and, consequently, packets will never reach the egress. The process-cpm-traffic-on-sap-down command can be configured under the VPLS SAP over a LAG to allow the LAG SAP to be reached even with a non-operational SAP.

If the service state is not operational or the egress SAP is not reachable via the forwarding plane, the traffic will never arrive on the SAP to be looped.

MPLS SDP bindings must be operationally up or the loopback function will fail.

In order to configure this functionality the operator is required to us use the tools hierarchy. In this specific case, the loopback tools supporting this functionality may be configured through CLI or through SNMP. However, these commands are never resident in the configuration. This means the loopback will survive high availability events that cause one CPM to change from standby to active, as well as ISSU function or IOM resets (hard or soft). However the function will not survive a complete node reboot.

In the case on SNMP, it is possible to configure a static mac address for the mac swap function without actually invoking the mac-swap. This is not possible through the CLI.

This function requires a minimum of IOM/IMM.

This feature is mutually exclusive with functions that use mirroring.

Figure 18 shows an example for placing sap 1/1/10:2.2 in service id 2 (an Epipe) in an active loopback mode with a mac-swap for all broadcast and multicast destined packets.

Figure 18:  Active Loopback Mode 

To stop the loopback, a simple stop command is required.

2.3.6. Promiscuous ETH-LBM Mode of Operation

ETH-CFM MEPs support the processing of ETH-CFM PDUs without interrupting the flow of service data. ETH-CFM processing identifies, processes, and responds to the appropriate ETH-CFM PDUs directed at the target MEP using domain-level logic comparison, equal to and lower. This behavior lends itself well to the testing of connectivity, performance monitoring, and path information. The pinpoint ETH-CFM processing logic also lends itself well to service activation testing streams encapsulated in ETH-LBM frames.

The lbm-svc-act-responder configuration option allocates additional resources to the associated MEP to process high-speed service activation streams encapsulated in ETH-LBM frames. When a MEP is created with this option, it streamlines the processing of the inbound ETH-LBM frame. This is accomplished by performing basic ETH-CFM header parsing, replacing the inbound ETH-LBM operational code (03) with the outbound ETH-LBR operational code (02), swapping source and destination MAC addresses, and reflecting any Data TLVs and other data contained in the PDU without validation. A MEP configured with the lbm-svc-act-responder configuration option will operate in promiscuous ETH-LBM mode.

Promiscuous ETH-LBM mode bypasses some checks and extended functions typically performed by a MEP. In this mode, the MEP will not validate the Layer 2 destination MAC address of the arriving ETH-LBM frame to ensure that it matches the MEP. ETH-LB system statistics and per-MEP statistics, as well as ETH-LB specific counters, will not be incremented. CFM debugging will not be available for these ETH-LB packets.

Only ETH-LBM PDUs at the same domain level as the MEP that is configured with the lbm-svc-act-responder function will access the additional resources required to accommodate high-speed service activation processing. Normal processing of ETH-CFM packets will occur for all other ETH-CFM PDUs that arrive on the MEP with the same domain level. The MEP will also process and terminate the lower levels as per normal processing. To ensure proper handling of the service activation stream encapsulated in the ETH-LBM PDU, the level of all ETH-LBM packets in the stream must equal that of the target MEP with the lbm-svc-act-responder command.

This mode of operation is supported for Up and Down MEPs in Epipe and VPLS services as well as for base router interfaces. This functionality requires a minimum of FP3 hardware.

2.5.1. OAM Considerations

Ethernet CFM can be enabled on each individual path under an Ethernet tunnel. Only down MEPs can be configured on each of them and CCM sessions can be enabled to monitor the liveliness of the path using interval as low as 10 msec. Different CCM intervals can be supported on the primary and secondary paths in an Ethernet tunnel.

MEPs can still be configured under the services independent of the Ethernet Tunnels.

The following rules control the interaction between the MEP defined under the eth-tunnel path and the MEP defined in the service:

2.5.2. QoS Considerations

When Ethernet tunnel is configured on two member ports located on different IOMs, the SAP queues and virtual schedulers will be created with the actual parameters on each IOM.

The protection mode '8031-1to1' (default) activates only the primary path at any point in time, guaranteeing the use of the desired QoS resources.

Ethernet tunnel CC messages transmitted over the SAP queues using the default egress QoS policy will use NC (network class) as a forwarding class. If user traffic is assigned to the NC forwarding class, it will compete for the same bandwidth resources with the Ethernet CCMs. As CCM loss could lead to unnecessary bouncing of the Ethernet tunnel, congestion of the queues associated with the NC traffic should be avoided. The operator must configure different QoS Policies to avoid congestion for the CCM forwarding class by controlling the amount of traffic assigned into the corresponding queue.

2.5.3. Mirroring and Lawful Intercept Considerations

Mirroring and Lawful Intercept (LI) cannot use the eth-tunnel as a source. Also, a SAP configured on an eth-tunnel cannot be used as mirror destination. The CLI blocks the above options. The SAP configured on the eth-tunnel, a filter associated with it and the member ports in the eth-tunnel> path context can be used as mirror and LI source.

2.5.4. Support Service and Solution Combinations

The Ethernet tunnels are supported Layer 2 service VLL, VPLS and B-VPLS instances. The following considerations apply:

2.5.5. LAG Emulation using Ethernet Tunnels

Ethernet Tunnels can provide G.8031 Ethernet APS protection as described in G.8031 Protected Ethernet Tunnels, or they can operate in a load-sharing manner providing an emulated LAG function. Moreover, as multiple Ethernet Tunnels can be provisioned on the same physical link(s), it is possible that two physical links could support one or more Ethernet Tunnels supporting APS protection for protected services whilst concurrently supporting one or more Ethernet Tunnels in load-sharing mode for non-protected services.

When Ethernet Tunnels have the protection type set to load-sharing, the precedence is configured to secondary, making the tunnels equal in order to implement load-sharing capability. A path threshold parameter allows the load-sharing group to be declared down if the number of paths drops equal to or lower than the threshold value. The ‘lag-emulation’ context provides access to conventional LAG parameters such as the adapt-qos mode (link, port-fair or distributed bandwidth distribution) and per-fp-ing-queuing to ensure that only one ingress queue is instantiated for every physical link supported on the same FP complex.

A typical use case for LAG emulation is to allow unprotected Ethernet services to capitalize on the LAG capability. RSTP and MSTP can also be used to network VPLS or B-VPLS over the Ethernet tunnels. LAG Emulation is also recommended when you use BGP-MH site support for Ethernet tunnels.

2.6.1. Overview of G.8032 Operation

R-APS messages that carry the G.8032 protocol are sent on dedicated protocol VLAN called the Ethernet Ring Protection (ERP) instance. In a revertive case, G.8032 Protocol ensures that one Ring Protection Link (RPL) owner blocks the RPL link. R-APS messages are periodically sent around the ring to inform other nodes in the Ring about the blocked port in the RPL owner node. In non-revertive mode any link may be the RPL. Y.1731 Ethernet OAM CC is the basis of the RAPs messages. Y.1731 CC messages are typically used by nodes in the ring to monitor the health of each link in the ring in both directions. However CC messages are not mandatory. Other link layer mechanisms could be considered – for example LOS (Loss of Signal) when the nodes are directly connected.

Initially each Ring Node blocks one of its links and notifies other nodes in the ring about the blocked link. Once a ring node in the ring learns that another link is blocked, the node unblocks its blocked link possibly causing FDB flush in all links of the ring for the affected service VLANs, controlled by the ring control instance. This procedure results in unblocking all links but the one link and the ring normal (or idle) state is reached. In revertive mode the RPL link will be the link that is blocked when all links are operable after the revert time. In non-revertive mode the RPL link is no different than other ring links. Revertive mode offers predictability particularly when there are multiple ring instances and the operator can control which links are block on the different instances. Each time there is a topology change that affects reachability, the nodes may flush the FDB and MAC learning takes place for the affected service VLANs, allowing forwarding of packets to continue. Figure 22 depicts this operational state:

Figure 22:  0-1 G.8032 Ring in the Initial State 

When a ring failure occurs, a node or nodes detecting the failure (enabled by Y.1731 OAM CC monitoring) send R-APS message in both directions. This allows the nodes at both ends of the failed link to block forwarding to the failed link preventing it from becoming active. In revertive mode, the RPL Owner then unblocks the previously blocked RPL and triggers FDB flush for all nodes for the affected service instances. The ring is now in protecting state and full ring connectivity is restored. MAC learning takes place to allow Layer 2 packet forwarding on a ring. Figure 23 depicts the failed link scenario.

Figure 23:  0-1 G.8032 Ring in the Protecting State 

Once the failed link recovers, the nodes that blocked the link again send the R-APS messages indicating no failure this time. This in turn triggers RPL owner to block the RPL link and indicate the blocked RPL link the ring in R-APS message, which when received by the nodes at the recovered link cause them to unblock that link and restore connectivity (again all nodes in the ring perform FDB flush and MAC learning takes place). The ring is back in the normal (or idle) state.

Within each path, Y.1731 Maintenance Entity Group (MEG) Endpoints (MEPs) are used to exchange R-APS specific information (specifically to co-ordinate switchovers) as well as optionally fast Continuity Check Messages (CCM) providing an inherent fault detection mechanism as part of the protocol. Failure detection of a ring path by one of the mechanisms triggers to activate the protection links. Upon failure, re-convergence times are a dependent on the failure detection mechanisms. In the case of Y.1731, the CCM transmit interval determines the response time. The router supports message timers as low as 10 milliseconds (also 100 ms) so the restoration times are comparable to SONET/SDH. Alternatively, 802.3ah (Ethernet in the First Mile) or simple Loss of Signal can act as a trigger for a protection switch where appropriate. In case of direct connectivity between the nodes, there is no need to use Ethernet CC messaging for liveliness detection.

Revertive and non-revertive behaviors are supported. The Ring protection link (RPL) is configured and Ethernet-rings can be configured to revert to the RPL upon recovery.

G.8032 supports multiple data channels (VIDs) or instances per ring control instance (R-APS tag). G.8032 also supports multiple control instances such that each instance can support RPLs on different links providing for a load balancing capability however once services have been assigned to one instance the rest of the services that need to be interconnected to those services must be on the same instance. In other words each data instance is a separate data VLAN on the same physical topology. When there is any one link failure or any one node failure in the ring, G.8032 protocols are capable of restoring traffic between all remaining nodes in these data instances.

Ethernet R-APS can be configured on any port configured for access mode using dot1q, q-in-q encapsulation enabling support for Ethernet R-APS protected services on the service edge towards the customer site, or within the Ethernet backbone. ELINE services (using PBB Epipes with the B-VPLS configured with Ethernet rings), E-LAN services, and E-Tree data services can be afforded Ethernet R-APS protection and, although the Ethernet ring providing the protection uses a ring for protection the services are configured independent of the Ring properties. The intention of this is to cause minimum disruption to the service during Ethernet R-APS failure detection and recovery.

In the implementation, the Ethernet Ring is built from a VPLS service on each node with VPLS SAPs that provides Ring path with SAPs. As a result, most of the VPLS SAP features are available on Ethernet rings if desired. This results in a fairly feature rich ring service.

The control tag defined under each Ethernet-ring is used for encapsulating and forwarding the CCMs and the G.8032 messages used for the protection function. If a failure of a link or node affects an active Ethernet ring segment, the services will fail to receive the CCMs exchanged on that segment or will receive a fault indication from the Link Layer OAM module. CCMs are optional but MEPs are always configured to provide G.8032 control. Note that the forwarding of CCMs and G.8032 R-APS messages continues in the control VPLS even if the service or its SAPs are administratively shut down. The Ethernet ring instance can be shut down if it is desired to stop the operation of the ring on a node.

For fault detection using CCMs three CC messages plus a configurable hold-off timer must be missed for a fault to be declared on the associated path. The latter mechanism is required to accommodate the existence of additional, 50 ms resiliency mechanism in the optical layer. After it receives the fault indication, the protection module will declare the associated ring link down and the G.8032 state machine will send the appropriate messages to open the RPL and flush the learned addresses.

Flushing is triggered by the G.8032 state machine and the router implementation allows flooding of traffic during the flushing interval to expedite traffic recovery.

Figure 24 illustrates a resilient Ring Service. In the example a PBB ring (solid line) using VID 500 carries 2 service VLANs on I-SID 1000 and 1001 for Service VIDs (Dot1q 100 and QinQ 400.1 respectively.) The RPL for the PBB ring is between A and B where B is the RPL owner. Also illustrated is a QinQ service on the (dotted line) ring that uses Dot1q VID 600 for the ring to connect service VLAN 100.50. The two rings have RPLs on different nodes which allow a form of load balancing. The example serves to illustrate that service encapsulations and ring encapsulation can be mixed in various combinations. Also note that neither of the rings is closed loop. A ring can restore connectivity when any one node or link fails to all remaining nodes within the 50 ms transfer time (signaling time after detection).

Figure 24:  0-3 Ring Example 

Sample Configuration:

2.6.2. Ethernet Ring Sub-Rings

Ethernet Sub-Rings offer a dual redundant way to interconnect rings. The router supports Sub-Rings connected to major rings and a sub-ring connected to a VPLS (LDP based) for access rings support in VPLS networks. Figure 25 illustrates a Major Ring and Sub-Ring scenario. In this scenario, any link can fail in either ring (ERP1 or ERP2) and each ring is protected. Furthermore, the sub ring (ERP2) relies on the major Ring (ERP1) as part of its protection for the traffic from C and D. The nodes C and D are configured as inter connection nodes.

Figure 25:  0-4 G.8032 Sub-Ring 

Sub-Rings and Major Rings run similar state machines for the ring logic, however there are some differences. When Sub-Rings protect a link, the flush messages are propagated to the major ring. (A special configuration allows control of this option on the router.) When major rings change topology, the flush is propagated around the major ring and does not continue to any sub-rings. The reason for this is that Major Rings are completely connected but Sub-Rings are dependent on another ring or network for full connectivity. The topology changes need to be propagated to the other ring or network usually. Sub-Rings offer the same capabilities as major rings in terms of control and data so that all link resource may be utilized.

2.6.2.1. Virtual and Non-Virtual Channel

The 7450 ESS, 7750 SR, and 7950 XRS support both the virtual channel and non-virtual channel for sub-ring control communication. In the virtual channel mode, a dedicated VID, other than the major ring RAPs control channel is configured as a data instance on the major ring. This allows the sub-ring control messages and state machine logic to behave similar to a major ring. In the non-virtual channel mode, the sub-ring is only connected by the RAPs control channels on the sub-ring itself. This mode offers slightly less redundancy in the RAPs messaging than the virtual channel mode since sub-ring RAPs messages are not propagated across the major ring. When non-virtual link is configured, the protocol allows RPL messages over the sub-ring blocked link.

Figure 26:  0-5 Sub-Ring Configuration Example 

Sub-ring configuration is similar to major ring configuration and consists of three parts: Ethernet-ring instance configuration, control VPLS configuration, and data VPLS configuration (data instance or data channel). The Ethernet-ring configuration of a sub-ring is tied to a major ring and only one path is allowed.

Note: A split horizon group is mandatory to ensure that Sub-Ring control messages from the major ring are only passed to the sub-ring control.

As with a major ring, the forwarding of CCMs and G.8032 R-APS messages continues in the control VPLS even if the service or its SAPs are administratively shut down. The Ethernet ring instance can be shut down if it is desired to stop the operation of the ring on a node.

The data VPLS can be configured on the major ring, and in the example, shares the same VID (SAP encapsulation) on both the major ring and the sub-ring to keep data on the same VLAN ID everywhere.

Note: Like other services in the router, the encapsulation VID is controlled by SAP configuration and the association to the controlling ring is by the Ethernet-ring, ring-id.

The following illustrates a sample sub-ring configuration on Node C:

eth-ring 2

        description "Ethernet Sub Ring on Ring 1"

        sub-ring virtual-link // Using a virtual link

            interconnect ring-id 1 // Link to Major Ring 1

                propagate-topology-change 

            exit

        exit

        path a 1/1/3 raps-tag 100 // Ring control uses VID 100

            eth-cfm

                mep 9 domain 1 association 4

                    ccm-enable

                    control-mep

                    no shutdown

                exit

            exit

            no shutdown

        exit

        no shutdown

    exit

If the sub-ring had been configured as a non-virtual-link, the sub-ring configuration above and on all the other sub-ring nodes for this sub-ring would become:

        sub-ring non-virtual-link // Not using a virtual link

 

# Control Channel for the Major Ring ERP1 illustrates that Major ring 

# control is still separate from Sub-ring control

  vpls 10 customer 1 create

      description "Control VID 10 for Ring 1 Major Ring"

      stp shutdown

      sap 1/1/1:10 eth-ring 1 create

          stp shutdown

          exit

      sap 1/1/4:10 eth-ring 1 create

          stp shutdown 

          exit

      no shutdown

  exit 

 

# Data configuration for the Sub-Ring

 

  vpls 11 customer 1 create

      description "Data on VID 11 for Ring 1"

      stp shutdown 

      sap 1/1/1:11 eth-ring 1 create // VID 11 used for ring

          stp shutdown 

      exit

      sap 1/1/4:11 eth-ring 1 create

          stp shutdown 

      exit

      sap 1/1/3:11 eth-ring 2 create // Sub-ring data

          stp shutdown 

      exit

      sap 3/2/1:1 create 

      description "Local Data SAP"

          stp shutdown 

      no shutdown

  exit

 

# Control Channel for the Sub-Ring using a virtual link. This is 

# a data channel as far as Ring 1 configuration. Other Ring 1 

# nodes also need this VID to be configured. 

 

  vpls 100 customer 1 create

      description "Control VID 100 for Ring 2 Interconnection"

      split-horizon-group "s1" create //Ring Split horizon Group

      exit

      stp shutdown 

      sap 1/1/1:100 split-horizon-group "s1" eth-ring 1 create

          stp shutdown 

      exit

      sap 1/1/4:100 split-horizon-group "s1" eth-ring 1 create

          stp shutdown 

      exit

      sap 1/1/3:100 eth-ring 2 create

          stp shutdown 

      exit

      no shutdown

  exit 

If the sub-ring had been configured as a non-virtual-link, the configuration above would then become:

  vpls 100 customer 1 create

      description "Control VID 100 for Ring 2 Interconnection"

      sap 1/1/3:100 eth-ring 2 create

          stp shutdown 

      exit

      no shutdown

  exit 

The 7450 ESS, 7750 SR, and 7950 XRS allow for a special configuration of the non-virtual link sub-ring that can be homed to a VPLS service illustrated in Figure 27. This is an economical way to have a redundant ring connection to a VPLS service. This is currently supported only for dot1Q and QinQ sub-rings and only on LDP based VPLS. The primary application for this is access rings that require resiliency. This configuration shows the configuration for a sub-ring at an interconnection node without a virtual channel and interconnected to a VPLS. A VPLS service 1 is used to terminate the ring control. The Ethernet ring data SAP appears in the associated LDP based VPLS service 5.

Figure 27:  0-6 Sub-Ring Homed to VPLS 

The following is a sample sub-ring configuration for VPLS (at PE1):

eth-ring 1

      description "Ethernet Ring 1"

      guard-time 20

      no revert-time

      rpl-node nbr

      sub-ring non-virtual-link

          interconnect vpls // VPLS is interconnection type

              propagate-topology-change 

          exit

      exit

      path a 1/1/3 raps-tag 1.1

          description "Ethernet Ring : 1 Path on LAG"

          eth-cfm

          mep 8 domain 1 association 8

               ccm-enable

               control-mep

               no shutdown

            exit

        exit

        no shutdown

    exit

    no shutdown

exit

 

# Configuration for the ring control interconnection termination:  

  vpls 1 customer 1 create

      description "Ring 1 Control termination"

      stp shutdown

      sap 1/1/3:1.1 eth-ring 1 create //path a control

          stp shutdown

      exit

      no shutdown

  exit

 

# Configuration for the ring data into the LDP based VPLS Service

 

  vpls 5 customer 1 create

      description "VPLS Service at PE1"

      stp

          no shutdown

      exit

      sap 1/1/3:2.2 eth-ring 1 create

          stp shutdown

      exit

      sap 1/1/5:1 create

      exit

      mesh-sdp 5001:5 create //sample LDP MPLS LSPs

      exit

      mesh-sdp 5005:5 create

      exit

      mesh-sdp 5006:5 create

      exit

          

      no shutdown

  exit

Ethernet-rings and sub-rings offer a way to build a scalable resilient Ethernet transport network. Figure 28 illustrates a hierarchical ring network using PBB where dual homed services are connected to a PBB based Ethernet ring network.

Figure 28:  0-7 Multi Ring Hierarchy 

The major rings are connected by sub-rings to the top level major ring. These sub-rings require virtual channel and will not work with non-virtual channel. Ring flushing is contained to major rings, or in the case of a sub-ring link or node failure, to the sub-ring and the directly attached major rings.

2.6.3. OAM Considerations

Ethernet CFM is enabled by configuring MEPs on each individual path under an Ethernet ring. Only down MEPs can be configured on each of them and optionally, CCM sessions can be enabled to monitor the liveliness of the path using interval of 10 or 100 msec. Different CCM intervals can be supported on the path a and path b in an Ethernet ring. CFM is optional if hardware supports Loss of Signal (LOS) for example, which is controlled by configuring no-ccm-enable.

Up MEPs on service SAPs which multicast into the service and monitor the active path may be used to monitor services.

When Ethernet ring is configured on two ports located on different cards, the SAP queues and virtual schedulers will be created with the actual parameters on each card.

Ethernet ring CC messages transmitted over the SAP queues using the default egress QoS policy will use NC (network class) as a forwarding class. If user traffic is assigned to the NC forwarding class, it will compete for the same bandwidth resources with the Ethernet CCMs. As CCM loss could lead to unnecessary switching of the Ethernet ring, congestion of the queues associated with the NC traffic should be avoided. The operator must configure different QoS Policies to avoid congestion for the CCM forwarding class by controlling the amount of traffic assigned into the corresponding queue.

Details of the Ethernet ring applicability in the services solution can be found in the respective sections of the 7450 ESS, 7750 SR, 7950 XRS, and VSR Layer 2 Services and EVPN Guide: VLL, VPLS, PBB, and EVPN.

2.6.4. Support Service and Solution Combinations

The Ethernet rings are supported Layer 2 service, VPLS, I-VPLS, R-VPLS and B-VPLS instances. The following considerations apply:

2.10.1. NGE Overview

The network group encryption (NGE) feature enables end-to-end encryption of MPLS services, Layer 3 user traffic, and IP/MPLS control traffic. NGE is an encryption method that uses a group-based keying security architecture, which removes the need to configure individual encryption tunnels to achieve network-wide encryption.

NGE relies on the NSP NFM-P to manage the network and apply encryption to specific MPLS services, Layer 3 user traffic, or control plane traffic depending on the security requirements of the network. Operators designate traffic types that require added security and then apply NGE to those traffic types using the NSP NFM-P. The NSP NFM-P also acts as the network-wide NGE key manager, downloading encryption and authentication keys to nodes and performing hitless rekeying of the network at operator-defined intervals. For more information about managing NGE within a network, see the NSP NFM-P User Guide.

Figure 29 shows an NGE network with NSP NFM-P services, control plane configuration, and key management.

Figure 29:  NGE Network with NSP NFM-P Management 

NGE provides three main types of encryption to secure an IP/MPLS network:

Note: Refer to the 7450 ESS, 7750 SR, 7950 XRS, and VSR Router Configuration Guide for information about configuring NGE on router interfaces. Refer to the 7450 ESS, 7750 SR, and VSR Triple Play Service Delivery Architecture Guide for information about configuring group encryption on the WLAN-GW group interface.

NGE is supported on the following platforms:

WLAN-GW group interfaces enabled with NGE is further supported on the following platforms:

2.10.1.1. NGE Key Groups and Encryption Partitions

NGE allows a tiered approach to managing encryption keys in a network using key groups by configuring services or router interfaces to use specific key groups, depending on security policies for the service and network topology.

Figure 30 shows a typical application of NGE key-group partitioning in which there are several critical levels (tiers) of security that need to be considered. In this example, the protection of Distribution Automation and Field Area Network (DA/FAN) equipment are less critical than the Transmission or Distribution Substation network equipment. Ensure that nodes more at risk of a security breach do not contain more critical information than is necessary. Therefore, encryption keys for the sensitive portions of the network (such as control center traffic) should not be available on nodes that are at risk. The NGE feature enables operators to partition and distribute encryption keys among different services, NGE domains, or nodal groups in a network. NGE partitions are enabled by configuring different key groups per security partition and applying those key groups as needed.

Figure 30:  Key-Group Partitioning 

Another application of key-group partitioning allows different parts of an organization to have their own method of end-to-end communication without the need to share encryption keys between each organization. If two partitions need to communicate between themselves, gateway nodes configured with both key groups allow inter-organization traffic flows between the key group partitions as needed.

2.10.2. Key Groups

Users can partition the network based on security requirements by organizing encryption keys into distinct key groups. A key group contains the following elements:

Figure 31 illustrates the use of key groups (KGs), SAs, and security parameter indices (SPIs). The VSR-1 and VSR-2 both have the same set of key groups configured. One path uses key group 1 (KG1) and the other uses key group 2 (KG2). Each key group contains the elements listed above. KG1 has four live keys, SPI_1 through SPI_4, and SPI_3 is the active outbound SA. The active outbound SA is identified by its SPI, and this SPI is embedded in the NGE packet.

Each SA listed in a key group, indexed by an SPI, specifies a single key for encryption and a single key for authentication. Packets transmitted or received that reference a particular SPI use the keys in the SA for that SPI when performing encryption and authentication.

Before enabling encryption, key groups must be configured on the node. Only after a key group is configured can it be assigned to an SDP or VPRN services.

Figure 31:  Key Groups and a Typical NGE Packet  

2.10.2.2. Security Associations

Each key group has a list of up to four security associations (SAs). An SA is a reference to a pair of encryption and authentication keys that are used to decrypt and authenticate packets received by the node and to encrypt packets leaving the node.

For encrypted ingress traffic, any of the four SAs in the key group can be used for decryption if there is a match between the SPI in the traffic and the SPI in the SA. For egress traffic, only one of the SAs can be used for encryption and is designated as the active outbound SA. Figure 31 illustrates these relationships.

As shown in Figure 31, each SA is referenced by an SPI value, which is included in packets during encryption and authentication. SPI values must be numerically unique throughout all SAs in all key groups. If an SPI value is configured in one key group and an attempt is made to configure the same SPI value in another key group, the configuration is blocked.

Note: Keys are entered in clear text using the security-association command. After configuration, they are never displayed in their original, clear text form. Keys are displayed in an encrypted form, which is indicated by the system-appended crypto keyword when an info command is run. The NGE node also includes the crypto keyword with an admin>save operation so that the NGE node can decrypt the keys when reloading a configuration database. For security reasons, keys encrypted on one node are not usable on other nodes (that is, keys are not exchangeable between nodes).

2.10.2.2.1. Active Outbound SA

The active outbound SA is specified by the SPI referencing the specific SA used to encrypt and authenticate packets egressing the node for the SDP or service using the key group. The SPI value for the active outbound SA is included in the ESP header of packets being encrypted and authenticated.

2.10.3. Services Encryption

NGE provides the ability to encrypt MPLS services using key groups that are configured against these services. These services include:

For services that use SDPs, all tunnels may be either MPLS LSPs (RSVP-TE, LDP, or static LSP), or GRE or MPLSoUDP tunnels.

For MP-BGP services, auto-bind-tunnel is supported using LDP, GRE, MPLSoUDP, RSVP-TE, or MPLS (LDP or RSVP-TE).

2.10.3.1. Services Encryption Overview

NGE adds a global encryption label to the label stack for encrypting MPLS services. The global encryption label must be a unique network-wide label; in other words, the same label must be used on all nodes in the network that require NGE services. The label must be configured on individual nodes before NGE can become operational on those nodes.

The global encryption label is used to identify packets that have an NGE-encrypted payload and is added to the bottom of the stack. This allows network elements such as LSRs, ABRs, ASBRs, and RRs to forward NGE packets without needing to understand NGE or to know that the contents of these MPLS packets are encrypted. Only when a destination PE receives a packet that needs to be understood at the service layer does the PE check for an encryption label, and then decrypt the packet.

After the global encryption label is set, it should not be changed. If the label must be changed without impacting traffic, all key groups in the system should first be deleted. Next, the label should be changed, and then all key groups should be reconfigured.

The NSP NFM-P helps to coordinate the distribution of the global encryption label and ensures that all nodes in the network are using the same global encryption label.

Figure 32 illustrates the NGE MPLS, GRE, or MPLSoUDP label stack.

Figure 32:  NGE MPLS/GRE/MPLSoUDP Label Stack  

Figure 33 illustrates VPRN and PW (with control word) packet formats using NGE.

Figure 33:  NGE and Packet Formats  

2.10.3.2. Assigning Key Groups to Services

Assigning key groups to services requires configuring an inbound and outbound key group for directional processing on a per-service basis (see Figure 34).

Figure 34:  Inbound and Outbound Key-Group Assignments 

The outbound key group identifies which key group to use for traffic that egresses the node for the service. The inbound key group ensures that ingress traffic is using the correct key group for the service.

If the inbound key group is not set, the node ensures that packets are either unencrypted or are using one of the valid key groups configured in the system.

In most deployment scenarios, the inbound and outbound key groups are identical; however, it is possible to configure different key groups as the outbound and the inbound key groups, as this is not checked by the node.

Including an inbound and outbound direction when assigning key groups to services allows users to:

1. gracefully enable and disable NGE for services
2. move services from one key-group domain to another domain without halting encryption

The NGE feature makes use of the NSP NFM-P to help manage the assignment of key groups to services on a network-wide basis. Refer to the NSP NFM-P User Guide for more information.

2.10.3.5. WLAN-GW Encryption

NGE is supported on the WLAN-GW to provide encryption of traffic to and from WLAN APs that support NGE, such as the 7705 SAR-Hm. As shown in Figure 35, the application uses Epipe pseudowire services, as described in Pseudowire Switching for NGE Traffic and Pseudowire Control Word for NGE Traffic, with L2oMPLSoGRE transport and NGE applied to the GRE-SDP.

Figure 35:  Terminating NGE-Protected WLAN AP Traffic Destined to the WLAN-GW 

In Figure 35, the same key-group, KG1, is configured on:

1. the WLAN-GW gw-address under the IES or VPRN soft-GRE group interface for the vMS-ISA
2. the GRE SDPs that are bound to any WLAN AP Epipes that are terminating soft-GRE services on the WLAN-GW group interface.

Traffic from an authenticated user on the SAR-Hm WLAN AP is encrypted and an NGE label is added to the packet after the Epipe service label. The packet format is shown in Figure 35.

The WLAN-GW group interface is configured with the same inbound and outbound key-group as the GRE-SDP used for the Epipe from the WLAN AP. Any L2oMPLSoGRE packet received by the WLAN-GW on the NGE-enabled group interface must be encrypted with NGE per the above format. All other supported WLAN-GW packet types (that is, those not using L2oMPLSoGRE) are not impacted by the NGE configuration and can pass through the WLAN-GW without NGE (such as L2oGRE packets).

2.10.3.7. NGE and RFC 3107

When RFC 3107 is enabled on the node and NGE traffic is crossing the Area Border Router (ABR) between two VPRN domains, the same key group must be used between the two domains.

Note: It is the responsibility of the network operator to ensure key group consistency across the (ABR).

2.10.4. NGE Packet Overhead and MTU Considerations

NGE adds overhead packets to services. Table 8 shows the additional overhead for the worst-case scenario of MPLS services encryption. Table 9 shows the additional overhead for the worst-case scenario of router interface. Additional overhead depends on which encryption and authentication algorithms are chosen.

For MP-BGP-based VPRNs, the total is 77 bytes because the control word copy is not required.

For Layer 3 packets for router interface encryption, the total is 73 bytes because the encryption label and control word copy are not required.

The overhead values in Table 8 must be considered for services that are supported by NGE.

Note: Currently, the port MTU has a default value of 1572 bytes. This value is too low for outbound traffic when NGE is enabled. Users must configure new MTU values to adjust for the overhead associated with NGE, as outlined in Table 10 for MPLS-based and GRE-based services. For details on configuring MTU, refer to the “MTU Configuration Guidelines” section in the 7450 ESS, 7750 SR, 7950 XRS, and VSR Interface Configuration Guide.

The calculations in Table 10 show how NGE overhead affects SDP MTU and service MTU values for MPLS-based, GRE-based, and VPRN-based services. The calculations are with and without NGE enabled.

When an unencrypted Layer 3 packet ingresses the node and routing determines that the egress interface is a router interface NGE-enabled interface, the node calculates whether the packet size will be greater than the MTU of the egress interface after the router interface NGE overhead is added. If the packet cannot be forwarded out from the network interface, an ICMP message is sent back to the sender and the packet is dropped. Users must configure new MTU values to adjust for the overhead associated with NGE.

If an IP exception ACL that matches the ingressing packet exists on the egress interface, the MTU check applied to the ingress packet includes the router interface NGE overhead. This is because the ingress interface cannot determine which IP exceptions are configured on the egress interface, and therefore the worst-case MTU check that includes the router interface NGE overhead is performed.

2.10.5. Statistics

Statistics specific to NGE are counted for the following main areas:

2.10.6. Remote Network Monitoring Support

Remote network Monitoring (RMON) can be used in conjunction with NGE statistics to provide event and alarm reporting. This can be used by customers to detect security breaches of NGE traffic flows and provide real-time reporting of such events.

Threshold crossing alerts and alarms using RMON are supported for SNMP MIB objects, including NGE.

2.10.7. Configuration Notes

This section describes NGE configuration guidelines and caveats. For more information about configuring NGE using the NSP NFM-P, see the NSP NFM-P User Guide.

Consider the following caveats when performing NGE configuration tasks.

To enable NGE for an SDP or VPRN service:

To enable NGE for a router interface:

To change NGE from one key group to another key group for an SDP or VPRN service:

To change NGE from one key group to another key group for a router interface:

To disable NGE for an SDP or VPRN service:

To disable NGE for a router interface:

2.12.1. Phase 1: Core Network Construction

Before the services are provisioned, the following tasks should be completed:

2.12.2. Phase 2: Service Administration

Perform preliminary policy and SDP configurations to control traffic flow, operator access, and to manage fault conditions and alarm messages, the following tasks should be completed:

2.12.3. Phase 3: Service Provisioning

2.13.1. General

Service provisioning tasks are typically performed prior to provisioning a subscriber service and can be logically separated into two main functional areas: core tasks and subscriber tasks.

Core tasks include the following:

Subscriber services tasks include the following: