5. Ethernet Virtual Private Networks (EVPNs)

5.1. Overview and EVPN Applications

EVPN is an IETF technology per RFC 7432, BGP MPLS-Based Ethernet VPN, that uses a new BGP address family and allows VPLS services to be operated as IP-VPNs, where the MAC addresses and the information to set up the flooding trees are distributed by BGP.

EVPN is defined to fill the gaps of other L2VPN technologies such as VPLS. The main objective of the EVPN is to build E-LAN services in a similar way to RFC 4364 IP-VPNs, while supporting MAC learning within the control plane (distributed by MP-BGP), efficient multi-destination traffic delivery, and active-active multi-homing.

EVPN can be used as the control plane for different data plane encapsulations. The Nokia implementation supports the following data planes:

  1. EVPN for VXLAN overlay tunnels (EVPN-VXLAN)
    EVPN for VXLAN overlay tunnels (EVPN-VXLAN), being the Data Center Gateway (DC GW) function the main application for this feature. In such application VXLAN is expected within the Data Center and VPLS sdp-bindings or SAPs are expected for the connectivity to the WAN. R-VPLS and VPRN connectivity to the WAN is also supported.
    The EVPN-VXLAN functionality is standardized in IETF Draft draft-ietf-bess-evpn-overlay.
  2. EVPN for MPLS tunnels (EVPN-MPLS)
    EVPN for MPLS tunnels (EVPN-MPLS), where PEs are connected by any type of MPLS tunnel. EVPN-MPLS is generally used as an evolution for VPLS services in the WAN, being Data Center Interconnect one of the main applications.
    The EVPN-MPLS functionality is standardized in RFC 7432.
  3. EVPN for PBB over MPLS tunnels (PBB-EVPN),
    PEs are connected by PBB over MPLS tunnels in this data plane. It is usually used for large scale E-LAN and E-Line services in the WAN.
    The PBB-EVPN functionality is standardized in RFC 7623.

The 7750 SR, 7450 ESS, or 7950 XRS EVPN VXLAN implementation is integrated in the Nuage Data Center architecture, where the router serves as the DC GW.

Refer to the Nuage Networks Virtualized Service Platform Guide for more information about the Nuage Networks architecture and products. The following sections describe the applications supported by EVPN in the 7750 SR, 7450 ESS, or 7950 XRS implementation.

5.1.1. EVPN for VXLAN Tunnels in a Layer 2 DC GW (EVPN-VXLAN)

Figure 138 shows the use of EVPN for VXLAN overlay tunnels on the 7750 SR, 7450 ESS, or 7950 XRS when it is used as a Layer 2 DC GW.

Figure 138:  Layer 2 DC PE with VPLS to the WAN 

DC providers require a DC GW solution that can extend tenant subnets to the WAN. Customers can deploy the NVO3-based solutions in the DC, where EVPN is the standard control plane and VXLAN is a predominant data plane encapsulation. The Nokia DC architecture (Nuage) uses EVPN and VXLAN as the control and data plane solutions for Layer 2 connectivity within the DC and so does the SR OS.

While EVPN VXLAN will be used within the DC, most service providers use VPLS and H-VPLS as the solution to extend Layer 2 VPN connectivity. Figure 138 shows the Layer 2 DC GW function on the 7750 SR, 7450 ESS, and 7950 XRS routers, providing VXLAN connectivity to the DC and regular VPLS connectivity to the WAN.

The WAN connectivity will be based on VPLS where SAPs (null, dot1q, and qinq), Spoke-SDPs (FEC type 128 and 129), and mesh-SDPs are supported.

The DC GWs can provide multi-homing resiliency through the use of BGP multi-homing.

EVPN-MPLS can also be used in the WAN. In this case, the Layer 2 DC GW function provides translation between EVPN-VXLAN and EVPN-MPLS. EVPN multi-homing can be used to provide DC GW redundancy.

If point-to-point services are needed in the DC, SR OS supports the use of EVPN-VPWS for VXLAN tunnels, including multi-homing, according to RFC8214.

5.1.2. EVPN for VXLAN Tunnels in a Layer 2 DC with Integrated Routing Bridging Connectivity on the DC GW

Figure 139 shows the use of EVPN for VXLAN overlay tunnels on the 7750 SR, 7450 ESS, or 7950 XRS when the DC provides Layer 2 connectivity and the DC GW can route the traffic to the WAN through an R-VPLS and linked VPRN.

Figure 139:  GW IRB on the DC PE for an L2 EVPN/VXLAN DC 

In some cases, the DC GW must provide a Layer 3 default gateway function to all the hosts in a specified tenant subnet. In this case, the VXLAN data plane will be terminated in an R-VPLS on the DC GW, and connectivity to the WAN will be accomplished through regular VPRN connectivity. The 7750 SR, 7450 ESS, and 7950 XRS support IPv4 and IPv6 interfaces as default gateways in this scenario.

5.1.3. EVPN for VXLAN Tunnels in a Layer 3 DC with Integrated Routing Bridging Connectivity among VPRNs

Figure 140 shows the use of EVPN for VXLAN tunnels on the 7750 SR, 7450 ESS, or 7950 XRS when the DC provides distributed Layer 3 connectivity to the DC tenants.

Figure 140:  GW IRB on the DC PE for an L3 EVPN/VXLAN DC 

Each tenant will have several subnets for which each DC Network Virtualization Edge (NVE) provides intra-subnet forwarding. An NVE may be a Nuage VSG, VSC/VRS, or any other NVE in the market supporting the same constructs, and each subnet normally corresponds to an R-VPLS. For example, in Figure 140, subnet 10.20.0.0 corresponds to R-VPLS 2001 and subnet 10.10.0.0 corresponds to R-VPLS 2000. In this example, the NVE provides inter-subnet forwarding too, by connecting all the local subnets to a VPRN instance. When the tenant requires L3 connectivity to the IP-VPN in the WAN, a VPRN is defined in the DC GWs, which connects the tenant to the WAN. That VPRN instance will be connected to the VPRNs in the NVEs by means of an IRB (Integrated Routing and Bridging) backhaul R-VPLS. This IRB backhaul R-VPLS provides a scalable solution because it allows L3 connectivity to the WAN without the need for defining all of the subnets in the DC GW.

The 7750 SR, 7450 ESS (in mixed mode), and 7950 XRS DC GW support the IRB backhaul R-VPLS model, where the R-VPLS runs EVPN-VXLAN and the VPRN instances exchange IP prefixes (IPv4 and IPv6) through the use of EVPN. Interoperability between the EVPN and IP-VPN for IP prefixes is also fully supported.

5.1.4. EVPN for VXLAN Tunnels in a Layer 3 DC with EVPN-Tunnel Connectivity among VPRNs

Figure 141 shows the use of EVPN for VXLAN tunnels on the 7750 SR, 7450 ESS (in mixed mode), or 7950 XRS, when the DC provides distributed Layer 3 connectivity to the DC tenants and the VPRN instances are connected through EVPN tunnels.

Figure 141:  EVPN-Tunnel GW IRB on the DC PE for an L3 EVPN/VXLAN DC 

The solution described in section EVPN for VXLAN Tunnels in a Layer 3 DC with Integrated Routing Bridging Connectivity among VPRNs provides a scalable IRB backhaul R-VPLS service where all the VPRN instances for a specified tenant can be connected by using IRB interfaces. When this IRB backhaul R-VPLS is exclusively used as a backhaul and does not have any SAPs or SDP-bindings directly attached, the solution can be optimized by using EVPN tunnels.

EVPN tunnels are enabled using the evpn-tunnel command under the R-VPLS interface configured on the VPRN. EVPN tunnels provide the following benefits to EVPN-VXLAN IRB backhaul R-VPLS services:

  1. Easier provisioning of the tenant service. If an EVPN tunnel is configured in an IRB backhaul R-VPLS, there is no need to provision the IRB IPv4 addresses on the VPRN. This makes the provisioning easier to automate and saves IP addresses from the tenant space.
    Note:

    IPv6 interfaces do not require the provisioning of an IPv6 Global Address; a Link Local Address is automatically assigned to the IRB interface.

  2. Higher scalability of the IRB backhaul R-VPLS. If EVPN tunnels are enabled, multicast traffic is suppressed in the EVPN-VXLAN IRB backhaul R-VPLS service (it is not required). As a result, the number of VXLAN binds in IRB backhaul R-VPLS services with EVPN-tunnels can be much higher.

This optimization is fully supported by the 7750 SR, 7450 ESS (in mixed mode), and 7950 XRS.

5.1.5. EVPN for MPLS Tunnels in E-LAN Services

Figure 142 shows the use of EVPN for MPLS tunnels on the 7750 SR, 7450 ESS, and 7950 XRS. In this case, EVPN is used as the control plane for E-LAN services in the WAN.

Figure 142:  EVPN for MPLS in VPLS Services 

EVPN-MPLS is standardized in RFC 7432 as an L2VPN technology that can fill the gaps in VPLS for E-LAN services. A significant number of service providers offering E-LAN services today are requesting EVPN for their multi-homing capabilities, as well as the optimization EVPN provides. EVPN supports all-active multi-homing (per-flow load-balancing multi-homing) as well as single-active multi-homing (per-service load-balancing multi-homing).

EVPN is a standard-based technology that supports all-active multi-homing, and although VPLS already supports single-active multi-homing, EVPN's single-active multi-homing is perceived as a superior technology due to its mass-withdrawal capabilities to speed up convergence in scaled environments.

EVPN technology provides a number of significant benefits, including:

  1. superior multi-homing capabilities
  2. an IP-VPN-like operation and control for E-LAN services
  3. reduction and (in some cases) suppression of the BUM (broadcast, Unknown unicast, and Multicast) traffic in the network
  4. simple provision and management
  5. new set of tools to control the distribution of MAC addresses and ARP entries in the network

The SR OS EVPN-MPLS implementation is compliant with RFC 7432.

EVPN-MPLS can also be enabled in R-VPLS services with the same feature-set that is described for VXLAN tunnels in sections EVPN for VXLAN Tunnels in a Layer 3 DC with Integrated Routing Bridging Connectivity among VPRNs and EVPN for VXLAN Tunnels in a Layer 3 DC with EVPN-Tunnel Connectivity among VPRNs.

5.1.6. EVPN for MPLS Tunnels in E-Line Services

The MPLS network used by EVPN for E-LAN services can also be shared by E-Line services using EVPN in the control plane. EVPN for E-Line services (EVPN-VPWS) is a simplification of the RFC 7432 procedures, and it is supported in compliance with RFC 8214.

5.1.7. EVPN for MPLS Tunnels in E-Tree Services

The MPLS network used by E-LAN and E-Line services can also be shared by Ethernet-Tree (E-Tree) services using the EVPN control plane. EVPN E-Tree services use the EVPN control plane extensions described in IETF RFC 8317 and are supported on the 7750 SR, 7450 ESS, and 7950 XRS.

5.1.8. EVPN for PBB over MPLS Tunnels (PBB-EVPN)

Figure 143 shows the use of EVPN for MPLS tunnels on the 7750 SR, 7450 ESS, and 7950 XRS. In this case, EVPN is used as the control plane for E-LAN services in the WAN.

Figure 143:  EVPN for PBB over MPLS  

EVPN for PBB over MPLS (hereafter called PBB-EVPN) is specified in RFC 7623. It provides a simplified version of EVPN for cases where the network requires very high scalability and does not need all the advanced features supported by EVPN-MPLS (but still requires single-active and all-active multi-homing capabilities).

PBB-EVPN is a combination of 802.1ah PBB and RFC 7432 EVPN and reuses the PBB-VPLS service model, where BGP-EVPN is enabled in the B-VPLS domain. EVPN is used as the control plane in the B-VPLS domain to control the distribution of BMACs and setup per-ISID flooding trees for I-VPLS services. The learning of the CMACs, either on local SAPs/SDP-bindings or associated with remote BMACs, is still performed in the data plane. Only the learning of BMACs in the B-VPLS is performed through BGP.

The SR OS PBB-EVPN implementation supports PBB-EVPN for I-VPLS and PBB-Epipe services, including single-active and all-active multi-homing.

5.2. EVPN for VXLAN Tunnels and Cloud Technologies

This section provides information about EVPN for VXLAN tunnels and cloud technologies.

5.2.1. Introduction to VXLAN

The SR OS and Nuage solution for DC supports VXLAN (Virtual eXtensible Local Area Network) overlay tunnels as per RFC 7348.

VXLAN addresses the data plane needs for overlay networks within virtualized data centers accommodating multiple tenants. The main attributes of the VXLAN encapsulation are:

  1. VXLAN is an overlay network encapsulation used to carry MAC traffic between VMs over a logical Layer 3 tunnel.
  2. Avoids the Layer 2 MAC explosion, because VM MACs are only learned at the edge of the network. Core nodes simply route the traffic based on the destination IP (which is the system IP address of the remote PE or VTEP-VXLAN Tunnel End Point).
  3. Supports multi-path scalability through ECMP (to a remote VTEP address, based on source UDP port entropy) while preserving the Layer 2 connectivity between VMs. xSTP is no longer needed in the network.
  4. Supports multiple tenants, each with their own isolated Layer 2 domain. The tenant identifier is encoded in the VNI field (VXLAN Network Identifier) and allows up to 16M values, as opposed to the 4k values provided by the 802.1q VLAN space.

Figure 144 shows an example of the VXLAN encapsulation supported by the Nokia implementation.

Figure 144:  VXLAN Frame Format 

As shown in Figure 144, VXLAN encapsulates the inner Ethernet frames into VXLAN + UDP/IP packets. The main pieces of information encoded in this encapsulation are:

  1. VXLAN header (8 bytes)
    1. Flags (8 bits) where the I flag is set to 1 to indicate that the VNI is present and valid. The rest of the flags (“Reserved” bits) are set to 0.
    2. Includes the VNI field (24-bit value) or VXLAN network identifier. It identifies an isolated Layer 2 domain within the DC network.
    3. The rest of the fields are reserved for future use.
  2. UDP header (8 bytes)
    1. Where the destination port is a well-known UDP port assigned by IANA (4789).
    2. The source port is derived from a hashing of the inner source and destination MAC/IP addresses that the 7750 SR, 7450 ESS, or 7950 XRS does at ingress. This will create an “entropy” value that can be used by the core DC nodes for load balancing on ECMP paths.
    3. The checksum will be set to zero.
  3. Outer IP and Ethernet headers (34 or 38 bytes)
    1. The source IP and source MAC will identify the source VTEP. That is, these fields will be populated with the PE’s system IP and chassis MAC address.
      Note:

      The source MAC address will be changed on all the IP hops along the path, as is usual in regular IP routing.

    2. The destination IP will identify the remote VTEP (remote system IP) and will be the result of the destination MAC lookup in the service Forwarding Database (FDB).
      Note:

      All remote MACs will be learned by the EVPN BGP and associated with a remote VTEP address and VNI.

Some considerations related to the support of VXLAN on the 7750 SR, 7450 ESS, and 7950 XRS are:

  1. VXLAN is only supported on network or hybrid ports with null or dot1q encapsulation.
  2. VXLAN is supported on Ethernet/LAG and POS/APS.
  3. IPv4 and IPv6 unicast addresses are supported as VTEPs.
  4. By default, system IP addresses are supported, as VTEPs, for originating and terminating VXLAN tunnels. Non-system IPv4 and IPv6 addresses are supported by using a Forwarding Path Extension (FPE).

5.2.1.1. VXLAN ECMP and LAG

The DC GW supports ECMP load balancing to reach the destination VTEP. Also, any intermediate core node in the Data Center should be able to provide further load balancing across ECMP paths because the source UDP port of each tunneled packet is derived from a hash of the customer inner packet. The following must be considered:

  1. ECMP for VXLAN is supported on VPLS services, but not for BUM traffic. Unicast spraying will be based on the packet contents.
  2. ECMP for VXLAN on R-VPLS services is supported for VXLAN IPv6 tunnels.
  3. ECMP for VXLAN IPv4 tunnels on R-VPLS is only supported if the command config>service>vpls>allow-ip-int-bind>vxlan-ipv4-tep-ecmp is enabled on the R-VPLS.
  4. In the above cases where ECMP is not supported (BUM traffic in VPLS and VXLAN IPv4 on R-VPLS if not enabled), each VXLAN binding is tied to a single (different) ECMP path, so in a normal deployment with a reasonable number of remote VTEPs, there should be a fair distribution of the traffic across the paths. In other words, only per-VTEP load-balancing is supported, instead of per-flow load-balancing.
  5. LAG spraying based on the packet hash is supported in all the cases (VPLS unicast, VPLS BUM, and R-VPLS).

5.2.1.2. VXLAN VPLS Tag Handling

The following describes the behavior on the 7750 SR, 7450 ESS, and 7950 XRS with respect to VLAN tag handling for VXLAN VPLS services:

  1. Dot1q, QinQ, and null SAPs, as well as regular VLAN handling procedures at the WAN side, are supported on VXLAN VPLS services.
  2. No “vc-type vlan” like VXLAN VNI bindings are supported. Therefore, at the egress of the VXLAN network port, the router will not add any inner VLAN tag on top of the VXLAN encapsulation, and at the ingress network port, the router will ignore any VLAN tag received and will consider it as part of the payload.

5.2.1.3. VXLAN MTU Considerations

For VXLAN VPLS services, the network port MTU must be at least 50 Bytes (54 Bytes if dot1q) greater than the Service-MTU to allow enough room for the VXLAN encapsulation.

The Service-MTU is only enforced on SAPs, (any SAP ingress packet with MTU greater than the service-mtu will be discarded) and not on VXLAN termination (any VXLAN ingress packet will make it to the egress SAP regardless of the configured service-mtu).

Note:

The router will never fragment or reassemble VXLAN packets. In addition, the router always sets the DF (Do not Fragment) flag in the VXLAN outer IP header.

5.2.1.4. VXLAN QoS

VXLAN is a network port encapsulation; therefore, the QoS settings for VXLAN are controlled from the network QoS policies.

5.2.1.4.1. Ingress

The network ingress QoS policy can be applied either to the network interface over which the VXLAN traffic arrives or under vxlan/network/ingress within the EVPN service.

Regardless of where the network QoS policy is applied, the ingress network QoS policy is used to classify the VXLAN packets based on the outer dot1p (if present), then the outer DSCP, to yield an FC/profile.

If the ingress network QoS policy is applied to the network interface over which the VXLAN traffic arrives then the VXLAN unicast traffic uses the network ingress queues configured on FP where the network interface resides. QoS control of BUM traffic received on the VXLAN tunnels is possible by separately redirecting these traffic types to policers within an FP ingress network queue group. This QoS control uses the per forwarding class fp-redirect-group parameter together with broadcast-policer, unknown-policer, and mcast-policer within the ingress section of a network QoS policy. This QoS control applies to all BUM traffic received for that forwarding class on the network IP interface on which the network QoS policy is applied.

The ingress network QoS policy can also be applied within the EVPN service by referencing an FP queue group instance, as follows:

configure
    service
        vpls <service-id>
            vxlan vni <vni-id>
                network
                    ingress
                        qos <network-policy-id>
                            fp-redirect-group <queue-group-name>
                                     instance <instance-id>

In this case, the redirection to a specific ingress FP queue group applies as a single entity (per forwarding class) to all VXLAN traffic received only by this service. This overrides the QoS applied to the related network interfaces for traffic arriving on VXLAN tunnels in that service but does not affect traffic received on a Spoke-SDP in the same service. It is possible to also redirect unicast traffic to a policer using the per forwarding class fp-redirect-group policer parameter, as well as the BUM traffic as above, within the ingress section of a network QoS policy. The use of ler-use-dscp, ip-criteria and ipv6-criteria statements are ignored if configured in the ingress section of the referenced network QoS policy. If the instance of the named queue group template referenced in the qos command is not configured on an FP receiving the VXLAN traffic, then the traffic uses the ingress network queues or queue group related to the network interface.

5.2.1.4.2. Egress

On egress, there is no need to specify “remarking” in the policy to mark the DSCP. This is because the VXLAN adds a new IPv4 header, and the DSCP will be always marked based on the egress network qos policy.

5.2.1.5. VXLAN Ping

A new VXLAN troubleshooting tool, VXLAN Ping, is available to verify VXLAN VTEP connectivity. The VXLAN Ping command is available from interactive CLI and SNMP.

This tool allows the operator to specify a wide range of variables to influence how the packet is forwarded from the VTEP source to VTEP termination. The ping function requires the operator to specify a different test-id (equates to originator handle) for each active and outstanding test. The required local service identifier from which the test is launched will determine the source IP (the system IP address) to use in the outer IP header of the packet. This IP address is encoded into the VXLAN header Source IP TLV. The service identifier will also encode the local VNI. The outer-ip-destination must equal the VTEP termination point on the remote node, and the dest-vni must be a valid VNI within the associated service on the remote node. The outer source IP address is automatically detected and inserted in the IP header of the packet. The outer source IP address uses the IPv4 system address by default.

If the VTEP is created using a non-system source IP address via the vxlan-src-vtep command, the outer source IP address uses the address specified by vxlan-src-vtep. The remainder of the variables are optional.

The VXLAN PDU will be encapsulated in the appropriate transport header and forwarded within the overlay to the appropriate VTEP termination. The VXLAN router alert (RA) bit will be set to prevent forwarding OAM PDU beyond the terminating VTEP. Since handling of the router alert bit was not defined in some early releases of VXLAN implementations, the VNI Informational bit (I-bit) is set to “0” for OAM packets. This indicates that the VNI is invalid, and the packet should not be forwarded. This safeguard can be overridden by including the i-flag-on option that sets the bit to “1”, valid VNI. Ensure that OAM frames meant to be contained to the VTEP are not forwarded beyond its endpoints.

The supporting VXLAN OAM ping draft includes a requirement to encode a reserved IEEE MAC address as the inner destination value. However, at the time of implementation, that IEEE MAC address had not been assigned. The inner IEEE MAC address will default to 00:00:00:00:00:00, but may be changed using the inner-l2 option. Inner IEEE MAC addresses that are included with OAM packets will not be learned in the local Layer 2 forwarding databases.

The echo responder will terminate the VXLAN OAM frame, and will take the appropriate response action, and include relevant return codes. By default, the response is sent back using the IP network as an IPv4 UDP response. The operator can chose to override this default by changing the reply-mode to overlay. The overlay return mode will force the responder to use the VTEP connection representing the source IP and source VTEP. If a return overlay is not available, the echo response will be dropped by the responder.

Support is included for:

  1. IPv4 VTEP
  2. Optional specification of the outer UDP Source, which helps downstream network elements along the path with ECMP to hash to flow to the same path
  3. Optional configuration of the inner IP information, which helps the operator test different equal paths where ECMP is deployed on the source. A test will only validate a single path where ECMP functions are deployed. The inner IP information is processed by a hash function, and there is no guarantee that changing the IP information between tests will select different paths.
  4. Optional end system validation for a single L2 IEEE MAC address per test. This function checks the remote FDB for the configured IEEE MAC Address. Only one end system IEEE MAC Address can be configured per test.
  5. Reply mode UDP (default) or Overlay
  6. Optional additional padding can be added to each packet. There is an option that indicates how the responder should handle the pad TLV. By default, the padding will not be reflected to the source. The operator can change this behavior by including reflect-pad option. The reflect-pad option is not supported when the reply mode is set to UDP.
  7. Configurable send counts, intervals, times outs, and forwarding class

The VXLAN OAM PDU includes two timestamps. These timestamps are used to report forward direction delay. Unidirectional delay metrics require accurate time of day clock synchronization. Negative unidirectional delay values will be reported as “0.000”. The round trip value includes the entire round trip time including the time that the remote peer takes to process that packet. These reported values may not be representative of network delay.

The following example commands and outputs show how the VXLAN Ping function can be used to validate connectivity. The echo output includes a new header to better describe the VxLAN ping packet headers and the various levels.

oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 interval
0.1 send-count 10 
 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Profile
In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1
 
! ! ! ! ! ! ! ! ! ! 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 1.097ms, avg = 2.195ms, max = 2.870ms, stddev = 0.735ms
round-trip-delay min = 1.468ms, avg = 1.693ms, max = 2.268ms, stddev = 0.210ms
 
 
 
oam vxlan-ping test-id 2 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 outer-
ip-source-udp 65000 outer-ip-ttl 64 inner-l2 d0:0d:1e:00:00:01 inner-ip-source
 192.168.1.2 inner-ip-destination 127.0.0.8 reply-mode overlay send-count 20 interval
 1 timeout 3 padding 1000 reflect-pad fc nc profile out 
 
TestID 2, Service 1, DestVNI 2, ReplyMode overlay, IFlag Off, PadSize 1000, ReflectPad
 Yes, SendCount 20, Interval 1, Timeout 3
Outer: SourceIP 10.20.1.3, SourcePort 65000, DestIP 10.20.1.4, TTL 64, FC nc, Profile
 out
Inner: DestMAC d0:0d:1e:00:00:01, SourceIP 192.168.1.2, DestIP 127.0.0.8
 
===================================================================================
======================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
===================================================================================
======================================
 
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=1.733ms fwd
-time=0.302ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.549ms fwd
-time=1.386ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=3.243ms fwd
-time=0.643ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.551ms fwd
-time=2.350ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.644ms fwd
-time=1.080ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.670ms fwd
-time=1.307ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.636ms fwd
-time=0.490ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.649ms fwd
-time=0.005ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.401ms fwd
-time=0.685ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.634ms fwd
-time=0.373ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=11 ttl=255 rtt-time=1.559ms fwd
-time=0.679ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=12 ttl=255 rtt-time=1.666ms fwd
-time=0.880ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=13 ttl=255 rtt-time=1.629ms fwd
-time=0.669ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=14 ttl=255 rtt-time=1.280ms fwd
-time=1.029ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=15 ttl=255 rtt-time=1.458ms fwd
-time=0.268ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=16 ttl=255 rtt-time=1.659ms fwd
-time=0.786ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=17 ttl=255 rtt-time=1.636ms fwd
-time=1.071ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=18 ttl=255 rtt-time=1.568ms fwd
-time=2.129ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=19 ttl=255 rtt-time=1.657ms fwd
-time=1.326ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=20 ttl=255 rtt-time=1.762ms fwd
-time=1.335ms. rc=4
 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
   20 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 0.005ms, avg = 0.939ms, max = 2.350ms, stddev = 0.577ms
round-trip-delay min = 1.280ms, avg = 1.679ms, max = 3.243ms, stddev = 0.375ms
 
 
 
oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 interval 0.1 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Profile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1
 
2 2 2 2 2 2 2 2 2 2 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present(1), 10 end-system not present(2)
forward-delay min = 0.467ms, avg = 0.979ms, max = 1.622ms, stddev = 0.504ms
round-trip-delay min = 1.501ms, avg = 1.597ms, max = 1.781ms, stddev = 0.088ms
 
 
 
oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Profile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1
 
===================================================================================
======================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
mac=1 End System Present, mac=2 End System Not Present
===================================================================================
======================================
 
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=2.883ms fwd
-time=4.196ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.596ms fwd
-time=1.536ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=1.698ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.687ms fwd
-time=1.766ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.679ms fwd
-time=0.799ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.678ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.709ms fwd
-time=0.031ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.757ms fwd
-time=1.441ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.613ms fwd
-time=2.570ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.631ms fwd
-time=2.130ms. rc=4 mac=2
 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present, 10 end-system not present
forward-delay min = 0.000ms, avg = 1.396ms, max = 4.196ms, stddev = 1.328ms
round-trip-delay min = 1.596ms, avg = 1.793ms, max = 2.883ms, stddev = 0.366ms

5.2.1.6. EVPN-VXLAN Routed VPLS Multicast Routing Support

IPv4 and IPv6 multicast routing is supported in an EVPN-VXLAN VPRN routed VPLS service through its IP interface when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side, sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 Multicast Routing Support for more details.

5.2.1.7. IGMP and MLD Snooping on VXLAN

The delivery of IP multicast in VXLAN services can be optimized with IGMP and MLD snooping. IGMP and MLD snooping are supported in EVPN-VXLAN VPLS services and in EVPN-VXLAN VPRN R-VPLS services. When enabled, IGMP and MLD reports will be snooped on SAPs or SDP-bindings, but also on VXLAN bindings, to create or modify entries in the MFIB for the VPLS service.

When configuring IGMP and MLD snooping in EVPN-VXLAN VPLS services, consider the following:

  1. To enable IGMP snooping in the VPLS service on VXLAN, use the config>service>vpls>igmp-snooping no shutdown command.
  2. To enable MLD snooping in the VPLS service on VXLAN, use the config>service>vpls>mld-snooping no shutdown command.
  3. The VXLAN bindings only support basic IGMP/MLD snooping functionality. Features configurable under SAPs or SDP-bindings are not available for VXLAN (VXLAN bindings are configured with the default values used for SAPs and SDP bindings). Since there is no specific IGMP or MLD snooping settings for VXLAN bindings, a specified VXLAN binding will only become a dynamic mrouter when it receives IGMP or MLD queries and will add a specified multicast group to the MFIB when it receives an IGMP or MLD report for that group.
  4. The show service id igmp-snooping, clear service id igmp-snooping, show service id mld-snooping, and clear service id mld-snooping commands are also available for VXLAN bindings.
    Note:

    MLD snooping uses MAC-based forwarding. See MAC-Based IPv6 Multicast Forwarding for more details.

    The following CLI commands show how the system displays IGMP snooping information and statistics on VXLAN bindings (the equivalent MLD output is similar).
*A:PE1# show service id 1 igmp-snooping port-db vxlan vtep  192.0.2.72 vni 1 detail 
 
===============================================================================
IGMP Snooping VXLAN 192.0.2.72/1 Port-DB for service 1
===============================================================================
-------------------------------------------------------------------------------
IGMP Group 239.0.0.1
-------------------------------------------------------------------------------
Mode             : exclude              Type             : dynamic
Up Time          : 0d 19:07:05          Expires          : 137s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
No sources.
-------------------------------------------------------------------------------
IGMP Group 239.0.0.2
-------------------------------------------------------------------------------
Mode             : include              Type             : dynamic
Up Time          : 0d 19:06:39          Expires          : 0s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
10.0.0.232      0d 19:06:39  137s     dynamic  Fwd              
-------------------------------------------------------------------------------
Number of groups: 2
===============================================================================
 
*A:PE1# show service id 1 igmp-snooping 
statistics vxlan vtep  192.0.2.72 vni 1        
 
===============================================================================
IGMP Snooping Statistics for VXLAN 192.0.2.72/1 (service 1)
===============================================================================
Message Type            Received      Transmitted   Forwarded
-------------------------------------------------------------------------------
General Queries         0             0             556
Group Queries           0             0             0
Group-Source Queries    0             0             0
V1 Reports              0             0             0
V2 Reports              0             0             0
V3 Reports              553           0             0
V2 Leaves               0             0             0
Unknown Type            0             N/A           0
-------------------------------------------------------------------------------
Drop Statistics
-------------------------------------------------------------------------------
Bad Length               : 0
Bad IP Checksum          : 0
Bad IGMP Checksum        : 0
Bad Encoding             : 0
No Router Alert          : 0
Zero Source IP           : 0
Wrong Version            : 0
Lcl-Scope Packets        : 0
Rsvd-Scope Packets       : 0
 
Send Query Cfg Drops     : 0
Import Policy Drops      : 0
Exceeded Max Num Groups  : 0
Exceeded Max Num Sources : 0
Exceeded Max Num Grp Srcs: 0
MCAC Policy Drops        : 0
===============================================================================
*A:PE1# show service id 1 mfib 
===============================================================================
Multicast FIB, Service 1
===============================================================================
Source Address  Group Address         SAP or SDP Id               Svc Id   Fwd/Blk
-------------------------------------------------------------------------------
*               *                     sap:1/1/1:1              Local    Fwd
*               239.0.0.1             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
10.0.0.232      239.0.0.2             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 3
===============================================================================
 

5.2.1.8. PIM Snooping on VXLAN

PIM snooping for IPv4 and IPv6 are supported in an EVPN-VXLAN service. The snooping operation is similar to that within a VPLS service (see PIM Snooping for VPLS) and supports both PIM snooping and PIM proxy modes.

PIM snooping for IPv4 is enabled using the config>service>vpls>pim-snooping command.

PIM snooping for IPv6 is enabled using the config>service>vpls>pim-snooping no ipv6-multicast-disable command.

When using PIM snooping for IPv6, the default forwarding is MAC-based with optional support for SG-based (see IPv6 Multicast Forwarding). SG-based forwarding requires FP3- or higher-based hardware.

It is not possible to configure max-num-groups for VXLAN bindings.

5.2.1.9. Static VXLAN Termination in Epipe Services

By default, the system IP address is used to terminate and generate VXLAN traffic. The following configuration example shows an Epipe service that supports static VXLAN termination:

config service epipe 1 customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan vni 100 create
    egr-vtep 192.0.2.1
      oper-group op-grp-1
    exit
  no shutdown

Where:

  1. vxlan vni vni create specifies the ingress VNI the router will use to identify packets for the service. The following considerations apply.
    1. In services that use EVPN, the configured VNI is only used as the ingress VNI to identify packets that belong to the service. Egress VNIs are learned from the BGP EVPN. In the case of Static VXLAN, the configured VNI is also used as egress VNI (because there is no BGP EVPN control plane).
    2. The configured VNI is unique in the system, and as a result, it can only be configured in one service (VPLS or Epipe).
  2. egr-vtep ip-address specifies the remote VTEP the router will use when encapsulating frames into VXLAN packets. The following consideration apply.
    1. When the PE receives VXLAN packets, the source VTEP is not checked against the configured egress VTEP.
    2. The IP-address must be present in the global routing table so that the VXLAN destination is operationally up.
  3. oper-group may be added under egr-vtep. The expected behavior for the operational group and service status is as follows.
    1. If the egr-vtep entry is not present in the routing table, the VXLAN destination (in the show service id vxlan command) and the provisioned operational group under egr-vtep will go into the operationally down state.
    2. The service goes down if the Epipe SAP goes down, but it is not affected if the VXLAN destination goes down.
    3. If the service is admin shutdown, then in addition to the SAP, the VXLAN destination and the oper-group will also go into the operationally down state.
    Note:

    The operational group configured under egr-vtep cannot be monitored on the SAP of the Epipe where it is configured.

The following features are not supported by Epipe services with VXLAN destinations.

  1. per-service hashing
  2. SDP-binds
  3. PBB context
  4. BGP-VPWS
  5. BGP-EVPN
  6. Spoke-SDP-FEC
  7. PW-port

5.2.1.10. Non-System IPv4 and IPv6 VXLAN Termination in VPLS, R-VPLS, and Epipe Services

By default, only VXLAN packets with the same IP destination address as the system IPv4 address of the router can be terminated and processed for a subsequent MAC lookup. A router can simultaneously terminate VXLAN tunnels destined for its system IP address and three additional non-system IPv4 or IPv6 addresses, which can be on the base router or VPRN instances. This section describes the configuration requirements for services to terminate VXLAN packets destined for a non-system loopback IPv4 or IPv6 address on the base router or VPRN.

Perform the following steps to configure a service with non-system IPv4 or IPv6 VXLAN termination:

  1. Create the FPE (see FPE Creation).
  2. Associate the FPE with VXLAN termination (see FPE Association with VXLAN Termination).
  3. Configure the router loopback interface (see VXLAN Router Loopback Interface).
  4. Configure VXLAN termination (non-system) VTEP addresses (see VXLAN Termination VTEP Addresses).
  5. Add the service configuration (see VXLAN Services).

FPE Creation

A Forwarding Path Extension (FPE) is required to terminate non-system IPv4 or IPv6 VXLAN tunnels.

In a non-system IPv4 VXLAN termination, the FPE function is used for additional processing required at ingress (VXLAN tunnel termination) only, and not at egress (VXLAN tunnel origination).

If the IPv6 VXLAN terminates on a VPLS or Epipe service, the FPE function is used at ingress only, and not at egress.

For R-VPLS services terminating IPv6 VXLAN tunnels and also for VPRN VTEPs, the FPE is used for the egress as well as the VXLAN termination function. In the case of R-VPLS, an internal static SDP is created to allow the required extra processing.

See section “Forwarding Path Extension” of the 7450 ESS, 7750 SR, 7950 XRS, and VSR Interface Configuration Guide for information about FPE configuration and functions.

FPE Association with VXLAN Termination

The FPE must be associated with the VXLAN termination application. The following sample configuration shows two FPEs and their corresponding association. FPE 1 uses the base router and FPE 2 is configured for VXLAN termination on VPRN 10.

configure
   fwd-path-ext
       fpe 1 create
             path pxc pxc-1
             vxlan-termination
       fpe 2 create
             path pxc pxc-2
             vxlan-termination router 10

VXLAN Router Loopback Interface

Create the interface that will terminate and originate the VXLAN packets. The interface is created as a router interface, which is added to the Interior Gateway Protocol (IGP) and used by the BGP as the EVPN NLRI next hop.

Because the system cannot terminate the VXLAN on a local interface address, a subnet must be assigned to the loopback interface and not a host IP address that is /32 or /128. In the following example, all the addresses in subnet 11.11.11.0/24 (except 11.11.11.1, which is the interface IP) and subnet 10.1.1.0/24 (except 10.1.1.1) can be used for tunnel termination. The subnet is advertised using the IGP and is configured on either the base router or a VPRN. In the example, two subnets are assigned, in the base router and VPRN 10 respectively.

configure 
  router 
       interface “lo1” 
            loopback
            address 10.11.11.1/24 
       isis
            interface “lo1”
                 passive
                 no shutdown
configure
    service
        vprn 10 customer 1 create
            interface “lo1”
                loopback
                address 10.1.1.1/24
            isis 
                interface “lo1” 
                    passive 
                    no shutdown 

A local interface address cannot be configured as a VXLAN tunnel-termination IP address in the CLI, as shown in the following example.

*A:PE-3# configure service system vxlan tunnel-termination 192.0.2.3 fpe 1 create
MINOR: SVCMGR #8353 VXLAN Tunnel termination IP address cannot be configured -
 IP address in use by another application or matches a local interface IP address

The subnet can be up to 31 bits. For example, to use 10.11.11.1 as the VXLAN termination address, the subnet should be configured and advertised as shown in the following sample configuration.

 interface "lo1"
            address 10.11.11.0/31
            loopback
            no shutdown
        exit
        isis 0
            interface "lo1"
                passive
                no shutdown
            exit
            no shutdown
        exit

It is not a requirement for the remote PEs and NVEs to have the specific /32 or /128 IP address in their RTM to resolve the BGP EVPN NLRI next hop or forward the VXLAN packets. An RTM with a subnet that contains the remote VTEP can also perform these tasks.

Note:

The system does not check for a pre-existing local base router loopback interface with a subnet corresponding to the VXLAN tunnel termination address. If a tunnel termination address is configured and the FPE is operationally up, the system starts terminating VXLAN traffic and responding ICMP messages for that address. The following conditions are ignored in this scenario:

  1. the presence of a loopback interface in the base router
  2. the presence of an interface with the address contained in the configured subnet, and no loopback

The following sample output includes an IPv6 address in the base router. It could also be configured in a VPRN instance.

configure 
  router 
      interface "lo1" 
           loopback
           address 10.11.11.1/24 
           ipv6
               address 2001:db8::/127
           exit
      isis
           interface "lo1"
               passive
               no shutdown

VXLAN Termination VTEP Addresses

The service>system>vxlan>tunnel-termination context allows the user to configure non-system IP addresses that can terminate the VXLAN and their corresponding FPEs.

As shown in the following example, an IP address may be associated with a new or existing FPE already terminating the VXLAN. The list of addresses that can terminate the VXLAN can include IPv4 and IPv6 addresses.

config service system vxlan# 
       tunnel-termination 10.11.11.1 fpe 1 create
       tunnel-termination 2001:db8:1000::1 fpe 1 create 
        
config service vprn 10 vxlan# 
       tunnel-termination 10.1.1.2 fpe 2 create

The tunnel-termination command creates internal loopback interfaces that can respond to ICMP requests. In the following sample output, an internal loopback is created when the tunnel termination address is added (for 10.11.11.1 and 2001:db8:1000::1). The internal FPE router interfaces created by the VXLAN termination function are also shown in the output. Similar loopback and interfaces are created for tunnel termination addresses in a VPRN (not shown).

*A:PE1# show router interface 
===============================================================================
Interface Table (Router: Base)
===============================================================================
Interface-Name                   Adm       Opr(v4/v6)  Mode    Port/SapId
   IP-Address                                                  PfxState
-------------------------------------------------------------------------------
_tmnx_fpe_1.a                    Up        Up/Up       Network pxc-2.a:1
   fe80::100/64                                                PREFERRED
_tmnx_fpe_1.b                    Up        Up/Up       Network pxc-2.b:1
   fe80::101/64                                                PREFERRED
_tmnx_vli_vxlan_1_131075         Up        Up/Up       Network loopback
   10.11.11.1/32                                               n/a
   2001:db8:1000::1                                            PREFERRED
   fe80::6cfb:ffff:fe00:0/64                                   PREFERRED
lo1                              Up        Up/Down     Network loopback
   10.11.11.0/31                                               n/a
system                           Up        Up/Down     Network system
   1.1.1.1/32                                                  n/a
<snip>

VXLAN Services

By default, the VXLAN services use the system IP address as the source VTEP of the VXLAN encapsulated frames. The vxlan-src-vtep command in the service>vpls or service>epipe context enables the system to use a non-system IPv4 or IPv6 address as the source VTEP for the VXLAN tunnels in that service.

A different vxlan-src-vtep can be used for different services, as shown in the following example where two different services use different non-system IP addresses as source VTEPs.

configure service vpls 1 
   vxlan-src-vtep 10.11.11.1
 
configure service vpls 2
   vxlan-src-vtep 2001:db8:1000::1

In addition, if a vxlan-src-vtep is configured and the service uses EVPN, the IP address is also used to set the BGP NLRI next hop in EVPN route advertisements for the service.

Note:

The BGP EVPN next hop can be overridden by the use of export policies based on the following rules.

  1. A BGP peer policy can override a next hop pushed by the vxlan-src-vtep configuration.
  2. If the VPLS service is IPv6 (that is, the vxlan-src-vtep is IPv6) and a BGP peer export policy is configured with next-hop-self, the BGP next-hop is overridden with an IPv6 address auto-derived from the IP address of the system. The auto-derivation is based on RFC 4291. For example, ::ffff:10.20.1.3 is auto-derived from system IP 10.20.1.3.
  3. The policy checks the address type of the next hop provided by the vxlan-src-vtep command. If the command provides an IPv6 next hop, the policy is unable use an IPv4 address to override the IPv6 address provided by the vxlan-src-vtep command.

After the preceding steps are performed to configure a VXLAN termination, the VPLS, R-VPLS, or Epipe service can be used normally, except that the service will terminate VXLAN tunnels with a non-system IPv4 or IPv6 destination address (in the base router or a VPRN instance) instead of the system IP address only.

The FPE vxlan-termination function creates internal router interfaces and loopbacks that are displayed by the show commands. When configuring IPv6 VXLAN termination on an R-VPLS service, as well as the internal router interfaces and loopbacks, the system will create internal SDP bindings for the required egress processing. The following output shows an example of an internal FPE-type SDP binding created for IPv6 R-VPLS egress processing.

*A:PE1# show service sdp-using  
===============================================================================
SDP Using
===============================================================================
SvcId      SdpId              Type   Far End              Opr   I.Label E.Label
                                                          State         
-------------------------------------------------------------------------------
2002       17407:2002         Fpe    fpe_1.b              Up    262138  262138
-------------------------------------------------------------------------------
Number of SDPs : 1
-------------------------------------------------------------------------------
===============================================================================

When BGP EVPN is used, the BGP peer over which the EVPN-VXLAN updates are received can be an IPv4 or IPv6 peer, regardless of whether the next-hop is an IPv4 or IPv6 address.

The same VXLAN tunnel termination address cannot be configured on different router instances; that is, on two different VPRN instances or on a VPRN and the base router.

5.2.2. EVPN for Overlay Tunnels

This section describes the specifics of EVPN for non-MPLS Overlay tunnels.

5.2.2.1. BGP-EVPN Control Plane for VXLAN Overlay Tunnels

The IETF Draft draft-ietf-bess-evpn-overlay describes EVPN as the control plane for overlay-based networks. The 7750 SR, 7450 ESS, and 7950 XRS support a subset of the routes and features described in RFC 7432 that are required for the DC GW function. In particular, EVPN-specific multi-homing capabilities are not supported for VXLAN. However, multi-homing can be supported by using regular BGP multi-homing based on the L2VPN BGP address family.

Figure 145 shows the EVPN MP-BGP NLRI, required attributes and extended communities, and two route types supported for the DC GW Layer 2 applications:

  1. route type 3 – Inclusive Multicast Ethernet Tag route
  2. route type 2 – MAC/IP advertisement route
    Figure 145:  EVPN-VXLAN Required Routes and Communities 

EVPN Route Type 3 – Inclusive Multicast Ethernet Tag Route

Route type 3 is used to set up the flooding tree (BUM flooding) for a specified VPLS service in the data center. The received inclusive multicast routes add entries to the VPLS flood list in the 7750 SR, 7450 ESS, and 7950 XRS. The tunnel types supported in an EVPN route type 3 when BGP-EVPN MPLS is enabled are ingress replication, P2MP MLDP, and composite tunnels.

Ingress Replication (IR) and Assisted Replication (AR) are supported for VXLAN tunnels. See Layer 2 Multicast Optimization for VXLAN (Assisted-Replication) for more information about the AR.

If ingress-repl-inc-mcast-advertisement is enabled, a route type 3 is generated by the router per VPLS service as soon as the service is in an operationally up state. The following fields and values are used:

  1. Route Distinguisher: taken from the RD of the VPLS service within the BGP context
    Note: The RD can be configured or derived from the bgp-evpn evi value.
  2. Ethernet Tag ID: 0
  3. IP address length: always 32
  4. Originating router’s IP address: carries the system address (IPv4 only)
    Note: By default, the IP address of the Originating router is derived from the system IP address. However, this can be overridden by the config>service>vpls>bgp-evpn>incl-mcast-orig-ip <ip-address> command for the Ingress Replication (and mLDP if MPLS is used) tunnel type.
  5. PMSI Tunnel Attribute (PTA):
    1. Tunnel type = Ingress replication (6) or Assisted Replication (10)
      1. Flags—Leaf not required.
      2. MPLS label—Carries the VNI configured in the VPLS service. Only one VNI can be configured per VPLS service.
      3. Tunnel endpoint—Equal to the system IP address.
      As shown in Figure 146, additional flags are used in the PTA when the service is configured for AR.
      Figure 146:  PMSI Attribute Flags Field for AR 
      The Flags field is defined as a Type field (for AR) with two new flags that are defined as follows:
      1. T is the AR Type field (2 bits):
        —   00 (decimal 0) = RNVE (non-AR support)
        —   01 (decimal 1) = AR REPLICATOR
        —   10 (decimal 2) = AR LEAF
      2. The U and BM flags defined in IETF Draft draft-ietf-bess-evpn-optimized-ir are not used in the SR OS.
    Table 78 describes the inclusive multicast route information sent per VPLS service when the router is configured as assisted-replication replicator (AR-R) or assisted-replication leaf (AR-L). A Regular Network Virtualization Edge device (RNVE) is defined as an EVPN-VXLAN router that does not support (or is not configured for) Assisted-Replication.
    Note: For AR-R, two inclusive multicast routes may be advertised if ingress-repl-inc-mcast-advertisement is enabled: a route with tunnel-type IR, tunnel-id = IR IP (generally system-ip) and a route with tunnel-type AR, tunnel-id = AR IP (the address configured in the assisted-replication-ip command).
    Table 78:  AR-R AND AR-L Routes and Usage 

    AR Role

    Function

    Inclusive Mcast Routes Advertisement

    AR-R

    Assists AR-LEAFs

    1. IR included in the Mcast route (uses IR IP) if ingress-repl-inc-mcast-advertisement is enabled
    2. AR included in the Mcast route (uses AR IP, tunnel type=AR, T=1)

    AR-LEAF

    Sends BM only to AR-Rs

    IR inclusive multicast route (IR IP, T=2) if ingress-repl-inc-mcast-advertisement is enabled

    RNVE

    Non-AR support

    IR inclusive multicast route (IR IP) if ingress-repl-inc-mcast-advertisement is enabled

EVPN Route Type 2 – MAC/IP Advertisement Route

The 7750 SR, 7450 ESS, and 7950 XRS will generate this route type for advertising MAC addresses. The router will generate MAC advertisement routes for the following:

  1. Learned MACs on SAPs or sdp-bindings – if mac-advertisement is enabled
  2. Conditional static MACs – if mac-advertisement is enabled
  3. unknown-mac-routes – if unknown-mac-route is enabled, there is no bgp-mh site in the service or there is a (single) DF site

The route type 2 generated by a router uses the following fields and values:

  1. Route Distinguisher: taken from the RD of the VPLS service within the BGP context
    Note:

    The RD can be configured or derived from the bgp-evpn evi value.

  2. Ethernet Segment Identifier (ESI): Value = 0:0:0:0:0:0:0:0:0:0.
  3. Ethernet Tag ID: 0.
  4. MAC address length: always 48
  5. MAC Address:
    1. is 00:00:00:00:00:00 for the Unknown MAC route address.
    2. is different from 00:…:00 for the rest of the advertised MACs.
  6. IP address and IP address length:
    1. is the IP address associated with the MAC being advertised with a length of 32 (or 128 for IPv6).
    2. if the MAC address is the Unknown MAC route, the IP address length is zero and the IP omitted.
    3. in general, any MAC route without IP has IPL=0 (IP length) and the IP is omitted.
    4. when received, any IPL value not equal to zero, 32, or 128 will make discard the route.
  7. MPLS Label 1: carries the VNI configured in the VPLS service. Only one VNI can be configured per VPLS.
  8. MPLS Label 2: 0
  9. MAC Mobility extended community: used for signaling the sequence number in case of mac moves and the sticky bit in case of advertising conditional static MACs. If a MAC route is received with a MAC mobility ext-community, the sequence number and the sticky bit are considered for the route selection.

When EVPN is used in an IRB backhaul R-VPLS that connects all the VPRN instances for a specified tenant and there is a need to advertise IP prefixes in EVPN, a separate route type is used: route-type 5 IP prefix route.

EVPN Route Type 5 – IP Prefix Route

Figure 147 shows the IP prefix route or route-type 5.

Figure 147:  EVPN Route-Type 5  

The router will generate this route type for advertising IP prefixes in EVPN. The router will generate IP Prefix advertisement routes for:

  1. IP prefixes existing in a VPRN linked to the IRB backhaul R-VPLS service.

The route-type 5 generated by a router uses the following fields and values:

  1. Route Distinguisher: taken from the RD configured in the IRB backhaul R-VPLS service within the BGP context
  2. Ethernet Segment Identifier (ESI): Value = 0:0:0:0:0:0:0:0:0:0
  3. Ethernet Tag ID: 0
  4. IP address length: Any value in the 0 to 128 range
  5. IP address: any valid IPv4 or IPv6 address
  6. GW IP address: can carry two different values:
    1. if different from zero, the route-type 5 carries the primary IP interface address of the VPRN behind which the IP prefix is known. This is the case for the regular IRB backhaul R-VPLS model.
    2. if 0.0.0.0, the route-type 5 is sent with a MAC next-hop extended community that will carry the VPRN interface MAC address. This is the case for the EVPN tunnel R-VPLS model.
  7. MPLS Label: carries the VNI configured in the VPLS service. Only one VNI can be configured per VPLS service.

All the routes in EVPN-VXLAN will be sent with the RFC 5512 tunnel encapsulation extended community, with the tunnel type value set to VXLAN.

5.2.2.2. EVPN for VXLAN in VPLS Services

The EVPN-VXLAN service is designed around the current VPLS objects and the additional VXLAN construct.

Figure 138 shows a DC with a Layer 2 service that carries the traffic for a tenant who wants to extend a subnet beyond the DC. The DC PE function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where a VPLS instance exists for that particular tenant. Within the DC, the tenant will have VPLS instances in all the Network Virtualization Edge (NVE) devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The VPLS instances in the redundant DC GW and the DC NVEs will be connected by VXLAN bindings. BGP-EVPN will provide the required control plane for such VXLAN connectivity.

The DC GW routers will be configured with a VPLS per tenant that will provide the VXLAN connectivity to the Nuage VPLS instances. On the router, each tenant VPLS instance will be configured with:

  1. The WAN-related parameters (saps, Spoke-SDPs, mesh-sdps, bgp-ad, and so on).
  2. The BGP-EVPN and VXLAN (VNI) parameters. The following CLI output shows an example for an EVPN-VXLAN VPLS service.
 
*A:DGW1>config>service>vpls# info 
----------------------------------------------
            description "vxlan-service"
            vxlan instance 1 vni 1 create
            exit
            bgp
                route-distinguisher 65001:1
                route-target export target:65000:1 import target:65000:1
            exit
            bgp-evpn
                unknown-mac-route
                mac-advertisement 
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            sap 1/1/1:1 create
            exit
            no shutdown
----------------------------------------------

The bgp-evpn context specifies the encapsulation type (only vxlan is supported) to be used by EVPN and other parameters like the unknown-mac-route and mac-advertisement commands. These commands are typically configured in three different ways:

  1. no unknown-mac-route and mac-advertisement (default option) — The router will advertise new learned MACs (on the SAPs or sdp-bindings) or new conditional static MACs.
  2. unknown-mac-route and no mac-advertisement — The router will only advertise an unknown-mac-route as long as the service is operationally up (if no BGP-MH site is configured in the service) or the router is the DF (if BGP-MH is configured in the service).
  3. unknown-mac-route and mac-advertisement — The router will advertise new learned MACs, conditional static MACs, and the unknown-mac-route. The unknown-mac-route will only be advertised under the preceding described conditions.

Other parameters related to EVPN or VXLAN are:

  1. Mac duplication parameters
  2. vxlan vni: Defines the VNI that the router will use in the EVPN routes generated for the VPLS service.

After the VPLS is configured and operationally up, the router will send/receive inclusive multicast Ethernet Tag routes, and a full-mesh of VXLAN connections will be automatically created. These VXLAN “auto-bindings” can be characterized as follows:

  1. The VXLAN auto-bindings model is based on an IP-VPN-like design, where no SDPs or SDP-binding objects are created by or visible to the user. The VXLAN auto-binds are composed of remote VTEPs and egress VNIs, and can be displayed with the following command:
A:DCGW# show service id 1 vxlan 
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 1
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address            Egress VNI  Num. MACs     Mcast   Oper         L2  
                                                          State       PBR
-------------------------------------------------------------------------------
192.0.2.71                      1        1        BUM      Up          No
192.0.2.72                      1        0        BUM      Up          No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2
-------------------------------------------------------------------------------
===============================================================================
  1. The VXLAN bindings observe the VPLS split-horizon rule. This is performed automatically without the need for any split-horizon configuration.
  2. BGP Next-Hop Tracking for EVPN is fully supported. If the BGP next-hop for a specified received BGP EVPN route disappears from the routing table, the BGP route will not be marked as “used” and the respective entry in show service id vxlan will be removed.

After the flooding domain is setup, the routers and DC NVEs start advertising MAC addresses, and the routers can learn MACs and install them in the FDB. Some considerations are the following:

  1. All the MAC addresses associated with remote VTEP/VNIs are always learned in the control plane by EVPN. Data plane learning on VXLAN auto-bindings is not supported.
  2. When unknown-mac-route is configured, it will be generated when no (BGP-MH) site is configured, or a site is configured AND the site is DF in the PE.
    Note:

    The unknown-mac-route will not be installed in the FDB (therefore, will not show up in the show service id x fdb detail command).

  3. While the router can be configured with only one VNI (and signals a single VNI per VPLS), it can accept any VNI in the received EVPN routes as long as the route-target is properly imported. The VTEPs and VNIs will show up in the FDB associated with MAC addresses:
A:PE65# show service id 1000 fdb detail 
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1000      00:00:00:00:00:01 vxlan-1:                 Evpn     10/05/13 23:25:57
                            192.0.2.63:1063                   
1000      00:00:00:00:00:65 sap:1/1/1:1000           L/30     10/05/13 23:25:57
1000      00:ca:ca:ca:ca:00 vxlan-1:                 EvpnS    10/04/13 17:35:43
                            192.0.2.63:1063                   
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

5.2.2.2.1. Resiliency and BGP Multi-Homing

The DC overlay infrastructure relies on IP tunneling, that is, VXLAN; therefore, the underlay IP layer resolves failure in the DC core. The IGP should be optimized to get the fastest convergence.

From a service perspective, resilient connectivity to the WAN may be provided by BGP-Multi-homing.

5.2.2.2.2. Use of bgp-evpn, bgp-ad, and Sites in the Same VPLS Service

All bgp-evpn (control plane for a VXLAN DC), bgp-ad (control plane for MPLS-based Spoke-SDPs connected to the WAN), and one site for BGP multi-homing (control plane for the multi-homed connection to the WAN) can be configured in one service in a specified system. If that is the case, the following considerations apply:

  1. The configured BGP route-distinguisher and route-target are used by BGP for the two families, that is, evpn and l2vpn. If different import/export route targets are to be used per family, vsi-import/export policies must be used.
  2. The pw-template-binding command under BGP, does not have any effect on evpn or bgp-mh. It is only used for the instantiation of the bgp-ad Spoke-SDPs.
  3. If the same import/export route-targets are used in the two redundant DC GWs, VXLAN binding as well as a fec129 Spoke-SDP binding will be established between the two DGWs, creating a loop. To avoid creating a loop, the router will allow the establishment of an EVPN VXLAN binding and an sdp-binding to the same far-end, but the sdp-binding will be kept operationally down. Only the VXLAN binding will be operationally up.

5.2.2.2.3. Use of the unknown-mac-route

This section describes the behavior of the EVPN-VXLAN service in the router when the unknown-mac-route and BGP-MH are configured at the same time.

The use of EVPN, as the control plane of NVO networks in the DC, provides a significant number of benefits as described in IETF Draft draft-ietf-bess-evpn-overlay.

However, there is a potential issue that must be addressed when a VPLS DCI is used for an NVO3-based DC: all the MAC addresses learned from the WAN side of the VPLS must be advertised by BGP EVPN updates. Even if optimized BGP techniques like RT-constraint are used, the number of MAC addresses to advertise or withdraw (in case of failure) from the DC GWs can be difficult to control and overwhelming for the DC network, especially when the NVEs reside in the hypervisors.

The 7750 SR, 7450 ESS, and 7950 XRS solution to this issue is based on the use of an unknown-mac-route address that is advertised by the DC PEs. By using this unknown-mac-route advertisement, the DC tenant may decide to optionally turn off the advertisement of WAN MAC addresses in the DC GW, therefore, reducing the control plane overhead and the size of the FDB tables in the NVEs.

The use of the unknown-mac-route is optional and helps to reduce the amount of unknown-unicast traffic within the data center. All the receiving NVEs supporting this concept will send any unknown-unicast packet to the owner of the unknown-mac-route, as opposed to flooding the unknown-unicast traffic to all other NVEs that are part of the same VPLS.

Note:

Although the router can be configured to generate and advertise the unknown-mac-route, the router will never honor the unknown-mac-route and will flood to the TLS-flood list when an unknown-unicast packet arrives at an ingress SAP or SDP-binding.

The use of the unknown-mac-route assumes the following:

  1. A fully virtualized DC where all the MACs are control-plane learned, and learned previous to any communication (no legacy TORs or VLAN connected servers).
  2. The only exception is MACs learned over the SAPs/SDP-bindings that are part of the BGP-MH WAN site-id. Only one site-id is supported in this case.
  3. No other SAPs/SDP-bindings out of the WAN site-id are supported, unless only static MACs are used on those SAPs/SDP-bindings.

Therefore, when unknown-mac-route is configured, it will only be generated when one of the following applies:

  1. No site is configured and the service is operationally up.
  2. A BGP-MH site is configured AND the DC GW is Designated Forwarder (DF) for the site. In case of BGP-MH failover, the unknown-mac-route will be withdrawn by the former DF and advertised by the new DF.

5.2.2.3. EVPN for VXLAN in R-VPLS Services

Figure 139 shows a DC with a Layer 2 service that carries the traffic for a tenant who extends a subnet within the DC, while the DC GW is the default gateway for all the hosts in the subnet. The DC GW function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where an R-VPLS instance exists for that particular tenant. Within the DC, the tenant will have VPLS instances in all the NVE devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The WAN connectivity will be based on existing IP-VPN features.

In this model, the DC GW routers will be configured with a R-VPLS (bound to the VPRN that provides the WAN connectivity) per tenant that will provide the VXLAN connectivity to the Nuage VPLS instances. This model provides inter-subnet forwarding for L2-only TORs and other L2 DC NVEs.

On the router:

  1. The VPRN will be configured with an interface bound to the backhaul R-VPLS. That interface will be a regular IP interface (IP address configured or possibly a Link Local Address if IPv6 is added).
  2. The VPRN can support other numbered interfaces to the WAN or even to the DC.
  3. The R-VPLS will be configured with the BGP, BGP-EVPN and VXLAN (VNI) parameters.

On the Nuage VSGs and NVEs:

  1. Regular VPLS service model with BGP EVPN and VXLAN parameters.

Other considerations:

  1. Route-type 2 routes with MACs and IPs will be advertised. Some considerations about MAC+IP and ARP/ND entries are:
    1. The 7750 SR will advertise its IRB MAC+IP in a route type 2 route and possibly the VRRP vMAC+vIP if it runs VRRP and the 7750 SR is the master. In both cases, the MACs will be advertised as static MACs, therefore, protected by the receiving PEs.
    2. If the 7750 SR VPRN interface is configured with one or more additional secondary IP addresses, they will all be advertised in routes type 2, as static MACs.
    3. The 7750 SR will process route-type 2 routes as usual, populating the FDB with the received MACs and the VPRN ARP/ND table with the MAC and IPs, respectively.
      Note:

      ND entries received from the EVPN are installed as "Router" entries. The ARP/ND entries coming from the EVPN will be tagged as “EVPN”:

    1. When a VPLS containing proxy-ARP/proxy-ND entries is bound to a VPRN (allow-ip-int-bind) all the proxy-ARP/proxy-ND entries are moved to the VPRN ARP/ND table. ARP/ND entries will be also moved to proxy-ARP/proxy-ND entries if the VPLS is unbound.
    2. EVPN will not program EVPN-received ARP/ND entries if the receiving VPRN has no IP addresses for the same subnet. The entries will be added when the IP address for the same subnet is added.
    3. Static ARP/ND entries have precedence over dynamic and EVPN ARP/ND entries.
  2. VPRN interface binding to VPLS service will bring down the VPRN interface operational status, if the VPRN interface mac or the VRRP mac matches a static-mac or OAM mac configured in the associated VPLS service. If that is the case, a trap will be generated.
  3. Redundancy will be handled by VRRP. The 7750 SR master will advertise vMAC and vIP, as discussed, including the mac mobility extended community and the sticky bit.

EVPN-enabled R-VPLS services are also supported on IES interfaces.

5.2.2.3.1. EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes

Figure 140 shows a Layer 3 DC model, where a VPRN is defined in the DC GWs, connecting the tenant to the WAN. That VPRN instance will be connected to the VPRNs in the NVEs by means of an IRB backhaul R-VPLS. Since the IRB backhaul R-VPLS provides connectivity only to all the IRB interfaces and the DC GW VPRN is not directly connected to all the tenant subnets, the WAN ip-prefixes in the VPRN routing table must be advertised in EVPN. In the same way, the NVEs will send IP prefixes in EVPN that will be received by the DC GW and imported in the VPRN routing table.

Note:

To generate or process IP prefixes sent or received in EVPN route type 5, support for IP route advertisement must be enabled in BGP-EVPN using the bgp-evpn>ip-route-advertisement command. This command is disabled by default and must be explicitly enabled. The command is tied to the allow-ip-int-bind command required for R-VPLS, and it is not supported on an R-VPLS linked to IES services.

Local router interface host addresses are not advertised in EVPN by default. To advertise them, the ip-route-advertisement incl-host command must be enabled. For example:

===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                         Active     Metric
-------------------------------------------------------------------------------
10.1.1.0/24                                   Local   Local     00h00m11s  0
       if                                              Y            0
10.1.1.100/32                                 Local   Host      00h00m11s  0
       if                                              Y            0
==============================================================================

For the case displayed by the output above, the behavior is the following:

  1. ip-route-advertisement only local subnet (default) - 10.1.1.0/24 is advertised
  2. ip-route-advertisement incl-host local subnet, host - 10.1.1.0/24 and 10.1.1.100/32 are advertised

Below is an example of VPRN (500) with two IRB interfaces connected to backhaul R-VPLS services 501 and 502 where EVPN-VXLAN runs:

vprn 500 customer 1 create            
            ecmp 4
            route-distinguisher 65072:500
            auto-bind-tunnel
                resolution-filter
                resolution-filter gre ldp rsvp
            vrf-target target:65000:500
            interface "evi-502" create
                address 10.20.20.72/24
                vpls "evpn-vxlan-502"
                exit
            exit
            interface "evi-501" create
                address 10.10.10.72/24
                vpls "evpn-vxlan-501"
                exit
            exit
            no shutdown
vpls 501 name “evpn-vxlan-501” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 501 create
            exit
            bgp
                route-distinguisher 65072:501
                route-target export target:65000:501 import target:65000:501
            exit
            bgp-evpn
                ip-route-advertisement incl-host
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit                  
            exit
            no shutdown
        exit
vpls 502 name “evpn-xvlan-502” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 502 create
            exit
            bgp
                route-distinguisher 65072:502
                route-target export target:65000:502 import target:65000:502
            exit
            bgp-evpn
                ip-route-advertisement incl-host
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit
 

When the above commands are enabled, the router will:

  1. Receive route-type 5 routes and import the IP prefixes and associated IP next-hops into the VPRN routing table.
    1. If the route-type 5 is successfully imported by the router, the prefix included in the route-type 5 (for example, 10.0.0.0/24), will be added to the VPRN routing table with a next-hop equal to the GW IP included in the route (for example, 192.0.0.1. that refers to the IRB IP address of the remote VPRN behind which the IP prefix sits).
    2. When the router receives a packet from the WAN to the 10.0.0.0/24 subnet, the IP lookup on the VPRN routing table will yield 192.0.0.1 as the next-hop. That next-hop will be resolved to a MAC in the ARP table and the MAC resolved to a VXLAN tunnel in the FDB table
      Note:

      IRB MAC and IP addresses are advertised in the IRB backhaul R-VPLS in routes type 2.

  2. Generate route-type 5 routes for the IP prefixes in the associated VPRN routing table.
    1. For example, if VPRN-1 is attached to EVPN R-VPLS 1 and EVPN R-VPLS 2, and R-VPLS 2 has bgp-evpn ip-route-advertisement configured, the 7750 SR will advertise the R-VPLS 1 interface subnet in one route-type 5.
  3. Routing policies can filter the imported and exported IP prefix routes accordingly.

The VPRN routing table can receive routes from all the supported protocols (BGP-VPN, OSPF, IS-IS, RIP, static routing) as well as from IP prefixes from EVPN, as shown below:

*A:PE72# show router 500 route-table                      
===============================================================================
Route Table (Service: 500)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.20.20.0/24                                 Local   Local     01d11h10m  0
       evi-502                                                      0
10.20.20.71/32                                Remote  BGP EVPN  00h02m26s  169
       10.10.10.71                                                  0
10.10.10.0/24                                Remote  Static    00h00m05s  5
       10.10.10.71                                                  1
10.16.0.1/32                                 Remote  BGP EVPN  00h02m26s  169
       10.10.10.71                                                  0
-------------------------------------------------------------------------------
No. of Routes: 4

The following considerations apply:

  1. The route Preference for EVPN IP prefixes is 169.
    1. BGP IP-VPN routes have a preference of 170 by default, therefore, if the same route is received from the WAN over BGP-VPRN and from BGP-EVPN, then the EVPN route will be preferred.
  2. When the same route-type 5 prefix is received from different GW IPs, ECMP is supported if configured in the VPRN.
  3. All routes in the VPRN routing table (as long as they do not point back to the EVPN R-VPLS interface) are advertised via EVPN.

Although the description above is focused on IPv4 interfaces and prefixes, it applies to IPv6 interfaces too. The following considerations are specific to IPv6 VPRN R-VPLS interfaces:

  1. IPv4 and IPv6 interfaces can be defined on R-VPLS IP interfaces at the same time (dual-stack).
  2. The user may configure specific IPv6 Global Addresses on the VPRN R-VPLS interfaces. If a specific Global IPv6 Address is not configured on the interface, the Link Local Address interface MAC/IP will be advertised in a route type 2 as soon as IPv6 is enabled on the VPRN R-VPLS interface.
  3. Routes type 5 for IPv6 prefixes will be advertised using either the configured Global Address or the implicit Link Local Address (if no Global Address is configured).
    If more than one Global Address is configured, normally the first IPv6 address will be used as GW IP. The "first IPv6 address" refers to the first one on the list of IPv6 addresses shown via show router <id> interface <interface> IPv6 or via SNMP.
    The rest of the addresses will be advertised only in MAC-IP routes (Route Type 2) but not used as GW IP for IPv6 prefix routes.

5.2.2.3.2. EVPN for VXLAN in EVPN Tunnel R-VPLS Services

Figure 141 shows an L3 connectivity model that optimizes the solution described in EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes. Instead of regular IRB backhaul R-VPLS services for the connectivity of all the VPRN IRB interfaces, EVPN tunnels can be configured. The main advantage of using EVPN tunnels is that they don't need the configuration of IP addresses, as regular IRB R-VPLS interfaces do.

In addition to the ip-route-advertisement command, this model requires the configuration of the config>service>vprn>if>vpls <name> evpn-tunnel.

Note:

EVPN tunnels can be enabled independently of the ip-route-advertisement command, however, no route-type 5 advertisements will be sent or processed. Neither command, evpn-tunnel and ip-route-advertisement, is supported on R-VPLS services linked to IES interfaces.

The example below shows a VPRN (500) with an EVPN-tunnel R-VPLS (504):

        vprn 500 customer 1 create
            ecmp 4
            route-distinguisher 65071:500
            auto-bind-tunnel
                resolution-filter
                resolution-filter gre ldp rsvp
            vrf-target target:65000:500
            interface "evi-504" create
                vpls "evpn-vxlan-504"
                    evpn-tunnel
                exit
            exit
            no shutdown
        exit
        vpls 504 name “evpn-vxlan-504” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 504 create
            exit
            bgp
                route-distinguisher 65071:504
                route-target export target:65000:504 import target:65000:504
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit

A specified VPRN supports regular IRB backhaul R-VPLS services as well as EVPN tunnel R-VPLS services.

Note:

EVPN tunnel R-VPLS services do not support SAPs or SDP-binds.

The process followed upon receiving a route-type 5 on a regular IRB R-VPLS interface differs from the one for an EVPN-tunnel type:

  1. IRB backhaul R-VPLS VPRN interface:
    1. When a route-type 2 that includes an IP prefix is received and it becomes active, the MAC/IP information is added to the FDB and ARP tables. This can be checked with the show>router>arp command and the show>service>id>fdb detail command.
    2. When route -type 5 is received and becomes active for the R-VPLS service, the IP prefix is added to the VPRN routing table, regardless of the existence of a route-type 2 that can resolve the GW IP address. If a packet is received from the WAN side and the IP lookup hits an entry for which the GW IP (IP next-hop) does not have an active ARP entry, the system will use ARP to get a MAC. If ARP is resolved but the MAC is unknown in the FDB table, the system will flood into the TLS multicast list. Routes type 5 can be checked in the routing table with the show>router>route-table command and the show>router>fib command.
  2. EVPN tunnel R-VPLS VPRN interface:
    1. When route -type 2 is received and becomes active, the MAC address is added to the FDB (only).
    2. When a route-type 5 is received and active, the IP prefix is added to the VPRN routing table with next-hop equal to EVPN tunnel: GW-MAC.
      For example, ET-d8:45:ff:00:01:35, where the GW-MAC is added from the GW-MAC extended community sent along with the route-type 5.
      If a packet is received from the WAN side, and the IP lookup hits an entry for which the next-hop is a EVPN tunnel: GW-MAC, the system will look up the GW-MAC in the FDB. Usually a route-type 2 with the GW-MAC is previously received so that the GW-MAC can be added to the FDB. If the GW-MAC is not present in the FDB, the packet will be dropped.
    3. IP prefixes with GW-MACs as next-hops are displayed by the show router command, as shown below:
 
*A:PE71# show router 500 route-table 
===============================================================================
Route Table (Service: 500)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.20.20.72/32                                Remote  BGP EVPN  00h23m50s  169
       10.10.10.72                                                  0
10.30.30.0/24                                 Remote  BGP EVPN  01d11h30m  169
       evi-504 (ET-d8:45:ff:00:01:35)                               0
10.10.10.0/24                                Remote  BGP VPN   00h20m52s  170
       192.0.2.69 (tunneled)                                        0
10.1.0.0/16                                  Remote  BGP EVPN  00h22m33s  169
       evi-504 (ET-d8:45:ff:00:01:35)                               0
-------------------------------------------------------------------------------
No. of Routes: 4
 

The GW-MAC as well as the rest of the IP prefix BGP attributes are displayed by the show>router>bgp>routes>evpn>ip-prefix command.

*A:Dut-A# show router bgp routes evpn ip-prefix prefix 3.0.1.6/32 detail
===============================================================================
BGP Router ID:10.20.1.1        AS:100         Local AS:100       
===============================================================================
Legend -
Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup
 
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
-------------------------------------------------------------------------------
Original Attributes
 
Network        : N/A
Nexthop        : 10.20.1.2
From           : 10.20.1.2
Res. Nexthop   : 192.168.19.1
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                  
Connector      : None
Community      : target:100:1 mac-nh:00:00:01:00:01:02
                 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 10.20.1.2
Flags          : Used  Valid  Best  IGP 
Route Source   : Internal              
AS-Path        : No As-Path
EVPN type      : IP-PREFIX             
ESI            : N/A                    Tag            : 1
Gateway Address: 00:00:01:00:01:02     
Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1
MPLS Label    : 262140
Route Tag      : 0xb
Neighbor-AS    : N/A
Orig Validation: N/A                   
Source Class   : 0                      Dest Class     : 0
 
Modified Attributes
 
Network        : N/A                 
Nexthop        : 10.20.1.2
From           : 10.20.1.2
Res. Nexthop   : 192.168.19.1
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                  
Connector      : None
Community      : target:100:1 mac-nh:00:00:01:00:01:02
                 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 10.20.1.2
Flags          : Used  Valid  Best  IGP 
Route Source   : Internal              
AS-Path        : 111
EVPN type      : IP-PREFIX             
ESI            : N/A                    Tag            : 1
Gateway Address: 00:00:01:00:01:02     
Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1
MPLS Label    : 262140
Route Tag      : 0xb
Neighbor-AS    : 111
Orig Validation: N/A                   
Source Class   : 0                      Dest Class     : 0
 
-------------------------------------------------------------------------------
Routes : 1
===============================================================================
 

EVPN tunneling is also supported on IPv6 VPRN interfaces. When sending IPv6 prefixes from IPv6 interfaces, the GW-MAC in the route type 5 (IP-prefix route) is always zero. If no specific Global Address is configured on the IPv6 interface, the routes type 5 for IPv6 prefixes will always be sent using the Link Local Address as GW-IP. The following example output shows an IPv6 prefix received via BGP EVPN.

*A:PE71# show router 30 route-table ipv6 
 
===============================================================================
IPv6 Route Table (Service: 30)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
2001:db8:1000::/64                            Local   Local     00h01m19s  0
       int-PE-71-CE-1                                               0
2001:db8:2000::1/128                          Remote  BGP EVPN  00h01m20s  169
       fe80::da45:ffff:fe00:6a-"int-evi-301"                        0
-------------------------------------------------------------------------------
No. of Routes: 2
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
 
*A:PE71# show router bgp routes evpn ipv6-prefix prefix 2001:db8:2000::1/128 hunt 
===============================================================================
 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked
 Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup
 
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : N/A
Nexthop        : 192.0.2.69
From           : 192.0.2.69
Res. Nexthop   : 192.168.19.2
Local Pref.    : 100                    Interface Name : int-71-69
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                   
Connector      : None
Community      : target:64500:301 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.69
Flags          : Used  Valid  Best  IGP  
Route Source   : Internal
AS-Path        : No As-Path
EVPN type      : IP-PREFIX              
ESI            : N/A                    Tag            : 301
Gateway Address: fe80::da45:ffff:fe00:* 
Prefix         : 2001:db8:2000::1/128   Route Dist.    : 192.0.2.69:301
MPLS Label     : 0                      
Route Tag      : 0                      
Neighbor-AS    : N/A
Orig Validation: N/A                    
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 00h41m17s              
 
-------------------------------------------------------------------------------
RIB Out Entries
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 1
=============================================================================== 

5.2.2.4. EVPN-VPWS for VXLAN Tunnels

BGP-EVPN Control Plane for EVPN-VPWS

EVPN-VPWS uses route-type 1 and route-type 4; it does not use route-types 2, 3 or 5. Figure 148 shows the encoding of the required extensions for the Ethernet A-D per-EVI routes. The encoding follows the guidelines described in RFC 8214.

Figure 148:  EVPN VPWS BGP Extensions 

If the advertising PE has an access SAP-SDP or Spoke-SDP that is not part of an Ethernet Segment (ES), then the PE populates the fields of the AD per-EVI route with the following values:

  1. Ethernet Tag ID field is encoded with the value configured by the user in the service>bgp-evpn>local-ac-name>eth-tag value command.
  2. RD and MPLS label values are encoded as specified in RFC 7432. For VXLAN, the MPLS field encodes the VXLAN VNI.
  3. ESI is 0.
  1. The route is sent along an EVPN L2 attributes extended community, as specified in RFC 8214, where:
    1. type and subtype are 0x06 and 0x04 as allocated by IANA
    2. flag C is set if a control word is configured in the service. C will always be zero for VXLAN tunnels
    3. P and B flags are zero
    4. L2 MTU is encoded with a service mtu configured in the Epipe service

If the advertising PE has an access SAP-SDP or Spoke-SDP that is part of an ES, the AD per-EVI route is sent with the information described above, with the following minor differences:

  1. The ESI encodes the corresponding non-zero value.
  2. The P and B flags are set in the following cases:
    1. All-active multi-homing
      1. All PEs that are part of the ES always set the P flag.
      2. The B flag is never set in the all-active multi-homing ES case.
    2. Single-active multi-homing
      1. Only the DF PE sets the P bit for an EVI and the remaining PEs send it as P=0.
      2. Only the backup DF PE sets the B bit.
        If more than two PEs are present in the same single-active ES, the backup PE is the winner of a second DF election (excluding the DF). The remaining non-DF PEs send B=0.

Also, ES and AD per-ES routes are advertised and processed for the Ethernet-Segment, as described in RFC 7432 Ethernet Segments. The ESI label sent with the AD per-ES route is used by BUM traffic on VPLS services; it is not used for Epipe traffic.

EVPN-VPWS for VXLAN Tunnels in Epipe Services

BGP-EVPN can be enabled in Epipe services with either SAPs or Spoke-SDPs at the access, as shown in Figure 149.

Figure 149:  EVPN-MPLS VPWS 

EVPN-VPWS is supported in VXLAN networks that also run EVPN-VXLAN in VPLS services. From a control plane perspective, EVPN-VPWS is a simplified point-to-point version of RFC 7432 for E-Line services for the following reasons:

  1. EVPN-VPWS does not use inclusive multicast, MAC or IP routes or IP-Prefix routes.
  2. AD Ethernet per-EVI routes are used to advertise the local attachment circuit identifiers at each side of the VPWS instance. The attachment circuit identifiers are configured as local and remote Ethernet tags. When an AD per-EVI route is imported and the Ethernet tag matches the configured remote Ethernet tag, an EVPN destination is created for the Epipe.

In the following configuration example, Epipe 2 is an EVPN-VPWS service between PE2 and PE4 (as shown in Figure 149).

PE2>config>service>epipe(2)#
-----------------------
vxlan vni 2 instance 1 create
exit
bgp
exit
bgp-evpn
  evi 2
  local-ac-name "AC-1"
    eth-tag 100
  remote-ac-name "AC-2"
    eth-tag 200
  vxlan bgp 1 vxlan-instance 1
    ecmp 2
    no shutdown
sap 1/1/1:1 create
PE4>config>service>epipe(2)#
-----------------------
vxlan vni 2 instance 1 create
exit
bgp
exit
bgp-evpn
  evi 2
  local-ac-name "AC-2"
    eth-tag 200
  remote-ac-name "AC-1"
    eth-tag 100
  vxlan bgp 1 vxlan-instance 1
    ecmp 2
    no shutdown
spoke-sdp 1:1

The following considerations apply for the above example configuration:

  1. The EVI is used to auto-derive the route-target or route-distinguisher of the service. The EVI values must be unique in the system regardless of the type of service they are assigned to (EPIPE or VPLS).
  2. Support for the following BGP-EVPN commands in Epipe services is the same as in VPLS services:
    1. vxlan bgp 1 vxlan-instance 1
    2. vxlan send-evpn-encap
    3. vxlan shutdown
    4. vxlan ecmp
  3. The following BGP-EVPN commands identify the local and remote attachment circuits, with the configured Ethernet tags encoded in the advertised and received AD Ethernet per-EVI routes:
    1. local-ac-name name
    2. local-ac-name name eth-tag tag-value; where tag-value is 1 to 16777215
    3. remote-ac-name name
    4. remote-ac-name name eth-tag tag-value; where tag-value is 1 to 16777215
    5. Changes to remote Ethernet tags are allowed without shutting down BGP-EVPN VXLAN or the Epipe service. The local AC Ethernet tag value cannot be changed without BGP-EVPN VXLAN shutdown.
    6. Both local and remote Ethernet tags are mandatory to bring up the Epipe service.

EVPN-VPWS Epipes can also be configured with the following characteristics:

  1. Access attachment circuits can be SAPs or Spoke-SDPs. Only manually configured Spoke-SDPs are supported; BGP-VPWS and endpoints are not supported. The VC switching configuration is not supported on BGP-EVPN enabled pipes.
  2. EVPN-VPWS Epipes can advertise the Layer 2 (service) MTU and check its consistency as follows:
    1. The advertised MTU value will be taken from the configured service MTU in the Epipe service.
    2. The received L2 MTU will be compared to the local value. In case of a mismatch between the received MTU and the configured service MTU, the system will not set up the EVPN destination; as a result, the service will not come up.
    3. The system will not check the network port MTU value.
    4. If the received L2 MTU value is 0, the MTU is ignored.

Using A/S PW and MC-LAG with EVPN-VPWS Epipes

The use of A/S PW (for access Spoke-SDPs) and MC-LAG (for access SAPs) provides an alternative redundant solution for EVPN-VPWS that do not use the EVPN multi homing procedures described in RFC 8214. Figure 150 shows the use of both mechanisms in a single Epipe.

Figure 150:  A/S PW and MC-LAG Support on EVPN-VPWS 

In Figure 150, an A/S PW connects the CE to PE1 and PE2 (left-hand side of the diagram), and an MC-LAG connects the CE to PE3 and PE4 (right-hand side of the diagram). As EVPN multi homing is not used, there are no AD per-ES routes or ES routes. The redundancy is handled as follows:

  1. PE1 and PE2 are configured with EPIPE-1, where a Spoke-SDP connects the service in each PE to the access CE. The local AC Ethernet tag is 1 and the remote AC Ethernet tag is 2 (in PE1/PE2).
  2. PE3 and PE4 are configured with EPIPE-1, where each PE has a lag SAP that belongs to a previously-configured MC-LAG construct. The local AC Ethernet tag is 2 and the remote AC Ethernet tag is 1.
  3. An endpoint and A/S PW is configured on the CE on the left-hand side of the diagram. PE1/PE2 are able to advertise Ethernet tag 1 based on the operating status or the forwarding status of the Spoke-SDP.
    For example, if PE1 receives a standby PW status indication from the CE and the previous status was forward, it will withdraw the AD EVI route for Ethernet tag 1. If PE2 receives a forward PW status indication and the previous status was standby or down, it will advertise the AD EVI route for Ethernet tag 1.
  4. The user can configure MC-LAG for access SAPs using the example configuration of PE3 and PE4, as shown in Figure 150. In this case, the MC-LAG will determine which chassis is active and which is standby.
    If PE4 becomes the standby chassis, the entire LAG port will be brought down. As a result, the SAP will go operationally down and PE4 will withdraw any previous AD EVI routes for Ethernet tag 2.
    If PE3 becomes the active chassis, the LAG port becomes operationally up. As a result, the SAP and the PE3 will advertise the AD per-EVI route for Ethernet tag 2.

EVPN Multi-homing for EVPN-VPWS Services

EVPN multi homing is supported for EVPN-VPWS Epipe services with the following considerations:

  1. Single-active and all-active multi-homing is supported for SAPs and Spoke-SDPs.
  2. ESs can be shared between the Epipe (MPLS and VXLAN) and VPLS (MPLS) services for LAGs, ports, and SDPs.
  3. A split-horizon function is not required because there is no traffic between the Designated Forwarder (DF) and the non-DF for Epipe services. As a result, the ESI label is never used, and the ethernet-segment multi-homing single-active no-esi-label and ethernet-segment source-bmac-lsb commands do not affect Epipe services.
  4. The local Ethernet tag values must match on all PEs that are part of the same ES, regardless of the multi homing mode. The PEs in the ES use the AD per-EVI routes from the peer PEs to validate the PEs as DF election candidates for a specific EVI.

The DF election for Epipes that is defined in an all-active multi homing ES is not relevant because all PEs in the ES behave in the same way as follows:

  1. All PEs send P=1 on the AD per-EVI routes.
  2. All PEs can send upstream and downstream traffic, regardless of whether the traffic is unicast, multicast, or broadcast (all traffic is treated as unicast in the Epipe services).
    Therefore, the following tools command shows "N/A" when all-active multi-homing is configured.
    *A:PE-2# tools dump service system bgp-evpn ethernet-segment "ESI-12" evi 6000 df
    [03/18/2016 20:31:35] All Active VPWS - DF N/A

Aliasing is supported for traffic sent to an ES destination. If ECMP is enabled on the ingress PE, per-flow load balancing is performed to all PEs that advertise P=1. The PEs that advertise P=0, are not considered as next hops for an ES destination.

Note:

The ingress PE will load balance the traffic if shared queuing or ingress policing is enabled on the access SAPs.

Although DF election is not relevant for Epipes in an all-active multi homing ES, it is essential for the following forwarding and backup functions in a single-active multihoming ES.

  1. The PE elected as DF is the primary PE for the ES in the Epipe. The primary PE unblocks the SAP or Spoke-SDP for upstream and downstream traffic; the remaining PEs in the ES bring their ES SAPs or Spoke-SDPs operationally down.
  2. The DF candidate list is built from the PEs sending ES routes for the same ES and is pruned for a specific service, depending on the availability of the AD per-ES and per-EVI routes.
  3. When the SAP or Spoke-SDPs that are part of the ES come up, the AD per-EVI routes are sent with P=0 and B=0. The remote PEs do not start sending traffic until the DF election process is complete and the ES activation timer is expired, and the PEs advertise AD per-EVI routes with P and B bits other than zero.
  4. The backup PE function is supported as defined in RFC 8214. The primary PE, backup, or none status is signaled by the PEs (part of the same single-active MH ES) in the P or B flags of the EVPN L2 attributes extended community. Figure 151 shows the advertisement and use of the primary, backup, or none indication by the PEs in the ES.
    Figure 151:  EVPN-VPWS Single-active Multi-homing 
    As specified in RFC 7432, the remote PEs in VPLS services have knowledge of the primary PE in the remote single-active ES, based on the advertisement of the MAC or IP routes because only the DF will learn and advertise MAC or IP routes.
    Because there are no MAC or IP routes in EVPN-VPWS, the remote PEs can forward the traffic based on the P/B bits. The process is described in the following list:
    1. The DF PE for an EVI (PE1) sends P=1 and B=0.
    2. For each ES or EVI, a second DF election is run among the PEs in the backup candidate list to elect the backup PE. The backup PE sends P=0 and B=1 (PE2).
    3. All remaining multi homing PEs send P=0 and B=0 (PE3 and PE4).
    4. At the remote PEs (PE5), the P and B flags are used to identify the primary and backup PEs within the ES destination. The traffic is then sent to the primary PE, provided that it is active.
  1. When a remote PE receives the withdrawal of an Ethernet AD per-ES (or per-EVI) route from the primary PE, the remote PE immediately switches the traffic to the backup PE for the affected EVIs.
  2. The backup PE takes over immediately without waiting for the ES activation timer to bring up its SAP or Spoke-SDP.
  3. The BGP-EVPN MPLS ECMP setting also governs the forwarding in single-active multi homing, regardless of the single-active multi homing bit in the AD per-ES route received at the remote PE (PE5).
    1. PE5 always sends the traffic to the primary remote PE (the owner of the P=1 bit). In case of multiple primary PEs and ECMP>1, PE5 will load balance the traffic to all primary PEs, regardless of the multi homing mode.
    2. If the last primary PE withdraws its AD per-EVI or per-ES route, PE5 sends the traffic to the backup PE or PEs. In case of multiple backup PEs and ECMP>1, PE1 load balances the traffic to the backup PEs.

5.2.2.4.1. EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes

Figure 140 shows a Layer 3 DC model, where a VPRN is defined in the DC GWs, connecting the tenant to the WAN. That VPRN instance will be connected to the VPRNs in the NVEs by means of an IRB backhaul R-VPLS. Since the IRB backhaul R-VPLS provides connectivity only to all the IRB interfaces and the DC GW VPRN is not directly connected to all the tenant subnets, the WAN ip-prefixes in the VPRN routing table must be advertised in EVPN. In the same way, the NVEs will send IP prefixes in EVPN that will be received by the DC GW and imported in the VPRN routing table.

Note:

To generate or process IP prefixes sent or received in EVPN route type 5, the support for IP route advertisement must be enabled in BGP-EVPN. This is performed through the bgp-evpn>ip-route-advertisement command. This command s disabled by default and must be explicitly enabled. The command is tied to the allow-ip-int-bind command required for R-VPLS, and it is not supported on R-VPLS linked to IES services.

Local router interface host addresses are not advertised in EVPN by default. To advertise them, the ip-route-advertisement incl-host command must be enabled. For example:

===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                         Active     Metric
-------------------------------------------------------------------------------
10.1.1.0/24                                   Local   Local     00h00m11s  0
       if                                              Y            0
10.1.1.100/32                                 Local   Host      00h00m11s  0
       if                                              Y            0
==============================================================================

For the case displayed by the output above, the behavior is the following:

  1. ip-route-advertisement only local subnet (default) - 10.1.1.0/24 is advertised
  2. ip-route-advertisement incl-host local subnet, host - 10.1.1.0/24 and 10.1.1.100/32 are advertised

Below is an example of VPRN (500) with two IRB interfaces connected to backhaul R-VPLS services 501 and 502 where EVPN-VXLAN runs:

vprn 500 customer 1 create            
            ecmp 4
            route-distinguisher 65072:500
            auto-bind-tunnel
                resolution-filter
                resolution-filter gre ldp rsvp
            vrf-target target:65000:500
            interface "evi-502" create
                address 10.20.20.72/24
                vpls "evpn-vxlan-502"
                exit
            exit
            interface "evi-501" create
                address 10.10.10.72/24
                vpls "evpn-vxlan-501"
                exit
            exit
            no shutdown
vpls 501 name “evpn-vxlan-501” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 501 create
            exit
            bgp
                route-distinguisher 65072:501
                route-target export target:65000:501 import target:65000:501
            exit
            bgp-evpn
                ip-route-advertisement incl-host
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit                  
            exit
            no shutdown
        exit
vpls 502 name “evpn-vxlan-502” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 502 create
            exit
            bgp
                route-distinguisher 65072:502
                route-target export target:65000:502 import target:65000:502
            exit
            bgp-evpn
                ip-route-advertisement incl-host
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit
 

When the above commands are enabled, the router will:

  1. Receive route-type 5 routes and import the IP prefixes and associated IP next-hops into the VPRN routing table.
    1. If the route-type 5 is successfully imported by the router, the prefix included in the route-type 5 (for example, 10.0.0.0/24), will be added to the VPRN routing table with a next-hop equal to the GW IP included in the route (for example, 192.0.0.1. that refers to the IRB IP address of the remote VPRN behind which the IP prefix sits).
    2. When the router receives a packet from the WAN to the 10.0.0.0/24 subnet, the IP lookup on the VPRN routing table will yield 192.0.0.1 as the next-hop. That next-hop will be resolved to a MAC in the ARP table and the MAC resolved to a VXLAN tunnel in the FDB table
      Note:

      IRB MAC and IP addresses are advertised in the IRB backhaul R-VPLS in routes type 2.

  2. Generate route-type 5 routes for the IP prefixes in the associated VPRN routing table.
    1. For example, if VPRN-1 is attached to EVPN R-VPLS 1 and EVPN R-VPLS 2, and R-VPLS 2 has bgp-evpn ip-route-advertisement configured, the 7750 SR will advertise the R-VPLS 1 interface subnet in one route-type 5.
  3. Routing policies can filter the imported and exported IP prefix routes accordingly.

The VPRN routing table can receive routes from all the supported protocols (BGP-VPN, OSPF, IS-IS, RIP, static routing) as well as from IP prefixes from EVPN, as shown below:

*A:PE72# show router 500 route-table                      
===============================================================================
Route Table (Service: 500)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.20.20.0/24                                 Local   Local     01d11h10m  0
       evi-502                                                      0
10.20.20.71/32                                Remote  BGP EVPN  00h02m26s  169
       10.10.10.71                                                  0
10.10.10.0/24                                Remote  Static    00h00m05s  5
       10.10.10.71                                                  1
10.16.0.1/32                                 Remote  BGP EVPN  00h02m26s  169
       10.10.10.71                                                  0
-------------------------------------------------------------------------------
No. of Routes: 4

The following considerations apply:

  1. The route Preference for EVPN IP prefixes is 169.
    1. BGP IP-VPN routes have a preference of 170 by default, therefore, if the same route is received from the WAN over BGP-VPRN and from BGP-EVPN, then the EVPN route will be preferred.
  2. When the same route-type 5 prefix is received from different GW IPs, ECMP is supported if configured in the VPRN.
  3. All routes in the VPRN routing table (as long as they do not point back to the EVPN R-VPLS interface) are advertised via EVPN.

Although the description above is focused on IPv4 interfaces and prefixes, it applies to IPv6 interfaces too. The following considerations are specific to IPv6 VPRN R-VPLS interfaces:

  1. IPv4 and IPv6 interfaces can be defined on R-VPLS IP interfaces at the same time (dual-stack).
  2. The user may configure specific IPv6 Global Addresses on the VPRN R-VPLS interfaces. If a specific Global IPv6 Address is not configured on the interface, the Link Local Address interface MAC/IP will be advertised in a route type 2 as soon as IPv6 is enabled on the VPRN R-VPLS interface.
  3. Routes type 5 for IPv6 prefixes will be advertised using either the configured Global Address or the implicit Link Local Address (if no Global Address is configured).
    If more than one Global Address is configured, normally the first IPv6 address will be used as GW IP. The "first IPv6 address" refers to the first one on the list of IPv6 addresses shown via show router <id> interface <interface> IPv6 or via SNMP.
    The rest of the addresses will be advertised only in MAC-IP routes (Route Type 2) but not used as GW IP for IPv6 prefix routes.

5.2.2.4.2. EVPN for VXLAN in EVPN Tunnel R-VPLS Services

Figure 141 shows an L3 connectivity model that optimizes the solution described in EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes. Instead of regular IRB backhaul R-VPLS services for the connectivity of all the VPRN IRB interfaces, EVPN tunnels can be configured. The main advantage of using EVPN tunnels is that they don't need the configuration of IP addresses, as regular IRB R-VPLS interfaces do.

In addition to the ip-route-advertisement command, this model requires the configuration of the config>service>vprn>if>vpls <name> evpn-tunnel.

Note:

evpn-tunnel can be enabled independently of ip-route-advertisement, however, no route-type 5 advertisements will be sent or processed in that case. Neither command, evpn-tunnel and ip-route-advertisement, is supported on R-VPLS services linked to IES interfaces.

The example below shows a VPRN (500) with an EVPN-tunnel R-VPLS (504):

        vprn 500 customer 1 create
            ecmp 4
            route-distinguisher 65071:500
            auto-bind-tunnel
                resolution-filter
                resolution-filter gre ldp rsvp
            vrf-target target:65000:500
            interface "evi-504" create
                vpls "evpn-vxlan-504"
                    evpn-tunnel
                exit
            exit
            no shutdown
        exit
        vpls 504 name “evpn-vxlan-504” customer 1 create
            allow-ip-int-bind
            vxlan instance 1 vni 504 create
            exit
            bgp
                route-distinguisher 65071:504
                route-target export target:65000:504 import target:65000:504
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit

A specified VPRN supports regular IRB backhaul R-VPLS services as well as EVPN tunnel R-VPLS services.

Note:

EVPN tunnel R-VPLS services do not support SAPs or SDP-binds.

The process followed upon receiving a route-type 5 on a regular IRB R-VPLS interface differs from the one for an EVPN-tunnel type:

  1. IRB backhaul R-VPLS VPRN interface:
    1. When a route-type 2 that includes an IP prefix is received and it becomes active, the MAC/IP information is added to the FDB and ARP tables. This can be checked with the show>router>arp command and the show>service>id>fdb detail command.
    2. When route-type 5 is received and becomes active for the R-VPLS service, the IP prefix is added to the VPRN routing table, regardless of the existence of a route-type 2 that can resolve the GW IP address. If a packet is received from the WAN side and the IP lookup hits an entry for which the GW IP (IP next-hop) does not have an active ARP entry, the system will use ARP to get a MAC. If ARP is resolved but the MAC is unknown in the FDB table, the system will flood into the TLS multicast list. Routes type 5 can be checked in the routing table with the show>router>route-table command and the show>router>fib command.
  2. EVPN tunnel R-VPLS VPRN interface:
    1. When route-type 2 is received and becomes active, the MAC address is added to the FDB (only).
    2. When a route-type 5 is received and active, the IP prefix is added to the VPRN routing table with next-hop equal to EVPN tunnel: GW-MAC.
      For example, ET-d8:45:ff:00:01:35, where the GW-MAC is added from the GW-MAC extended community sent along with the route-type 5.
      If a packet is received from the WAN side, and the IP lookup hits an entry for which the next-hop is a EVPN tunnel: GW-MAC, the system will look up the GW-MAC in the FDB. Usually a route-type 2 with the GW-MAC is previously received so that the GW-MAC can be added to the FDB. If the GW-MAC is not present in the FDB, the packet will be dropped.
    3. IP prefixes with GW-MACs as next-hops are displayed by the show router command, as shown below:
 
*A:PE71# show router 500 route-table 
===============================================================================
Route Table (Service: 500)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.20.20.72/32                                Remote  BGP EVPN  00h23m50s  169
       10.10.10.72                                                  0
10.30.30.0/24                                 Remote  BGP EVPN  01d11h30m  169
       evi-504 (ET-d8:45:ff:00:01:35)                               0
10.10.10.0/24                                Remote  BGP VPN   00h20m52s  170
       192.0.2.69 (tunneled)                                        0
10.1.0.0/16                                  Remote  BGP EVPN  00h22m33s  169
       evi-504 (ET-d8:45:ff:00:01:35)                               0
-------------------------------------------------------------------------------
No. of Routes: 4
 

The GW-MAC as well as the rest of the IP prefix BGP attributes are displayed by the show>router>bgp>routes>evpn>ip-prefix command.

*A:Dut-A# show router bgp routes evpn ip-prefix prefix 3.0.1.6/32 detail
===============================================================================
BGP Router ID:10.20.1.1        AS:100         Local AS:100       
===============================================================================
Legend -
Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup
 
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
-------------------------------------------------------------------------------
Original Attributes
 
Network        : N/A
Nexthop        : 10.20.1.2
From           : 10.20.1.2
Res. Nexthop   : 192.168.19.1
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                  
Connector      : None
Community      : target:100:1 mac-nh:00:00:01:00:01:02
                 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 10.20.1.2
Flags          : Used  Valid  Best  IGP 
Route Source   : Internal              
AS-Path        : No As-Path
EVPN type      : IP-PREFIX             
ESI            : N/A                    Tag            : 1
Gateway Address: 00:00:01:00:01:02     
Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1
MPLS Label    : 262140
Route Tag      : 0xb
Neighbor-AS    : N/A
Orig Validation: N/A                   
Source Class   : 0                      Dest Class     : 0
 
Modified Attributes
 
Network        : N/A                 
Nexthop        : 10.20.1.2
From           : 10.20.1.2
Res. Nexthop   : 192.168.19.1
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                  
Connector      : None
Community      : target:100:1 mac-nh:00:00:01:00:01:02
                 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 10.20.1.2
Flags          : Used  Valid  Best  IGP 
Route Source   : Internal              
AS-Path        : 111
EVPN type      : IP-PREFIX             
ESI            : N/A                    Tag            : 1
Gateway Address: 00:00:01:00:01:02     
Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1
MPLS Label    : 262140
Route Tag      : 0xb
Neighbor-AS    : 111
Orig Validation: N/A                   
Source Class   : 0                      Dest Class     : 0
 
-------------------------------------------------------------------------------
Routes : 1
===============================================================================
 

EVPN tunneling is also supported on IPv6 VPRN interfaces. When sending IPv6 prefixes from IPv6 interfaces, the GW-MAC in the route type 5 (IP-prefix route) is always zero. If no specific Global Address is configured on the IPv6 interface, the routes type 5 for IPv6 prefixes will always be sent using the Link Local Address as GW-IP. The following example output shows an IPv6 prefix received via BGP EVPN.

*A:PE71# show router 30 route-table ipv6 
 
===============================================================================
IPv6 Route Table (Service: 30)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
2001:db8:1000::/64                            Local   Local     00h01m19s  0
       int-PE-71-CE-1                                               0
2001:db8:2000::1/128                          Remote  BGP EVPN  00h01m20s  169
       fe80::da45:ffff:fe00:6a-"int-evi-301"                        0
-------------------------------------------------------------------------------
No. of Routes: 2
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
 
*A:PE71# show router bgp routes evpn ipv6-prefix prefix 2001:db8:2000::1/128 hunt 
===============================================================================
 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked
 Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup
 
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : N/A
Nexthop        : 192.0.2.69
From           : 192.0.2.69
Res. Nexthop   : 192.168.19.2
Local Pref.    : 100                    Interface Name : int-71-69
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                   
Connector      : None
Community      : target:64500:301 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.69
Flags          : Used  Valid  Best  IGP  
Route Source   : Internal
AS-Path        : No As-Path
EVPN type      : IP-PREFIX              
ESI            : N/A                    Tag            : 301
Gateway Address: fe80::da45:ffff:fe00:* 
Prefix         : 2001:db8:2000::1/128   Route Dist.    : 192.0.2.69:301
MPLS Label     : 0                      
Route Tag      : 0                      
Neighbor-AS    : N/A
Orig Validation: N/A                    
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 00h41m17s              
 
-------------------------------------------------------------------------------
RIB Out Entries
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 1
=============================================================================== 

5.2.3. DC GW integration with the Nuage Virtual Services Directory (VSD)

The Nuage VSD (Virtual Services Directory) provides automation in the Nuage DC. The VSD is a programmable policy and analytics engine. It provides a flexible and hierarchical network policy framework that enables IT administrators to define and enforce resource policies.

The VSD contains a multi-tenant service directory that supports role-based administration of users, computing, and network resources. The VSD also manages network resource assignments such as IP addresses and ACLs.

To communicate with the Nuage controllers and gateways (including the 7750 SR, 7450 ESS, or 7950 XRS DC GW), VSD uses an XMPP (eXtensible Messaging and Presence Protocol) communication channel. The router can receive service parameters from the Nuage VSD through XMPP and add them to the existing VPRN/VPLS service configuration.

Note:

The service must be pre-provisioned in the router using the CLI, SNMP, or other supported interfaces. The VSD will only push a limited number of parameters into the configuration. This router – VSD integration model is known as a Static-Dynamic provisioning model, because only a few parameters are dynamically pushed by VSD, as opposed to a Fully Dynamic model, where the entire service can be created dynamically by VSD.

The router – VSD integration comprises the following building blocks:

  1. An XMPP interface to the DC XMPP server, through which the router can discover the Data Center Nuage VSDs and select a specified VSD for each VPLS/VPRN service.
  2. The configuration of vsd-domains on those services where VSD will dynamically provision parameters. As part of the static provisioning of a service, the user will configure a domain name (that will be used between VSD and 7750 SR) using a new CLI command vsd-domain name. Any parameters sent by the VSD for an existing service will contain the vsd-domain. Based on that tag, the router will add the required configuration changes to the correct service.
  3. The dynamic provisioning of parameters in the following four use-cases:
    1. L2-DOMAIN: To attach a service at the gateway to a Layer 2 (Ethernet) domain in the data center with no routing at the gateway, a VPLS service should be associated with a vsd-domain of type l2-domain. When the appropriate configuration for the domain is present/added at the VSD, the VSD will dynamically add the VXLAN VNI and BGP export and import route-targets to exchange DC EVPN routes with the VPLS service.
    2. L2-DOMAIN-IRB: To attach a service at the gateway to a Layer 2 (Ethernet) domain in the data center with routing at the gateway, an R-VPLS service should be associated with a vsd-domain of type l2-domain-irb. When the appropriate configuration for the domain is present/added at the VSD, the VSD will dynamically add the VXLAN VNI and BGP export and import route-targets to exchange DC EVPN routes with the R-VPLS service.
    3. VRF-GRE: To attach a service at the gateway to a layer 3 domain (with GRE transport) in the data center, a VPRN service should be associated with a vsd-domain of type vrf-gre. When the appropriate configuration for the domain is present/added at the VSD, the VSD will dynamically add the BGP export and import route-targets to exchange DC IP VPN routes with the VPRN service.
    4. VRF-VXLAN: To attach a service at the gateway to a layer 3 domain (with VXLAN transport) in the data center, an R-VPLS service (linked to an EVPN-tunnel with ip-route-advertisement enabled) should be associated with a vsd-domain of type vrf-vxlan. When the appropriate configuration for the domain is present/added at the VSD, the VSD will dynamically add the VXLAN VNI and BGP export and import route-targets to exchange DC EVPN routes with the backhaul R-VPLS connected to the data center VPRN service.

These building blocks are described in more detail in the following subsections.

5.2.3.1. XMPP Interface on the DC GW

The Extensible Messaging and Presence Protocol (XMPP) is an open technology for real-time communication using XML (Extensible Markup Language) as the base format for exchanging information. The XMPP provides a way to send small pieces of XML from one entity to another in close to real time.

In a Nuage DC, an XMPP ejabberd server will have an interface to the Nuage VSD as well as the Nuage VSC/VSG and the 7750 SR, 7450 ESS, or 7950 XRS DC GW.

Figure 152 shows the basic XMPP architecture in the data center. While a single XMPP server is represented in the diagram, XMPP allows for easy server clustering and performs message replication to the cluster. It is similar to how BGP can scale and replicate the messages through the use of route reflectors.

Also the VSD is represented as a single server, but a cluster of VSD servers (using the same data base) will be a very common configuration in a DC.

Figure 152:  Basic XMPP Architecture 

In the Nuage solution, each XMPP client, including the 7750 SR, 7450 ESS, and 7950 XRS, is referred to with a JID (JabberID) in the following format: username@xmppserver.domain. The xmppserver.domain points to the XMPP Server.

To enable the XMPP interface on the 7750 SR, 7450 ESS, or 7950 XRS, the following command must be added to indicate to which XMPP server address the DC GW has to register, as well as the router’s JID:

*A:PE-2# configure system xmpp server
  - server <xmpp-server-name> [domain-name <fqdn>] [username <user-name>]
    [password <password>] [create] [service-name <service-name>]
  - no server <xmpp-server-name>
  - server <xmpp-server-name> [domain-name <fqdn>] [username <user-name>]
    [password <password>] [create] [router <router-instance>]
 <xmpp-server-name>   : [32 chars max]
 <fqdn>               : [256 chars max]
 <user-name>          : [32 chars max]
 <password>           : [32 chars max]
 <create>             : keyword - mandatory while creating an entry.
 <router-instance>    : <router-name>|<vprn-svc-id>
                        router-name    - "Base"|"management"
                        vprn-svc-id    - [1..2147483647]
 <service-name>       : [64 chars max]
 [no] shutdown        - Administratively enable or disable XMPP server

Where:

  1. [domain-name <fqdn>] is the domain portion of the JID.
  2. <user-name> and <password> is the username:password portion of the JID of the router acting as an XMPP client. Plain/MD5/anonymous authentication is supported.
  3. The user can choose not to configure the username portion of the JID. In that case, an in-band registration will be attempted, using the chassis MAC as username.
  4. The user has the option to try to establish an XMPP TCP session over a router instance by using the router router-instance command. The router name can be “Base”, “management”, or a given VPRN service identifier.
  5. When the xmpp server is properly configured and no shutdown, the 7750 SR will try to establish a TCP session with the XMPP server through the management interface first. If it fails to establish communication, the 7750 SR will use an in-band communication and will use its system IP as source IP address. Shutdown will not remove the dynamic configs in all the services. No server will remove all the dynamic configs in all the services.
  6. Only one xmpp server can be configured.
Note:

The DNS must be configured on the router so that the XMPP server name can be resolved. XMPP relies on the Domain Name System (DNS) to provide the underlying structure for addressing, instead of using raw IP addresses. The DNS is configured using the following bof commands: bof primary-dns, bof secondary-dns, bof dns-domain.

After the XMPP server is properly configured, the router can generate or receive XMPP stanza elements, such as presence and IQ (Information/Query) messages. IQ messages are used between the VSD and the router to request and receive configuration parameters. The status of the XMPP communication channel can be checked with the following command:

Dut# show system xmpp server "vsd1-hy" 
 
===============================================================================
XMPP Server Table
===============================================================================
XMPP FQDN          : vsd1-hy.alu.us
XMPP Admin User    : csproot
XMPP Oper User     : csproot
State Lst Chg Since: 0d 02:56:44        State              : Functional
Admin State        : Up                 Connection Mode    : outOfBand
Auth Type          : md5
IQ Tx.             : 47                 IQ Rx.             : 47
IQ Error           : 0                  IQ Timed Out       : 0
IQ Min. Rtt        : 0 ms               IQ Max. Rtt        : 180 ms
IQ Ack Rcvd.       : 47                 
Push Updates Rcvd  : 1                  VSD list Upd Rcvd  : 12
Msg Tx.            : 27                 Msg Rx.            : 27
Msg Ack. Rx.       : 27                 Msg Error          : 0
Msg Min. Rtt       : 0 ms               Msg Max. Rtt       : 180 ms
Sub Tx.            : 1                  UnSub Tx.          : 0
Msg Timed Out      : 0                  
 
===============================================================================
 

In addition to the XMPP server, the router must be configured with a VSD system-id that uniquely identifies the router in the VSD:

*B:Dut>config>system>vsd# info 
----------------------------------------------
                system-id "SR12U-46-PE"
---------------------------------------------- 

After the above configuration is complete, the router will subscribe to a VSD XMPP PubSub node to discover the available VSD servers. Then, the router will be discovered in the VSD UIs. On the router, the available VSD servers can be shown with the following command.

B:Dut#show system xmpp vsd 
 
===============================================================================
Virtual Services Directory Table
===============================================================================
Id User Name                    Uptime             Status
-------------------------------------------------------------------------------
1  cna@vsd1-hy.alu-srpm.us/nua* 0d 00:44:36        Available
-------------------------------------------------------------------------------
No. of VSD's: 1
===============================================================================
* indicates that the corresponding row element may have been truncated.
*B:Dut#show system xmpp vsd 1 
 
===============================================================================
VSD Server Table
===============================================================================
VSD User Name      : cna@vsd1-hy.alu-srpm.us/nuage
Uptime             : 0d 00:44:39        Status             : Available
Msg Tx.            : 16                 Msg Rx.            : 10
Msg Ack. Rx.       : 4                  Msg Error          : 6
Msg TimedOut       : 0                  Msg MinRtt         : 80 ms
Msg MaxRtt         : 240 ms             
 
===============================================================================

5.2.3.2. Overview of the Static-Dynamic VSD Integration Model

In the Static-Dynamic integration model, the DC and DC GW management entities can be the same or different. The DC GW operator will provision the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN. VSD will only push the required parameters so that those WAN services can be attached to the existing DC domains.

Figure 153 shows the workflow for the attachment of the WAN services defined on the DC GW to the DC domains.

Figure 153:  WAN Services Attachment Workflow 

The Static-Dynamic VSD integration model can be summarized in the steps shown in Figure 153 and described in the following procedure.

  1. The WAN or SP (Service Provider) administrator (which can be also the DC or Cloud Service Provider administrator) provisions the WAN services with all the parameters required for the connectivity to the WAN. This configuration is performed through the regular management interfaces, for example, CLI or SNMP. In the example above, there are two services created by the SP:
    1. VPRN 1 – associated with vsd-domain Ent-1, which is a VRF-GRE domain.
    2. VPLS 2 – associated with vsd-domain Ent-2, which is an L2-DOMAIN
    Note:

    The parameters between brackets “[..]” are not configured at this step. They will be pushed by the VSD through XMPP.

  2. The router communicates with the VSD through the XMPP channel and lets VSD know about its presence and available domains: Ent-1 and Ent-2. In the VSD’s User Interface (UI), the router will show up as DC GW with its System ID, personality (for example, router) and the available WAN resources, that is, vsd-domains Ent-1 and Ent-2.
  3. At VSD, the Cloud Service Provider administrator will assign the available WAN resources to Enterprises defined in VSD. In this example, VRF-GRE Ent-1 will be assigned to Enterprise-1 and L2-DOMAIN Ent-2 to Enterprise-2.
  4. Each Enterprise administrator will have visibility of their own assigned WAN resource and will attach it to an existing DC Domain, assuming that both the DC domain and WAN resource are compatible. For instance, a VRF-GRE domain can only be attached to an L3 domain in the DC that uses GRE as transport.
  5. When the Enterprise administrator attaches the WAN resource to the DC domain, VSD will send the required configuration parameters to the DC GW through the XMPP channel:
    1. In the case of the VRF-GRE domain, VSD will only send the vrf-target required for the service attachment to the DC domain.
    2. In the case of the L2-DOMAIN, VSD will send the route-target (in the service>bgp or vsi-import/export contexts) as well as the vxlan vni and the bgp-evpn vxlan no shutdown commands.
    WAN resources can also be detached from the DC domains.

5.2.3.3. VSD-Domains and Association to Static-Dynamic Services

In the Static-Dynamic integration model, VSD can only provision certain parameters in VPLS and/or VPRN services. When VSD and the DC GW exchange XMPP messages for a specified service, they use vsd-domains to identify those services. A vsd-domain is a tag that will be used by the 7750 SR, 7450 ESS, or 7950 XRS router and the VSD to correlate a configuration to a service. When redundant DC GWs are available, the vsd-domain for the same service can have the same or a different name in the two redundant DC GWs.

There are four different types of vsd-domains that can be configured in the router:

  1. L2-DOMAIN – it will be associated with a VPLS service in the router and, in VSD, it will be attached to an existing Nuage L2-DOMAIN. This type of domain will be used for extending Layer 2 subnets to the WAN connected to the DC GW.
  2. L2-DOMAIN-IRB – it will be associated with a R-VPLS service in the router and, in VSD, it will be attached to an existing Nuage L2-DOMAIN. In this case, the DC GW will be the default gateway for all the VMs and hosts in the Nuage L2-DOMAIN.
  3. VRF-GRE – this domain type will be associated with a VPRN service in the router that uses GRE tunnels and MP-BGP VPN-IPv4 to provide connectivity to the DC. In VSD, it will be attached to an existing Nuage L3-DOMAIN, when GRE is configured as tunnel-type for L3-DOMAINs.
  4. VRF-VXLAN – this domain type will be associated with a router R-VPLS service (connected to a VPRN with an evpn-tunnel VPLS interface) that uses VXLAN tunnels and EVPN to provide connectivity to the DC. In VSD, it will be attached to an existing Nuage L3-DOMAIN, when VXLAN is configured as the tunnel-type for L3-DOMAINs.

The domains will be configured in the config>service# context and assigned to each service.

# configure service vsd domain 
  - domain <name> [type {l2-domain|vrf-gre|vrf-vxlan|l2-domain-irb}] [create]
  - no domain <name>
 <name>               : [32 chars max]
 <create>             : keyword
 [no] description     - Set VSD Domain Description
 [no] shutdown        - Administratively enable/disable the domain
 

5.2.3.3.1. VSD-Domain Type L2-DOMAIN

L2-DOMAIN VSD-domains will be associated with VPLS services configured without a route-target and vxlan VNI. VSD will configure the route-target and VNI in the router VPLS service. Some considerations related to L2-DOMAINs are:

  1. ip-route-advertisement and allow-ip-int-bind commands are not allowed in this type of domain. An example of configuration for an L2-DOMAIN association is shown below:
*B:Dut>config>service# info
...
      vsd
            domain nuage_501 type l2-domain create
                description "nuage_501_l2_domain"
                no shutdown
            exit 
*B:Dut>config>service# vpls 501 
*B:Dut>config>service>vpls# info 
----------------------------------------------
            bgp
                route-distinguisher 192.0.2.2:52
                vsi-import “policy-1”
                vsi-export “policy-1”
            exit
            bgp-evpn
            exit
            sap 1/1/1:501 create
            exit
            spoke-sdp 10:501 create
                no shutdown
            exit
            vsd-domain "nuage_501"
            no shutdown
----------------------------------------------
  1. The VSD will push a dynamic vxlan vni and route-target that the router will add to the VPLS service. For the route-target, the system will check whether the VPLS service has a configured policy:
    1. If there is no vsi-import/export policy, the received dynamic route-target will be added in the vpls>bgp context, and will be used for all the BGP families in the service.
    2. If there is a vsi-import/export policy, the dynamic route-target will be added to the policy, in an auto-created community that will be shown with the following format: “_VSD_svc-id”. That community will be added to dynamically created entries 1000 and 2000 in the first policy configured in the service vsi-import and vsi-export commands. This allows the user to allocate entries before entries 1000 and 2000 in case other modifications have to be made (user entries would have an action next-entry). An example of the auto-generated entries is shown below:
*A:PE# show router policy "policy-1" 
    entry 900 # manual entry
        from
          as-path “null"  
          family evpn
        exit
        action next-entry
          local-preference 500
        exit
    exit
    entry 1000 # automatic VSD-generated entry
        from
            community "_VSD_1"
            family evpn
        exit
        action accept
        exit
    exit
    entry 2000 # automatic VSD-generated entry
        from
            family evpn
        exit
        action accept
            community add "_VSD_1"
        exit
    exit

5.2.3.3.2. VSD-Domain Type L2-DOMAIN-IRB

L2-DOMAIN-IRB VSD-domains will be associated with R-VPLS services configured without a static route-target and vxlan VNI. VSD will configure the dynamic route-target and VNI in the router VPLS service. The same considerations described for L2-DOMAINs apply to L2-DOMAIN-IRB domains with one exception: allow-ip-int-bind is now allowed.

5.2.3.3.3. VSD-Domain Type VRF-GRE

VRF-GRE VSD-domains will be associated with VPRN services configured without a static route-target. In this case, the VSD will push a route-target that the router will add to the VPRN service. The system will check whether the VPRN service has a configured policy:

  1. If there is no vrf-import policy, the received dynamic route-target will be added in the vprn> context.
  2. If there is a vrf-import policy, the dynamic route-target will be added to the policy, in an auto-created community that will be shown with the following format: “VSD_svc-id” in a similar way as in L2-DOMAINs.
    Note:

    In cases where a vrf-import policy is used, the user will provision the WAN route-target statically in a vrf-export policy. This route-target will also be used for the routes advertised to the DC.

An example of the auto-generated entry is shown below:

*A:PE# show router policy "policy-1" 
    entry 1000 # automatic VSD-generated entry
        from
            community "_VSD_1"
            family vpn-ipv4
        exit
        action accept
        exit
    exit

5.2.3.3.4. VSD-Domain Type VRF-VXLAN

VRF-VXLAN VSD-domains will be associated with R-VPLS services configured without a static route-target and vxlan VNI. VSD will configure the dynamic route-target and VNI in the router VPLS service. Some considerations related to VRF-VXLAN domains are:

  1. ip-route-advertisement, allow-ip-int-bind, as well as the VPRN evpn-tunnel commands are now required for this type of VSD-domain. An example of configuration for a VRF-VXLAN association is shown below:
*A:Dut>config>service# info 
<snip>
     vsd
            domain L3Domain-1 type vrf-vxlan create
                description "L3Domain-example"
                no shutdown
            exit
*A:Dut>config>service# vpls 20003 
*A:Dut>config>service>vpls# info 
----------------------------------------------
            allow-ip-int-bind
            bgp
                route-distinguisher 65000:20003
            exit
            bgp-evpn
                ip-route-advertisement
            exit
            stp
                shutdown
            exit
            vsd-domain "L3Domain-1"
            no shutdown
----------------------------------------------
*A:sr7L2-47-PE4# configure service vprn 20002 
*A:sr7L2-47-PE4>config>service>vprn# info 
----------------------------------------------
            route-distinguisher 65000:20002
            auto-bind-tunnel
                resolution-filter
                resolution-filter gre ldp rsvp
            vrf-target target:10:10
            interface "toDC" create
                vpls "vpls-20003"
                    evpn-tunnel
                exit
            exit
            no shutdown
  1. The VSD will push a dynamic vxlan VNI and route-target that the router will add to the VPLS service. For the route-target, the system will check whether the VPLS service has a configured policy and will push the route-target either in the service context or the vsi-import/export policies, as described in the section for L2-DOMAINs.

The following commands help show the association between the 7750 SR, 7450 ESS, and 7950 XRS router services and VSD-domains, as well as statistics and configuration errors sent/received to/from VSD.

*A:Dut# show service service-using vsd 
===========================================
Services-using VSD Domain
===========================================
Svc Id     Domain
-------------------------------------------
501        nuage_501
200001     MyL2Domain
20003      MyL3Domain
-------------------------------------------
Number of services using VSD Domain: 3
===========================================
*A:Dut# show service vsd domain "MyL3Domain" 
 
===============================================================================
VSD Information
===============================================================================
Name               : MyL3Domain
Description        : MyL3Domain-example
Type               : vrfVxlan                     Admin State   : inService
Last Error To Vsd  : (Not Specified)
Last Error From Vsd: (Not Specified)
 
Statistics
-------------------------------------------------------------------------------
Last Cfg Chg Evt   : 02/06/2015 01:28:30          Cfg Chg Evts  : 671
Last Cfg Update    : 02/06/2015 02:58:41          Cfg Upd Rcvd  : 3
Last Cfg Done      : 02/06/2015 02:58:41          
Cfg Success        : 667                          Cfg Failed    : 0
===============================================================================
 
*A:Dut# show service vsd domain "MyL3Domain" association 
 
============================================================
Service VSD Domain
============================================================
Svc Id      Svc Type  Domain Type    Domain Admin   Origin
------------------------------------------------------------
20003       vpls      vrfVxlan       inService      manual
------------------------------------------------------------
Number of entries: 1
============================================================

5.2.3.4. Fully-Dynamic VSD Integration Model

In the Static-Dynamic VSD integration model, the VPLS/VPRN service, as well as most of the parameters, must be provisioned "statically" through usual procedures (CLI, SNMP, and so on). VSD will "dynamically" send the parameters that are required for the attachment of the VPLS/VPRN service to the L2/L3 domain in the Data Center. In the Fully-Dynamic VSD integration model, the entire VPLS/VPRN service configuration will be dynamically driven from VSD and no static configuration is required. Through the existing XMPP interface, the VSD will provide the 7750 SR, 7450 ESS, and 7950 XRS routers with a handful of parameters that will be translated into a service configuration by a python-script. This python-script provides an intermediate abstraction layer between VSD and the router, translating the VSD data model into a 7750 SR, 7450 ESS, or 7950 XRS CLI data model.

In this Fully-Dynamic integration model, the DC and DC GW management entities are usually the same. The DC GW operator will provision the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN and the DC. VSD will push the required parameters so that the router can create the service completely and get it attached to an existing DC domain.

The workflow of the Fully-Dynamic integration model is shown in Figure 154.

Figure 154:  Fully-Dynamic VSD Integration Model Workflow 

The Fully-Dynamic VSD integration model can be summarized in the steps shown in Figure 154 and described in the following procedure.

  1. The 7750 SR, 7450 ESS, or 7950 XRS administrator only needs to provision parameters required for connectivity to the VSD, a service-range, and configure the python script/policy in the system. Provisioning of service parameters is not required.
    The service-range defines the service identifiers to include for VSD provisioned services and, once configured, they are protected from CLI changes. The vsd-policy defines the script to be used:
    *A:PE1>config>python# info 
    ----------------------------------------------
            python-script "l2-domain_services" create
                primary-url "ftp://1.1.1.1/cf2/l2domain_service.py"
                no shutdown
            exit
            python-policy "py-l2" create
                description "Python script to create L2 domains"
                vsd script "l2-domain_services"
            exit
    ----------------------------------------------
     
    *A:PE1>config>service>vsd# info 
    ----------------------------------------------
      service-range 64000 to 65000
    ----------------------------------------------
    When the router boots up or the gateway configuration is changed, the router sends a message to the VSD indicating its capabilities:
    1. System-ID
    2. Name and gateway type
    The VSD uses this information to register the router on its list of router GWs.
    When registered, the VSD and router exchange messages where the VSD communicates its list of service-names and their domain-type to the router. Based on this list, the router sends an XMPP IQ message to request the configuration of a specified service.
    The 7750 SR, 7450 ESS, or 7950 XRS router will periodically audit the VSD and request a “DIFF” list of Full-Dynamic VSD domains. The VSD keeps a “DIFF” list of domains, that contains the Fully-Dynamic domain names for which the VSD has not received an IQ request from the router for a long time.
    The 7750 SR, 7450 ESS, or 7950 XRS CLI user can also audit the VSD to get the DIFF list, or even the “FULL” list of all the domains in the VSD. The following command triggers this audit: tools perform service vsd fd-domain-sync {full | diff}.
  2. Concurrently at the VSD, the Cloud Service Provider administrator will assign WAN resources to Enterprises defined in the VSD. In this example, a VRF-GRE domain will be assigned to Enterprise-1.
  3. Each Enterprise administrator will have visibility of their own assigned WAN resource and will attach it to an existing DC Domain, assuming that both the DC domain and WAN resource are compatible. For instance, a VRF-GRE domain can only be attached to an L3 domain in the DC that uses GRE as transport.
  4. When the Enterprise administrator attaches the service requested by the 7750 SR, 7450 ESS, or 7950 XRS router to the DC domain, the VSD will send the required configuration parameters for that service to the DC GW through the XMPP channel in an IQ Service message, including the following information:
    1. Service name and service type, where the type can be:
      1. L2-DOMAIN
      2. L2-DOMAIN-IRB
      3. VRF-GRE
      4. VRF-VXLAN
    2. Configuration type— Static (for Static-Dynamic model) or Dynamic (for Fully-Dynamic model).
    3. Internal route-target (RT-i) — Used to export/import the BGP routes sent/received from/to the DC route-reflector.
    4. External route-target (RT-e) — Used to export/import the BGP routes sent/received from/to the WAN route-reflector. The value can be the same as the RT-i.
    5. VNI (VXLAN Network Identifier) — Used to configure the EVPN-VXLAN VPLS service on the router (if the domain type is L2-DOMAIN, L2-DOMAIN-IRB, or VRF-VXLAN).
    6. Metadata — A collection of 'opaque' <key=value> pairs including the rest of the service parameters required for the service configuration at the router.
      Note:

      The keys or values do not need to follow any specific format. The python script interprets and translates them into the router data model.

    7. Python-policy— Used by the router to find the Python script that will translate the VSD parameters into configuration.
  5. When the 7750 SR, 7450 ESS, or 7950 XRS router receives the IQ Service message, it builds a string with all the parameters and passes it to the Python module. The Python module is responsible for creating and activating the service, and, therefore, provides connectivity between the tenant domain and the WAN.
    Note:

    The python-script cannot access all the CLI commands and nodes in the system. A white-list of nodes and commands is built and Python will only have access to those nodes/commands. The list can be displayed using the following command: tools dump service vsd-services command-list.

    In addition to the white-list, the user can further restrict the allowed CLI nodes to the VSD by using a separate CLI user for the XMPP interface, and associating that user to a profile where the commands are limited. The CLI user for the XMPP interface is configurable:
    config>system>security>cli-script>authorization>
       vsd   
    [no] cli-user <username>
    When the system executes a python-script for VSD commands, the vsd cli-user profile is checked to allow the resulting commands. A single CLI user is supported for VSD, therefore, the operator can configure a single 'profile' to restrict (even further than the whitelist) the CLI commands that can be executed by the VSD Python scripts.
    No cli-user means that the system will not perform any authorization checks and the VSD scripts can access any commands that are supported in the white-list.

5.2.3.4.1. Python Script Implementation Details

A python-script provides an intermediate abstraction layer between VSD and the router, translating the VSD data model into the 7750 SR, 7450 ESS, or 7950 XRS router CLI data model. VSD will use metadata key=value parameters to provision service-specific attributes on the 7750. The XMPP messages get to the 7750 and are passed transparently to the Python module so that the CLI is generated and the service created.

Note:

The CLI generated by the python-script is not saved in the configuration file; it can be displayed by running the info include-dynamic command on the service contexts. The configuration generated by the python-script is protected and cannot be changed by a CLI user.

The following example shows how the python-script works for a VSD generated configuration:

  1. The following configuration is added to the 7750 SR. In this case, py-l2 is the python-policy received from VSD that will call the l2domain_service.py python script:
*A:PE1>config>python# info 
----------------------------------------------
        python-script "l2-domain_services" create
            primary-url "ftp://1.1.1.1/cf2/l2domain_service.py"
            no shutdown
        exit
        python-policy "py-l2" create
            description "Python script to create L2 domains"
            vsd script "l2-domain_services"
        exit
----------------------------------------------
*A:PE1>config>service>vsd# info 
----------------------------------------------
  service-range 64000 to 65000
----------------------------------------------
  1. VSD will send metadata containing the service parameters. This opaque parameter string will be passed to the python script and is composed of tag=value pairs, with the following format:
  2. In addition, other information provided by the VSD, (domain, vni, rt-i, rt-e, and service type) is bundled with the metadata string and passed to the python script. For example:
{'rt': 'target:64000:64000', 'rte': 'target:64000:64000', 'domain': 'L2-DOMAIN-
1', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1, sap=1/1/
10:3000'}

The user should consider the following:

  1. The python script is solely responsible for generating the configuration; no configuration aspects of the Static-Dynamic model are used. The python script must be written in a manner similar to those used by RADIUS Dynamic Business Services. Currently, RADIUS Dynamic Business Services and the Fully-Dynamic VSD model are mutually exclusive, one or the other can operate on the same system, but not both at the same time.
  2. The following scripts must be defined in order to set up, modify, revert, and tear down the configuration for a service: setup_script(), modify_script(), revert_script(), and teardown_script(). These names must always be the same in all scripts. The revert_script() is only required if the modify_script() is defined, the latter being optional.
  3. When the configuration for a new domain name is received from the VSD, the metadata and the VSD parameters are concatenated into a single dictionary and setup_script() is called. Within the python script:
    1. The VSD UI parameters are referenced as vsdParams['rt'], vsdParams ['domain'], and so on.
    2. The metadata parameters are referenced as vsdParams['metadata'].
  4. When the startup script is executed, the config>service>vsd>domain is created outside the script context before running the actual script. The teardown script will remove the vsd domain.
    1. If a specified setup_script() fails, the teardown_script() is invoked.
  5. When subsequent configuration messages are received from the VSD, the new parameter list is generated again from the VSD message and compared to the last parameter list that was successfully executed.
    1. If the two strings are identical, no action is taken.
    2. If there is a difference between the strings, the modify_script() function is called.
    3. If the modify_script() fails, the revert_script() is invoked. The teardown_script() is invoked if the revert_script() fails or does not exist.
  6. The python-policy is always present in the attributes received from VSD; if the VSD user does not include a policy name, VSD will include 'default' as the python-policy. Hence, care must be taken to ensure that the 'default' policy is always configured in the 7750.
  7. If the scripts are incorrect, teardown and modify procedures could leave orphaned objects. An admin mode (enable-vsd-config) is available to enable an administrator to clean up these objects; it is strictly meant for cleaning orphaned objects only.
    Note:

    The CLI configured services cannot be modified when the user is in enable-vsd-config mode.

  8. Unless the CLI user enters the enable-vsd-config mode, changes of the dynamic created services are not allowed in the CLI. However, changes through SNMP are allowed.
  9. The command tools perform service vsd evaluate-script is introduced to allow the user to test their setup and to modify and tear down python scripts in a lab environment without the need to be connected to a VSD. The successful execution of the command for action setup will create a vsd domain and the corresponding configuration, just as the system would do when the parameters are received from VSD.

The following example shows the use of the tools perform service vsd evaluate-script command:

*A:PE1# tools perform service vsd evaluate-script domain-name "L2-DOMAIN-5" type l2-
domain action setup policy "py-l2" vni 64000 rt-i target:64000:64000 rt-
e target:64000:64000 metadata "rd=1:1, sap=1/1/10:3000"   
 
Success
 

The following example output shows a python-script that can set up or tear down L2-DOMAINs.

*A:PE1# show python python-script "l2-domain_services" source-in-use 
 
===============================================================================
Python script "l2-domain_services"
===============================================================================
Admin state   : inService
Oper state    : inService
Primary URL   : ftp://1.1.1.1/timos86/cses-V71/cf2/l2domain_service.py
Secondary URL : (Not Specified)
Tertiary URL  : (Not Specified)
Active URL    : primary
 
-------------------------------------------------------------------------------
Source (dumped from memory)
-------------------------------------------------------------------------------
      1 from alc import dyn
      2 
      3 def setup_script(vsdParams):
      4     
      5     print ("These are the VSD params: " + str(vsdParams))
      6     servicetype = vsdParams.get('servicetype')
      7     vni = vsdParams.get('vni')
      8     metadata = vsdParams['metadata']
      9     print ("VSD metadata: " + str(metadata))
     10     metadata = dict(e.split('=') for e in metadata.split(','))
     11     print ("Modified metadata: " + str(metadata))
     12     vplsSvc_id = dyn.select_free_id("service-id")
     13     print ("this is the free svc id picked up by the system: " + vplsSvc_id)
     14      
     15     if servicetype == "L2DOMAIN":
     16         rd = metadata['rd']
     17         sap_id = metadata[' sap']
     18         print ('servicetype, VPLS id, rd, sap:', servicetype, vplsSvc_id, rd
, sap_id)
     19         dyn.add_cli("""
     20           configure service
     21               vpls %(vplsSvc_id)s name evi%(vplsSvc_id)s customer 1 create
     22                   description vpls%(vplsSvc_id)s
     23                   bgp
     24                       route-distinguisher %(rd)s
     25                       route-target %(rt)s
     26                   exit
     27                   vxlan instance 1 vni %(vni)s create
     28                   exit
     29                   bgp-evpn 
     30                       evi %(vplsSvc_id)s
     31                       vxlan bgp 1 vxlan-instance 1
     32                           no shutdown
     33                       exit
     34                   exit
     35                   sap %(sap_id)s create
     36                   exit
     37                   no shutdown
     38                   exit
     39               exit
     40           exit
     41         """ % {'vplsSvc_id' : vplsSvc_id, 'vni' : vsdParams['vni'], 'rt' : v
sdParams['rt'], 'rd' : metadata['rd'], 'sap_id' : sap_id})
     42         # L2DOMAIN returns setupParams: vplsSvc_id, servicetype, vni, sap
     43         return {'vplsSvc_id' : vplsSvc_id, 'servicetype' : servicetype, 'vni
' : vni, 'sap_id' : sap_id}
     44 
     45 
     46 #---------------------------------------------------------------------------
     47 
     48 def teardown_script(setupParams):
     49     print ("These are the teardown_script setupParams: " + str(setupParams))
     50     servicetype = setupParams.get('servicetype')   
     51     if servicetype == "L2DOMAIN":
     52       dyn.add_cli("""
     53         configure service
     54             vpls %(vplsSvc_id)s
     55                 no description 
     56                 bgp-evpn 
     57                     vxlan 
     58                         shutdown
     59                     exit
     60                     no evi
     61                     exit
     62                 no vxlan instance 1 vni %(vni)s
     63                 bgp
     64                    no route-distinguisher 
     65                    no route-target 
     66                 exit
     67                 no bgp
     68                 no bgp-evpn
     69                 sap %(sap_id)s
     70                    shutdown
     71                    exit
     72                 no sap %(sap_id)s
     73                 shutdown
     74                 exit
     75                 no vpls %(vplsSvc_id)s
     76             exit
     77         exit                  
     78       """ % {'vplsSvc_id' : setupParams['vplsSvc_id'], 'vni' : setupParams['
vni'], 'sap_id' : setupParams['sap_id']})
     79       return setupParams
     80 
     81 
     82 d = { "script" : (setup_script, None, None, teardown_script) }
     83 
     84 dyn.action(d)
===============================================================================

For instance, assuming that the VSD sends the following:

{'rt': 'target:64000:64000', 'rte': 'target:64000:64000', 'domain': 'L2-DOMAIN-
5', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1, sap=1/1/
10:3000 '}

The system will execute the setup script as follows:

123 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup
"dyn-script req=setup: L2-DOMAIN-5
  state=init->waiting-for-setup
"
 
124 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup
"dyn-script req=setup: L2-DOMAIN-5
  state=waiting-for-setup->generating-setup
"
 
125 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base Python Output
"Python Output: l2-domain_services
These are the VSD params: {'rt': 'target:64000:64000', 'rte': 'target:64000:6400
0', 'domain': '', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1
, sap=1/1/10:3000 '}
VSD metadata: rd=1:1, sap=1/1/10:3000 
Modified metadata: {'rd': '1:1', ' sap': '1/1/10:3000 '}
this is the free svc id picked up by the system: 64000
('servicetype, VPLS id, rd, sap:', 'L2DOMAIN', '64000', '1:1', '1/1/10:3000 ')
"
Success
 
126 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base Python Result
"Python Result: l2-domain_services
"
 
127 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup
"dyn-script req=setup: L2-DOMAIN-5
  state=generating-setup->executing-setup
"
 
128 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script cli 1/1
"dyn-script cli 1/1: script:L2-DOMAIN-5(cli 698 dict 0->126)
 
          configure service
              vpls 64000 name “evi64000” customer 1 create
                  description vpls64000
                  bgp
                      route-distinguisher 1:1
                      route-target target:64000:64000
                  exit
                  vxlan instance 1 vni 64000 create
                  exit
                  bgp-evpn 
                      evi 64000
                      vxlan bgp 1 vxlan-instance 1
                          no shutdown
                      exit
                  exit
                  sap 1/1/10:3000  create
                  exit
                  no shutdown
                  exit
              exit
          exit
        "
 
143 2015/06/16 23:35:40.23 UTC MINOR: DEBUG #2001 Base dyn-script req=setup
"dyn-script req=setup: L2-DOMAIN-5
  state=executing-setup->established"

5.2.3.4.2. Provisioning Filters using the VSD Fully Dynamic Model

IP, IPv6, and MAC filters can be configured from the VSD within the context of the Fully Dynamic XMPP provisioning model for VPRN and VPLS services.

The VSD filters or filter entries are intended for use in two DC environments:

  1. Dedicated DC GW Model
    The DC GW services and filter policies in this DC environment are completely owned and self-managed by the Nuage VSD. In this model, the filter cannot be changed and/or deleted by any management or policy interface other than the VSD; changes are not saved in the configuration file.
    To enable the VSD to configure a filter, the python setup script must contain a config filter ip/ipv6/mac-filter _tmnx_vsd_<filter_id> create statement. The VSD exclusively manages the removal and change of such filters.
    The following excerpt shows a sample setup script to create a filter.
    def setup_script(vsdParams):
    <snip>
           filter_id = metadata[' filter'] 
    <snip>
           dyn.add_cli("""
                  config filter ip-filter _tmnx_vsd_%(filter_id)s create
           entry 10 create
                  match protocol tcp
                        dst-port eq 80
                  exit 
                 action
                   forward
                 exit
    <snip>
  2. PE/BNG + DC GW Combination Model
    The filter and point of embedding insertion in this DC environment is owned by a WAN controller. In this model, the entries in the embedded filter are populated by the VSD.
    The WAN controller creates a filter and the embedding point (through a management interface other than the VSD) by using the config filter ip/ipv6/mac-filter <id> embed-filter vsd _tmnx_vsd_<filter-id> command. When this command is run, a filter with the name _tmnx_vsd_<filter-id> will be auto-generated; the python scripts can use that name to create entries driven by the VSD.

5.2.4. Layer 2 Multicast Optimization for VXLAN (Assisted-Replication)

The Assisted-Replication feature for IPv4 VXLAN tunnels (both Leaf and Replicator functions) is supported in compliance with the non-selective mode described in IETF Draft draft-ietf-bess-evpn-optimized-ir.

The Assisted-Replication feature is a Layer 2 multicast optimization feature that helps software-based PE and NVEs with low-performance replication capabilities to deliver broadcast and multicast Layer 2 traffic to remote VTEPs in the VPLS service.

The EVPN and proxy-ARP/ND capabilities can reduce the amount of broadcast and unknown unicast in the VPLS service; ingress replication is sufficient for most use cases in this scenario. However, when multicast applications require a significant amount of replication at the ingress node, software-based nodes struggle due to their limited replication performance. By enabling the Assisted-Replication Leaf function on the software-based SR-series router, all the broadcast and multicast packets are sent to a 7x50 router configured as a Replicator, which replicates the traffic to all the VTEPs in the VPLS service on behalf of the Leaf. This guarantees that the broadcast or multicast traffic is delivered to all the VPLS participants without any packet loss caused by performance issues.

The Leaf or Replicator function is enabled per VPLS service by the config>service>vpls>vxlan>assisted-replication {replicator|leaf} command. In addition, the Replicator requires the configuration of an Assisted-Replication IP (AR-IP) address. The AR-IP loopback address indicates whether the received VXLAN packets have to be replicated to the remote VTEPs. The AR-IP address is configured using the config>service>system>vxlan>assisted-replication-ip <ip-address> command.

Based on the assisted-replication {replicator|leaf} configuration, the SR-series router can behave as a Replicator (AR-R), Leaf (AR-L), or Regular Network Virtualization Edge (RNVE) router. An RNVE router does not support the Assisted-Replication feature. Because it is configured with no assisted replication, the RNVE router ignores the AR-R and AR-L information and replicates to its flooding list where VTEPs are added based on the regular ingress replication routes.

5.2.4.1. Replicator (AR-R) Procedures

An AR-R configuration is shown in the following example.

*A:PE-2>config>service>system>vxlan# info 
----------------------------------------------
            assisted-replication-ip 10.2.2.2
----------------------------------------------
*A:PE-2>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication replicator
            exit
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
<snip>
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with tunnel-type = AR, type (T) = AR-R, and tunnel-id = originating-ip = next-hop = assisted-replication-ip (IP address 10.2.2.2 in the preceding example). In addition to the AR route, the AR-R sends a regular IR route if ingress-repl-inc-mcast-advertisement is enabled.

Note:

You should disable the ingress-repl-inc-mcast-advertisement command if the AR-R does not have any SAP or SDP-bindings and is used solely for Assisted-Replication functions.

The AR-R builds a flooding list composed of ACs (SAPs and SDP-bindings) and VXLAN tunnels to remote nodes in the VPLS. All objects in the flooding list are broadcast/multicast (BM) and unknown unicast (U) capable. The following sample output of the show service id vxlan command shows that the VXLAN destinations in the flooding list are tagged as “BUM”.

*A:PE-2# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: replicator
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
192.0.2.3                               4000        0           BUM   Up    No
192.0.2.5                               4000        0           BUM   Up    No
192.0.2.6                               4000        0           BUM   Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

When the AR-R receives a BUM packet on an AC, the AR-R forwards the packet to its flooding list (including the local ACs and remote VTEPs).

When the AR-R receives a BM packet on a VXLAN tunnel, it checks the IP DA of the underlay IP header and performs the following BM packet processing.

Note:

The AR-R function is only relevant to BM packets; it does not apply to unknown unicast packets. If the AR-R receives unknown unicast packets, it sends them to the flooding list, skipping the VXLAN tunnels.

  1. If the destination IP matches its AR-IP, the AR-R forwards the BM packet to its flooding list (ACs and VXLAN tunnels). The AR-R performs source suppression to ensure that the traffic is not sent back to the originating Leaf.
  2. If the destination IP matches its regular VXLAN termination IP (IR-IP), the AR-R skips all the VXLAN tunnels from the flooding list and only replicates to the local ACs. This is the default Ingress Replication (IR) behavior.

5.2.4.2. Leaf (AR-L) procedures

An AR-L is configured as shown in the following example.

A:PE-3>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication leaf replicator-activation-time 30
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
                mpls
                    shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:4000 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with a tunnel-type = IR, type (T) = AR-L and tunnel-id = originating-ip = next-hop = IR-IP (IP address terminating VXLAN normally, either system-ip or vxlan-src-vtep address).

The AR-L builds a single flooding list per service but controlled by the BM and U flags. These flags are displayed in the following show service id vxlan command sample output.

A:PE-3# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: leaf      Replicator-Activation-Time: 30
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
10.2.2.2                                 4000        0           BM    Up    No
10.4.4.4                                 4000        0           -     Up    No
192.0.2.2                               4000        0           U     Up    No
192.0.2.5                               4000        0           U     Up    No
192.0.2.6                               4000        0           U     Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 5
-------------------------------------------------------------------------------
===============================================================================

The AR-L creates the following VXLAN destinations when it receives and selects a Replicator-AR route or the Regular-IR routes.

  1. A VXLAN destination to each remote PE that sent an IR route. These bindings have the U flag set.
  2. A VXLAN destination to the selected AR-R. These bindings have only the BM flag set; the U flag is not set.
  3. The non-selected AR-Rs create a binding with flag “-” (in the CPM) that is displayed by the show service id vxlan command. Although the VXLAN destinations to non-selected AR-Rs do not carry any traffic, the destinations count against the total limit and must be considered when accounting for consumed VXLAN destinations in the router.

The BM traffic is only sent to the selected AR-R, whereas the U (unknown unicast) traffic is sent to all the destinations with the U flag.

The AR-L performs per-service load-balancing of the BM traffic when two or more AR-Rs exist in the same service. The AR Leaf creates a list of candidate PEs for each AR-R (ordered by IP and VNI; candidate 0 being the lowest IP and VNI). The replicator is selected out of a modulo function of the service-id and the number of replicators, as shown in the following sample output.

A:PE-3# show service id 4000 vxlan assisted-replication replicator 
===============================================================================
Vxlan AR Replicator Candidates
===============================================================================
VTEP Address           Egress VNI     In Use  In Candidate List Pending Time
-------------------------------------------------------------------------------
10.2.2.2                4000           yes     yes               0
10.4.4.4                4000           no      yes               0
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

A change in the number of Replicator-AR routes (for example, if a route is withdrawn or a new route appears) affects the result of the hashing, which may cause a different AR-R to be selected.

Note:

An AR-L waits for the configured replicator-activation-time before sending the BM packets to the AR-R. In the interim, the AR-L uses regular ingress replication procedures. This activation time allows the AR-R to program the Leaf VTEP. If the timer is zero, the AR-R may receive packets from a not-yet-programmed source VTEP, in which case it will discard the packets.

The following list summarizes other aspects of the AR-L behavior.

  1. When a Leaf receives a BM packet on an AC, it sends the packet to its flood list that includes access SAP or SDP-bindings and VXLAN destinations with BM or BUM flags. If a single AR-R is selected, only a VXLAN destination will include the BM flags.
  2. Control plane-generated BM packets, such as ARP/ND (when proxy-ARP/ND is enabled) or Eth-CFM, follow the behavior of regular data plane BM packets.
  3. When a Leaf receives an unknown unicast packet on an AC, it sends the packet to the flood-list, skipping the AR destination because the U flag is set to 0. To avoid packet re-ordering, the unknown unicast packets do not go through the AR-R.
  4. When a Leaf receives a BUM packet on an overlay tunnel, it forwards the packet to the flood list, skipping the VXLAN tunnels (that is, the packet is sent to the local ACs and never to a VXLAN tunnel). This is the default IR behavior.
  5. When the last Replicator-AR route is withdrawn, the AR-L removes the AR destination from the flood list and falls back to ingress replication.

Figure 155 shows the expected replication behavior for BM traffic when received at the access on an AR-R, AR-L, or RNVE router. Unknown unicast follows regular ingress replication behavior regardless of the role of the ingress node for the specific service.

Figure 155:  AR BM Replication Behavior for a BM Packet 

5.2.4.3. Assisted-Replication Interaction with Other VPLS Features

The Assisted-Replication feature has the following limitations.

  1. The following features are not supported on the same service where the Assisted-Replication feature is enabled.
    1. Aggregate QoS per VNI
    2. VXLAN IPv6 transport
    3. IGMP-snooping
  2. Assisted-Replication Leaf and Replicator functions are mutually exclusive within the same VPLS service.
  3. The Assisted-Replication feature is supported with IPv4 non-system-ip VXLAN termination. However, the configured assisted-replication-ip (AR-IP) must be different from the tunnel termination IP address.
  4. The AR-IP address must be a /32 loopback interface on the base router.
  5. The Assisted-Replication feature is only supported in EVPN-VXLAN services (VPLS with bgp-evpn vxlan enabled). Although services with a combination of EVPN-MPLS and EVPN-VXLAN are supported, the Assisted-Replication configuration is only relevant to the VXLAN.

5.2.5. DC GW Policy Based Forwarding/Routing to an EVPN ESI (Ethernet Segment Identifier)

The Nuage VSP (Virtual Services Platform) supports a service chaining function that ensures traffic traverses a number of services (also known as Service Functions) between application hosts (FW, LB, NAT, IPS/IDS, and so on.) if the operator meeds to do so. In the DC, tenants want the ability to specify these functions and their sequence, so that services can be added or removed without requiring changes to the underlying application.

This service chaining function is built based on a series of policy based routing/forwarding redirecting rules that are automatically coordinated and abstracted by the Nuage VSD (Virtual Services Directory). From a networking perspective, the packets are 'hop-by-hop' redirected based on the location of the corresponding SF (Service Function) in the DC fabric. The location of the SF is specified by its VTEP and VNI and is advertised by BGP-EVPN along with an Ethernet Segment Identifier (ESI) that is uniquely associated with the SF.

Refer to the Nuage VSP documentation for more information about the Nuage Service Chaining solution.

The 7750 SR, 7450 ESS, or 7950 XRS can be integrated as the first hop in the chain in a Nuage DC. This service chaining integration is intended to be used as described in the following three use-cases.

5.2.5.1. Policy Based Forwarding in VPLS Services for Nuage Service Chaining Integration in L2-Domains

Figure 156 shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on VPLS services. In this example, the DC GW, PE1, is connected to an L2-DOMAIN that exists in the DC and must redirect the traffic to the Service Function SF-1. The regular Layer 2 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

Figure 156:  PBF to ESI Function 

An operator must configure a PBF match/action filter policy entry in an IPv4 or MAC ingress access or network filter deployed on a VPLS interface using CLI/SNMP/NETCONF management interfaces. The PBF target is the first service function in the chain (SF-1) that is identified by an Ethernet Segment Identifier.

In the example shown in Figure 156, the PBF filter will redirect the matching packets to ESI 0x01 in VPLS-1.

Note:

Figure 156 represents ESI as ‘0x01’ for simplicity; in reality, the ESI is a 10-byte number.

As soon as the redirection target is configured and associated with the vport connected to SF-1, the Nuage VSC (Virtual Services Controller, or the remote PE3 in the example) advertises the location of SF-1 via an Auto-Discovery Ethernet Tag route (route type 1) per-EVI. In this AD route, the ESI associated with SF-1 (ESI 0x01) is advertised along with the VTEP (PE3's IP) and VNI (VNI-1) identifying the vport where SF-1 is connected. PE1 will send all the frames matching the ingress filter to PE3's VTEP and VNI-1.

Note:

When packets get to PE3, VNI-1 (the VNI advertised in the AD route) will indicate that a cut-through switching operation is needed to deliver the packets straight to the SF-1 vport, without the need for a regular MAC lookup.

The following filter configuration shows an example of PBF rule redirecting all the frames to an ESI.

A:PE1>config>filter>mac-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                action
                    forward esi ff:00:00:00:00:00:00:00:00:01 service-id 301
                exit
            exit

When the filter is properly applied to the VPLS service (VPLS-301 in this example), it will show 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported.

A:PE1# show filter mac 1 
===============================================================================
Mac Filter
===============================================================================
Filter Id   : 1                                Applied         : Yes
Scope       : Template                         Def. Action     : Forward
Entries     : 1                                Type            : normal
Description : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : Mac
-------------------------------------------------------------------------------
Entry       : 10                               FrameType       : Ethernet
Description : (Not Specified)
Log Id      : n/a                              
Src Mac     : Undefined
Dest Mac    : Undefined
Dot1p       : Undefined                        Ethertype       : Undefined
DSAP        : Undefined                        SSAP            : Undefined
Snap-pid    : Undefined                        ESnap-oui-zero  : Undefined
Match action: Forward (ESI) Active             
  ESI       : ff:00:00:00:00:00:00:00:00:01
  Svc Id    : 301                              
PBR Down Act: Forward (entry-default)          
Ing. Matches: 3 pkts
Egr. Matches: 0 pkts
===============================================================================
 
A:PE1# show service id 301 es-pbr 
===============================================================================
L2 ES PBR
===============================================================================
ESI                           Users      Status
                                         VTEP:VNI
-------------------------------------------------------------------------------
ff:00:00:00:00:00:00:00:00:01 1          Active
                                         192.0.2.72:7272
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

Details of the received AD route that resolves the filter forwarding are shown in the following 'show router bgp routes' command.

A:PE1# show router bgp routes evpn auto-
disc esi ff:00:00:00:00:00:00:00:00:01                                    
===============================================================================
 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup
 Origin codes  : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
 
u*>i  192.0.2.72:100      ff:00:00:00:00:00:00:00:00:01 192.0.2.72
      0                                                 VNI 7272
 
-------------------------------------------------------------------------------
Routes : 1
=============================================================

This AD route, when used for PBF redirection, is added to the list of EVPN-VXLAN bindings for the VPLS service and shown as 'L2 PBR' type:

A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.72             301            1            Yes     Up           No
192.0.2.72             7272           0            No      Up           Yes
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

If the AD route is withdrawn, the binding will disappear and the filter will be inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

Note:

ES-based PBF filters can be applied only on services with the default bgp (vxlan) instance (instance 1).

5.2.5.2. Policy Based Routing in VPRN Services for Nuage Service Chaining Integration in L2-DOMAIN-IRB Domains

Figure 157 shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on L2-DOMAIN-IRB domains. In this example, the DC GW, PE1, is connected to an L2-DOMAIN-IRB that exists in the DC and must redirect the traffic to the Service Function SF-1 with IP address 10.10.10.1. The regular Layer 3 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

Figure 157:  PBR to ESI Function 

In this case, an operator must configure a PBR match/action filter policy entry in an IPv4 ingress access or network filter deployed on IES/VPRN interface using CLI, SNMP or NETCONF management interfaces. The PBR target identifies first service function in the chain (ESI 0x01 in Figure 157, identifying where the Service Function is connected and the IPv4 address of the SF) and EVPN VXLAN egress interface on the PE (VPRN routing instance and R-VPLS interface name). The BGP control plane together with ESI PBR configuration are used to forward the matching packets to the next-hop in the EVPN-VXLAN data center chain (through resolution to a VNI and VTEP). If the BGP control plane information is not available, the packets matching the ESI PBR entry will be, by default, forwarded using regular routing. Optionally, an operator can select to drop the packets when the ESI PBR target is not reachable.

The following filter configuration shows an example of a PBR rule redirecting all the matching packets to an ESI.

*A:PE1>config>filter>ip-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                match 
                    dst-ip 10.10.10.253/32
                exit 
                action
                    forward esi ff:00:00:00:00:21:5f:00:df:e5 sf-ip 10.10.10.1 vas-
interface "evi-301" router 300
                exit
                pbr-down-action-override filter-default-action
            exit 
----------------------------------------------

In this use case, the following are required in addition to the ESI: the sf-ip (10.10.10.1 in the example above), router instance (300), and vas-interface.

The sf-ip is used by the system to know which inner MAC DA it has to use when sending the redirected packets to the SF. The SF-IP will be resolved to the SF MAC following regular ARP procedures in EVPN-VXLAN.

The router instance may be the same as the one where the ingress filter is configured or may be different: for instance, the ingress PBR filter can be applied on an IES interface pointing at a VPRN router instances that is connected to the DC fabric.

The vas-interface refers to the R-VPLS interface name through which the SF can be found. The VPRN instance may have more than one R-VPLS interface, therefore, it is required to specify which R-VPLS interface to use.

When the filter is properly applied to the VPRN or IES service (VPRN-300 in this example), it will show 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported and the SF-IP resolved to a MAC address.

*A:PE1# show filter ip 1 
 
===============================================================================
IP Filter
===============================================================================
Filter Id    : 1                                Applied        : Yes
Scope        : Template                         Def. Action    : Forward
System filter: Unchained                        
Radius Ins Pt: n/a                              
CrCtl. Ins Pt: n/a                              
RadSh. Ins Pt: n/a                              
PccRl. Ins Pt: n/a                              
Entries      : 1                                
Description  : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry        : 10
Description  : (Not Specified)
Log Id       : n/a                              
Src. IP      : 0.0.0.0/0
Src. Port    : n/a
Dest. IP     : 10.16.0.253/32
Dest. Port   : n/a
Protocol     : Undefined                        Dscp           : Undefined
ICMP Type    : Undefined                        ICMP Code      : Undefined
Fragment     : Off                              Src Route Opt  : Off
Sampling     : Off                              Int. Sampling  : On
IP-Option    : 0/0                              Multiple Option: Off
TCP-syn      : Off                              TCP-ack        : Off
Option-pres  : Off                              
Egress PBR   : Undefined                        
Match action : Forward (ESI) Active
  ESI        : ff:00:00:00:00:21:5f:00:df:e5    
  SF IP      : 10.10.10.1
  VAS If name: evi-301                          
  Router     : 300                              
PBR Down Act : Forward (filter-default-action) Ing. Matches : 3 pkts (318 bytes)
Egr. Matches : 0 pkts
===============================================================================
 
*A:PE1# show service id 300 es-pbr 
===============================================================================
L3 ES PBR
===============================================================================
SF IP              ESI                                 Users Status
                   Interface                                 MAC
                                                             VTEP:VNI
-------------------------------------------------------------------------------
10.10.10.1         ff:00:00:00:00:21:5f:00:df:e5       1     Active
                   evi-301                                   d8:47:01:01:00:0a
                                                             192.0.2.71:7171
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
=================================================================================

In the FDB for the R-VPLS 301, the MAC address is associated with the VTEP and VNI specified by the AD route, and not by the MAC/IP route anymore. When a PBR filter with a forward action to an ESI and SF-IP (Service Function IP) exists, a MAC route is auto-created by the system and this route has higher priority that the remote MAC, or IP routes for the MAC (see section BGP and EVPN route selection for EVPN routes).

The following shows that the AD route creates a new EVPN-VXLAN binding and the MAC address associated with the SF-IP uses that 'binding':

*A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.71             301            0            Yes     Up           No
192.0.2.71             7171           1            No      Up           No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================
*A:PE1# show service id 301 fdb detail 
===============================================================================
Forwarding Database, Service 301
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
301       d8:45:ff:00:00:6a vxlan-1:                 EvpnS    06/15/15 21:55:27
                            192.0.2.69:301
301       d8:47:01:01:00:0a vxlan-1:                 EvpnS    06/15/15 22:32:56
                            192.0.2.71:7171
301       d8:48:ff:00:00:6a cpm                      Intf     06/15/15 21:54:12
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

For Layer 2, if the AD route is withdrawn or the SF-IP ARP not resolved, the filter will be inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

5.3. EVPN for MPLS Tunnels

This section provides information about EVPN for MPLS tunnels.

5.3.1. BGP-EVPN Control Plane for MPLS Tunnels

Table 79 lists all the EVPN routes supported in 7750 SR, 7450 ESS, or 7950 XRS SR OS and their usage in EVPN-VXLAN, EVPN-MPLS, and PBB-EVPN.

Note:

Route type 1 is not required in PBB-EVPN as per RFC 7623.

Table 79:  EVPN Routes and Usage 

EVPN Route

Usage

EVPN-VXLAN

EVPN-MPLS

PBB-EVPN

Type 1 - Ethernet Auto-Discovery route (A-D)

Mass-withdraw, ESI labels, Aliasing

(Only EVPN-VPWS)

Y

Type 2 - MAC/IP Advertisement route

MAC/IP advertisement, IP advertisement for ARP resolution

Y

Y

Y

Type 3 - Inclusive Multicast Ethernet Tag route

Flooding tree setup (BUM flooding)

Y

Y

Y

Type 4 - Ethernet Segment route

ES discovery and DF election

(Only EVPN-VPWS)

Y

Y

Type 5 - IP Prefix advertisement route

IP Routing

Y

Y

RFC 7432 describes the BGP-EVPN control plane for MPLS tunnels. If EVPN multi-homing is not required, two route types are needed to set up a basic EVI (EVPN Instance): MAC/IP Advertisement and the Inclusive Multicast Ethernet Tag routes. If multi-homing is required, the Ethernet Segment and the Auto-Discovery routes are also needed.

The route fields and extended communities for route types 2 and 3 are shown in Figure 145. BGP-EVPN Control Plane for VXLAN Overlay Tunnels The changes compared to their use in EVPN-VXLAN are described below.

EVPN Route Type 3 – Inclusive Multicast Ethernet Tag Route

As in EVPN-VXLAN, route type 3 is used for setting up the flooding tree (BUM flooding) for a specified VPLS service. The received inclusive multicast routes add entries to the VPLS flood list in the 7750 SR, 7450 ESS, and 7950 XRS. Ingress replication, p2mp mLDP, and composite tunnels are supported as tunnel types in route type 3 when BGP-EVPN MPLS is enabled

The following route values are used for EVPN-MPLS services:

  1. Route Distinguisher: taken from the RD of the VPLS service within the BGP context. The RD can be configured or derived from the bgp-evpn evi value.
  2. Ethernet Tag ID: 0.
  3. IP address length: always 32.
  4. Originating router's IP address: carries the system address (IPv4 only).
  5. PMSI attribute: the PMSI attribute can have different formats depending on the tunnel type enabled in the service.
    1. Tunnel type = Ingress replication (6)
      The route is referred to as an Inclusive Multicast Ethernet Tag IR (IMET-IR) route and the PMSI Tunnel Attribute (PTA) fields are populated as follows:
      1. Flags—Leaf not required.
      2. MPLS label—Carries the MPLS label allocated for the service in the high-order 20 bits of the label field.
        Unless bgp-evpn mpls ingress-replication-bum-label is configured in the service, the MPLS label used will be the same as that used in the MAC/IP routes for the service.
      3. Tunnel endpoint—Equal to the originating IP address.
    2. Tunnel type=p2mp mLDP (2)
      The route is referred to as an IMET-P2MP route and its PTA fields are populated as follows.
      1. Flags—Leaf not required.
      2. MPLS label—0.
      3. Tunnel endpoint—Includes the route node address and an opaque number. This is the tunnel identifier that the leaf-nodes will use to join the mLDP P2MP tree.
    3. Tunnel type=Composite tunnel (130)
      The route is referred to as an IMET-P2MP-IR route and its PTA fields are populated as follows.
      1. Flags—Leaf not required.
      2. MPLS label 1— 0.
      3. Tunnel endpoint identifier will include the following:
        MPLS label2—Non-zero, downstream allocated label (like any other IR label). The leaf-nodes will use the label to set up an EVPN-MPLS destination to the root and add it to the default-multicast list.
        mLDP tunnel identifier—The route node address and an opaque number. This is the tunnel identifier that the leaf-nodes will use to join the mLDP P2MP tree.

IMET-P2MP-IR routes are used in EVIs with a few root nodes and a significant number of leaf-only PEs. In this scenario, a combination of P2MP and IR tunnels can be used in the network, such that the root nodes use P2MP tunnels to send broadcast, Unknown unicast, and Multicast traffic but the leaf-PE nodes use IR to send traffic to the roots. This use-case is documented in IETF RFC 8317 and the main advantage it offers is the significant savings in P2MP tunnels that the PE/P routers in the EVI need to handle (as opposed to a full mesh of P2MP tunnels among all the PEs in an EVI).

In this case, the root PEs will signal a special tunnel type in the PTA, indicating that they intend to transmit BUM traffic using an mLDP P2MP tunnel but they can also receive traffic over an IR evpn-mpls binding. An IMET route with this special "composite" tunnel type in the PTA is called an IMET-P2MP-IR route and the encoding of its PTA is shown in Figure 158.

Figure 158:  Composite p2mp mLDP and IR Tunnels—PTA 

EVPN Route Type 2 - MAC/IP Advertisement Route

The 7750 SR, 7450 ESS, or 7950 XRS router generates this route type for advertising MAC addresses (and IP addresses if proxy-ARP/proxy-ND is enabled). The router generates MAC advertisement routes for the following:

  1. Learned MACs on SAPs or sdp-bindings—if mac-advertisement is enabled.
  2. Conditional static MACs—if mac-advertisement is enabled.
    Note:

    The unknown-mac-route is not supported for EVPN-MPLS services.

The route type 2 generated by a router uses the following fields and values:

  1. Route Distinguisher: Taken from the RD of the VPLS service within the BGP context. The RD can be configured or derived from the bgp-evpn evi value.
  2. Ethernet Segment Identifier (ESI): Zero for MACs learned from single-homed CEs and different from zero for MACs learned from multi-homed CEs.
  3. Ethernet Tag ID: 0.
  4. MAC address length: Always 48.
  5. MAC Address learned or statically configured.
  6. IP address and IP address length:
    1. It will be the IP address associated with the MAC being advertised with a length of 32 (or 128 for IPv6).
    2. In general, any MAC route without IP will have IPL=0 (IP length) and the IP will be omitted.
    3. When received, any IPL value not equal to zero, 32, or 128 will discard the route.
    4. MPLS Label 1: Carries the MPLS label allocated by the system to the VPLS service. The label value is encoded in the high-order 20 bits of the field and will be the same label used in the routes type 3 for the same service unless bgp-evpn mpls ingress-replication-bum-label is configured in the service.
  7. MPLS Label 2: 0.
  8. The MAC Mobility extended community: Used for signaling the sequence number in case of mac moves and the sticky bit in case of advertising conditional static MACs. If a MAC route is received with a MAC mobility ext-community, the sequence number and the 'sticky' bit are considered for the route selection.

When EVPN multi-homing is enabled in the system, two more routes are required. Figure 159 shows the fields in routes type 1 and 4 and their associated extended communities.

Figure 159:  EVPN Routes Type 1 and 4  

EVPN Route Type 1 - Ethernet Auto-discovery Route (AD route)

The 7750 SR, 7450 ESS, or 7950 XRS router generates this route type for advertising for multi-homing functions. The system can generate two types of AD routes:

  1. Ethernet AD route per-ESI (Ethernet Segment ID)
  2. Ethernet AD route per-EVI (EVPN Instance)

The Ethernet AD per-ESI route generated by a router uses the following fields and values:

  1. Route Distinguisher: Taken from the system level RD or service level RD.
  2. Ethernet Segment Identifier (ESI): Will contain a 10-byte identifier as configured in the system for a specified ethernet-segment.
  3. Ethernet Tag ID: MAX-ET (0xFFFFFFFF). This value is reserved and used only for AD routes per ESI.
  4. MPLS label: 0.
  5. ESI Label Extended community: Includes the single-active bit (0 for all-active and 1 for single-active) and ESI label for all-active multi-homing split-horizon.
  6. Route-target extended community: Taken from the service level RT or an RT-set for the services defined on the ethernet-segment.

The system can either send a separate Ethernet AD per-ESI route per service, or a few Ethernet AD per-ESI routes aggregating the route-targets for multiple services. While both alternatives will inter-operate, RFC 7432 states that the EVPN Auto-Discovery per-ES route must be sent with a set of route-targets corresponding to all the EVIs defined on the ethernet-segment. Either option can be enabled using the command: config>service>system>bgp-evpn#ad-per-es-route-target <[evi-rt] | [evi-rt-set route-distinguisher <ip-address>]>

The default option ad-per-es-route-target evi-rt configures the system to send a separate AD per-ES route per service. When enabled, the evi-rt-set option allows the aggregation of routes: A single AD per-ES route with the associated RD (ip-address:1) and a set of EVI route-targets will be advertised (to a maximum of 128). When the number of EVIs defined in the ethernet-segment is significant (hence the number of route-targets), the system will send more than one route. For example:

  1. AD per-ES route for evi-rt-set 1 will be sent with RD ip-address:1
  2. AD per-ES route for evi-rt-set 2 will be sent with RD ip-address:2
Note:

When evi-rt-set is configured, no vsi-export policies are possible on the services defined on the ethernet-segment. If vsi-export policies are configured for a service, the system will send an individual AD per-ES route for that service. The maximum standard BGP update size is 4KB, with a maximum of 2KB for the route-target extended community attribute.

The Ethernet AD per-EVI route generated by a router uses the following fields and values:

  1. Route Distinguisher: Taken from the service level RD.
  2. Ethernet Segment Identifier (ESI): Will contain a 10-byte identifier as configured in the system for a specified ethernet-segment.
  3. Ethernet Tag ID: 0.
  4. MPLS label: Encodes the unicast label allocated for the service (high-order 20 bits).
  5. Route-target extended community: Taken from the service level RT.
Note:

The AD per-EVI route is not sent with the ESI label Extended Community.

EVPN Route Type 4 - Ethernet Segment Route (ES route)

The router generates this route type for multi-homing ES discovery and DF (Designated Forwarder) election.

  1. Route Distinguisher: Taken from the service level RD.
  2. Ethernet Segment Identifier (ESI): Will contain a 10-byte identifier as configured in the system for a specified ethernet-segment.
  3. ES-import route-target community: The value is automatically derived from the MAC address portion of the ESI. This extended community is treated as a route-target and is supported by RT-constraint (route-target BGP family).

EVPN Route Type 5 - IP Prefix Route

IP Prefix Routes are also supported for MPLS tunnels. The route fields for route type 5 are shown in Figure 147. The 7750 SR, 7450 ESS (mixed mode), or 7950 XRS router will generate this route type for advertising IP prefixes in EVPN using the same fields that are described in section BGP-EVPN Control Plane for VXLAN Overlay Tunnels, with the following exceptions:

  1. MPLS Label—Carries the MPLS label allocated for the service
  2. This route will be sent with the RFC 5512 tunnel encapsulation extended community with the tunnel type value set to MPLS

RFC 5512 - BGP Tunnel Encapsulation Extended Community

The following routes are sent with the RFC 5512 BGP Encapsulation Extended Community: MAC/IP, Inclusive Multicast Ethernet Tag, and AD per-EVI routes. ES and AD per-ESI routes are not sent with this Extended Community.

The router processes the following BGP Tunnel Encapsulation tunnel values registered by IANA for RFC 5512:

  1. VXLAN encapsulation: 8.
  2. MPLS encapsulation: 10.

Any other tunnel value will make the route 'treat-as-withdraw'.

If the encapsulation value is MPLS, the BGP will validate the high-order 20-bits of the label field, ignoring the low-order 4 bits. If the encapsulation is VXLAN, the BGP will take the entire 24-bit value encoded in the MPLS label field as the VNI.

If the encapsulation extended community (as defined in RFC 5512) is not present in a received route, BGP will treat the route as an MPLS or VXLAN-based configuration of the config>router>bgp>neighbor# def-recv-evpn-encap [mpls | vxlan] command. The command is also available at the bgp and group levels.

5.3.2. EVPN for MPLS Tunnels in VPLS Services (EVPN-MPLS)

EVPN can be used in MPLS networks where PEs are interconnected through any type of tunnel, including RSVP-TE, Segment-Routing TE, LDP, BGP, Segment Routing IS-IS, Segment Routing OSPF, or MPLSoUDP. As with VPRN services, the selection of the tunnel to be used in a VPLS service (with BGP-EVPN MPLS enabled) is based on the auto-bind-tunnel command.

EVPN-MPLS is modeled similar to EVPN-VXLAN, that is, using a VPLS service where EVPN-MPLS 'bindings' can coexist with SAPs and SDP-bindings. The following shows an example of a VPLS service with EVPN-MPLS.

*A:PE-1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls-service"
  bgp
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create

The user will configure a bgp-evpn context where vxlan must be shutdown and mpls no shutdown. In addition to the mpls no shutdown command, the minimum set of commands to be configured to set up the EVPN-MPLS instance are the evi and the auto-bind-tunnel resolution commands. However, the user can configure some other options. The most relevant configuration options are described below.

evi {1..65535} — This EVPN identifier is unique in the system and will be used for the service-carving algorithm used for multi-homing (if configured) and auto-deriving route-target and route-distinguishers in the service. It can be used for EVPN-MPLS and EVPN-VXLAN services.

If this EVPN identifier is not specified, the value will be zero and no route-distinguisher or route-targets will be auto-derived from it. If specified and no other route-distinguisher/route-target are configured in the service:, then the following applies:

  1. The route-distinguisher is derived from: <system_ip>:evi
  2. The route-target is derived from: <autonomous-system>:evi
Note:

When the vsi-import/export polices are configured, the route-target must be configured in the policies and those values take preference over the auto-derived route-targets. The operational route-target for a service will be displayed by the show service id x bgp command. If the bgp-ad>vpls-id is configured in the service, the vpls-id derived route-target takes precedence over the evi-derived route-target.

When the evi is configured, a config>service>vpls>bgp node (even empty) is required to allow the user to see the correct information on the show service id 1 bgp and show service system bgp-route-distinguisher commands.

Although not mandatory, if no multi-homing is configured, the configuration of an evi is enforced for EVPN services with SAPs/SDP-bindings in an ethernet-segment. See the 'EVPN multi-homing' section for more information about ethernet-segments.

The following options are specific to EVPN-MPLS (and defined on bgp-evpn>mpls):

  1. control-word: Required as per RFC 7432 to avoid frame disordering. The user can enable/disable it so that interoperability to other vendors can be guaranteed.
  2. auto-bind-tunnel: Allows the user to decide what type of MPLS transport tunnels will be used for a particular instance. The command will be used in the same way as it is used in VPRN services.
    For bgp-evpn mpls, ‘bgp’ is explicitly added to the resolution-filter in EVPN (‘bgp’ is implicit in VPRNs).
  3. force-vlan-vc-forwarding: This command will allow the system to preserve the VLAN ID and pbits of the service-delimiting qtag in a new tag added in the customer frame before sending it to the EVPN core.
    Note:

    This command may be used in conjunction with the sap ingress vlan-translation command. If so, the configured translated VLAN ID will be the VLAN ID sent to the EVPN binds as opposed to the service-delimiting tag VLAN ID. If the ingress SAP/binding is ‘null’-encapsulated, the output VLAN ID and pbits will be zero.

  4. split-horizon-group: This command allows the association of a user-created split horizon group to all the EVPN-MPLS destinations. See the EVPN and VPLS integration section for more information.
  5. ecmp: When this command is set to a value greater than 1, aliasing is activated to the remote PEs that are defined in the same all-active multi-homing ethernet-segment. See the EVPN multi-homing section for more information.
  6. ingress-replication-bum-label: This command is only enabled when the user wants the PE to advertise a label for BUM traffic (Inclusive Multicast routes) that is different from the label advertised for unicast traffic (with the MAC/IP routes). This is useful to avoid potential transient packet duplication in all-active multi-homing.

In addition to these options, the following bgp-evpn commands are also available for EVPN-MPLS services:

  1. [no] mac-advertisement
  2. mac-duplication and settings

When EVPN-MPLS is established among some PEs in the network, EVPN unicast and multicast 'bindings' are created on each PE to the remote EVPN destinations. A specified ingress PE will create:

  1. A unicast EVPN-MPLS destination binding to a remote egress PE as soon as a MAC/IP route is received from that egress PE.
  2. A multicast EVPN-MPLS destination binding to a remote egress PE, if and only if the egress PE advertises an Inclusive Multicast Ethernet Tag Route with a BUM label. That is only possible if the egress PE is configured with ingress-replication-bum-label.

Those bindings, as well as the MACs learned on them, can be checked through the following show commands. In the following example, the remote PE(192.0.2.69) is configured with no ingress-replication-bum-label and PE(192.0.2.70) is configured with ingress-replication-bum-label. Hence, Dut has a single EVPN-MPLS destination binding to PE(192.0.2.69) and two bindings (unicast and multicast) to PE(192.0.2.70).

*A:Dut# show service id 1 evpn-mpls  
 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                 Transport                                
-------------------------------------------------------------------------------
192.0.2.69      262118        1           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.70      262139        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.70      262140        1           No              06/11/2015 19:59:03
                ldp                                        
192.0.2.72      262140        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.72      262141        1           No              06/11/2015 19:59:03
                ldp                                        
192.0.2.73      262139        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.254     262142        0           Yes             06/11/2015 19:59:03
                bgp                                        
-------------------------------------------------------------------------------
Number of entries : 7
-------------------------------------------------------------------------------
===============================================================================
 
*A:Dut# show service id 1 fdb detail 
 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 21:53:48
                            192.0.2.69:262118
1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 19:59:57
                            192.0.2.70:262140
1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 19:59:57
                            192.0.2.72:262141
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

5.3.2.1. EVPN and VPLS Integration

The 7750 SR, 7450 ESS, or 7950 XRS router SR OS EVPN implementation supports draft-ietf-bess-evpn-vpls-seamless-integ so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service. Since EVPN will not be deployed in green-field networks, this feature is useful for the integration between both technologies and even for the migration of VPLS services to EVPN-MPLS.

The following behavior enables the integration of EVPN and sdp-bindings in the same VPLS network:

a) Systems with EVPN endpoints and sdp-bindings to the same far-end bring down the sdp-bindings.

  1. The router will allow the establishment of an EVPN endpoint and a SDP-binding to the same far-end but the SDP-binding will be kept operationally down. Only the EVPN endpoint will be operationally up. This is true for Spoke-SDPs (manual and BGP-AD) and mesh-sdps. It is also possible between VXLAN and SDP-bindings.
  2. If there is an existing EVPN endpoint to a specified far-end and a Spoke-SDP establishment is attempted, the Spoke-SDP will be setup but kept down with an operational flag indicating that there is an EVPN route to the same far-end.
  3. If there is an existing Spoke-SDP and a valid/used EVPN route arrives, the EVPN endpoint will be setup and the Spoke-SDP will be brought down with an operational flag indicating that there is an EVPN route to the same far-end.
  4. In the case of an SDP-binding and EVPN endpoint to different far-end IPs on the same remote PE, both links will be up. This can happen if the SDP-binding is terminated in an IPv6 address or IPv4 address different from the system address where the EVPN endpoint is terminated.

b) The user can add Spoke-SDPs and all the EVPN-MPLS endpoints in the same split horizon group (SHG).

  1. A CLI command is added under the bgp-evpn>mpls> context so that the EVPN-MPLS endpoints can be added to a split horizon group:
    1. bgp-evpn>mpls> [no] split-horizon-group <group-name>
  2. The bgp-evpn mpls split-horizon-group must reference a user-configured split horizon group. User-configured split horizon groups can be configured within the service context. The same group-name can be associated with SAPs, Spoke-SDPs, pw-templates, pw-template-bindings, and EVPN-MPLS endpoints.
  3. If the split-horizon-group command in bgp-evpn>mpls> is not used, the default split horizon group (that contains all the EVPN endpoints) is still used, but it will not be possible to refer to it on saps/Spoke-SDPs.
  4. SAPs and SDP-bindings that share the same split horizon group of the EVPN-MPLS provider-tunnel will be brought operationally down if the point-to-multipoint tunnel is operationally up.

c) The system disables the advertisement of MACs learned on Spoke-SDPs/saps that are part of an EVPN split horizon group.

  1. When the saps and/or Spoke-SDPs (manual or BGP-AD-discovered) are configured within the same split horizon group as the EVPN endpoints, MAC addresses will still be learned on them, but they will not be advertised in EVPN.
  2. The preceding statement is also true if proxy-ARP/proxy-ND is enabled and an IP->MAC pair is learned on a SAP or SDP-binding that belongs to the EVPN split horizon group.
  3. The SAPs and/or Spoke-SDPs added to an EVPN split horizon group should not be part of any EVPN multi-homed ES. If that happened, the PE would still advertise the AD per-EVI route for the SAP and/or Spoke-SDP, attracting EVPN traffic that could not possibly be forwarded to that SAP and/or sdp-binding.
  4. Similar to the preceding statement, a split horizon group composed of SAPs/sdp-bindings used in a BGP-MH site should not be configured under bgp-evpn>mpls>split-horizon-group. This misconfiguration would prevent traffic being forwarded from the EVPN to the BGP-MH site, regardless of the DF/NDF state.
    Figure 160 shows an example of EVPN-VPLS integration.
    Figure 160:  EVPN-VPLS Integration 
    An example CLI configuration for PE1, PE5, and PE2 is provided below.
*A:PE1>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  split-horizon-group "SHG-1" create 
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 
  bgp-ad
    no shutdown
    vpls-id 65000:1
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      split-horizon-group SHG-1
  spoke-sdp 12:1 create
  exit
  sap 1/1/1:1 create
  exit
 
*A:PE5>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 # auto-created SHG
  bgp-ad
    no shutdown
    vpls-id 65000:1
  spoke-sdp 52:1 create
  exit
 
*A:PE2>config>service# info 
----------------------------------------------
vpls 1 customer 1 create
  end-point CORE create
    no suppress-standby-signaling
  spoke-sdp 21:1 end-point CORE
    precedence primary
  spoke-sdp 25:1 end-point CORE
  1. PE1, PE3, and PE4 have BGP-EVPN and BGP-AD enabled in VPLS-1. PE5 has BGP-AD enabled and PE2 has active/standby Spoke-SDPs to PE1 and PE5.
    In this configuration:
    1. PE1, PE3, and PE4 will attempt to establish BGP-AD Spoke-SDPs, but they will be kept operationally down as long as there are EVPN endpoints active among them.
    2. BGP-AD Spoke-SDPs and EVPN endpoints are instantiated within the same split horizon group, for example, SHG-1.
    3. Manual Spoke-SDPs from PE1 and PE5 to PE2 are not part of SHG-1.
  2. EVPN MAC advertisements:
    1. MACs learned on FEC128 Spoke-SDPs are advertised normally in EVPN.
    2. MACs learned on FEC129 Spoke-SDPs are not advertised in EVPN (because they are part of SHG-1, which is the split horizon group used for bgp-evpn>mpls). This prevents any data plane MACs learned on the SHG from being advertised in EVPN.
  3. BUM operation on PE1:
    1. When CE1 sends BUM, PE1 will flood to all the active bindings.
    2. When CE2 sends BUM, PE2 will send it to PE1 (active Spoke-SDP) and PE1 will flood to all the bindings and saps.
    3. When CE5 sends BUM, PE5 will flood to the three EVPN PEs. PE1 will flood to the active Spoke-SDP and saps, never to the EVPN PEs because they are part of the same SHG.

5.3.2.2. Auto-Derived Route-Distinguisher (RD) in Services with Multiple BGP Families

In a VPLS service, multiple BGP families and protocols can be enabled at the same time. When bgp-evpn is enabled, bgp-ad and bgp-mh are supported as well. A single RD is used per service and not per BGP family/protocol.

The following rules apply:

  1. The VPLS RD is selected based on the following precedence:
    1. Manual RD or auto-rd always take precedence when configured.
    2. If no manual/auto-rd configuration, the RD is derived from the bgp-ad>vpls-id.
    3. If no manual/auto-rd/vpls-id configuration, the RD is derived from the bgp-evpn>evi, except for bgp-mh, which does not support evi-derived RD.
    4. If no manual/auto-rd/vpls-id/evi configuration, there will not be RD and the service will fail.
  2. The selected RD (see above rules) will be displayed by the Oper Route Dist field of the show service id bgp command.
  3. The service supports dynamic RD changes, for instance, the CLI allows the vpls-id be changed dynamically, even if it is used to auto-derive the service RD for bgp-ad, bgp-vpls, or bgp-mh.
    Note:

    When the RD changes, the active routes for that VPLS will be withdrawn and re-advertised with the new RD.

  4. If one of the mechanisms to derive the RD for a specified service is removed from the configuration, the system will select a new RD based on the above rules. For example, if the vpls-id is removed from the configuration, the routes will be withdrawn, the new RD selected from the evi, and the routes re-advertised with the new RD.
    Note:

    This reconfiguration will fail if the new RD already exists in a different VPLS/epipe.

  5. Because the vpls-id takes precedence over the evi when deriving the RD automatically, adding evpn to an existing bgp-ad service will not impact the existing RD - this is important to support bgp-ad to evpn migration.

5.3.2.3. EVPN Multi-Homing in VPLS Services

EVPN multi-homing implementation is based on the concept of the ethernet-segment. An ethernet-segment is a logical structure that can be defined in one or more PEs and identifies the CE (or access network) multi-homed to the EVPN PEs. An ethernet-segment is associated with port, LAG, or SDP objects and is shared by all the services defined on those objects. In the case of virtual Ethernet segments, individual VID or VC-ID ranges can be associated to the port, LAG, or SDP objects defined in the ethernet-segment.

Each ethernet-segment has a unique identifier called esi (Ethernet Segment Identifier) that is 10 bytes long and is manually configured in the router.

Note:

The esi is advertised in the control plane to all the PEs in an EVPN network; therefore, it is very important to ensure that the 10-byte esi value is unique throughout the entire network. Single-homed CEs are assumed to be connected to an ethernet-segment with esi = 0 (single-homed ethernet-segments are not explicitly configured).

This section describes the behavior of the EVPN multi-homing implementation in an EVPN-MPLS service.

5.3.2.3.1. EVPN All-Active Multi-Homing

As described in RFC 7432, all-active multi-homing is only supported on access LAG SAPs and it is mandatory that the CE is configured with a LAG to avoid duplicated packets to the network. LACP is optional.

Three different procedures are implemented in 7750 SR, 7450 ESS, and 7950 XRS SR OS to provide all-active multi-homing for a specified ethernet-segment:

  1. DF (Designated Forwarder) election
  2. Split-horizon
  3. Aliasing

Figure 161 shows the need for DF election in all-active multi-homing.

Figure 161:  DF Election 

The DF election in EVPN all-active multi-homing avoids duplicate packets on the multi-homed CE. The DF election procedure is responsible for electing one DF PE per ESI per service; the rest of the PEs being non-DF for the ESI and service. Only the DF will forward BUM traffic from the EVPN network toward the ES SAPs (the multi-homed CE). The non-DF PEs will not forward BUM traffic to the local ethernet-segment SAPs.

Note:

BUM traffic from the CE to the network and known unicast traffic in any direction is allowed on both the DF and non-DF PEs.

Figure 162 shows the EVPN split-horizon concept for all-active multi-homing.

Figure 162:  Split-Horizon 

The EVPN split-horizon procedure ensures that the BUM traffic originated by the multi-homed PE and sent from the non-DF to the DF, is not replicated back to the CE (echoed packets on the CE). To avoid these echoed packets, the non-DF (PE1) will send all the BUM packets to the DF (PE2) with an indication of the source ethernet-segment. That indication is the ESI Label (ESI2 in the example), previously signaled by PE2 in the AD per-ESI route for the ethernet-segment. When PE2 receives an EVPN packet (after the EVPN label lookup), the PE2 will find the ESI label that will identify its local ethernet-segment ESI2. The BUM packet will be replicated to other local CEs but not to the ESI2 SAP.

Figure 163 shows the EVPN aliasing concept for all-active multi-homing.

Figure 163:  Aliasing  

Because CE2 is multi-homed to PE1 and PE2 using an all-active ethernet-segment, 'aliasing' is the procedure by which PE3 can load-balance the known unicast traffic between PE1 and PE2, even if the destination MAC address was only advertised by PE1 as in the example. When PE3 installs MAC1 in the FDB, it will associate MAC1 not only with the advertising PE (PE1) but also with all the PEs advertising the same esi (ESI2) for the service. In this example, PE1 and PE2 advertise an AD per-EVI route for ESI2, therefore, the PE3 installs the two next-hops associated with MAC1.

Aliasing is enabled by configuring ECMP greater than 1 in the bgp-evpn mpls context.

5.3.2.3.1.1. All-Active Multi-Homing Service Model

The following shows an example PE1 configuration that provides all-active multi-homing to the CE2 shown in Figure 163.

*A:PE1>config>lag(1)# info 
----------------------------------------------
  mode access
  encap-type dot1q
  port 1/1/2
  lacp active administrative-key 1 system-id 00:00:00:00:00:22
  no shutdown
 
*A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.1.1.1:0
  ethernet-segment "ESI2" create
    esi 01:12:12:12:12:12:12:12:12:12
    multi-homing all-active
    service-carving
    lag 1 
    no shutdown
 
*A:PE1>config>redundancy>evpn-multi-homing# info 
----------------------------------------------
    boot-timer 120
    es-activation-timer 10
 
*A:PE1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls-service with all-active multihoming"
  bgp
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap lag-1:1 create 
  exit
 

In the same way, PE2 is configured as follows:

*A:PE1>config>lag(1)# info 
----------------------------------------------
  mode access
  encap-type dot1q
  port 1/1/1
  lacp active administrative-key 1 system-id 00:00:00:00:00:22
  no shutdown
 
*A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.1.1.1:0
  ethernet-segment "ESI12" create
    esi 01:12:12:12:12:12:12:12:12:12
    multi-homing all-active
    service-carving
    lag 1 
    no shutdown
 
*A:PE1>config>redundancy>evpn-multi-homing# info 
----------------------------------------------
    boot-timer 120
    es-activation-timer 10
 
*A:PE1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls-service with all-active multihoming"
  bgp
    route-distinguisher 65001:60
    route-target target:65000:60
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap lag-1:1 create 
  exit

The preceding configuration will enable the all-active multi-homing procedures. The following must be considered:

  1. The ethernet-segment must be configured with a name and a 10-byte esi:
    1. config>service>system>bgp-evpn#ethernet-segment <es_name> create
    2. config>service> system>bgp-evpn>ethernet-segment# esi <value>
  2. When configuring the esi, the system enforces the 6 high-order octets after the type to be different from zero (so that the auto-derived route-target for the ES route is different from zero). Other than that, the entire esi value must be unique in the system.
  3. Only a LAG can be associated with the ethernet-segment. This LAG will be exclusively used for EVPN multi-homing. Other LAG ports in the system can be still used for MC-LAG and other services.
  4. When the LAG is configured on PE1 and PE2, the same admin-key, system-priority, and system-id must be configured on both PEs, so that CE2 responds as though it is connected to the same system.
  5. The same ethernet-segment may be used for EVPN-MPLS and PBB-EVPN services.
    Note:

    The source-bmac-lsb attribute must be defined for PBB-EVPN (so that it will only be used in PBB-EVPN, and ignored by EVPN). Other than EVPN-MPLS and PBB-EVPN I-VPLS/Epipe services, no other Layer 2 services are allowed in the same ethernet-segment (regular VPLS or EVPN-VXLAN SAPs defined on the ethernet-segment will be kept operationally down).

  6. Only one sap per service can be part of the same ethernet-segment.

5.3.2.3.1.2. ES Discovery and DF Election Procedures

The ES (Ethernet Segment) discovery and DF election is implemented in three logical steps, as shown in Figure 164.

Figure 164:  ES Discovery and DF Election 
Step 1 - ES Advertisement and Discovery

Ethernet-segment ESI-1 is configured as per the previous section, with all the required parameters. When ethernet-segment no shutdown is executed, PE1 and PE2 will advertise an ES route for ESI-1. They will both include the route-target auto-derived from the MAC portion of the configured ESI. If the route-target address family is configured in the network, this will allow the RR to keep the dissemination of the ES routes under control.

In addition to the ES route, PE1 and PE2 will advertise AD per-ESI routes and AD per-EVI routes.

  1. AD per-ESI routes will announce the ethernet-segment capabilities, including the mode (single-active or all-active) as well as the ESI label for split-horizon.
  2. AD per-EVI routes are advertised so that PE3 knows what services (EVIs) are associated with the ESI. These routes are used by PE3 for its aliasing and backup procedures.
Step 2 - DF Election

When ES routes exchange between PE1 and PE2 is complete, both run the DF election for all the services in the ethernet-segment.

PE1 and PE2 elect a Designated Forwarder (DF) per <ESI, service>. The default DF election mechanism in 7750 SR, 7450 ESS, and 7950 XRS SR OS is service-carving (as per RFC 7432). The following applies when enabled on a specified PE:

  1. An ordered list of PE IPs where ESI-1 resides is built. The IPs are gotten from the Origin IP fields of all the ES routes received for ESI-1, as well as the local system address. The lowest IP will be considered ordinal '0' in the list.
  2. The local IP can only be considered a "candidate" after successful ethernet-segment no shutdown for a specified service.
    Note:

    The remote PE IPs must be present in the local PE's RTM so that they can participate in the DF election.

  3. A PE will only consider a specified remote IP address as candidate for the DF election algorithm for a specified service if, as well as the ES route, the corresponding AD routes per-ESI and per-EVI for that PE have been received and properly activated.
  4. All the remote PEs receiving the AD per-ES routes (for example, PE3), will interpret that ESI-1 is all-active if all the PEs send their AD per-ES routes with the single-active bit = 0. Otherwise, if at least one PE sends an AD route per-ESI with the single-active flag set or the local ESI configuration is single-active, the ESI will behave as single-active.
  5. An es-activation-timer can be configured at the redundancy>bgp-evpn-multi-homing>es-activation-timer level or at the service>system>bgp-evpn>eth-seg>es-activation-timer level. This timer, which is 3 seconds by default, delays the transition from non-DF to DF for a specified service, after the DF election has run.
    1. This use of the es-activation-timer is different from zero and minimizes the risks of loops and packet duplication due to "transient" multiple DFs.
    2. The same es-activation-timer should be configured in all the PEs that are part of the same ESI. It is up to the user to configure either a long timer to minimize the risks of loops/duplication or even es-activation-timer=0 to speed up the convergence for non-DF to DF transitions. When the user configures a specific value, the value configured at ES level supersedes the configured global value.
  6. The DF election is triggered by the following events:
    1. config>service>system>bgp-evpn>eth-seg# no shutdown triggers the DF election for all the services in the ESI.
    2. Reception of a new update/withdrawal of an ES route (containing an ESI configured locally) triggers the DF election for all the services in the ESI.
    3. Reception of a new update/withdrawal of an AD per-ES route (containing an ESI configured locally) triggers the DF election for all the services associated with the list of route-targets received along with the route.
    4. Reception of a new update of an AD per-ES route with a change in the ESI-label extended community (single-active bit or MPLS label) triggers the DF election for all the services associated with the list of route-targets received along with the route.
    5. Reception of a new update/withdrawal of an AD route per-EVI (containing an ESI configured locally) triggers the DF election for that service.
  1. When the PE boots up, the boot-timer will allow the necessary time for the control plane protocols to come up before bringing up the ethernet-segment and running the DF algorithm. The boot-timer is configured at system level - config>redundancy>bgp-evpn-multi-homing# boot-timer - and should use a value long enough to allow the IOMs and BGP sessions to come up before exchanging ES routes and running the DF election for each EVI/ISID.
    1. The system will not advertise ES routes until the boot timer expires. This will guarantee that the peer ES PEs don't run the DF election either until the PE is ready to become the DF if it needs to.
    2. The following show command displays the configured boot-timer as well as the remaining timer if the system is still in boot-stage.
A:PE1# show redundancy bgp-evpn-multi-homing 
===============================================================================
Redundancy BGP EVPN Multi-homing Information
===============================================================================
Boot-Timer              : 10 secs                 
Boot-Timer Remaining    : 0 secs                  
ES Activation Timer     : 3 secs                  
===============================================================================
  1. When service-carving mode auto is configured (default mode), the DF election algorithm will run the function [V(evi) mod N(peers) = i(ordinal)] to identify the DF for a specified service and ESI, as described in the following example:
    1. As shown in Figure 164, PE1 and PE2 are configured with ESI-1. Given that V(10) mod N(2) = 0, PE1 will be elected DF for VPLS-10 (because its IP address is lower than PE2's and it is the first PE in the candidate list).
      Note:

      The algorithm takes the configured evi in the service as opposed to the service-id itself. The evi for a service must match in all the PEs that are part of the ESI. This guarantees that the election algorithm is consistent across all the PEs of the ESI. The evi must be always configured in a service with saps/sdp-bindings that are created in an ES.

  2. A manual service-carving option is allowed so that the user can manually configure for which evi identifiers the PE is primary: service-carving mode manual / manual service <evi> to <evi>.
    1. The system will be the PE forwarding/multicasting traffic for the evi identifiers included in the configuration. The PE will be secondary (non-DF) for the non-specified evis.
    2. If a range is configured but the service-carving is not mode manual, then the range has no effect.
    3. Only two PEs are supported when service-carving mode manual is configured. If a third PE is configured with service-carving mode manual for an ESI, the two non-primary PEs will remain non-DF regardless of the primary status.
    4. For example, as shown in Figure 164: if PE1 is configured with service-carving manual evi 1 to 100 and PE2 with service-carving manual evi 101 to 200, then PE1 will be the primary PE for service VPLS 10 and PE2 the secondary PE.
  3. When service-carving is disabled, the lowest originator IP will win the election for a specified service and ESI:

config>service>system>bgp-evpn>ethernet-segment> mode off

The following show command displays the ethernet-segment configuration and DF status for all the EVIs and ISIDs (if PBB-EVPN is enabled) configured in the ethernet-segment.

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-1" all 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-1
Admin State             : Up                 Oper State         : Up
ESI                     : 01:00:00:00:00:71:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
Source BMAC LSB         : 71-71              
ES BMac Tbl Size        : 8                  ES BMac Entries    : 1
Lag Id                  : 1                  
ES Activation Timer     : 0 secs             
Exp/Imp Route-Target    : target:00:00:00:00:71:00
 
Svc Carving             : auto               
ES SHG Label            : 262142             
===============================================================================
===============================================================================
EVI Information 
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   no
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.69
1                                       192.0.2.72
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
===============================================================================
ISID Information 
===============================================================================
ISID                SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
20001               20001               0                   no
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================                                 
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
ISID                                    DF Address
-------------------------------------------------------------------------------
20001                                   192.0.2.69
20001                                   192.0.2.72
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
===============================================================================
BMAC Information 
===============================================================================
SvcId                                   BMacAddress
-------------------------------------------------------------------------------
20000                                   00:00:00:00:71:71
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
Step 3 - DF and Non-DF Service Behavior

Based on the result of the DF election or the manual service-carving, the control plane on the non-DF (PE1) will instruct the data path to remove the LAG SAP (associated with the ESI) from the default flooding list for BM traffic (unknown unicast traffic may still be sent if the EVI label is a unicast label and the source MAC address is not associated to the ESI). On PE1 and PE2, both LAG SAPs will learn the same MAC address (coming from the CE). For instance, in the following show commands, 00:ca:ca:ba:ce:03 is learned on both PE1 and PE2 access LAG (on ESI-1). However, PE1 learns the MAC as 'Learned' whereas PE2 learns it as 'Evpn'. This is due to the CE2 hashing the traffic for that source MAC to PE1. PE2 learns the MAC through EVPN but it associates the MAC to the ESI SAP, because the MAC belongs to the ESI.

*A:PE1# show service id 1 fdb detail 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:ca:ca:ba:ce:03 sap:lag-1:1              L/0      06/11/15 00:14:47
1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:06
                            192.0.2.70:262140
1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 00:09:39
                            192.0.2.72:262141
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================
 
*A:PE2# show service id 1 fdb detail 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:ca:ca:ba:ce:03 sap:lag-1:1              Evpn     06/11/15 00:14:47
1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 00:09:40
                            192.0.2.69:262141
1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:40
                            192.0.2.70:262140
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

When PE1 (non-DF) and PE2 (DF) exchange BUM packets for evi 1, all those packets will be sent including the ESI label at the bottom of the stack (in both directions). The ESI label advertised by each PE for ESI-1 can be displayed by the following command:

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-1" 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-1
Admin State             : Up                 Oper State         : Up
ESI                     : 01:00:00:00:00:71:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
Source BMAC LSB         : 71-71              
ES BMac Tbl Size        : 8                  ES BMac Entries    : 1
Lag Id                  : 1                  
ES Activation Timer     : 0 secs             
Exp/Imp Route-Target    : target:00:00:00:00:71:00
 
Svc Carving             : auto               
ES SHG Label            : 262142             
===============================================================================
*A:PE2# show service system bgp-evpn ethernet-segment name "ESI-1" 
 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-1
Admin State             : Up                 Oper State         : Up
ESI                     : 01:00:00:00:00:71:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
Source BMAC LSB         : 71-71              
ES BMac Tbl Size        : 8                  ES BMac Entries    : 0
Lag Id                  : 1                  
ES Activation Timer     : 20 secs            
Exp/Imp Route-Target    : target:00:00:00:00:71:00
 
Svc Carving             : auto               
ES SHG Label            : 262142             
===============================================================================

5.3.2.3.1.3. Aliasing

Following the example in Figure 164, if the service configuration on PE3 has ECMP > 1, PE3 will add PE1 and PE2 to the list of next-hops for ESI-1. As soon as PE3 receives a MAC for ESI-1, it will start load-balancing between PE1 and PE2 the flows to the remote ESI CE. The following command shows the FDB in PE3.

Note:

mac 00:ca:ca:ba:ce:03 is associated with the ethernet-segment eES:01:00:00:00:00:71:00:00:00:01 (esi configured on PE1 and PE2 for ESI-1).

*A:PE3# show service id 1 fdb detail 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:ca:ca:ba:ce:03 eES:                     Evpn     06/11/15 00:14:47
                            01:00:00:00:00:71:00:00:00:01
1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 00:09:18
                            192.0.2.69:262141
1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:18
                            192.0.2.70:262140
1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 00:09:39
                            192.0.2.72:262141
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

The following command shows all the EVPN-MPLS destination bindings on PE3, including the ES destination bindings.

The ethernet-segment eES:01:00:00:00:00:71:00:00:00:01 is resolved to PE1 and PE2 addresses:

*A:PE3# show service id 1 evpn-mpls 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                 Transport                                
-------------------------------------------------------------------------------
192.0.2.69      262140        0           Yes             06/10/2015 14:33:30
                ldp                                        
192.0.2.69      262141        1           No              06/10/2015 14:33:30
                ldp                                        
192.0.2.70      262139        0           Yes             06/10/2015 14:33:30
                ldp                                        
192.0.2.70      262140        1           No              06/10/2015 14:33:30
                ldp                                        
192.0.2.72      262140        0           Yes             06/10/2015 14:33:30
                ldp                                        
192.0.2.72      262141        1           No              06/10/2015 14:33:30
                ldp                                        
192.0.2.73      262139        0           Yes             06/10/2015 14:33:30
                ldp                                        
192.0.2.254     262142        0           Yes             06/10/2015 14:33:30
                bgp                                        
-------------------------------------------------------------------------------
Number of entries : 8
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                     TEP Address     Egr Label   Last Change
                                               Transport  
-------------------------------------------------------------------------------
01:00:00:00:00:71:00:00:00:01 192.0.2.69      262141      06/10/2015 14:33:30
                                              ldp          
01:00:00:00:00:71:00:00:00:01 192.0.2.72      262141      06/10/2015 14:33:30
                                              ldp          
01:74:13:00:74:13:00:00:74:13 192.0.2.73      262140      06/10/2015 14:33:30
                                              ldp          
-------------------------------------------------------------------------------
Number of entries : 3
-------------------------------------------------------------------------------
===============================================================================

PE3 will perform aliasing for all the MACs associated with that ESI. This is possible because PE1 is configured with ecmp parameter >1:

*A:PE3>config>service>vpls# info 
----------------------------------------------
            bgp
            exit
            bgp-evpn
                evi 1
                mpls bgp 1
                    ecmp 4
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            proxy-arp
                shutdown
            exit
            stp
                shutdown
            exit
            sap 1/1/1:2 create
            exit
            no shutdown

5.3.2.3.1.4. Network Failures and Convergence for All-Active Multi-Homing

Figure 165 shows the behavior on the remote PEs (PE3) when there is an ethernet-segment failure.

Figure 165:  All-Active Multi-Homing ES Failure 

The unicast traffic behavior on PE3 is as follows:

  1. PE3 can only forward MAC DA = CE2 to both PE1 and PE2 when the MAC advertisement route from PE1 (or PE2) and the set of Ethernet AD per-ES routes and Ethernet AD per-EVI routes from PE1 and PE2 are active at PE3.
  2. If there was a failure between CE2 and PE2, PE2 would withdraw its set of Ethernet AD and ES routes, then PE3 would forward traffic destined to CE2 to PE1 only. PE3 does not need to wait for the withdrawal of the individual MAC.
  3. The same behavior would be followed if the failure had been at PE1.
  4. If after (2), PE2 withdraws its MAC advertisement route, then PE3 treats traffic to MAC DA = CE2 as unknown unicast, unless the MAC had been previously advertised by PE1.

For BUM traffic, the following events would trigger a DF election on a PE and only the DF would forward BUM traffic after the esi-activation-timer expiration (if there was a transition from non-DF to DF).

  1. Reception of ES route update (local ES shutdown/no shutdown or remote route)
  2. New AD-ES route update/withdraw
  3. New AD-EVI route update/withdraw
  4. Local ES port/SAP/service shutdown
  5. Service carving range change (affecting the evi)
  6. Multi-homing mode change (single/all active to all/single-active)
5.3.2.3.1.4.1. Logical Failures on Ethernet Segments and Blackholes

Be aware of the effects triggered by certain 'failure scenarios'; some of these scenarios are shown in Figure 166:

Figure 166:  Blackhole Caused by SAP/SVC Shutdown 

If an individual VPLS service is shutdown in PE1 (the example is also valid for PE2), the corresponding LAG SAP will go operationally down. This event will trigger the withdrawal of the AD per-EVI route for that particular SAP. PE3 will remove PE1 of its list of aliased next-hops and PE2 will take over as DF (if it was not the DF already). However, this will not prevent the network from black-holing the traffic that CE2 'hashes' to the link to PE1. Traffic sent from CE2 to PE2 or traffic from the rest of the CEs to CE2 will be unaffected, so this situation is not easily detected on the CE.

The same result occurs if the ES SAP is administratively shutdown instead of the service.

Note:

When bgp-evpn mpls shutdown is executed, the sap associated with the ES will be brought operationally down (StandbyforMHprotocol) and so will the entire service if there are no other saps or sdp-bindings in the service. However, if there are other saps/sdp-bindings, the service will remain operationally up.

5.3.2.3.1.4.2. Transient Issues Due to MAC Route Delays

Some situations may cause potential transient issues to occur. These are shown in Figure 167 and explained below.

Figure 167:  Transient Issues Caused by “slow” MAC Learning 

Transient packet duplication caused by delay in PE3 to learn MAC1:

This scenario is illustrated by the diagram on the left in Figure 167. In an all-active multi-homing scenario, if a specified MAC address is not yet learned in a remote PE, but is known in the two PEs of the ES, for example, PE1 and PE2, the latter PEs might send duplicated packets to the CE.

In an all-active multi-homing scenario, if a specified MAC address (for example, MAC1), is not learned yet in a remote PE (for example, PE3), but it is known in the two PEs of the ES (for example, PE1 and PE2), the latter PEs might send duplicated packets to the CE.

This issue is solved by the use of ingress-replication-bum-label in PE1 and PE2. If configured, PE1/PE2 will know that the received packet is an unknown unicast packet, therefore, the NDF (PE1) will not send the packets to the CE and there will not be duplication.

Note:

Even without the ingress-replication-bum-label, this is only a transient situation that would be solved as soon as MAC1 is learned in PE3.

Transient blackhole caused by delay in PE1 to learn MAC1:

This case is illustrated by the diagram on the right in Figure 167. In an all-active multi-homing scenario, MAC1 is known in PE3 and aliasing is applied to MAC1. However, MAC1 is not known yet in PE1, the NDF for the ES. If PE3 hashing picks up PE1 as the destination of the aliased MAC1, the packets will be blackholed. This case is solved on the NDF by not blocking unknown unicast traffic that arrives with a unicast label (possible if PE1 and PE2 are configured using ingress-replication-bum-label).

As soon as PE1 learns MAC1, the blackhole will be resolved even if ingress-replication-bum-label is not used.

5.3.2.3.2. EVPN Single-Active Multi-Homing

The 7750 SR, 7450 ESS, and 7950 XRS SR OS supports single-active multi-homing on access LAG SAPs, regular SAPs, and Spoke-SDPs for a specified VPLS service. For LAG SAPs, the CE will be configured with a different LAG to each PE in the ethernet-segment (as opposed to a single LAG in an all-active multi-homing).

The following SR OS procedures support EVPN single-active multi-homing for a specified ethernet-segment:

  1. DF (Designated Forwarder) election
    As in all-active multi-homing, DF election in single-active multi-homing determines the forwarding for BUM traffic from the EVPN network to the ethernet-segment CE. Also, in single-active multi-homing, DF election also determines the forwarding of any traffic (unicast/BUM) and in any direction (to/from the CE).
  2. Backup PE
    In single-active multi-homing, the remote PEs do not perform aliasing to the PEs in the ethernet-segment. The remote PEs identify the DF based on the MAC routes and send the unicast flows for the ethernet-segment to the PE in the DF and program a backup PE as an alternative next-hop for the remote ESI in case of failure.
    This RFC 7432 procedure is known as 'Backup PE' and is shown in Figure 168 for PE3.
    Figure 168:  Backup PE 

5.3.2.3.2.1. Single-Active Multi-Homing Service Model

The following shows an example of PE1 configuration that provides single-active multi-homing to CE2, as shown in Figure 168.

*A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.1.1.1:0
  ethernet-segment "ESI2" create
    esi 01:12:12:12:12:12:12:12:12:12
    multi-homing single-active
    service-carving
    sdp 1 
    no shutdown
 
*A:PE1>config>redundancy>evpn-multi-homing# info 
----------------------------------------------
    boot-timer 120
    es-activation-timer 10
 
*A:PE1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls-service with single-active multihoming"
  bgp
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  spoke-sdp 1:1 create 
  exit
 

The PE2 example configuration for this scenario is as follows:

*A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.1.1.1:0
  ethernet-segment "ESI2" create
    esi 01:12:12:12:12:12:12:12:12:12
    multi-homing single-active
    service-carving
    sdp 2 
    no shutdown
 
*A:PE1>config>redundancy>evpn-multi-homing# info 
----------------------------------------------
    boot-timer 120
    es-activation-timer 10
 
*A:PE1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls-service with single-active multihoming"
  bgp
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  spoke-sdp 2:1 create 
  exit
 

In single-active multi-homing, the non-DF PEs for a specified ESI will block unicast and BUM traffic in both directions (upstream and downstream) on the object associated with the ESI. Other than that, single-active multi-homing is similar to all-active multi-homing with the following differences:

  1. The ethernet-segment will be configured for single-active: service>system>bgp-evpn>ethernet-segment>multi-homing single-active.
  2. The advertisement of the ESI-label in a per-ESI AD route is optional for single-active ethernet-segments. The user can control the no advertisement of the ESI label by using the following command: service>system>bgp-evpn>ethernet-segment>multi-homing single-active no-esi-label. By default, the ESI label is used for single-active ESs too.
  3. For single-active multi-homing, the ethernet-segment can be associated with a port and sdp, as well as a lag-id, as shown in Figure 168, where:
    1. port would be used for single-active sap redundancy without the need for lag.
    2. sdp would be used for single-active Spoke-SDP redundancy.
    3. lag would be used for single-active LAG redundancy
      Note:

      In this case, key, system-id, and system-priority must be different on the PEs that are part of the ethernet-segment).

  4. For single-active multi-homing, when the PE is non-DF for the service, the saps/Spoke-SDPs on the ethernet-segment will be down and show StandByForMHProtocol as the reason.
  5. From a service perspective, single-active multi-homing can provide redundancy to CEs (MHD, Multi-Homed Devices) or networks (MHN, Multi-Homed Networks) with the following setup:
    1. LAG with or without LACP
      In this case, the multi-homed ports on the CE will be part of the different LAGs (a LAG per multi-homed PE will be used in the CE). The non-DF PE for each service can signal that the sap is operationally down if eth-cfm fault-propagation-enable {use-if-tlv|suspend-ccm} is configured.
    2. Regular Ethernet 802.1q/ad ports
      In this case, the multi-homed ports on the CE/network will not be part of any LAG. Eth-cfm can also be used for non-DF indication to the multi-homed device/network.
    3. Active-standby PWs
      In this case, the multi-homed CE/network is connected to the PEs through an MPLS network and an active/standby Spoke-SDP per service. The non-DF PE for each service will make use of the LDP PW status bits to signal that the Spoke-SDP is operationally down on the PE side.

5.3.2.3.2.2. ES and DF Election Procedures

In all-active multi-homing, the non-DF keeps the SAP up, although it removes it from the default flooding list. In the single-active multi-homing implementation the non-DF will bring the SAP or SDP-binding operationally down. Refer to the ES Discovery and DF Election Procedures for more information.

The following show commands display the status of the single-active ESI-7413 in the non-DF. The associated Spoke-SDP is operationally down and it signals PW Status standby to the multi-homed CE:

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-7413"       
 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-7413
Admin State             : Up                 Oper State         : Up
ESI                     : 01:74:13:00:74:13:00:00:74:13
Multi-homing            : singleActive       Oper Multi-homing  : singleActive
Source BMAC LSB         : <none>             
Sdp Id                  : 4                  
ES Activation Timer     : 0 secs             
Exp/Imp Route-Target    : target:74:13:00:74:13:00
 
Svc Carving             : auto               
ES SHG Label            : 262141             
===============================================================================
*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-7413" evi 1 
===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  06/11/2015 20:05:32
===============================================================================
===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.70                              06/11/2015 20:05:20
192.0.2.73                              06/11/2015 20:05:32
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================
*A:PE1# show service id 1 base 
===============================================================================
Service Basic Information
===============================================================================
Service Id        : 1                   Vpn Id            : 0
Service Type      : VPLS                
Name              : (Not Specified)
Description       : (Not Specified)
 
<snip>
-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sap:1/1/1:1                              q-tag        9000    9000    Up   Up
sdp:4:13 S(192.0.2.74)                   Spok         0       8978    Up   Down
===============================================================================
* indicates that the corresponding row element may have been truncated.
 
 
*A:PE1# show service id 1 all | match Pw 
Local Pw Bits      : pwFwdingStandby
Peer Pw Bits       : None
 
*A:PE1# show service id 1 all | match Flag 
Flags              : StandbyForMHProtocol
Flags              : None
 

5.3.2.3.2.3. Backup PE Function

A remote PE (PE3 in Figure 168) will import the AD routes per ESI, where the single-active flag is set. PE3 will interpret that the ethernet-segment is single-active if at least one PE sends an AD route per-ESI with the single-active flag set. MACs for a specified service and ESI will be learned from a single PE, that is, the DF for that <ESI, EVI>.

The remote PE will install a single EVPN-MPLS destination (TEP, label) for a received MAC address and a backup next-hop to the PE for which the AD routes per-ESI and per-EVI are received. For instance, in the following command, 00:ca:ca:ba:ca:06 is associated with the remote ethernet-segment eES 01:74:13:00:74:13:00:00:74:13. That eES is resolved to PE(192.0.2.73), which is the DF on the ES.

*A:PE3# show service id 1 fdb detail 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:ca:ca:ba:ca:02 sap:1/1/1:2              L/0      06/12/15 00:33:39
1         00:ca:ca:ba:ca:06 eES:                     Evpn     06/12/15 00:33:39
                            01:74:13:00:74:13:00:00:74:13
1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 21:53:47
                            192.0.2.69:262118
1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 19:59:57
                            192.0.2.70:262140
1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 19:59:57
                            192.0.2.72:262141
-------------------------------------------------------------------------------
No. of MAC Entries: 5
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================
 
*A:PE3# show service id 1 evpn-mpls 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                 Transport                                
-------------------------------------------------------------------------------
192.0.2.69      262118        1           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.70      262139        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.70      262140        1           No              06/11/2015 19:59:03
                ldp                                        
192.0.2.72      262140        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.72      262141        1           No              06/11/2015 19:59:03
                ldp                                        
192.0.2.73      262139        0           Yes             06/11/2015 19:59:03
                ldp                                        
192.0.2.254     262142        0           Yes             06/11/2015 19:59:03
                bgp                                        
-------------------------------------------------------------------------------
Number of entries : 7
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                     TEP Address     Egr Label   Last Change
                                               Transport  
-------------------------------------------------------------------------------
01:74:13:00:74:13:00:00:74:13 192.0.2.73      262140      06/11/2015 19:59:03
                                              ldp          
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

If PE3 sees only two single-active PEs in the same ESI, the second PE will be the backup PE. Upon receiving an AD per-ES/per-EVI route withdrawal for the ESI from the primary PE, the PE3 will start sending the unicast traffic to the backup PE immediately.

If PE3 receives AD routes for the same ESI and EVI from more than two PEs, the PE will not install any backup route in the data path. Upon receiving an AD per-ES/per-EVI route withdrawal for the ESI, it will flush the MACs associated with the ESI.

5.3.2.3.2.4. Network Failures and Convergence for Single-Active Multi-Homing

Figure 169 shows the remote PE (PE3) behavior when there is an ethernet-segment failure.

Figure 169:  Single-Active Multi-Homing ES Failure 

The PE3 behavior for unicast traffic is as follows:

  1. PE3 forwards MAC DA = CE2 to PE2 when the MAC Advertisement Route came from PE2 and the set of Ethernet AD per-ES routes and Ethernet AD per-EVI routes from PE1 and PE2 are active at PE3.
  2. If there was a failure between CE2 and PE2, PE2 would withdraw its set of Ethernet AD and ES routes, then PE3 would immediately forward the traffic destined for CE2 to PE1 only (the backup PE). PE3 does not need to wait for the withdrawal of the individual MAC.
  3. After the (2) PE2 withdraws its MAC advertisement route, PE3 will treat traffic to MAC DA = CE2 as unknown unicast, unless the MAC has been previously advertised by PE1.

Also, a DF election on PE1 is triggered. In general, a DF election is triggered by the same events as for all-active multi-homing. In this case, the DF will forward traffic to CE2 when the esi-activation-timer expiration occurs (the timer kicks in when there is a transition from non-DF to DF).

5.3.3. P2MP mLDP tunnels for BUM traffic in EVPN-MPLS Services

P2MP mLDP tunnels for BUM traffic in EVPN-MPLS services are supported and enabled through the use of the provider-tunnel context. If EVPN-MPLS takes ownership over the provider-tunnel, bgp-ad is still supported in the service but it will not generate BGP updates, including the PMSI Tunnel Attribute. The following CLI example shows an EVPN-MPLS service that uses P2MP mLDP LSPs for BUM traffic.

*A:PE-1>config>service>vpls(vpls or b-vpls)# info 
----------------------------------------------
  description "evpn-mpls-service with p2mp mLDP"
  bgp-evpn
    evi 10
    no ingress-repl-inc-mcast-advertisement 
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  provider-tunnel
    inclusive
      owner bgp-evpn-mpls
      root-and-leaf
      mldp
      no shutdown
      exit
    exit
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create
 exit

When provider-tunnel inclusive is used in EVPN-MPLS services, the following commands can be used in the same way as for BGP-AD or BGP-VPLS services:

  1. data-delay-interval
  2. root-and-leaf
  3. mldp
  4. shutdown

The following commands are used by provider-tunnel in BGP-EVPN MPLS services:

  1. [no] ingress-repl-inc-mcast-advertisement
    This command allows you to control the advertisement of IMET-IR and IMET-P2MP-IR routes for the service. See BGP-EVPN Control Plane for MPLS Tunnels for a description of the IMET routes. The following considerations apply:
    1. If configured as no ingress-repl-inc-mcast-advertisement, the system will not send the IMET-IR or IMET-P2MP-IR routes, regardless of the service being enabled for bgp-evpn mpls or bgp-evpn vxlan.
    2. If configured as ingress-repl-inc-mcast-advertisement and the PE is root-and-leaf, the system will send an IMET-P2MP-IR route.
    3. If configured as ingress-repl-inc-mcast-advertisement and the PE is no root-and-leaf, the system will send an IMET-IR route.
    4. Default value is ingress-repl-inc-mcast-advertisement.
  2. [no] owner {bgp-ad|bgp-vpls|bgp-evpn-mpls}
    The owner of the provider tunnel must be configured. The default value is no owner. The following considerations apply:
    1. Only one of the protocols will support a provider tunnel in the service and it must be explicitly configured.
    2. bgp-vpls and bgp-evpn are mutually exclusive.
    3. While bgp-ad and bgp-evpn can coexist in the same service, only bgp-evpn can be the provider-tunnel owner in such cases.

Figure 170 shows the use of P2MP mLDP tunnels in an EVI with a root node and a few leaf-only nodes.

Figure 170:  EVPN Services with p2mp mLDP—Control Plane 

Consider the use-case of a root-and-leaf PE4 where the other nodes are configured as leaf-only nodes (no root-and-leaf). This scenario is handled as follows:

  1. If ingress-repl-inc-mcast-advertisement is configured, then as soon as the bgp-evpn mpls option is enabled, the PE4 sends an IMET-P2MP route (tunnel type mLDP), or optionally, an IMET-P2MP-IR route (tunnel type composite). IMET-P2MP-IR routes allow leaf-only nodes to create EVPN-MPLS multicast destinations and send BUM traffic to the root.
  2. If ingress-repl-inc-mcast-advertisement is configured, PE1/2/3 will not send IMET-P2MP routes; only IMET-IR routes will be sent.
    1. The root-and-leaf node will import the IMET-IR routes from the leaf nodes but it will only send BUM traffic to the P2MP tunnel as long as it is active.
    2. If the P2MP tunnel goes operationally down, the root-and-leaf node will start sending BUM traffic to the evpn-mpls multicast destinations
  3. When PE1/2/3 receive and import the IMET-P2MP or IMET-P2MP-IR from PE4, they will join the mLDP P2MP tree signaled by PE4. They will issue an LDP label-mapping message including the corresponding P2MP FEC.

As described in IETF Draft draft-ietf-bess-evpn-etree, mLDP and Ingress Replication (IR) can work in the same network for the same service; that is, EVI1 can have some nodes using mLDP (for example, PE1) and others using IR (for example, PE2). For scaling, this is significantly important in services that consist of a pair of root nodes sending BUM in P2MP tunnels and hundreds of leaf-nodes that only need to send BUM traffic to the roots. By using IMET-P2MP-IR routes from the roots, the operator makes sure the leaf-only nodes can send BUM traffic to the root nodes without the need to set up P2MP tunnels from the leaf nodes.

When both static and dynamic P2MP mLDP tunnels are used on the same router, Nokia recommends that the static tunnels use a tunnel ID lower than 8193. If a tunnel ID is statically configured with a value equal to or greater than 8193, BGP-EVPN may attempt to use the same tunnel ID for services with enabled provider-tunnel, and fail to set up an mLDP tunnel.

Inter-AS option C or seamless-MPLS models for non-segmented mLDP trees are supported with EVPN for BUM traffic. The leaf PE that joins an mLDP EVPN root PE supports Recursive and Basic Opaque FEC elements (types 7 and 1, respectively). Therefore, packet forwarding is handled as follows:

  1. The ABR or ASBR may leak the root IP address into the leaf PE IGP, which allows the leaf PE to issue a Basic opaque FEC to join the root.
  2. The ABR or ASBR may distribute the root IP using BGP label-ipv4, which results in the leaf PE issuing a Recursive opaque FEC to join the root.

For more information about mLDP opaque FECs, refer to the 7450 ESS, 7750 SR, 7950 XRS, and VSR Layer 3 Services Guide: IES and VPRN and the 7450 ESS, 7750 SR, 7950 XRS, and VSR MPLS Guide.

All-active multihoming and single-active with an ESI label multihoming are supported in EVPN-MPLS services together with P2MP mLDP tunnels. Both use an upstream-allocated ESI label, as described in RFC 7432 section 8.3.1.2, which is popped at the leaf PEs, resulting in the requirement that, in addition to the root PE, all EVPN-MPLS P2MP leaf PEs must support this capability (including the PEs not connected to the multihoming Ethernet segment).

5.3.4. EVPN-VPWS for MPLS Tunnels

This section contains information about EVPN-VPWS for MPLS tunnels.

5.3.4.1. BGP-EVPN Control Plane for EVPN-VPWS

EVPN-VPWS for MPLS tunnels uses the RFC 8214 BGP extensions described in EVPN-VPWS for VXLAN Tunnels, with the following differences for the Ethernet AD per-EVI routes:

  1. The MPLS field encodes an MPLS label as opposed to a VXLAN VNI.
  2. The C flag is set if the control word is configured in the service.

5.3.4.2. EVPN for MPLS Tunnels in Epipe Services (EVPN-VPWS)

The use and configuration of EVPN-VPWS services is described in EVPN-VPWS for VXLAN Tunnels with the following differences when the EVPN-VPWS services use MPLS tunnels instead of VXLAN.

When MPLS tunnels are used, the bgp-evpn>mpls context must be configured in the Epipe. As an example, if Epipe 2 is an EVPN-VPWS service that uses MPLS tunnels between PE2 and PE4, this would be its configuration:

PE2>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-ac-name "AC-1"
    eth-tag 200
 remote-ac-name "AC-2"
    eth-tag 200
  mpls bgp 1
    ecmp 2
    no shutdown
sap 1/1/1:1 create
PE4>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-ac-name "AC-2"
    eth-tag 200
  remote-ac-name "AC-1"
    eth-tag 100
  mpls bgp 1
    ecmp 2
    no shutdown
spoke-sdp 1:1

Where the following BGP-EVPN commands, specific to MPLS tunnels, are supported in the same way as in VPLS services:

  1. mpls auto-bind-tunnel
  2. mpls control-word
  3. mpls entropy-label
  4. mpls force-vlan-vc-forwarding
  5. mpls shutdown

EVPN-VPWS Epipes with MPLS tunnels can also be configured with the following characteristics:

  1. Access attachment circuits can be SAPs or Spoke-SDPs. Only manually configured Spoke-SDPs are supported; BGP-VPWS and endpoints are not supported. The VC switching configuration is not supported on BGP-EVPN enabled pipes.
  2. EVPN-VPWS Epipes for MPLS tunnels support control words and entropy labels. When a control word is configured, the PE will set the C bit in its AD per-EVI advertisement and send the control word in the data path. In this case, the PE will also expect the control word to be received. If there is a mismatch between the received control word and the configured control word, the system will not set up the EVPN destination; as a result, the service will not come up.
  3. EVPN-VPWS Epipes support force-qinq-vc-forwarding [c-tag-c-tag | s-tag-c-tag] under bgp-evpn>mpls and qinq-vlan-translation s-tag.c-tag on ingress QinQ SAPs.
    When QinQ VLAN translation is configured at the ingress QinQ SAP, the service-delimiting outer and inner VLAN values can be translated to the configured values. In order to preserve the translated QinQ tags in the payload when sending EVPN packets, the command force-qinq-vc-forwarding s-tag-c-tag must be configured. This translation and preservation behavior is aligned with the "normalization" concept described in draft-ietf-bess-evpn-vpws-fxc. The VLAN tag processing described in Epipe Service Pseudowire VLAN Tag Processing applies to EVPN destinations in EVPN-VPWS services too.

The following features, described in EVPN-VPWS for VXLAN Tunnels, are also supported for MPLS tunnels:

  1. Advertisement of the Layer-2 MTU and consistency checking of the MTU of the AD per-EVI routes.
  2. Use of A/S PW and MC-LAG at access.
  3. EVPN multi-homing, including:
    1. Single-active and all-active
    2. Regular or virtual ESs
    3. All existing DF election modes

5.3.5. EVPN for MPLS Tunnels in Routed VPLS Services

EVPN-MPLS and IP-prefix advertisement (enabled by the ip-route-advertisement command) are fully supported in routed VPLS services and provide the same feature-set as EVPN-VXLAN. The following capabilities are supported in a service where bgp-evpn mpls is enabled:

  1. R-VPLS with VRRP support on the VPRN or IES interfaces
  2. R-VPLS support including ip-route-advertisement with regular interfaces
    This includes the advertisement and process of ip-prefix routes defined in IETF Draft draft-ietf-bess-evpn-prefix-advertisement with the appropriate encoding for EVPN-MPLS.
  3. R-VPLS support including ip-route-advertisement with evpn-tunnel interfaces
  4. R-VPLS with IPv6 support on the VPRN or IES IP interface

IES interfaces do not support either ip-route-advertisement or evpn-tunnel.

5.3.5.1. EVPN-MPLS Multi-Homing and Passive VRRP

SAP and Spoke-SDP based Ethernet Segments are supported on R-VPLS services where bgp-evpn mpls is enabled.

Figure 171 shows an example of EVPN-MPLS multi-homing in R-VPLS services, with the following assumptions.

  1. There are two subnets for a specific customer (for example, EVI1 and EVI2 in Figure 171), and a VPRN is instantiated in all the PEs for efficient inter-subnet forwarding.
  2. A “backhaul” R-VPLS with evpn-tunnel mode enabled is used in the core to interconnect all the VPRNs. EVPN IP-prefix routes are used to exchange the prefixes corresponding to the two subnets.
  3. An all-active ES is configured for EVI1 on PE1 and PE2.
  4. A single-active ES is configured for EVI2 on PE3 and PE4.
Figure 171:  EVPN-MPLS Multi-Homing in R-VPLS Services 

In the example in Figure 171, the hosts connected to CE1 and CE4 could use regular VRRP for default gateway redundancy; however, this may not be the most efficient way to provide upstream routing.

For example, if PE1 and PE2 are using regular VRRP, the upstream traffic from CE1 may be hashed to the backup IRB VRRP interface, instead of being hashed to the master. The same thing may occur for single-active multi-homing and regular VRRP for PE3 and PE4. The traffic from CE4 will be sent to PE3, while PE4 may be the VRRP master. In that case, PE3 will have to send the traffic to PE4, instead of route it directly.

In both cases, unnecessary bandwidth between the PEs is used to get to the master IRB interface. In addition, VRRP scaling is limited if aggressive keepalive timers are used.

Because of these issues, passive VRRP is recommended as the best method when EVPN-MPLS multi-homing is used in combination with R-VPLS redundant interfaces.

Passive VRRP is a VRRP setting in which the transmission and reception of keepalive messages is completely suppressed, and therefore the VPRN interface always behaves as the master. Passive VRRP is enabled by adding the passive keyword to the VRRP instance at creation, as shown in the following examples:

config service vprn 1 interface int-1 vrrp 1 passive

config service vprn 1 interface int-1 ipv6 vrrp 1 passive

For example, if PE1, PE2, and PE5 in Figure 171 use passive VRRP, even if each individual R-VPLS interface has a different MAC/IP address, because they share the same VRRP instance 1 and the same backup IP, the three PEs will own the same virtual MAC and virtual IP address (for example, 00-00-5E-00-00-01 and 10.0.0.254). The virtual MAC is auto-derived from 00-00-5E-00-00-VRID per RFC 3768. The following is the expected behavior when passive VRRP is used in this example.

  1. All R-VPLS IRB interfaces for EVI1 have their own physical MAC/IP address; they will also own the same default gateway virtual MAC and IP address.
  2. All EVI1 hosts have a unique configured default gateway; for example, 10.0.0.254.
  3. When CE1 or CE2 send upstream traffic to a remote subnet, the packets are routed by the closest PE because the virtual MAC is always local to the PE.
    For example, the packets from CE1 hashed to PE1 will be routed at PE1. The packets from CE1 hashed to PE2 will be routed directly at PE2.
  4. Downstream packets (for example, packets from CE3 to CE1), are routed directly by the PE to CE1, regardless of the PE to which PE5 routed the packets.
    For example, the packets from CE3 sent to PE1 will be routed at PE1. The packets from CE3 sent to PE2 will be routed at PE2.
  5. In case of ES failure in one of the PEs, the traffic will be forwarded by the available PE.
    For example, if the packets routed by PE5 arrive at PE1 and the link to CE1 is down, then PE1 will send the packets to PE2. PE2 will forward the packets to CE1 even if the MAC source address of the packets matches PE2's virtual MAC address. Virtual MACs bypass the R-VPLS interface MAC protection.

The following list summarizes the advantages of using passive VRRP mode versus regular VRRP for EVPN-MPLS multi-homing in R-VPLS services.

  1. Passive VRRP does not require multiple VRRP instances to achieve default gateway load-balancing. Only one instance per R-VPLS, hence only one default gateway, is needed for all the hosts.
  2. The convergence time for link/node failures is not impacted by the VRRP convergence, as all the nodes in the VRRP instance are master routers.
  3. Passive VRRP scales better than VRRP, as it does not use keepalive or BFD messages to detect failures and allow the backup to take over.

5.3.6. PBB-EVPN

This section contains information on PBB-EVPN.

5.3.6.1. BGP-EVPN Control Plane for PBB-EVPN

PBB-EVPN uses a reduced subset of the routes and procedures described in RFC 7432. The supported routes are:

  1. ES routes
  2. MAC/IP routes
  3. Inclusive Multicast Ethernet Tag routes.

5.3.6.1.1. EVPN Route Type 3 - Inclusive Multicast Ethernet Tag Route

This route is used to advertise the ISIDs that belong to I-VPLS services as well as the default multicast tree. PBB-epipe ISIDs are not advertised in Inclusive Multicast routes. The following fields are used:

  1. Route Distinguisher: Taken from the RD of the B-VPLS service within the BGP context. The RD can be configured or derived from the bgp-evpn evi value.
  2. Ethernet Tag ID: Encodes the ISID for a specified I-VPLS.
  3. IP address length: Always 32.
  4. Originating router's IP address: Carries the system address (IPv4 only).
  5. PMSI attribute:
    1. Tunnel type = Ingress replication (6).
    2. Flags = Leaf no required.
    3. MPLS label: Carries the MPLS label allocated for the service in the high-order 20 bits of the label field.
      Note:

      This label will be the same label used in the BMAC routes for the same B-VPLS service unless bgp-evpn mpls ingress-replication-bum-label is configured in the B-VPLS service.

    4. Tunnel endpoint = equal to the originating IP address.
      Note:

      The mLDP P2MP tunnel type is supported on PBB-EPVN services, but it can be used in the default multicast tree only.

5.3.6.1.2. EVPN Route Type 2 - MAC/IP Advertisement Route (or BMAC Routes)

The 7750 SR, 7450 ESS, or 7950 XRS will generate this route type for advertising BMAC addresses for the following:

  1. Learned MACs on B-SAPs or B-SDP-bindings - if mac-advertisement is enabled.
  2. Conditional static MACs - if mac-advertisement is enabled.
  3. B-VPLS shared-BMACs (source-bmacs) and dedicated-BMACs (es-bmacs).

The route type 2 generated by the router uses the following fields and values:

  1. Route Distinguisher—Taken from the RD of the VPLS service within the BGP context. The RD can be configured or derived from the bgp-evpn evi value.
  2. Ethernet Segment Identifier (ESI):
    1. ESI = 0 for the advertisement of source-bmac, es-bmacs, sap-bmacs, or sdp-bmacs if no multi-homing or single-active multi-homing is used.
    2. ESI=MAX-ESI (0xFF..FF) in the advertisement of es-bmacs used for all-active multi-homing.
    3. ESI different from zero or MAX-ESI for learned BMACs on B-SAPs/SDP-bindings if EVPN multi-homing is used on B-VPLS SAPs and SDP-bindings.
  3. Ethernet Tag ID: 0.
    Note: A different Ethernet Tag value may be used only when send-bvpls-evpn-flush is enabled.
  4. MAC address length: Always 48.
  5. BMAC Address learned, configured, or system-generated.
  6. IP address length zero and IP address omitted.
  7. MPLS Label 1: carries the MPLS label allocated by the system to the B-VPLS service. The label value is encoded in the high-order 20 bits of the field and will be the same label used in the routes type 3 for the same service unless bgp-evpn mpls ingress-replication-bum-label is configured in the service.
  8. The MAC Mobility extended community:
    1. The mac mobility extended community is used in PBB-EVPN for CMAC flush purposes if per ISID load balancing (single-active multi-homing) is used and a source-bmac is used for traffic coming from the ESI.
      If there is a failure in one of the ES links, CMAC flush through the withdrawal of the BMAC cannot be done (other ESIs are still working); therefore, the mac mobility extended community is used to signal CMAC flush to the remote PEs.
    2. When a dedicated es-bmac per ESI is used, the mac flush can be based on the withdrawal of the BMAC from the failing node.
    3. es-bmacs will be advertised as static (sticky bit set).
    4. Source-bmacs will be advertised as static MACs (sticky bit set). In the case of an update, if advertised to indicate that CMAC flush is needed, the mac mobility extended community will be added to the BMAC route including a higher sequence number (than the one previously advertised) in addition to the sticky bit.

5.3.6.1.3. EVPN Route Type 4 - Ethernet Segment Route

This route type is used for DF election as described in section BGP-EVPN Control Plane for MPLS Tunnels.

Note:

The EVPN route type 1—Ethernet Auto Discovery route is not used in PBB-EVPN.

5.3.6.2. PBB-EVPN for I-VPLS and PBB Epipe Services

The 7750 SR, 7450 ESS, and 7950 XRS SR OS implementation of PBB-EVPN reuses the existing PBB-VPLS model, where N I-VPLS (or Epipe) services can be linked to a B-VPLS service. BGP-EVPN is enabled in the B-VPLS and the B-VPLS becomes an EVI (EVPN Instance). Figure 172 shows the PBB-EVPN model in the SR OS.

Figure 172:  PBB-EVPN for I-VPLS and PFF Epipe Services 

Each PE in the B-VPLS domain will advertise its source-bmac as either configured in (b)vpls>pbb>source-bmac or auto-derived from the chassis mac. The remote PEs will install the advertised BMACs in the B-VPLS FDB. If a specified PE is configured with an ethernet-segment associated with an I-VPLS or PBB Epipe, it may also advertise an es-bmac for the ethernet-segment.

In the example shown in Figure 172, when a frame with MAC DA = AA gets to PE1, a mac lookup is performed on the I-VPLS FDB and BMAC-34 is found. A BMAC lookup on the B-VPLS FDB will yield the next-hop (or next-hops if the destination is in an all-active ethernet-segment) to which the frame is sent. As in PBB-VPLS, the frame will be encapsulated with the corresponding PBB header. A label specified by EVPN for the B-VPLS and the MPLS transport label are also added.

If the lookup on the I-VPLS FDB fails, the system will send the frame encapsulated into a PBB packet with BMAC DA = Group BMAC for the ISID. That packet will be distributed to all the PEs where the ISID is defined and will contain the EVPN label distributed by the Inclusive Multicast routes for that ISID, as well as the transport label.

For PBB-Epipes, all the traffic is sent in a unicast PBB packet to the BMAC configured in the pbb-tunnel.

The following CLI output shows an example of the configuration of an I-VPLS, PBB-Epipe, and their corresponding B-VPLS.

*A:PE-1>config# 
 
service vpls 1 b-vpls create
  description "pbb-evpn-service"
  service-mtu 2000
  pbb   
    source-bmac 00:00:00:00:00:03 
  bgp
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create
 
 
*A:PE-1>config# 
 
service vpls 101 i-vpls create
  pbb
    backbone-vpls 1
  sap 1/2/1:101 create
  spoke-sdp 1:102 create
 
*A:PE-1>config# 
 
service epipe 102 create
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102
  sap 1/2/1:102 create

Configure the bgp-evpn context as described in section EVPN for MPLS Tunnels in VPLS Services (EVPN-MPLS).

Some EVPN configuration options are not relevant to PBB-EVPN and are not supported when bgp-evpn is configured in a B-VPLS; these are as follows:

  1. bgp-evpn> [no] ip-route-advertisement
  2. bgp-evpn> [no] unknown-mac-route
  3. bgp-evpn> vxlan [no] shutdown
  4. bgp-evpn>mpls>force-vlan-vc-forwarding

When bgp-evpn>mpls no shutdown is added to a specified B-VPLS instance, the following considerations apply:

  1. BGP-AD is supported along with EVPN in the same B-VPLS instance.
  2. The following B-VPLS and BGP-EVPN commands are fully supported:
    1. vpls>backbone-vpls
    2. vpls>backbone-vpls>send-flush-on-bvpls-failure
    3. vpls>backbone-vpls>source-bmac
    4. vpls>backbone-vpls>use-sap-bmac
    5. vpls>backbone-vpls>use-es-bmac (See PBB-EVPN Multi-Homing in I-VPLS and PBB Epipe Services for more information)
    6. vpls>isid-policies
    7. vpls>static-mac
    8. vpls>SAP or SDP-binding>static-isid
    9. bgp-evpn>mac-advertisement - this command will only have affect on the 'learned' BMACs on saps or sdp-bindings and not on the system BMAC or sap/es-bmacs being advertised.
    10. bgp-evpn>mac-duplication and settings.
    11. bgp-evpn>mpls>auto-bind-tunnel and options.
    12. bgp-evpn>mpls>ecmp
    13. bgp-evpn>mpls>control-word
    14. bgp-evpn>evi
    15. bgp-evpn>mpls>ingress-replication-bum-label

5.3.6.2.1. Flood Containment for I-VPLS Services

In general, PBB technologies in the 7750 SR, 7450 ESS, or 7950 XRS SR OS support a way to contain the flooding for a specified I-VPLS ISID, so that BUM traffic for that ISID only reaches the PEs where the ISID is locally defined. Each PE will create an MFIB per I-VPLS ISID on the B-VPLS instance. That MFIB supports SAP or SDP-bindings endpoints that can be populated by:

  1. MMRP in regular PBB-VPLS
  2. IS-IS in SPBM

In PBB-EVPN, B-VPLS EVPN endpoints can be added to the MFIBs using EVPN Inclusive Multicast Ethernet Tag routes.

The example in Figure 173 shows how the MFIBs are populated in PBB-EVPN.

Figure 173:  PBB-EVPN and I-VPLS Flooding Containment 

When the B-VPLS 10 is enabled, PE1 will advertise as follows:

  1. A BMAC route containing PE1's system BMAC (00:01 as configured in pbb>source-bmac) along with an MPLS label.
  2. An Inclusive Multicast Ethernet Tag route (IMET route) with Ethernet-tag = 0 that will allow the remote B-VPLS 10 instances to add an entry for PE1 in the default multicast list.
Note:

The MPLS label that will be advertised for the MAC routes and the inclusive multicast routes for a specified B-VPLS can be the same label or a different label. As in regular EVPN-MPLS, this will depend on the [no] ingress-replication-bum-label command.

When I-VPLS 2001 (ISID 2001) is enabled as per the CLI in the preceding section, PE1 will advertise as follows:

  1. An additional inclusive multicast route with Ethernet-tag = 2001. This will allow the remote PEs to create an MFIB for the corresponding ISID 2001 and add the corresponding EVPN binding entry to the MFIB.
    This default behavior can be modified by the configured isid-policy. For instance, for ISIDs 1-2000, configure as follows:
isid-policy
    entry 10 create
      no advertise-local 
      range 1 to 2000
      use-def-mcast

This configuration has the following effect for the ISID range:

  1. no advertise-local instructs the system to not advertise the local active ISIDs contained in the 1 to 2001 range.
  2. use-def-mcast instructs the system to use the default flooding list as opposed to the MFIB.

The ISID flooding behavior on B-VPLS saps and sdp-bindings is as follows:

  1. B-VPLS saps and sdp-bindings are only added to the TLS-multicast list and not to the MFIB list (unless static-isids are configured, which is only possible for saps/sdp-bindings and not BGP-AD Spoke-SDPs).
    As a result, if the system needs to flood ISID BUM traffic and the ISID is also defined in remote PEs connected through saps or Spoke-SDPs without static-isids, then an isid-policy must be configured for the ISID so that the ISID uses the default multicast list.
  2. When an isid-policy is configured and a range of ISIDs use the default multicast list, the remote PBB-EVPN PEs will be added to the default multicast list as long as they advertise an IMET route with an ISID included in the policy's ISID range. PEs advertising IMET routes with Ethernet-tag = 0 are also added to the default multicast list (7750 SR, 7450 ESS, or 7950 XRS SR OS behavior).
  3. The B-VPLS 10 also allows the ISID flooding to legacy PBB networks via B-SAPs or B-SDPs. The legacy PBB network BMACs will be dynamically learned on those saps/binds or statically configured through the use of conditional static-macs. The use of static-isids is required so that non-local ISIDs are advertised.
sap 1/1/1:1 create 
exit
spoke-sdp 1:1 create
  static-mac
      mac 00:fe:ca:fe:ca:fe create sap 1/1/1:1 monitor fwd-status
  static-isid
      range 1 isid 3000 to 5000 create
Note:

The configuration of PBB-Epipes does not trigger any IMET advertisement.

5.3.6.2.2. PBB-EVPN and PBB-VPLS Integration

The 7750 SR, 7450 ESS, and 7950 XRS SR OS EVPN implementation supports draft-ietf-bess-evpn-vpls-seamless-integ so that PBB-EVPN and PBB-VPLS can be integrated into the same network and within the same B-VPLS service.

All the concepts described in section EVPN and VPLS Integration are also supported in B-VPLS services so that B-VPLS SAP or SDP-bindings can be integrated with PBB-EVPN destination bindings. The features described in that section also facilitate a smooth migration from B-VPLS SDP-bindings to PBB-EVPN destination bindings.

5.3.6.2.3. PBB-EVPN Multi-Homing in I-VPLS and PBB Epipe Services

The 7750 SR, 7450 ESS, and 7950 XRS SR OS PBB-EVPN implementation supports all-active and single-active multi-homing for I-VPLS and PBB Epipe services.

PBB-EVPN multi-homing reuses the ethernet-segment concept described in section EVPN Multi-Homing in VPLS Services. However, unlike EVPN-MPLS, PBB-EVPN does not use AD routes; it uses BMACs for split-horizon checks and aliasing.

5.3.6.2.3.1. System BMAC Assignment in PBB-EVPN

RFC 7623 describes two types of BMAC assignments that a PE can implement:

  1. Shared BMAC addresses that can be used for single-homed CEs and a number of multi-homed CEs connected to ethernet-segments.
  2. Dedicated BMAC addresses per ethernet-segment.

In this document and in 7750 SR, 7450 ESS, and 7950 XRS SR OS terminology:

  1. A shared-bmac (in IETF) is a source-bmac as configured in service>(b)vpls>pbb>source-bmac
  2. A dedicated-bmac per ES (in IETF) is an es-bmac as configured in service>pbb>use-es-bmac

BMAC selection and use depends on the multi-homing model; for single-active mode, the type of BMAC will impact the flooding in the network as follows:

  1. All-active multi-homing requires es-bmacs.
  2. Single-active multi-homing can use es-bmacs or source-bmacs.
    1. The use of source-bmacs minimizes the number of BMACs being advertised but has a larger impact on CMAC flush upon ES failures.
    2. The use of es-bmacs optimizes the CMAC flush upon ES failures at the expense of advertising more BMACs.

5.3.6.2.3.2. PBB-EVPN all-active multi-homing service model

Figure 174 shows the use of all-active multi-homing in the 7750 SR, 7450 ESS, and 7950 XRS SR OS PBB-EVPN implementation.

Figure 174:  PBB-EVPN All-Active Multi-Homing 

For example, the following shows the ESI-1 and all-active configuration in PE3 and PE4. As in EVPN-MPLS, all-active multi-homing is only possible if a LAG is used at the CE. All-active multi-homing uses es-bmacs, that is, each ESI will be assigned a dedicated BMAC. All the PEs part of the ES will source traffic using the same es-bmac.

In Figure 174 and the following configuration, the es-bmac used by PE3 and PE4 will be BMAC-34, i.e. 00:00:00:00:00:34. The es-bmac for a specified ethernet-segment is configured by the source-bmac-lsb along with the (b-)vpls>pbb>use-es-bmac command.

Configuration in PE3:

*A:PE3>config>lag(1)# info 
----------------------------------------------
  mode access
  encap-type dot1q
  port 1/1/1
  lacp active administrative-key 32768
  no shutdown
 
*A:PE3>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.3.3.3:0
  ethernet-segment ESI-1 create
    esi 00:34:34:34:34:34:34:34:34:34
    multi-homing all-active
    service-carving auto
    lag 1
    source-bmac-lsb 00:34 es-bmac-table-size 8
    no shutdown 
 
*A:PE3>config>service>vpls 1(b-vpls)# info 
----------------------------------------------
  bgp    
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      ecmp 2
      auto-bind-tunnel resolution any
  pbb   
    source-bmac 00:00:00:00:00:03
    use-es-bmac
 
*A:PE3>config>service>vpls (i-vpls)# info 
----------------------------------------------
  pbb
    backbone-vpls 1
  sap lag-1:101 create
 
*A:PE1>config>service>epipe (pbb)# info 
----------------------------------------------
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102
  sap lag-1:102 create
 

Configuration in PE4:

*A:PE4>config>lag(1)# info 
----------------------------------------------
  mode access
  encap-type dot1q
  port 1/1/1
  lacp active administrative-key 32768
  no shutdown
 
*A:PE4>config>service>system>bgp-evpn# info 
----------------------------------------------
  route-distinguisher 10.4.4.4:0
  ethernet-segment ESI-1 create
    esi 00:34:34:34:34:34:34:34:34:34
    multi-homing all-active
    service-carving auto
    lag 1
    source-bmac-lsb 00:34 es-bmac-table-size 8
    no shutdown 
 
 
*A:PE4>config>service>vpls 1(b-vpls)# info 
----------------------------------------------
  bgp    
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      ecmp 2
      auto-bind-tunnel resolution any
  pbb   
    source-bmac 00:00:00:00:00:04
    use-es-bmac
 
*A:PE4>config>service>vpls (i-vpls)# info 
----------------------------------------------
  pbb
    backbone-vpls 1
  sap lag-1:101 create
 
*A:PE4>config>service>epipe (pbb)# info 
----------------------------------------------
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102
  sap lag-1:102 create

The above configuration will enable the all-active multi-homing procedures for PBB-EVPN.

Note:

The ethernet-segment ESI-1 can also be used for regular VPLS services.

The following considerations apply when the ESI is used for PBB-EVPN.

  1. ESI association: Only LAG is supported for all-active multi-homing. The following commands are used for the LAG to ESI association:
    1. config>service>system>bgp-evpn>ethernet-segment# lag <id>
    2. config>service>system>bgp-evpn>ethernet-segment# source-bmac-lsb <MAC-lsb> [es-bmac-table-size <size>]
    3. Where:
      1. The same ESI may be used for EVPN and PBB-EVPN services.
      2. For PBB-EVPN services, the source-bmac-lsb attribute is mandatory and ignored for EVPN-MPLS services.
      3. The source-bmac-lsb attribute must be set to a specific 2-byte value. The value must match on all the PEs part of the same ESI, for example, PE3 and PE4 for ESI-1. This means that the configured pbb>source-bmac on the two PEs for B-VPLS 1 must have the same 4 most significant bytes.
      4. The es-bmac-table-size parameter modifies the default value (8) for the maximum number of virtual BMACs that can be associated with the ethernet-segment, i.e. es-bmacs. When the source-bmac-lsb is configured, the associated es-bmac-table-size is reserved out of the total FDB space.
      5. When multi-homing all-active is configured within the ethernet-segment, only a LAG can be associated with it. The association of a port or an sdp will be restricted by the CLI.
  2. If service-carving is configured in the ESI, the DF election algorithm will be a modulo function of the ISID and the number of PEs part of the ESI, as opposed to a modulo function of evi and number of PEs (used for EVPN-MPLS).
  3. A service-carving mode manual option is added so that the user can control what PE is DF for a specified ISID. The PE will be DF for the configured ISIDs and non-DF for the non-configured ISIDs.
  4. DF election: An all-active Designated Forwarder (DF) election is also carried out for PBB-EVPN. In this case, the DF election defines which of the PEs of the ESI for a specified I-VPLS is the one able to send the downstream BUM traffic to the CE. Only one DF per ESI is allowed in the I-VPLS service, and the non-DF will only block BUM traffic and in the downstream direction.
  5. Split-horizon function: In PBB-EVPN, the split-horizon function to avoid echoed packets on the CE is based on an ingress lookup of the ES BMAC (as opposed to the ESI label in EVPN-MPLS). In Figure 174 PE3 sends packets using BMAC SA = BMAC-34. PE4 does not send those packets back to the CE because BMAC-34 is identified as the es-bmac for ESI-1.
  6. Aliasing: In PBB-EVPN, aliasing is based on the ES BMAC sent by all the PEs part of the same ESI. See the following section for more information. In Figure 174 PE1 performs load balancing between PE3 and PE4 when sending unicast flows to BMAC-34 (es-bmac for ESI-1).

In the configuration above, a PBB-Epipe is configured in PE3 and PE4, both pointing at the same remote pbb tunnel backbone-dest-mac. On the remote PE, i.e. PE1, the configuration of the PBB-Epipe will point at the es-bmac:

*A:PE1>config>service>epipe (pbb)# info 
----------------------------------------------
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:34 isid 102
  sap 1/1/1:102 create

When PBB-Epipes are used in combination with all-active multi-homing, Nokia recommends using bgp-evpn mpls ingress-replication-bum-label in the PEs where the ethernet-segment is created, that is in PE3 and PE4. This guarantees that in case of flooding in the B-VPLS service for the PBB Epipe, only the DF will forward the traffic to the CE.

Note:

PBB-Epipe traffic always uses BMAC DA = unicast; therefore, the DF cannot check whether the inner frame is unknown unicast or not based on the group BMAC. Therefore, the use of an EVPN BUM label is highly recommended.

Aliasing for PBB-epipes with all-active multi-homing only works if shared-queuing or ingress policing is enabled on the ingress PE epipe. In any other case, the IOM will send the traffic to a single destination (no ECMP will be used in spite of the bgp-evpn mpls ecmp setting).

All-active multi-homed es-bmacs are treated by the remote PEs as eES:MAX-ESI BMACs. The following example shows the FDB in B-VPLS 1 in PE1 as shown in Figure 174:

*A:PE1# show service id 1 fdb detail 
 
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1         00:00:00:00:00:03 eMpls:                   EvpnS    06/12/15 15:35:39
                            192.0.2.3:262138
1         00:00:00:00:00:04 eMpls:                   EvpnS    06/12/15 15:42:52
                            192.0.2.4:262130
1         00:00:00:00:00:34 eES:                     EvpnS    06/12/15 15:35:57
                            MAX-ESI
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================
 

The show service id evpn-mpls on PE1 shows that the remote es-bmac, i.e. 00:00:00:00:00:34, has two associated next-hops, i.e. PE3 and PE4:

*A:PE1# show service id 1 evpn-mpls 
 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                 Transport                                
-------------------------------------------------------------------------------
192.0.2.3       262138        1           Yes             06/12/2015 15:34:48
                ldp                                        
192.0.2.4       262130        1           Yes             06/12/2015 15:34:48
                ldp                                        
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
 
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                     TEP Address     Egr Label   Last Change
                                               Transport  
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
 
===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
VBMacAddr                   TEP Address     Egr Label     Last Change
                                             Transport    
-------------------------------------------------------------------------------
00:00:00:00:00:34           192.0.2.3       262138        06/12/2015 15:34:48
                                            ldp            
00:00:00:00:00:34           192.0.2.4       262130        06/12/2015 15:34:48
                                            ldp            
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
=============================================================================== 

5.3.6.2.3.3. Network failures and convergence for all-active multi-homing

ES failures are resolved by the PEs withdrawing the es-bmac. The remote PEs will withdraw the route and update their list of next-hops for a specified es-bmac.

No mac-flush of the I-VPLS FDB tables is required as long as the es-bmac is still in the FDB.

When the route corresponding to the last next-hop for a specified es-bmac is withdrawn, the es-bmac will be flushed from the B-VPLS FDB and all the CMACs associated with it will be flushed too.

The following events will trigger a withdrawal of the es-bmac and the corresponding next-hop update in the remote PEs:

  1. B-VPLS transition to operationally down status.
  2. Change of pbb>source-bmac.
  3. Change of es-bmac (or removal of pbb use-es-bmac).
  4. Ethernet-segment transition to operationally down status.
Note:

Individual saps going operationally down in an ES will not generate any BGP withdrawal or indication so that the remote nodes can flush their CMACs. This is solved in EVPN-MPLS by the use of AD routes per EVI; however, there is nothing similar in PBB-EVPN for indicating a partial failure in an ESI.

5.3.6.2.3.4. PBB-EVPN Single-Active Multi-Homing Service Model

In single-active multi-homing, the non-DF PEs for a specified ESI will block unicast and BUM traffic in both directions (upstream and downstream) on the object associated with the ESI. Other than that, single-active multi-homing will follow the same service model defined in the section 'PBB-EVPN all-active multi-homing service model' with the following differences:

  1. The ethernet-segment will be configured for single-active: service>system>bgp-evpn>ethernet-segment>multi-homing single-active.
  2. For single-active multi-homing, the ethernet-segment can be associated with a port and sdp, as well as a lag.
  3. From a service perspective, single-active multi-homing can provide redundancy to the following services and access types:
    1. I-VPLS LAG and regular SAPs
    2. I-VPLS active/standby Spoke-SDPs
    3. EVPN single-active multi-homing is supported for PBB-Epipes only in two-node scenarios with local switching.
  4. While all-active multi-homing only uses es-bmac assignment to the ES, single-active multi-homing can use source-bmac or es-bmac assignment. The system allows the following user choices per B-VPLS and ES:
    1. A dedicated es-bmac per ES can be used. In that case, the pbb>use-es-bmac command will be configured in the B-VPLS and the same procedures explained in PBB-EVPN all-active multi-homing service model will follow with one difference. While in all-active multi-homing all the PEs part of the ESI will source the PBB packets with the same source es-bmac, single-active multi-homing requires the use of a different es-bmac per PE.
    2. A non-dedicated source-bmac can be used. In this case, the user will not configure pbb>use-es-bmac and the regular source-bmac will be used for the traffic. A different source-bmac has to be advertised per PE.
    3. The use of source-bmacs or es-bmacs for single-active multi-homed ESIs has a different impact on CMAC flushing, as shown in Figure 175.
    Figure 175:  Source-Bmac Versus Es-Bmac CMAC Flushing  
    1. If es-bmacs are used as shown in the representation on the right in Figure 175, a less-impacting CMAC flush is achieved, therefore, minimizing the flooding after ESI failures. In case of ESI failure, PE1 will withdraw the es-bmac 00:12 and the remote PE3 will only flush the CMACs associated with that es-bmac (only the CMACs behind the CE are flushed).
    2. If source-bmacs are used, as shown on the left-hand side of Figure Figure 175, in case of ES failure, a BGP update with higher sequence number will be issued by PE1 and the remote PE3 will flush all the CMACs associated with the source-bmac. Therefore, all the CMACs behind the PE's B-VPLS will be flushed, as opposed to only the CMACs behind the ESI's CE.
  5. As in EVPN-MPLS, the non-DF status can be notified to the access CE or network:
    1. LAG with or without LACP: In this case, the multi-homed ports on the CE will not be part of the same LAG. The non-DF PE for each service may signal that the LAG sap is operationally down by using eth-cfm fault-propagation-enable {use-if-tlv|suspend-ccm}.
    2. Regular Ethernet 802.1q/ad ports: In this case, the multi-homed ports on the CE/network will not be part of any LAG. The non-DF PE for each service will signal that the sap is operationally down by using eth-cfm fault-propagation-enable {use-if-tlv|suspend-ccm}.
    3. Active-standby PWs: in this case, the multihomed CE/network is connected to the PEs through an MPLS network and an active/standby Spoke-SDP per service. The non-DF PE for each service will make use of the LDP PW status bits to signal that the Spoke-SDP is standby at the PE side. Nokia recommends that the CE suppresses the signaling of PW status standby.

5.3.6.2.3.5. Network Failures and Convergence for Single-Active Multihoming

ESI failures are resolved depending on the BMAC address assignment chosen by the user:

  1. If the BMAC address assignment is based on the use of es-bmacs, DF and non-DFs will send the es-bmac/ESI=0 for a specified ESI. Each PE will have a different es-bmac for the same ESI (as opposed to the same es-bmac on all the PEs for all-active). In case of an ESI failure on a PE:
    1. The PE will withdraw its es-bmac route triggering a mac-flush of all the CMACs associated with it in the remote PEs.
  2. If the BMAC address assignment is based on the use of source-bmac, DF and non-DFs will advertise their respective source-bmacs. In case of an ES failure:
    1. The PE will re-advertise its source-bmac with a higher sequence number (the new DF will not re-advertise its source-bmac).
    2. The far-end PEs will interpret a source-bmac advertisement with a different sequence number as a flush-all-from-me message from the PE detecting the failure. They will flush all the CMACs associated with that BMAC in all the ISID services.

The following events will trigger a CMAC flush notification. A 'CMAC flush notification' means the withdrawal of a specified BMAC or the update of BMAC with a higher sequence number (SQN). Both BGP messages will make the remote PEs flush all the CMACs associated with the indicated BMAC:

  1. B-VPLS transition to operationally down status. This will trigger the withdrawal of the associated BMACs, regardless of the use-es-bmac setting.
  2. Change of pbb>source-bmac. This will trigger the withdrawal and re-advertisement of the source-bmac, causing the corresponding CMAC flush in the remote PEs.
  3. Change of es-bmac (removal of pbb use-es-bmac). This will trigger the withdrawal of the es-bmac and re-advertisement of the new es-bmac.
  4. Ethernet-Segment (ES) transition to operationally down or admin-down status. This will trigger an es-bmac withdrawal (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
  5. Service Carving Range change for the ES. This will trigger an es-bmac update with higher SQN (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
  6. Change in the number of candidate PEs for the ES. This will trigger an es-bmac update with higher SQN (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
  7. In an ESI, individual saps/sdp-bindings or individual I-VPLS going operationally down will not generate any BGP withdrawal or indication so that the remote nodes can flush their CMACs. This is solved in EVPN-MPLS by the use of AD routes per EVI; however, there is nothing similar in PBB-EVPN for indicating a partial failure in an ESI.

5.3.6.2.3.6. PBB-Epipes and EVPN Multi-Homing

EVPN multi-homing is supported with PBB-EVPN Epipes, but only in a limited number of scenarios. In general, the following applies to PBB-EVPN Epipes:

  1. PBB-EVPN Epipes don't support Spoke-SDPs that are associated with EVPN Ethernet Segments (ES).
  2. PBB-EVPN Epipes support all-active EVPN multi-homing as long as no local-switching is required in the Epipe instance where the ES is defined.
  3. PBB-EVPN Epipes support single-active EVPN multi-homing only in a two-node case scenario.

Figure 176 shows the EVPN MH support in a three-node scenario.

Figure 176:  PBB-EVPN MH in a Three-Node Scenario 

EVPN MH support in a three-node scenario has the following characteristics:

  1. All-active EVPN multi-homing is fully supported (diagram on the left in Figure 176). CE1 might also be multi-homed to other PEs, as long as those PEs are not PE2 or PE3. In this case, PE1 Epipe's pbb-tunnel would be configured with the remote ES BMAC.
  2. Single-active EVPN multi-homing is not supported in a three (or more)-node scenario (diagram on the right in Figure 176). Since PE1's Epipe pbb-tunnel can only point at a single remote BMAC and single-active multi-homing requires the use of separate BMACs on PE2 and PE3, the scenario is not possible and not supported regardless of the ES association to port/LAG/sdps.
  3. Regardless of the EVPN multi-homing type, the CLI prevents the user from adding a Spoke-SDP to an Epipe, if the corresponding SDP is part of an ES.

Figure 177 shows the EVPN MH support in a two-node scenario.

Figure 177:  PBB-EVPN MH in a Two-Node Scenario 

EVPN MH support in a two-node scenario has the following characteristics, as shown in Figure 177:

  1. All-active multi-homing is not supported for redundancy in this scenario because PE1's pbb-tunnel cannot point at a locally defined ES-BMAC. This is represented in the left-most scenario in Figure 177.
  2. Single-active multi-homing is supported for redundancy in a two-node three or four SAP scenario, as displayed by the two right-most scenarios in Figure 177).
    In these two cases, the Epipe pbb-tunnel will be configured with the source BMAC of the remote PE node.
    When two saps are active in the same Epipe, local-switching is used to exchange frames between the CEs.

5.3.6.2.4. PBB-EVPN and Use of P2MP mLDP Tunnels for Default Multicast List

P2MP mLDP tunnels can also be used in PBB-EVPN services. The use of provider-tunnel inclusive MLDP is only supported in the B-VPLS default multicast list; that is, no per-ISID IMET-P2MP routes are supported. IMET-P2MP routes in a B-VPLS are always advertised with Ethernet tag zero. All-active EVPN multihoming is supported in PBB-EVPN services together with P2MP mLDP tunnels; however, single-active multihoming is not supported. This capability is only required on the P2MP root PEs within PBB-EVPN services using all-active multihoming.

B-VPLS supports the use of MFIBs for ISIDs using ingress replication. The following considerations apply when provider-tunnel is enabled in a B-VPLS service.

  1. Local I-VPLS or static-ISIDs configured on the B-VPLS will generate IMET-IR routes and MFIBs will be created per ISID by default.
  2. The default IMET-P2MP or IMET-P2MP-IR route sent with ethernet-tag = 0 will be issued depending on the ingress-repl-inc-mcast-advertisement command.
  3. The following considerations apply if an isid-policy is configured in the B-VPLS.
    1. A range of ISIDs configured with use-def-mcast will make use of the P2MP tree, assuming the node is configured as root-and-leaf.
    2. A range of ISIDs configured with advertise-local will make the system advertise IMET-IR routes for the local ISIDs included in the range.

The following example CLI output shows a range of ISIDs (1000-2000) that use the P2MP tree and the system does not advertise the IMET-IR routes for those ISIDs. Other local ISIDs will advertise the IMET-IR and will use the MFIB to forward BUM packets to the EVPN-MPLS destinations created by the IMET-IR routes.

*A:PE-1>config>service>vpls(b-vpls)# info 
----------------------------------------------
  service-mtu 2000
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  isid-policy
    entry 10 create
      use-def-mcast
      no advertise-local
      range 1000 to 2000
      exit
    exit  
  provider-tunnel
    inclusive
      owner bgp-evpn-mpls
      root-and-leaf
      mldp
      no shutdown
      exit
    exit
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create
 exit

5.3.6.2.5. PBB-EVPN ISID-Based CMAC-Flush

SR OS supports ISID-based CMAC-flush procedures for PBB-EVPN I-VPLS services where no single-active Ethernet segments are used. SR OS also supports CMAC-flush procedure where other redundancy mechanisms, such as BGP-MH, need these procedures to avoid blackholes caused by a SAP or Spoke-SDP failure.

The CMAC-flush procedures are enabled on the I-VPLS service using the config>service>vpls>pbb>send-bvpls-evpn-flush CLI command. The feature can be disabled on a per-SAP or per-Spoke-SDP basis by using the disable-send-bvpls-evpn-flush command in the config>service>vpls>sap or config>service>vpls>spoke-sdp context.

With the feature enabled on an I-VPLS service and a SAP or Spoke-SDP, if there is a SAP or Spoke-SDP failure, the router sends a CMAC-flush notification for the corresponding BMAC and ISID. The router receiving the notification flushes all the CMACs associated with the indicated BMAC and ISID when the config>service>vpls>bgp-evpn>accept-ivpls-evpn-flush command is enabled for the B-VPLS service.

The CMAC-flush notification consists of an EVPN BMAC route that is encoded as follows: the ISID to be flushed is encoded in the Ethernet Tag field and the sequence number is incremented with respect to the previously advertised route.

If send-bvpls-evpn-flush is configured on an I-VPLS with SAPs or Spoke-SDPs, one of the following rules must be observed.

  1. The disable-send-bvpls-evpn-flush option is configured on the SAPs or Spoke-SDPs.
  2. The SAPs or Spoke-SDPs are not on an ES.
  3. The SAPs or Spoke-SDPs are on an ES or vES with no src-bmac-lsb enabled.
  4. The no use-es-bmac is enabled on the B-VPLS.

ISID-based CMAC-flush can be enabled in I-VPLS services with ES or vES. If enabled, the expected interaction between the RFC 7623-based CMAC-flush and the ISID-based CMAC-flush is as follows.

  1. If send-bvpls-evpn-flush is enabled in an I-VPLS service, the ISID-based CMAC-flush overrides (replaces) the RFC 7623-based CMAC-flushing.
  2. For each ES, vES, or B-VPLS, the system checks for at least one I-VPLS service that does not have send-bvpls-evpn-flush enabled.
    1. If ISID-based CMAC-flush is enabled for all I-VPLS services, RFC 7623-based CMAC-flushing is not triggered; only ISID-based CMAC-flush notifications are generated.
    2. If at least one I-VPLS service is found with no ISID-based CMAC-flush enabled, then RFC 7623-based CMAC-flushing notifications are triggered based on ES events.
      ISID-based CMAC-flush notifications are also generated for I-VPLS services that have send-bvpls-evpn-flush enabled.

Figure 178 shows an example where the ISID-based CMAC-flush prevents blackhole situations for a CE that is using BGP-MH as the redundancy mechanism in the I-VPLS with an ISID of 3000.

Figure 178:  Per-ISID CMAC-Flush Following a SAP Failure 

When send-bvpls-evpn-flush is enabled, the I-VPLS service is ready to send per-ISID CMAC-flush messages in the form of BMAC/ISID routes. The first BMAC/ISID route for an I-VPLS service is sent with sequence number zero; subsequent updates for the same route increment the sequence number. A BMAC/ISID route for the I-VPLS is advertised or withdrawn in the following cases:

  1. during I-VPLS send-bvpls-evpn-flush configuration and deconfiguration
  2. during I-VPLS association and disassociation from the B-VPLS service
  3. during I-VPLS operational status change (up/down)
  4. during B-VPLS operational status change (up/down)
  5. during B-VPLS bgp-evpn mpls status change (no shutdown/shutdown)
  6. during B-VPLS operational source BMAC change
  7. if no disable-send-bvpls-evpn-flush is configured for a SAP or Spoke-SDP, upon a failure on that SAP or Spoke-SDP, the system will send a per-ISID CMAC-flush message; that is, a BMAC/ISID route update with an incremented sequence number

If the user explicitly configures disable-send-bvpls-evpn-flush for a SAP or Spoke-SDP, the system will not send per-ISID CMAC-flush messages for failures on that SAP or Spoke-SDP.

The B-VPLS on the receiving node must be configured with bgp-evpn>accept-ivpls-evpn-flush to accept and process CMAC-flush non-zero Ethernet-tag MAC routes. If the accept-ivpls-evpn-flush command is enabled (the command is disabled by default), the node accepts non-zero Ethernet-tag MAC routes (BMAC/ISID routes) and processes them. When a new BMAC/ISID update (with an incremented sequence number) for an existing route is received, the router will flush all the CMACs associated with that BMAC and ISID. The BMAC/ISID route withdrawals will also cause a CMAC-flush.

Note:

Only BMAC routes with the Ethernet Tag field set to zero are considered for BMAC installation in the FDB.

The following CLI example shows the commands that enable the CMAC-flush feature on PE1 and PE3.

*A:PE-1>config>service>vpls(i-vpls)# info 
----------------------------------------------
  pbb
    backbone-vpls 10
    send-bvpls-evpn-flush
    exit
  exit
  bgp
    route-distinguisher 65000:1
    vsi-export “vsi_export”
    vsi-import “vsi_import”
  exit
  site “CE-1” create
    site-id 1
    sap lag-1:1
    site-activation-timer 3
    no shutdown
  exit
  sap lag-1:1 create
    no disable-send-bvpls-evpn-flush
    no shutdown
    exit
<snip>
*A:PE-3>config>service>vpls(b-vpls 10)# info 
----------------------------------------------
<snip>
  bgp-evpn
    accept-ivpls-evpn-flush

In the preceding example, with send-bvpls-evpn-flush enabled on the I-VPLS service of PE1, a BMAC/ISID route (for pbb source-bmac address BMAC 00:..:01 and ISID 3000) is advertised. If the SAP goes operationally down, PE1 will send an update of the source BMAC address (00:..:01) for ISID 3000 with a higher sequence number.

With accept-ivpls-evpn-flush enabled on PE3’s B-VPLS service, PE3 flushes all CMACs associated with BMAC 00:01 and ISID 3000. The CMACs associated with other BMACs or ISIDs are retained in PE3’s FDB.

5.3.6.2.6. PBB-EVPN ISID-based Route Targets

Routers with PBB-EVPN services use the following route types to advertise the ISID of a specific service.

  1. Inclusive Multicast Ethernet Tag routes (IMET-ISID routes) are used to auto-discover ISIDs in the PBB-EVPN network. The routes encode the service ISID in the Ethernet Tag field.
  2. BMAC-ISID routes are only used when ISID-based CMAC-flush is configured. The routes encode the ISID in the Ethernet Tag field.

Although the preceding routes are only relevant for routers where the advertised ISID is configured, they are sent with the B-VPLS route-target by default. As a result, the routes are unnecessarily disseminated to all the routers in the B-VPLS network.

SR OS supports the use of per-ISID or group of ISID route-targets, which limits the dissemination of IMET-ISID or BMAC-ISID routes for a specific ISID to the PEs where the ISID is configured.

The config>service>(b-)vpls>isid-route-target>isid-range from [to to] [auto-rt | route-target rt] command allows the user to determine whether the IMET-ISID and BMAC-ISID routes are sent with the B-VPLS route-target (default option, no command), or a route-target specific to the ISID or range of ISIDs.

The following configuration example shows how to configure ISID ranges as auto-rt or with a specific route-target.

*A:PE-3>config>service>(b-)vpls>bgp-evpn# 
isid-route-target 
[no] isid-range <from> [to <to>] {auto-rt|route-target <rt>}
/* For example:
*A:PE-3>config>service>(b-)vpls>bgp-evpn# 
isid-route-target
 isid-range 1000 to 1999 auto-rt
 isid-range 2000 route-target target:65000:2000

The auto-rt option auto-derives a route-target per ISID in the following format:

<2-byte-as-number>:<4-byte-value>

Where: 4-byte-value = 0x30+ISID, as described in RFC 7623. Figure 179 shows the format of the auto-rt option.

Figure 179:  PBB-EVPN auto-rt ISID-based Route Target Format  

Where:

  1. If it is 2 bytes, then the AS number is obtained from the config>router>autonomous-system command. If the AS number exceeds the 2 byte limit, then the low order 16-bit value is used.
  2. A = 0 for auto-derivation
  3. Type = 3, which corresponds to an ISID-type route-target
  4. ISID is the 24-bit ISID
  5. The type and sub-type are 0x00 and 0x02.

If isid-route-target is enabled, the export and import directions for IMET-ISID and BMAC-ISID route processing are modified as follows.

  1. Exported IMET-ISID and BMAC-ISID routes
    1. For local I-VPLS ISIDs and static ISIDs, IMET-ISID routes are sent individually with an ISID-based route-target (and without a B-VPLS route-target) unless the ISID is contained in an ISID policy for which no advertise-local is configured.
    2. If both isid-route-target and send-bvpls-evpn-flush options are enabled for an I-VPLS, the BMAC-ISID route is also sent with the ISID-based route-target and no B-VPLS route-target.
    3. The isid-route-target command affects the IMET-ISID and BMAC-ISID routes only. The BMAC-0, IMET-0 (BMAC and IMET routes with Ethernet Tag == 0), and ES routes are not impacted by the command.
  2. Imported IMET-ISID and BMAC-ISID routes
    1. Upon enabling isid-route-target for a specific I-VPLS, the BGP starts importing IMET-ISID routes with ISID-based route-targets, and (assuming the bgp-evpn accept-ivpls-evpn-flush option is enabled) BMAC-ISID routes with ISID-based route-targets.
    2. The new ISID-based RTs are added for import operations when the I-VPLS is associated to the B-VPLS service (and not based on the I-VPLS operational status), or when the static-isid is added.
    3. The system does not maintain a mapping of the route-targets and ISIDs for the imported routes. For example, if I-VPLS 1 and 2 are configured with the isid-route-target option and IMET-ISID=2 route is received with a route-target corresponding to ISID=1, then BGP imports the route and the router processes it.
    4. The router does not check the format of the received auto-derived route-targets. The route is imported as long as the route-target is on the list of RTs for the B-VPLS.
  3. If the isid-route-target option is configured for one or more I-VPLS services, the vsi-import and vsi-export policies are blocked in the B-VPLS. BGP peer import and export policies are still allowed. Matching on the export ISID-based route-target is supported.

5.3.7. Virtual Ethernet Segments

SR OS supports virtual Ethernet Segments (vES) for EVPN multi-homing in accordance with draft-sajassi-bess-evpn-virtual-eth-segment.

Regular Ethernet segments can only be associated to ports, LAGs, and SDPs, which satisfies the redundancy requirements for CEs that are directly connected to the ES PEs by a port, LAG, or SDP. However, this implementation does not work when an aggregation network exists between the CEs and the ES PEs, which requires different ESs to be defined for the port or LAG of the SDP.

Figure 180 shows an example of how CE1 and CE2 use all-active multi-homing to the EVPN-MPLS network despite the third-party Ethernet aggregation network to which they are connected.

Figure 180:  All-Active Multi-Homing on vES  

The ES association can be made in a more granular way by creating a vES. A vES can be associated to the following:

  1. Qtag-ranges on dot1q ports or LAGs
  2. S-tag-ranges on qinq ports or LAGs
  3. C-tag-ranges per s-tag on qinq ports or LAGs
  4. VC-ID ranges on SDPs

The following CLI examples show the vES configuration options:

config>service>system>bgp-evpn#
...
ethernet-segment vES-1 virtual create
 lag 1 
 dot1q 
   qtag-range 100 to 200
...
ethernet-segment vES-2 virtual create
 port 1/1/1 
 qinq
   s-tag 1 c-tag-range 100 to 200
   s-tag-range 2 to 10
...
ethernet-segment vES-3 virtual create
 sdp 1 
 vc-id-range 1000 to 2000
...

Where:

  1. The virtual keyword creates an ES as defined in draft-sajassi-bess-evpn-virtual-eth-segment. The configuration of the dot1q or qinq nodes is allowed only when the ES is created as virtual.
  2. On the vES, the user must first create a port, LAG, or SDP before configuring a VLAN or VC-ID association. When added, the port/LAG type and encap-type is checked as follows:
    1. If the encap-type is dot1q, only the dot1q context configuration is allowed; the qinq context cannot be configured.
    2. If the encap-type is qinq, only the qinq node is allowed; the dot1q context cannot be configured.
    3. A dot1q, qinq, or vc-id range is required for the vES to become operationally active.
  3. The dot1q qtag-range <qtag1> [to qtag1] command determines which VIDs are associated with the vES on a specific dot1q port or LAG. The group of SAPs that match the configured port/LAG and VIDs will be part of the vES.
  4. The qinq s-tag-range <qtag1> [to qtag1] command determines which outer VIDs are associated with the vES on the qinq port or LAG.
  5. The qinq s-tag <qtag1> c-tag-range <qtag2> [to <qtag2] command determines which inner c-tags per s-tag is associated with the vES on the qinq port or LAG.
  6. The vc-id range <vcid> [to vc-id] command determines which VC ids are associated with the vES on the configured SDP.

Although qtag values 0, * and 1 to 4094 are allowed, the following considerations must be taken in to account when configuring a dot1q or qinq vES:

  1. Up to 8 dot1q or qinq ranges may be configured in the same vES.
  2. When configuring a qinq vES, a qtag included in a s-tag-range cannot be included in the s-tag qtag of the s-tag qtag1 c-tag-range qtag2 [to qtag2] command. For example, the following combination is not supported in the same vES:
    s-tag-range 350 to 500
    s-tag 500 c-tag-range 100 to 200 
    The following example shows a supported combination:
    *A:PE75>config>service>system>bgp-evpn>eth-seg>qinq# info
    ----------------------------------------------
                    s-tag-range 100 to 200
                    s-tag-range 300 to 400
                    s-tag 500 c-tag-range 100 to 200
                    s-tag 600 c-tag-range 100 to 200
                    s-tag 600 c-tag-range 150 to 200
  3. vES associations that contain qtags <0, *, null> are special and treated as follows:
    1. When a special qtag value is configured in the from value of the range, the to value must be the same.
    2. Qtag values <0, *> are only supported for the qtag-range and c-tag-range; they are not supported in the s-tag-range.
    3. The qtag “null” value is only supported in the c-tag-range if the s-tag is configured as “*”.

Table 80 lists examples of the supported qtag values between 1 to 4094.

Table 80:  Examples of Supported qtag Values 

vES Configuration for Port 1/1/1

SAP Association

dot1q qtag-range 100

1/1/1:100

dot1q qtag-range 100 to 102

1/1/1:100, 1/1/1:101, 1/1/1:102

qinq s-tag 100 c-tag-range 200

1/1/1:100.200

qinq s-tag-range 100

All the SAPs 1/1/1:100.x

where: x is a value between 1 to 4094, 0, *

qinq s-tag-range 100 to 102

All SAPs 1/1/1:100.x, 1/1/1:101.x, 1/1/1:102.x

where: x is a value between 1 to 4094, 0, *

Table 81 lists all the supported combinations that include qtag values <0, *, null>. Any other combination of these special values is not supported.

Table 81:  Examples of Supported Combinations 

vES Configuration for Port 1/1/1

SAP Association

dot1q qtag-range 0

1/1/1:0

dot1q qtag-range *

1/1/1:*

qinq s-tag 0 c-tag-range *

1/1/1:0.*

qinq s-tag * c-tag-range *

1/1/1:*.*

qinq s-tag * c-tag-range null

1/1/1:*.null

qinq s-tag x c-tag-range 0

1/1/1:x.0

where: x is a value between 1 to 4094

qinq s-tag x c-tag-range *

1/1/1:x.*

where: x is a value between 1 to 4094

On vESs, the single-active and all-active modes are supported for EVPN-MPLS VPLS, Epipe, and PBB-EVPN services. Single-active multi-homing is supported on port and SDP-based vESs, and all-active multi-homing is only supported on LAG-based vESs.

The following considerations apply if the vES is used with PBB-EVPN services:

  1. BMAC allocation procedures are the same as the regular ES procedures.
    Note: Two all-active vESs must use different ES-BMACs, even if they are defined in the same LAG
  2. The vES implements CMAC flush procedures described in RFC 7623. Optionally, the ISID-based CMAC flush can be used for cases where the single-active vES does not use ES-BMAC allocation.

5.3.8. Preference-Based and Non-Revertive DF Election

In addition to the ES service-carving modes auto and off, the manual mode also supports the preference-based algorithm with the non-revertive option, as described in draft-rabadan-bess-evpn-pref-df.

When ES is configured to use the preference-based algorithm, the ES route is advertised with the Designated Forwarder (DF) election extended community (sub-type 0x06). Figure 181 shows the DF election extended community.

Figure 181:  DF Election Extended Community  

In the extended community, a DF type 2 preference algorithm is advertised with a 2-byte preference value (32767 by default) if the preference-based manual mode is configured. The Don't Preempt Me (DP) bit is set if the non-revertive option is enabled.

The following CLI excerpt shows the relevant commands to enable the preference-based DF election on a specific ES (regular or virtual):

config>service>system>bgp-evpn>ethernet-segment#
...
service-carving mode {manual|auto|off}
  service-carving manual
    [no] preference [create] [non-revertive]
      value <value>
    exit
  [no] evi <evi> [to <evi>] 
  [no] isid <isid> [to <isid>]
# value 0..65535; default 32767
...

Where:

  1. The preference value can be changed on an active ES without shutting down the ES, and therefore, a new DF can be forced for maintenance or other reasons.
  2. The service-carving mode must be changed to manual mode to create the preference context.
  3. The preference command is supported on regular or virtual ES, regardless of the multi-homing mode (single-active or all-active) or the service type (VPLS, I-VPLS, or Epipe).
  4. By default, the highest-preference PE in the ES becomes the DF for an EVI or ISID, using the DP bit as the tiebreaker first (DP=1 wins over DP=0) and the lowest PE-IP as the last-resort tiebreaker. All the explicitly configured EVI or ISID ranges select the lowest preference PE as the DF (whereas the non-configured EVI or ISID values select the highest preference PE).
    This selection is displayed as Cfg Range Type: lowest-pref in the following show command example.
    *A:PE-2# show service system bgp-evpn ethernet-segment name "vES-23"      
    ===============================================================================
    Service Ethernet Segment
    ===============================================================================
    Name                    : vES-23
    Eth Seg Type            : Virtual            
    Admin State             : Enabled            Oper State         : Up
    ESI                     : 01:23:23:23:23:23:23:23:23:23
    Multi-homing            : allActive          Oper Multi-homing  : allActive
    ES SHG Label            : 262141             
    Source BMAC LSB         : 00-23              
    ES BMac Tbl Size        : 8                  ES BMac Entries    : 0
    Lag Id                  : 1                  
    ES Activation Timer     : 3 secs (default)   
    Svc Carving             : manual             Oper Svc Carving   : manual
    Cfg Range Type          : lowest-pref        
    -------------------------------------------------------------------------------
    DF Pref Election Information
    -------------------------------------------------------------------------------
    Preference     Preference     Last Admin Change        Oper Pref      Do No
    Mode           Value                                   Value          Preempt
    -------------------------------------------------------------------------------
    non-revertive  100            12/21/2016 15:16:38      100            Enabled
    -------------------------------------------------------------------------------
    EVI Ranges: <none>
    ISID Ranges: <none>
    ===============================================================================
  5. The EVI and ISID ranges configured on the service-carving context are not required to be consistent with any ranges configured for vESs.
  6. If the non-revertive option is configured, when the former DF comes back up after a failure and checks existing ES routes, it will advertise an operational preference and DP bit, which does not cause a DF switchover for the ES EVI/ISID values.

The following configuration example shows the use of the preference-based algorithm and non-revertive option in an ES defined in PE1 and PE2.

*A:PE-1>config>service>system>bgp-evpn# info 
----------------------------------------------
 ethernet-segment "ES1" create
   esi 01:00:00:00:00:12:00:00:00:01
   service-carving manual
       preference non-revertive create
         value 10000
       exit
       evi 2001 to 4000 
   exit
   multi-homing single-active
   port 1/1/1
   no shutdown
   
/* example of vpls 1 - similar config exists for evis 2-4000 */
*A:PE-1>config>service>vpls# info 
----------------------------------------------
 vxlan vni 1 create
 exit
 bgp-evpn
   evi 1
   mpls bgp 1
     ecmp 2
     auto-bind-tunnel
       resolution any
     exit
 sap 1/1/1:1 create
 no shutdown
----------------------------------------------
*A:PE-2>config>service>system>bgp-evpn# info 
----------------------------------------------
 ethernet-segment "ES1" create
   esi 01:00:00:00:00:12:00:00:00:01
   service-carving manual
       preference non-revertive create
         value 5000
       exit
       evi 2001 to 4000 
   exit
   multi-homing single-active
   port 1/1/1
   no shutdown
   
*A:PE-2>config>service>vpls# info 
----------------------------------------------
 vxlan vni 1 create
 exit
 bgp-evpn
   evi 1
   mpls bgp 1
     ecmp 2
     auto-bind-tunnel
       resolution any
     exit
 sap 1/1/1:1 create
 no shutdown
----------------------------------------------

Based on the configuration in the preceding example, the PE behavior is as follows:

  1. Assuming the ES is no shutdown on both PE1 and PE2, the PEs exchange ES routes, including the [Pref, DP-bit] in the DF election extended community.
  2. For EVIs 1 to 2000, PE2 is immediately promoted to NDF, whereas PE1 becomes the DF, and (following the es-activation-timer) brings up its SAP in EVIs 1 to 2000.
    For EVIs 2001 to 4000, the result is the opposite and PE2 becomes the DF.
  3. If port 1/1/1 on PE1 goes down, PE1 withdraws its ES route and PE2 becomes the DF for EVIs 1 to 2000.
  4. When port 1/1/1 on PE1 comes back up, PE1 compares its ES1 preference with the preferences in the remote PEs in ES1. PE1 advertises the ES route with an “in-use operational” Pref = 5000 and DP=0. Because PE2's Pref is the same as PE1's operational value, but PE2's DP=1, PE2 continues to be the DF for EVIs 1 to 4000.
    Note: The DP bit is the tiebreaker in case of equal Pref and regardless of the choice of highest or lowest preference algorithm.
  5. PE1's “in-use” Pref and DP will continue to be [5000,0] until one of the following conditions is true:
    1. PE2 withdraws its ES route, in which case PE1 will re-advertise its admin Pref and DP [10000,DP=1]
    2. The user changes PE1's Pref configuration

5.3.9. EVPN-MPLS Routed VPLS Multicast Routing Support

IPv4 multicast routing is supported in an EVPN-MPLS VPRN routed VPLS service through its IP interface, when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 Multicast Routing Support for more details.

5.3.10. IGMP Snooping in EVPN-MPLS and PBB EVPN Services

IGMP snooping is supported in EVPN-MPLS VPLS and PBB-EVPN I-VPLS (where BGP EVPN is running in the associated B-VPLS service) services. It is also supported in EVPN-MPLS VPRN R-VPLS services (it is not supported in EVPN-MPLS IES R-VPLS services). It is required in scenarios where the operator does not want to flood all of the IP multicast traffic to the access nodes or CEs, and only wants to deliver IP multicast traffic for which IGMP reports have been received.

The following points apply when IGMP snooping is configured in EVPN-MPLS VPLS or PBB-EVPN I-VPLS services.

  1. IGMP snooping is enabled using the config>service>vpls>igmp-snooping no shutdown command.
  2. Queries and reports received on SAP or SDP bindings are snooped and properly handled; they are sent to SAP or SDP bindings as expected.
  3. Queries and reports on EVPN-MPLS or PBB-EVPN B-VPLS destinations are handled as follows.
    1. If received from SAP or SDP bindings, the queries and reports are sent to all EVPN-MPLS and PBB-EVPN B-VPLS destinations, regardless of whether the service is using an ingress replication or mLDP provider tunnel.
    2. If received on an EVPN-MPLS or PBB-EVPN B-VPLS destination, the queries and reports are processed and propagated to access SAP or SDP bindings, regardless of whether the service is using an ingress replication or mLDP provider tunnel.
    3. EVPN-MPLS and PBB-EVPN B-VPLS destinations are is treated as a single IGMP snooping interface and is always added as an mrouter.
    4. The debug trace output displays one copy of messages being sent to all EVPN-MPLS and PBB-EVPN B-VPLS destinations (the trace does not show a copy for each destination) and displays messages received from all EVPN-MPLS and PBB-EVPN B-VPLS destinations as coming from a single EVPN-MPLS interface.
Note:

When IGMP snooping is enabled with P2MP LSPs, at least one EVPN-MPLS multicast destination must be established to enable the processing of IGMP messages by the system. The use of P2MP LSPs is not supported when sending IPv4 multicast into an EVPN-MPLS R-VPLS service from its IP interface.

In the following show command output, the EVPN-MPLS destinations are shown as part of the MFIB (when igmp-snooping is in a no shutdown state), and the EVPN-MPLS logical interface is shown as an mrouter.

*A:PE-2# show service id 2000 mfib 
===============================================================================
Multicast FIB, Service 2000
===============================================================================
Source Address  Group Address         SAP or SDP Id                   Svc Id   Fwd
                                                                            Blk
-------------------------------------------------------------------------------
*               *                     eMpls:192.0.2.3:262132       Local    Fwd
                                      eMpls:192.0.2.4:262136       Local    Fwd
                                      eMpls:192.0.2.5:262131       Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
*A:PE-2# show service id 2000 igmp-snooping base 
===============================================================================
IGMP Snooping Base info for service 2000
===============================================================================
Admin State : Up
Querier     : 10.0.0.3 on evpn-mpls
-------------------------------------------------------------------------------
SAP or SDP                   Oper MRtr Pim  Send Max   Max  Max   MVR       Num
Id                        Stat Port Port Qrys Grps  Srcs Grp   From-VPLS Grps
                                                         Srcs            
-------------------------------------------------------------------------------
sap:1/1/1:2000            Up   No   No   No   None  None None  Local     0
evpn-mpls                 Up   Yes  N/A  N/A  N/A   N/A  N/A   N/A       N/A
===============================================================================
 
 
*A:PE-4# show service id 2000 igmp-snooping mrouters 
 
===============================================================================
IGMP Snooping Multicast Routers for service 2000
===============================================================================
MRouter          SAP or SDP Id                 Up Time        Expires   Version
-------------------------------------------------------------------------------
10.0.0.3         evpn-mpls                  0d 00:38:49    175s      3
-------------------------------------------------------------------------------
Number of mrouters: 1
===============================================================================

The equivalent output for PBB-EVPN services is similar to the output above for EVPN-MPLS services, with the exception that the EVPN destinations are named "b-EVPN-MPLS".

5.3.10.1. Data-driven IGMP Snooping Synchronization with EVPN Multihoming

When single-active multihoming is used, the IGMP snooping state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IGMP messages to be received to instantiate the IGMP snooping state after the Ethernet Segment (ES) activation timer expires; this could result in an increased outage.

The outage can be reduced by using MCS synchronization, which is supported for IGMP snooping in both EVPN-MPLS and PBB-EVPN services (see Multi-Chassis Synchronization for Layer 2 Snooping States). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs. Also, IGMP snooping state can be synchronized only on a SAP.

An increased outage would also occur when using all-active EVPN multihoming. The IGMP snooping state on an ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE might not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, IGMP snooping messages can be synchronized between the PEs on an ES using data-driven IGMP snooping state synchronization, which is supported in EVPN-MPLS services, PBB-EVPN services, and EVPN-MPLS VPRN R-VPLS services. The IGMP messages received on an ES SAP or Spoke-SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES BMAC (for PBB-EVPN) and these are used to synchronize the IGMP snooping state on the ES SAP or Spoke-SDP on the receiving PE.

Data-driven IGMP snooping state synchronization is supported for both all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS and EVPN-MPLS VPRN R-VPLS services, and for all-active multihoming in PBB-EVPN services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability. PBB-EVPN with IGMP snooping using single-active multihoming is not supported.

Data-driven IGMP snooping state synchronization is also supported with P2MP mLDP LSPs in both EVPN-MPLS and PBB-EVPN services. When P2MP mLDP LSPs are used in EVPN-MPLS services, all PEs (including the PEs not connected to a multihomed ES) in the EVPN-MPLS service must be running an SR OS version supporting this capability with IGMP snooping enabled and all network interfaces must be configured on FP3 or higher-based line cards.

Figure 182 shows the processing of an IGMP message for EVPN-MPLS. In PBB-EVPN services, the ES BMAC is used instead of the ESI label to synchronize the state.

Figure 182:  Data-driven IGMP Snooping Synchronization with EVPN Multihoming 

Data-driven synchronization is enabled by default when IGMP snooping is enabled within an EVPN-MPLS service using all-active multihoming or single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If IGMP snooping MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP then MCS synchronization takes precedence over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS IGMP synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down which prevents state synchronization to the PE with the ES down, as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it might not have any IGMP snooping state until the next IGMP messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive will avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services.

  1. Upgrade SR OS on all ES PEs to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.
  2. Enable IGMP snooping in the required services on all ES PEs. Traffic loss will occur until all ES PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.
  3. There is no action required on the non-ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services.

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.
  2. For EVPN-MPLS:
    1. Enable IGMP snooping on all non-ES PEs.Traffic loss will occur until the first set of join/query messages are processed by the non-ES PEs.
    2. Then enable IGMP snooping on all ES PEs. Traffic loss will occur until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.
  3. For PBB-EVPN:
    1. Enable IGMP snooping on all ES PEs. Traffic loss will occur until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.
    2. There is no action required on the non-ES PEs.

To aid with troubleshooting, the debug packet output displays the IGMP packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows.

6 2017/06/16 18:00:07.819 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: TX packet on svc 1
  from chaddr 5e:00:00:16:d8:2e
  send towards ES:esi-1
  Port  : evpn-mpls
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1
 
 
4 2017/06/16 18:00:07.820 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: RX packet on svc 1
  from chaddr d8:2e:ff:00:01:41
  received via evpn-mpls on ES:esi-1
  Port  : sap lag-1:1
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1

5.3.11. PIM Snooping for IPv4 in EVPN-MPLS and PBB-EVPN Services

PIM Snooping for VPLS allows a VPLS PE router to build multicast states by snooping PIM protocol packets that are sent over the VPLS. The VPLS PE then forwards multicast traffic based on the multicast states. When all receivers in a VPLS are IP multicast routers running PIM, multicast forwarding in the VPLS is efficient when PIM snooping for VPLS is enabled.

PIM snooping for IPv4 is supported in EVPN-MPLS and PBB-EVPN I-VPLS (where BGP EVPN is running in the associated B-VPLS service) services. It is enabled using the following command (as IPv4 multicast is enabled by default):

Example:
configure service vpls <service-id> pim-snooping

PIM snooping on SAPs and Spoke-SDPs operates in the same way as in a plain VPLS service. However, EVPN-MPLS/PBB-EVPN B-VPLS destinations are treated as a single PIM interface, specifically:

  1. Hellos and join/prune messages from SAPs or SDPs are always sent to all EVPN-MPLS or PBB-EVPN B-VPLS destinations.
  2. As soon as a hello message is received from one PIM neighbor on an EVPN-MPLS or PBB-EVPN I-VPLS destination, then the single interface representing all EVPN-MPLS or PBB-EVPN I-VPLS destinations will have that PIM neighbor.
  3. The EVPN-MPLS or PBB-EVPN B-VPLS destination split horizon logic ensures that IP multicast traffic and PIM messages received on an EVPN-MPLS or PBB-EVPN B-VPLS destination are not forwarded back to other EVPN-MPLS or PBB-EVPN B-VPLS destinations.
  4. The debug trace output displays one copy of messages being sent to all EVPN-MPLS or PBB-EVPN B-VPLS destinations (the trace does not show a copy for each destination) and displays messages received from all EVPN-MPLS or PBB-EVPN B-VPLS destinations as coming from a single EVPN-MPLS interface.

PIM snooping for IPv4 is supported in EVPN-MPLS services using P2MP LSPs and PBB-EVPN I-VPLS services with P2MP LSPs in the associated B-VPLS service. When PIM snooping is enabled with P2MP LSPs, at least one EVPN-MPLS multicast destination is required to be established to enable the processing of PIM messages by the system.

Multi-chassis synchronization (MCS) of PIM snooping for IPv4 state is supported for both SAPs and Spoke-SDPs which can be used with single-active multihoming. Care should be taken when using *.null to define the range for a QinQ virtual Ethernet Segment if the associated SAPs are also being synchronized by MCS, as there is no equivalent MCS sync-tag support to the *.null range.

PBB-EVPN services operate in a similar way to regular PBB services, specifically:

  1. The multicast flooding between the I-VPLS and the B-VPLS works in a similar way as for PIM snooping for IPv4 with an I-VPLS using a regular B-VPLS. The first PIM join message received over the local B-VPLS from a B-VPLS SAP or SDP or EVPN destination will add all of the B-VPLS SAP or SDP or EVPN components into the related multicast forwarding table associated with that I-VPLS context. The multicast packets will be forwarded throughout the B-VPLS on the per ISID single tree.
  2. When a PIM router is connected to a remote I-VPLS instance over the B-VPLS infrastructure, its location is identified by the B-VPLS SAP, SDP or by the set of all EVPN destinations on which its PIM hellos are received. The location is also identified by the source BMAC address used in the PBB header for the PIM hello message (this is the BMAC associated with the B-VPLS instance on the remote PBB PE).

In EVPN-MPLS services, the individual EVPN-MPLS destinations appear in the MFIB but the information for each EVPN-MPLS destination entry will always be identical, as shown below:

*A:PE# show service id 1 mfib
===============================================================================
Multicast FIB, Service 1
===============================================================================
Source Address  Group Address         Port Id                      Svc Id   Fwd
                                                                            Blk
-------------------------------------------------------------------------------
*               239.252.0.1           sap:1/1/9:1                  Local    Fwd
                                      eMpls:1.1.1.2:262141         Local    Fwd
                                      eMpls:1.1.1.3:262141         Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
*A:PE#

Similarly for the PIM neighbors:

*A:PE# show service id 1 pim-snooping neighbor
===============================================================================
PIM Snooping Neighbors ipv4
===============================================================================
Port Id                 Nbr DR Prty     Up Time       Expiry Time     Hold Time
  Nbr Address
-------------------------------------------------------------------------------
SAP:1/1/9:1             1               0d 00:08:17   0d 00:01:29     105
  10.0.0.1
EVPN-MPLS               1               0d 00:27:26   0d 00:01:19     105
  10.0.0.2
EVPN-MPLS               1               0d 00:27:26   0d 00:01:19     105
  10.0.0.3
-------------------------------------------------------------------------------
Neighbors : 3
===============================================================================
*A:PE#

A single EVPN-MPLS interface is shown in the outgoing interface, as can be seen in the following output:

*A:PE# show service id 1 pim-snooping group detail
===============================================================================
PIM Snooping Source Group ipv4
===============================================================================
Group Address      : 239.252.0.1
Source Address     : *
Up Time            : 0d 00:07:07
Up JP State        : Joined             Up JP Expiry       : 0d 00:00:37
Up JP Rpt          : Not Joined StarG   Up JP Rpt Override : 0d 00:00:00
RPF Neighbor       : 10.0.0.1
Incoming Intf      : SAP:1/1/9:1
Outgoing Intf List : EVPN-MPLS, SAP:1/1/9:1
Forwarded Packets  : 0                  Forwarded Octets   : 0
-------------------------------------------------------------------------------
Groups : 1
===============================================================================
*A:PE#

An example of the debug trace output for a join received on an EVPN-MPLS destination is shown below:

A:PE1# debug service id 1 pim-snooping packet jp
A:PE1#
32 2016/12/20 14:21:22.68 CET MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 02:16:02.460] PIM-RX ifId 1071394 ifName EVPN-MPLS 10.0.0.3 -> 224.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd3eb
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.252.0.1/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>

The equivalent output for PBB-EVPN services is similar to that above for EVPN-MPLS services, with the exception that the EVPN destinations are named "b-EVPN-MPLS".

5.3.11.1. Data-driven PIM Snooping for IPv4 Synchronization with EVPN Multihoming

When single-active multihoming is used, PIM snooping for IPv4 state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IPv4 PIM messages to be received to instantiate the PIM snooping for IPv4 state after the ES activation timer expires, which could result in an increased outage.

This outage can be reduced by using MCS synchronization, which is supported for PIM snooping for IPv4 in both EVPN-MPLS and PBB-EVPN services (see Multi-Chassis Synchronization for Layer 2 Snooping States). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs.

An increased outage would also occur when using all-active EVPN multihoming. The PIM snooping for IPv4 state on an all-active ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE might not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, snooped IPv4 PIM messages should be synchronized between the PEs on an ES using data-driven PIM snooping for IPv4 state synchronization, which is supported in both EVPN-MPLS and PBB-EVPN services. The IPv4 PIM messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES BMAC (for PBB-EVPN) and are used to synchronize the PIM snooping for IPv4 state on the ES SAP or spoke SDP on the receiving PE.

Data-driven PIM snooping state synchronization is supported for all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability with PIM snooping for IPv4 enabled. It is also supported with P2MP mLDP LSPs in the EVPN-MPLS services, in which case all PEs (including the PEs not connected to a multihomed ES) must have PIM snooping for IPv4 enabled and all network interfaces must be configured on FP3 or higher-based line cards.

In addition, data-driven PIM snooping state synchronization is supported for all-active multihoming in PBB-EVPN services and with P2MP mLDP LSPs in PBB-EVPN services. All PEs participating in a multihomed ES, and all PEs using PIM proxy mode (including the PEs not connected to a multihomed ES) in the PBB-EVPN service must be running an SR OS version supporting this capability and must have PIM snooping for IPv4 enabled. PBB-EVPN with PIM snooping for IPv4 using single-active multihoming is not supported.

Figure 183 shows the processing of an IPv4 PIM message for EVPN-MPLS. In PBB-EVPN services, the ES BMAC is used instead of the ESI label to synchronize the state.

Figure 183:  Data-driven PIM Snooping for IPv4 Synchronization with EVPN Multihoming 

Data-driven synchronization is enabled by default when PIM snooping for IPv4 is enabled within an EVPN-MPLS service using all-active multihoming and single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If PIM snooping for IPv4 MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP or spoke SDP, then MCS synchronization takes preference over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS PIM synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down, which prevents state synchronization to the PE with the ES down as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it might not have any PIM snooping for IPv4 state until the next PIM messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive will avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling PIM snooping for IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services.

  1. PIM snooping mode
    1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.
    2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss will occur until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.
    3. There is no action required on the non-ES PEs.
  2. PIM proxy mode
    1. EVPN-MPLS
      1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.
      2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss will occur until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.
      3. There is no action required on the non-ES PEs.
    2. PBB-EVPN
      1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.
      2. Then enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss will occur until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.
      3. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss will occur until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling PIM snooping or IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services.

  1. PIM snooping mode
    1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.
    2. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss will occur until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.
    3. There is no action required on the non-ES PEs.
  2. PIM proxy mode
    1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.
    2. Then enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss will occur until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.
    3. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss will occur until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

In the above steps, when PIM snooping for IPv4 is enabled, the traffic loss can be reduced or eliminated by configuring a larger hold-time (up to 300 seconds), during which multicast traffic is flooded.

To aid with troubleshooting, the debug packet output displays the PIM packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows:

6 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimVplsFwdJPToEvpn
Forwarding to remote peer on bgp-evpn ethernet-segment esi-1"
7 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:37.040] PIM-TX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>
 
 
4 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimProcessPdu
Received from remote peer on bgp-evpn ethernet-segment esi-1, will be applied on
 lag-1:1
"
5 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:30.740] PIM-RX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>

5.3.12. EVPN E-Tree

This section contains information about EVPN E-Tree.

5.3.12.1. BGP EVPN Control Plane for EVPN E-Tree

BGP EVPN control plane is extended and aligned with IETF RFC 8317 to support EVPN E-Tree services. Figure 184 shows the main EVPN extensions for the EVPN E-Tree information model.

Figure 184:  EVPN E-Tree BGP Routes  

The following BGP extensions are implemented for EVPN E-Tree services.

  1. An EVPN E-Tree extended community (EC) sub-type 0x5 is defined. The following information is included.
    1. The lower bit of the Flags field contains the L bit (where L=1 indicates leaf-ac).
    2. The Leaf label contains a 20-bit MPLS label in the high-order 20 bits of the label field.
  2. The new E-Tree EC is sent with the following routes.
    1. AD per-ES per PE route for BUM egress filtering
      Each EVPN E-Tree capable PE advertises an AD per-ES route with the E-Tree EC, and the following information:
      1. Service RD and route-target
        If ad-per-es-route-target evi-rt-set is configured, then non-zero ESI AD per-ES routes (used for multi-homing) are sent per the evi-rt-set configuration, but E-Tree zero-ESI routes (used for E-Tree) are sent based on the default evi-rt configuration.
      2. ESI = 0
        Eth Tag = MAX-ET
        MPLS label = zero
    1. AD per-EVI route for root or leaf configuration consistency check as follows:
      1. The E-Tree EC is sent with the AD per-EVI routes for a specific ES. In this case, no validation is performed by the implementation, and the leaf indication is only used for troubleshooting on the remote PEs.
      2. The MPLS label value is zero.
      3. All ACs in each EVI for a specific ES must be configured as either a root or leaf-ac, but not a combination. In case of a configuration error, for example where the AC in PE1 is configured as root and in PE2 as leaf-ac, the remote PE3 will receive the AD per-EVI routes with inconsistent leaf indication. However, the unicast filtering remains unaffected and is still handled by the FDB lookup information.
    2. MAC or IP routes for known unicast ingress filtering as follows:
      1. An egress PE sends all MAC or IP routes learned over a leaf-ac SAP or Spoke-SDP with this E-Tree EC indicating that the MAC or IP belongs to a leaf-ac.
      2. The MPLS label value in the EC is 0.
      3. Upon receiving a route with E-Tree EC, the ingress PE imports the route and installs the MAC in the FDB with a leaf flag (if there is a leaf indication in the route). Any frame coming from a leaf-ac for which the MAC DA matches a leaf-ac MAC is discarded at the ingress.
      4. If two PEs send the same MAC with the same ESI but inconsistent root or leaf indication, the MAC is installed in the FDB as root.

5.3.12.2. EVPN for MPLS Tunnels in E-Tree Services

EVPN E-Tree services are modeled as VPLS services configured as E-Trees with bgp-evpn mpls enabled.

The following example CLI shows an excerpt of a VPLS E-Tree service with EVPN E-Tree service enabled.

*A:PE1>config>service>system>bgp-evpn#
  evpn-etree-leaf-label
*A:PE1>config>service# vpls 1 customer 1 etree create 
*A:PE1>config>service>vpls(etree)# info 
----------------------------------------------
  description "ETREE-enabled evpn-mpls-service"
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      ecmp 2
      auto-bind-tunnel resolution any
      ingress-replication-bum-label
  sap lag-1:1 leaf-ac create 
  exit
  sap 2/1/1:1 leaf-ac create 
  exit
  sap 2/2/2:1 create 
  exit
  spoke-sdp 3:1 leaf-ac create 
  exit

The following considerations apply to the configuration of the EVPN E-Tree service.

  1. The config>service>system>bgp-evpn# evpn-etree-leaf-label command is required prior to the configuration of any EVPN E-Tree service, and is relevant for EVPN E-Tree services only. The command allocates an E-Tree leaf label on the system and programs the ILM for this label. The ILM label ensures that in-flight traffic always has an ILM entry for lookup, therefore avoiding discards when the service is shutdown and subsequently no shutdown in a short period of time.
  2. The config>service# vpls x etree create command is compatible with the bgp-evpn mpls context.
  3. As in VPLS E-Tree services, an AC that is not configured as a leaf-ac is treated as root-ac.
  4. MAC addresses learned over a leaf-ac SAP or SDP binding are advertised as leaf MAC addresses.
  5. Any PE with one or more bgp-evpn enabled VPLS E-Tree service advertises an AD per-ES per-PE route with the leaf indication and leaf label that will be used for BUM egress filtering.
  6. Any leaf-ac SAP or SDP binding defined in an ES triggers the advertisement of an AD per-EVI route with the leaf indication.
  7. EVPN E-Tree services use the following CLI commands:
    1. sap sap-id leaf-ac create
    2. mesh-sdp sdp-id:vc-id leaf-ac create
    3. spoke-sdp sdp-id:vc-id leaf-ac create
  8. The root-leaf-tag command is blocked in VPLS E-Tree services where bgp-evpn mpls is enabled

5.3.12.3. EVPN E-Tree Operation

EVPN E-Tree supports all operations related to flows among local root-ac and leaf-ac objects in accordance with IETF Draft draft-ietf-bess-evpn-etree. This section describes the extensions required to forward to or from BGP-EVPN destinations.

5.3.12.3.1. EVPN E-Tree Known Unicast Ingress Filtering

Known unicast traffic forwarding is based on ingress PE filtering. Figure 185 shows an example of EVPN-E-Tree forwarding behavior for known unicast.

Figure 185:  EVPN E-Tree Known Unicast Ingress Filtering  

MAC addresses learned on leaf-ac objects are advertised in EVPN with their corresponding leaf indication.

In Figure 185, PE1 advertises MAC1 using the E-Tree EC and leaf indication, and PE2 installs MAC1 with a leaf flag in the FDB.

Assuming MAC DA is present in the local FDB (MAC1 in the FDB of PE2) when PE2 receives a frame, it is handled as follows.

  1. If the unicast frame enters a root-ac, the frame follows regular data plane procedures; that is, it is sent to the owner of the MAC DA (local SAP or SDP binding or remote BGP EVPN PE) without any filtering.
  2. If the unicast frame enters a leaf-ac, it is handled as follows.
    1. A MAC DA lookup is performed on the FDB.
    2. If there is a hit and the MAC was learned as an EVPN leaf (or from a leaf-ac), then the frame is dropped at ingress.
    3. The source MAC (MAC2) is learned and marked as a leaf-learned MAC. It is advertised by the EVPN with the corresponding leaf indication.
  3. A MAC received with a root and leaf indication from different PEs in the same ES is installed as root.

The ingress filtering for E-Tree leaf-to-leaf traffic requires the implementation of an extra leaf EVPN MPLS destination per remote PE (containing leaf objects) per E-Tree service. The ingress filtering for E-Tree leaf-to-leaf traffic is as follows.

  1. A separate EVPN MPLS bind is created for unicast leaf traffic in the service. The internal EVPN MPLS destination is created for each remote PE that contains a leaf and advertises at least one leaf MAC.
  2. The creation of the internal EVPN MPLS destination is triggered when a MAC route with L=1 in the E-Tree EC is received. Any EVPN E-Tree service can potentially use one extra EVPN MPLS destination for leaf unicast traffic per remote PE.
    The extra destination in the EVPN E-Tree service is for unicast only and it is not part of the flooding list. It is resource-accounted and displayed in the tools dump service evpn usage command, as shown in the following example output.
    A:PE-4# tools dump service evpn usage 
    vxlan-evpn-mpls usage statistics at 01/23/2017 00:53:14:
    MPLS-TEP                                        :             3
    VXLAN-TEP                                       :             0
    Total-TEP                                       :      3/ 16383
    Mpls Dests (TEP, Egress Label + ES + ES-BMAC)   :            10
    Mpls Etree Leaf Dests                           :             1
    Vxlan Dests (TEP, Egress VNI)                   :             0
    Total-Dest                                      :     10/196607
    Sdp Bind +  Evpn Dests                          :     13/245759
    ES L2/L3 PBR                                    :      0/ 32767
    Evpn Etree Remote BUM Leaf Labels               :             3
  3. MACs received with L=1 point to the EVPN MPLS destination, whereas root MACs point to the “root” destination.

5.3.12.3.2. EVPN E-Tree BUM Egress Filtering

BUM traffic forwarding is based on egress PE filtering. Figure 186 shows an example of EVPN E-Tree forwarding behavior for BUM traffic.

Figure 186:  EVPN E-Tree BUM Egress Filtering  

In Figure 186, BUM frames are handled as follows when they ingress PE or PE2.

  1. If the BUM frame enters a root-ac, the frame follows regular EVPN data plane procedures.
  2. If the BUM frame enters a leaf-ac, the frame handling is as follows:
    1. The frame is marked as leaf and forwarded or replicated to the egress IOM.
    2. At the egress IOM, the frame is flooded in the default multicast list subject to the following.
      1. Leaf entries are skipped when BUM traffic is forwarded. This prevents leaf-to-leaf BUM traffic forwarding.
      2. Traffic to remote BGP EVPN PEs is encapsulated with the EVPN label stack. If a leaf ESI label present for the far-end PE (L1 in Figure 186), the leaf ESI label is added at the bottom of the stack; the remaining stack will follow (including EVI label). If there is no leaf ESI label for the far-end egress PE, no additional label is added to the stack. This means that the egress PE does not have any E-Tree enabled service, but it can still work with the VPLS E-Tree service available in PE2.

The BUM-encapsulated packet is received on the network ingress interface at the egress PE or PE1. The packet is processed as follows.

  1. A normal ILM lookup is performed for each label (including the EVI label) in the stack.
  2. Further label lookups are performed when the EVI label ILM lookup is complete. If the lookup yields a leaf label, all the leaf-acs are skipped when flooding to the default-multicast list at the egress PE.

5.3.12.3.3. EVPN E-Tree Egress Filtering Based on MAC Source Address

The egress PE checks the MAC Source Address (SA) for traffic received without the leaf MPLS label. This check covers corner cases where the ingress PE sends traffic originating from a leaf-ac but without a leaf indication.

In Figure 186, PE2 receives a frame with MAC DA = MAC3 and MAC SA = MAC2. Because MAC3 is a root MAC, MAC lookup at PE2 allows the system to unicast the packet to PE1 without the leaf label. If MAC3 was no longer in PE1's FDB, PE1 would flood the frame to all the root and leaf-acs, despite the frame having originated from a leaf-ac.

To minimize and prevent leaf traffic from leaking to other leaf-acs (as described in the preceding case), the egress PE always performs a MAC SA check for all types of traffic. The data path performs MAC SA-based egress filtering as follows.

  1. An Ethernet frame may be treated as originating from a leaf-ac due to several reasons, which require the system to set a flag to indicate leaf traffic. The flag is set if one of the following conditions is true: if the frames arrive on a leaf SAP, if EVPN traffic arrives with a leaf label, or if a MAC SA is flagged as a leaf SA.
  2. After the flag is set, the action taken depends on the type of traffic.
    1. Unicast traffic: An FDB lookup is performed, and if the MAC DA FDB entry is marked as a leaf type, the frame is dropped to prevent leaf-to-leaf forwarding.
    2. BUM traffic: The flag is taken into account at the egress IOM and leaf-to-leaf forwarding is suppressed.

5.3.12.4. EVPN E-Tree and EVPN Multi-homing

EVPN E-Tree procedures support all-active and single-active EVPN multi-homing. Ingress filtering can handle MACs learned on ES leaf-ac SAP or SDP-bindings. If a MAC associated with an ES leaf-ac is advertised with a different E-Tree indication or if the AD per-EVI routes have inconsistent leaf indications, then the remote PEs performing the aliasing treat the MAC as root.

Figure 187 shows the expected behavior for multi-homing and egress BUM filtering.

Figure 187:  EVPN E-Tree BUM Egress Filtering and Multi-homing  

Multi-homing and egress BUM filtering in Figure 187 is handled as follows:

  1. BUM frames received on an ES leaf-ac are flooded to the EVPN based on EVPN E-Tree procedures. The leaf ESI label is sent when flooding to other PEs in the same ES, and additional labels are not added to the stack.
    When flooding in the default multicast list, the egress PE skips all the leaf-acs (including the ES leaf-acs) on the assumption that all ACs in a specific ES for a specified EVI have a consistent E-Tree configuration, and they send an AD per-EVI route with a consistent E-Tree indication.
  2. BUM frames received on an ES root-ac are flooded to the EVPN based on regular EVPN procedures. The regular ES label is sent for split-horizon when packets are sent to the DF or NDF PEs in the same ES. When flooding in the default multicast list, the egress PE skips the ES SAPs based on the ES label lookup.

If the PE receives an ES MAC from a peer that shares the ES and decides to install it against the local ES SAP that is oper-up, it checks the E-Tree configuration (root or leaf) of the local ES SAP against the received MAC route. The MAC route is processed as follows.

  1. If the E-Tree configuration does not match, then the MAC is not installed against any destination until the misconfiguration is resolved.
  2. If the SAP is oper-down, the MAC is installed against the EVPN destination to the peer.

5.3.12.5. PBB-EVPN E-Tree Services

SR OS supports PBB-EVPN E-Tree services in accordance with IETF RFC 8317. PBB-EVPN E-Tree services are modeled as PBB-EVPN services where some I-VPLS services are configured as etree and some of their SAP or Spoke-SDPs are configured as leaf-acs.

The procedures for the PBB-EVPN E-Tree are similar to those for the EVPN E-Tree, except that the egress leaf-to-leaf filtering for BUM traffic is based on the BMAC source address. Also, the leaf label and the EVPN AD routes are not used.

The PBB-EVPN E-Tree operation is as follows.

  1. When one or more I-VPLS E-Tree services are linked to a B-VPLS, the leaf backbone source MAC address (leaf-source-bmac parameter) is used for leaf-originated traffic in addition to the source B-VPLS MAC address (source-bmac parameter) that is used for sourcing root traffic.
  2. The leaf backbone source MAC address for PBB must be configured using the command config>service>pbb>leaf-source-bmac ieee-address prior to the configuration of any I-VPLS E-Tree service.
  3. The leaf-source-bmac address is advertised in a BMAC route with a leaf indication.
  4. Known unicast filtering occurs at the ingress PE. When a frame enters an I-VPLS leaf-ac, a MAC lookup is performed. If the CMAC DA is associated with a leaf BMAC, the frame is dropped.
  5. Leaf-to-leaf BUM traffic filtering occurs at the egress PE. When flooding BUM traffic with the BMAC SA matching a leaf BMAC, the egress PE skips the I-VPLS leaf-acs.

The following CLI example shows an I-VPLS E-Tree service that uses PBB-EVPN E-Tree. The leaf-source-bmac address must be configured prior to the configuration of the I-VPLS E-Tree. As is the case in regular E-Tree services, SAP and Spoke-SDPs that are not explicitly configured as leaf-acs are considered root-ac objects.

A:PE-2>config>service# info
----------------------------------------------
        pbb
            leaf-source-bmac 00:00:00:00:00:22
        exit
        vpls 1000 customer 1 b-vpls create
            service-mtu 2000
            bgp
            exit
            bgp-evpn
                evi 1000
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp                      
                shutdown
            exit
            no shutdown
        exit
        vpls 1001 customer 1 i-vpls etree create
            pbb
                backbone-vpls 1000
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:1001 leaf-ac create
                no shutdown
            exit
            sap 1/1/1:1002 create
                no shutdown
            exit
            no shutdown
        exit

The following considerations apply to PBB-EVPN E-Trees and multi-homing.

  1. All-active multi-homing is not supported on leaf-ac I-VPLS SAPs.
  2. Single-active multi-homing is supported on leaf-ac I-VPLS SAPs and Spoke-SDPs.
  3. ISID- and RFC 7623-based CMAC-flush are supported in addition to PBB-EVPN E-Tree services and single-active multi-homing.

5.3.13. MPLS Entropy Label and Hash Label

The router supports the MPLS entropy label (RFC 6790) and the Flow Aware Transport label (known as the hash label) (RFC 6391). These labels allow LSR nodes in a network to load-balance labeled packets in a much more granular fashion than allowed by simply hashing on the standard label stack. The entropy label can be enabled on bgp-evpn services (VPLS and Epipe). See the 7450 ESS, 7750 SR, 7950 XRS, and VSR MPLS Guide for further information.

5.3.14. Inter-AS Option B and Next-Hop-Self Route-Reflector for EVPN-MPLS

Inter-AS Option B and Next-Hop-Self Route-Reflector (VPN-NH-RR) functions are supported for the BGP-EVPN family in the same way both functions are supported for IP-VPN families.

A typical use-case for EVPN Inter-AS Option B or EVPN VPN-NH-RR is Data Center Interconnect (DCI) networks, where cloud and service providers are looking for efficient ways to extend their Layer 2 and Layer 3 tenant services beyond the data center and provide a tighter DC-WAN integration. While the instantiation of EVPN services in the DC GW to provide this DCI connectivity is a common model, some operators use Inter-AS Option B or VPN-NH-RR connectivity to allow the DC GW to function as an ASBR or ABR respectively, and the services are only instantiated on the edge devices.

Figure 188 shows a DCI example where the EVPN services in two DCs are interconnected without the need for instantiating services on the DC GWs.

Figure 188:  EVPN Inter-AS Option B or VPN-NH-RR Model 

The ASBRs or ABRs connect the DC to the WAN at the control plane and data plane levels where the following considerations apply.

  1. From a control plane perspective, the ASBRs or ABRs perform the following tasks:
    1. accept EVPN-MPLS routes from a BGP peer
      EVPN-VXLAN routes are not supported.
    2. extract the MPLS label from the EVPN NLRI or attribute and program a label swap operation on the IOM
    3. re-advertise the EVPN-MPLS route to the BGP peer in the other Autonomous Systems (ASs) or IGP domains
      The re-advertised route will have a Next-Hop-Self and a new label encoded for those routes that came with a label.
  2. From a data plan perspective, the ASBRs and ABRs terminate the ingress transport tunnel, perform an EVPN label swap operation, and send the packets on to an interface (if E-BGP is used) or a new tunnel (if I-BGP is used).
  3. The ASBR or ABR resolves the EVPN routes based on the existing bgp next-hop-resolution command for family vpn, where vpn refers to EVPN, VPN-IPv4, and VPN-IPv6 families.
*A:ABR-1# configure router bgp next-hop-resolution labeled-routes transport-tunnel family vpn resolution-filter 
  - resolution-filter
 [no] bgp             - Use BGP tunnelling for next hop resolution
 [no] ldp             - Use LDP tunnelling for next hop resolution
 [no] rsvp            - Use RSVP tunnelling for next hop resolution
 [no] sr-isis         - Use sr-isis tunnelling for next hop resolution
 [no] sr-ospf         - Use sr-ospf for next hop resolution
 [no] sr-te           - Use sr-te for next hop resolution
 [no] udp             - Use udp for next hop resolution

Refer to the 7450 ESS, 7750 SR, 7950 XRS, and VSR Unicast Routing Protocols Guide for more information about the next-hop resolution of BGP-labeled routes.

Inter-AS Option B for EVPN services on ABSRs and VPN-NH-RR on ABRs re-use the existing commands enable-inter-as-vpn and enable-rr-vpn-forwarding respectively. The two commands enable the ASBR or ABR function for both EVPN and IP-VPN routes. These two features can be used with the following EVPN services:

  1. EVPN-MPLS Epipe services (EVPN-VPWS)
  2. EVPN-MPLS VPLS services
  3. EVPN-MPLS R-VPLS services
  4. PBB-EVPN and PBB-EVPN E-Tree services
  5. EVPN-MPLS E-Tree services
  6. PE and ABR functions (EVPN services and enable-rr-vpn-forwarding), which are both supported on the same router
  7. PE and ASBR functions (EVPN services and enable-inter-as-vpn), which are both supported on the same router

The following sub-sections clarify some aspects of EVPN when used in an Inter-AS Option B or VPN-NH-RR network.

5.3.14.1. Inter-AS Option B and VPN-NH-RR Procedures on EVPN Routes

When enable-rr-vpn-forwarding or enable-inter-as-vpn is configured, only EVPN-MPLS routes are processed for label swap and the next hop is changed. EVPN-VXLAN routes are re-advertised without a change in the next hop.

The following shows how the router processes and re-advertises the different EVPN route types. Refer to BGP-EVPN Control Plane for MPLS Tunnels for detailed information about the route fields.

  1. Auto-discovery (AD) routes (Type 1)
    For AD per EVI routes, the MPLS label is extracted from the route NLRI. The route is re-advertised with Next-Hop-Self (NHS) and a new label. No modifications are made for the remaining attributes.
    For AD per ES routes, the MPLS label in the NLRI is zero. The route is re-advertised with NHS and the MPLS label remains zero. No modifications are made for the remaining attributes.
  2. MAC/IP routes (Type 2)
    The MPLS label (Label-1) is extracted from the NLRI. The route is re-advertised with NHS and a new Label-1. No modifications are made for the remaining attributes.
  3. Inclusive Multicast Ethernet Tag (IMET) routes (Type 3)
    Because there is no MPLS label present in the NLRI, the MPLS label is extracted from the PMSI Tunnel Attribute (PTA) if needed, and the route is then re-advertised with NHS, with the following considerations.
    1. For IMET routes with tunnel-type Ingress Replication, the router extracts the IR label from the PTA. The router programs the label swap and re-advertises the route with a new label in the PTA.
    2. For tunnel-type P2MP mLDP, the router re-advertises the route with NHS. No label is extracted; therefore, no swap operation occurs.
    3. For tunnel-type Composite, the IR label is extracted from the PTA, the swap operation is programmed and the route re-advertised with NHS. A new label is encoded in the PTA’s IR label with no other changes in the remaining fields.
    4. For tunnel-type AR, the routes are always considered VXLAN routes and are re-advertised with the next-hop unchanged.
  4. Ethernet-Segment (ES) routes (Type 4)
    Because ES routes do not contain an MPLS label, the route is re-advertised with NHS and no modifications to the remaining attributes. Although an ASBR or ABR will re-advertise ES routes, EVPN multi-homing for ES PEs located in different ASs or IGMP domains is not supported.
  5. IP-Prefix routes (Type 5)
    The MPLS label is extracted from the NLRI and the route is re-advertised with NHS and a new label. No modifications are made to the remaining attributes.

5.3.14.2. BUM Traffic in Inter-AS Option B and VPN-NH-RR Networks

Inter-AS Option B and VPN-NH-RR support the use of non-segmented trees for forwarding BUM traffic in EVPN.

For ingress replication and non-segmented trees, the ASBR or ABR performs an EVPN BUM label swap without any aggregation or further replication. This concept is shown in Figure 189.

Figure 189:  VPN-NH-RR and Ingress Replication for BUM Traffic 

In Figure 189, when PE2, PE3, and PE4 advertise their IMET routes, the ABRs re-advertise the routes with NHS and a different label. However, IMET routes are not aggregated; therefore, PE1 sets up three different EVPN multicast destinations and sends three copies of every BUM packet, even if they are sent to the same ABR. This example is also applicable to ASBRs and Inter-AS Option B.

P2MP mLDP may also be used with VPN-NH-RR, but not with Inter-AS Option B. The ABRs, however, will not aggregate or change the mLDP root IP addresses in the IMET routes. The root IP addresses must be leaked across IGP domains. For example, if PE2 advertises an IMET route with mLDP or composite tunnel type, PE1 will be able to join the mLDP tree if the root IP is leaked into PE1’s IGP domain.

5.3.14.3. EVPN Multi-Homing in Inter-AS Option B and VPN-NH-RR Networks

In general, EVPN multi-homing is supported in Inter-AS Option B or VPN-NH-RR networks with the following limitations.

  1. An ES PE can only process a remote ES route correctly if the received next hop and origination IP address match. EVPN multi-homing is not supported when the ES PEs are in different ASs or IGP domains, or if there is an NH-RR peering the ES PEs and overriding the ES route next hops.
  2. EVPN multi-homing Ethernet segments are not supported on EVPN PEs that are also ABRs or ASBRs.
  3. Mass-withdraw based on the AD per-ES routes is not supported for a PE that is in a different AS or IGP domain that the ES PEs. Figure 190 shows an EVPN multi-homing scenario where the ES PEs, PE2 and PE3, and the remote PE, PE1, are in different ASs or IGP domains.
Figure 190:  EVPN Multi-Homing with Inter-AS Option B or VPN-NH-RR 

In Figure 190, PE1’s aliasing and backup functions to the remote ES-1 are supported. However, PE1 cannot identify the originating PE for the received AD per-ES routes because they are both arriving with the same next hop (ASBR/ABR4) and RDs may not help to correlate each AD per-ES route to a given PE. Therefore, if there is a failure on PE2’s ES link, PE1 cannot remove PE2 from the destinations list for ES-1 based on the AD per-ES route. PE1 must wait for the AD per-EVI route withdrawals to remove PE2 from the list. In summary, when the ES PEs and the remote PE are in different ASs or IGP domains, per-service withdrawal based on AD per-EVI routes is supported, but mass-withdrawal based on AD per-ES routes is not supported.

5.3.14.4. EVPN E-Tree in Inter-AS Option B and VPN-NH-RR Networks

Unicast procedures known to EVPN-MPLS E-Tree are supported in Inter-AS Option B or VPN-NH-RR scenarios, however, the BUM filtering procedures are affected.

As described in EVPN E-Tree, leaf-to-leaf BUM filtering is based on the Leaf Label identification at the egress PE. In a non-Inter-AS or non-VPN-NH-RR scenario, EVPN E-tree AD per-ES (ESI-0) routes carrying the Leaf Label are distinguished by the advertised next hop. In Inter-AS or VPN-NH-RR scenarios, all the AD per-ES routes are received with the ABR or ASBR next hop. Therefore, AD per-ES routes originating from different PEs would all have the same next hop, and the ingress PE would not be able to determine which leaf label to use for a given EVPN multicast destination.

A simplified EVPN E-Tree solution is supported, where an E-Tree Leaf Label is not installed in the IOM if the PE receives more than one E-Tree AD per-ES route, with different RDs, for the same next hop. In this case, leaf BUM traffic is transmitted without a Leaf Label and the leaf-to-leaf traffic filtering depends on the egress source MAC filtering on the egress PE. See EVPN E-Tree Egress Filtering Based on MAC Source Address.

PBB-EVPN E-tree services are not affected by Inter-AS or VPN-NH-RR scenarios, as AD per-ES routes are not used.

5.4. General EVPN Topics

This section provides information on general topics related to EVPN.

5.4.1. ARP/ND Snooping and Proxy Support

VPLS services support proxy-ARP (Address Resolution Protocol) and proxy-ND (Neighbor Discovery) functions that can be enabled or disabled independently per service. When enabled (proxy-ARP/proxy-ND no shutdown), the system will populate the corresponding proxy-ARP/proxy-ND table with IP->MAC entries learned from the following sources:

  1. EVPN-received IP->MAC entries
  2. User-configured static IP->MAC entries
  3. Snooped dynamic IP->MAC entries (learned from ARP/GARP/NA messages received on local SAPs/SDP-bindings)

In addition, any ingress ARP or ND frame on a SAP or SDP-binding will be intercepted and processed. ARP requests and Neighbor Solicitations will be answered by the system if the requested IP address is present in the proxy table.

Figure 191 shows an example of how proxy-ARP is used in an EVPN network. Proxy-ND would work in a similar way. The MAC address notation in the diagram is shortened for readability.

Figure 191:  Proxy-ARP Example Usage in an EVPN Network 

PE1 is configured as follows:

*A:PE1>config>service>vpls# info 
----------------------------------------------
    vxlan instance 1 vni 600 create
        exit
        bgp
            route-distinguisher 192.0.2.71:600
            route-target export target:64500:600 import target:64500:600
    exit
    bgp-evpn
        vxlan bgp 1 vxlan-instance 1
            no shutdown
        exit
    exit
    proxy-arp
        age-time 600
        send-refresh 200
        dup-detect window 3 num-moves 3 hold-down max anti-spoof-
mac 00:ca:ca:ca:ca:ca
        dynamic-arp-populate
          no shutdown
            exit
            sap 1/1/1:600 create
            exit
no shutdown               
----------------------------------------------

Figure 191 shows the following steps, assuming proxy-ARP is no shutdown on PE1 and PE2, and the tables are empty:

  1. ISP-A sends ARP-request for (10.10.)10.3.
  2. PE1 learns the MAC 00:01 in the FDB as usual and advertises it in EVPN without any IP. Optionally, the MAC can be configured as a CStatic mac, in which case it will be advertised as protected. If the MAC is learned on a SAP or SDP-binding where auto-learn-mac-protect is enabled, the MAC will also be advertised as protected.
  3. The ARP-request is sent to the CPM where:
    1. An ARP entry (IP 10.1'MAC 00:01) is populated into the proxy-ARP table.
    2. EVPN advertises MAC 00:01 and IP 10.1 in EVPN with the same SEQ number and Protected bit as the previous route-type 2 for MAC 00:01.
    3. A GARP is also issued to other SAPs/SDP-bindings (assuming they are not in the same split horizon group as the source). If garp-flood-evpn is enabled, the GARP message is also sent to the EVPN network.
    4. The original ARP-request can still be flooded to the EVPN or not based on the unknown-arp-request-flood-evpn command.
  4. Assuming PE1 was configured with unknown-arp-request-flood-evpn, the ARP-request is flooded to PE2 and delivered to ISP-B. ISP-B replies with its MAC in the ARP-reply. The ARP-reply is finally delivered to ISP-A.
  5. PE2 will learn MAC 00:01 in the FDB and the entry 10.1'00:01 in the proxy-ARP table, based on the EVPN advertisements.
  6. When ISP-B replies with its MAC in the ARP-reply:
    1. MAC 00:03 is learned in FDB at PE2 and advertised in EVPN.
    2. MAC 00:03 and IP 10.3 are learned in the proxy-ARP table and advertised in EVPN with the same SEQ number as the previous MAC route.
    3. ARP-reply is unicasted to MAC 00:01.
  7. EVPN advertisements are used to populate PE1's FDB (MAC 00:03) and proxy-ARP (IP 10.3—>MAC 00:03) tables as mentioned in 5.

From this point onward, the PEs reply to any ARP-request for 00:01 or 00:03, without the need for flooding the message in the EVPN network. By replying to known ARP-requests / Neighbor Solicitations, the PEs help to significantly reduce the flooding in the network.

Use the following commands to customize proxy-ARP/proxy-ND behavior:

  1. dynamic-arp-populate and dynamic-nd-populate
    Enables the addition of dynamic entries to the proxy-ARP or proxy-ND table (disabled by default). When executed, the system will populate proxy-ARP/proxy-ND entries from snooped GARP/ARP/NA messages on SAPs/SDP-bindings in addition to the entries coming from EVPN (if EVPN is enabled). These entries will be shown as dynamic.
  2. static <IPv4-address> <mac-address> and static <IPv4-address> <mac-address> and static <ipv6-address> <mac-address> {host|router}
    Configures static entries to be added to the table.
    Note:

    A static IP->MAC entry requires the addition of the MAC address to the FDB as either learned or CStatic (conditional static mac) in order to become active (Status > active).

  3. age-time <60..86400> (seconds)
    Specifies the aging timer per proxy-ARP/proxy-ND entry. When the aging expires, the entry is flushed. The age is reset when a new ARP/GARP/NA for the same IP—>MAC is received.
  4. send-refresh <120..86400> (seconds)
    If enabled, the system will send ARP-request/Neighbor Solicitation messages at the configured time, so that the owner of the IP can reply and therefore refresh its IP—>MAC (proxy-ARP entry) and MAC (FDB entry).
  5. table-size [1..16384]
    Enables the user to limit the number of entries learned on a specified service. By default, the table-size limit is 250.
    The unknown ARP-requests, NS, or the unsolicited GARPs and NA messages can be configured to be flooded or not in an EVPN network with the following commands:
    1. proxy-arp [no] unknown-arp-request-flood-evpn
    2. proxy-arp [no] garp-flood-evpn
    3. proxy-nd [no] unknown-ns-flood-evpn
    4. proxy-nd [no] host-unsolicited-na-flood-evpn
    5. proxy-nd [no] router-unsolicited-na-flood-evpn
  6. dup-detect [anti-spoof-mac <mac-address>] window <minutes> num-moves <count> hold-down <minutes|max>
    Enables a mechanism that detects duplicate IPs and ARP/ND spoofing attacks. The working of the dup-detect command can be summarized as follows:
    1. Attempts (relevant to dynamic and EVPN entry types) to add the same IP (different MAC) are monitored for <window> minutes and when <count> is reached within that window, the proxy-ARP/proxy-ND entry for the IP is suspected and marked as duplicate. An alarm is also triggered.
    2. The condition is cleared when hold-down time expires (max does not expire) or a clear command is issued.
    3. If the anti-spoof-mac is configured, the proxy-ARP/proxy-ND offending entry's MAC is replaced by this <mac-address> and advertised in an unsolicited GARP/NA for local SAP or SDP-bindings and in EVPN to remote PEs.
    4. This mechanism assumes that the same anti-spoof-mac is configured in all the PEs for the same service and that traffic with destination anti-spoof-mac received on SAPs/SDP-bindings will be dropped. An ingress MAC filter has to be configured in order to drop traffic to the anti-spoof-mac.

Table 82 shows the combinations that will produce a Status = Active proxy-arp entry in the table. The system will only reply to proxy-ARP requests for active entries. Any other combination will result in a Status = inActv entry. If the service is not active, the proxy-arp entries will not be active either, regardless of the FDB entries

Note:

A static entry is active in the FDB even when the service is down.

Table 82:  Proxy-arp Entry combinations 

Proxy-arp Entry Type

FDB Entry Type (for the same MAC)

Dynamic

learned

Static

learned

Dynamic

CStatic/Static

Static

CStatic/Static

EVPN

EVPN

Duplicate

When proxy-ARP/proxy-ND is enabled on services with all-active multi-homed ethernet-segments, a proxy-arp entry type 'EVPN' might be associated with a 'learned' FDB entry (because the CE can send traffic for the same MAC to all the multi-homed PEs in the ES). If that is the case, the entry will be inactive, as per Table 82.

5.4.1.1. Proxy-ARP/ND Periodic Refresh, Unsolicited Refresh and Confirm-Messages

When proxy-ARP/proxy-ND is enabled, the system starts populating the proxy table and responding to ARP-requests/NS messages. To keep the active IP->MAC entries alive and ensure that all the host/routers in the service update their ARP/ND caches, the system may generate the following three types of ARP/ND messages for a specified IP->MAC entry:

  1. Periodic refresh messages (ARP-requests or NS for a specified IP):
    These messages are activated by the send-refresh command and their objective is to keep the existing FDB and Proxy-ARP/ND entries alive, in order to minimize EVPN withdrawals and re-advertisements.
  2. Unsolicited refresh messages (unsolicited GARP or NA messages):
    These messages are sent by the system when a new entry is learned or updated. Their objective is to update the attached host/router caches.
  3. Confirm messages (unicast ARP-requests or unicast NS messages):
    These messages are sent by the system when a new MAC is learned for an existing IP. The objective of the confirm messages is to verify that a specified IP has really moved to a different part of the network and is associated with the new MAC. If the IP has not moved, it will force the owners of the duplicate IP to reply and cause dup-detect to kick in.

5.4.1.2. Proxy-ND and the Router Flag in Neighbor Advertisement messages

RFC 4861 describes the use of the (R) or "Router" flag in NA messages as follows:

  1. A node capable of routing IPv6 packets must reply to NS messages with NA messages where the R flag is set (R=1).
  2. Hosts must reply with NA messages where R=0.

The use of the "R" flag in NA messages impacts how the hosts select their default gateways when sending packets "off-link". Therefore, it is important that the proxy-ND function on the 7750 SR, 7450 ESS, or 7950 XRS must meet one of the following criteria:

  1. Either provide the appropriate R flag information in proxy-ND NA replies
  2. Flood the received NA messages if it cannot provide the appropriate R flag when replying

Due to the use of the "R" flag, the procedure for learning proxy-ND entries and replying to NS messages differs from the procedures for proxy-ARP in IPv4: the router or host flag will be added to each entry, and that will determine the flag to use when responding to a NS.

5.4.1.3. Procedure to Add the R Flag to a Specified Entry

The procedure to add the R flag to a specified entry is as follows:

  1. Dynamic entries are learned based on received NA messages. The R flag is also learned and added to the proxy-ND entry so that the appropriate R flag is used in response to NS requests for a specified IP.
  2. Static entries are configured as host or router as per the command [no] static <ip-address> <ieee-address> {host | router}.
  3. EVPN entries are learned from BGP and the command evpn-nd-advertise {host | router} determines the R flag added to them.
  4. In addition, the evpn-nd-advertise {host | router} command will indicate what static and dynamic IP->MAC entries the system will advertise in EVPN. If evpn-nd-advertise router is configured, the system should flood the received unsolicited NA messages for hosts. This is controlled by the [no] host-unsolicited-na-flood-evpn command. The opposite is also recommended so that the evpn-nd-advertise host is configured with the router-unsolicited-na-flood-evpn.

5.4.1.4. Proxy-ARP/ND Mac-List for Dynamic Entries

SR OS supports the association of configured MAC lists with a configured dynamic proxy-ARP or proxy-ND IP address. The actual proxy-ARP or proxy-ND entry is not created until an ARP or Neighbor Advertisement message is received for the IP and one of the MACs in the associated MAC-list. This is in accordance with IETF Draft draft-ietf-bess-evpn-proxy-arp-nd. which states that a proxy-ARP or proxy-ND IP entry can be associated to one MAC among a list of allowed MACs.

The following example shows the use of MAC lists for dynamic entries.

A:PE-2>config>service#
  proxy-arp-nd
    mac-list ISP-1 create 
      mac 00:de:ad:be:ef:01 
      mac 00:de:ad:be:ef:02 
      mac 00:de:ad:be:ef:03
 
A:PE-2>config>service>vpls>proxy-arp#
  dynamic 1.1.1.1 create
    mac-list ISP-1
    resolve 30
 
A:PE-2>config>service>vpls>proxy-nd#
  dynamic 2001:db8:1000::1 create
    mac-list ISP-1 
    resolve 30

Where:

  1. A dynamic IP (dynamic ip create) is configured and associated to a MAC list (mac-list name).
  2. The MAC list is created in the config>service context and can be reused by multiple configured dynamic IPs as follows:
    1. in different services
    2. in the same service, for proxy-ARP and proxy-ND entries
  3. If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.
  4. The same MAC list can be applied to multiple configured dynamic entries even within the same service.
  5. The new proxy-ARP and proxy-ND entries behave as dynamic entries and are displayed as type dyn in the show commands.
    The following output sample displays the entry corresponding to the configured dynamic IP.
    *A:PE-2# show service id 1 proxy-arp detail                                  
    -------------------------------------------------------------------------------
    Proxy Arp
    -------------------------------------------------------------------------------
    Admin State       : enabled             
    Dyn Populate      : enabled             
    Age Time          : 900 secs            Send Refresh      : 300 secs
    Table Size        : 250                 Total             : 1
    Static Count      : 0                   EVPN Count        : 0
    Dynamic Count     : 1                   Duplicate Count   : 0
    Dup Detect
    -------------------------------------------------------------------------------
    Detect Window     : 3 mins              Num Moves         : 5
    Hold down         : 9 mins              
    Anti Spoof MAC    : None
    EVPN
    -------------------------------------------------------------------------------
    Garp Flood        : enabled             Req Flood         : enabled
    Static Black Hole : disabled            
    -------------------------------------------------------------------------------
    ===============================================================================
    VPLS Proxy Arp Entries
    ===============================================================================
    IP Address          Mac Address       Type Status    Last Update
    -------------------------------------------------------------------------------
    1.1.1.1            00:de:ad:be:ef:01  dyn active    02/23/2016 09:05:49
    -------------------------------------------------------------------------------
    Number of entries : 1
    ===============================================================================
    *A:PE-2# show service proxy-arp-nd mac-list "ISP-1" associations 
    ===============================================================================
    MAC List Associations
    ===============================================================================
    Service Id                    IP Addr
    -------------------------------------------------------------------------------
    1                             1.1.1.1
    1                             2001:db8:1000::1
    -------------------------------------------------------------------------------
    Number of Entries: 2
    ===============================================================================

Although no new proxy-ARP or proxy-ND entries are created when a dynamic IP is configured, the router triggers the following resolve procedure.

  1. The router sends a resolve message with a configurable frequency of 1 to 60 minutes; the default value is 5 minutes.
    The resolve message is an ARP-request or NS message flooded to all the non-EVPN endpoints in the service.
  2. The router sends resolve messages at the configured frequency until a dynamic entry for the IP is created.
    Note:

    The dynamic entry is created only if an ARP, GARP, or NA message is received for the configured IP, and the associated MAC belongs to the configured MAC list of the IP. If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.

After a dynamic entry (with a MAC address included in the list) is successfully created, its behavior (for send-refresh, age-time, and other activities) is the same as a configured dynamic entry with the following exceptions.

  1. Regular dynamic entries may override configured dynamic entries, but static or EVPN entries cannot override configured dynamic entries.
  2. If the corresponding MAC is flushed from the FDB after the entry is successfully created, the entry becomes inactive in the proxy-ARP or proxy-ND table and the resolve process is restarted.
  3. If the MAC list is changed, all the IPs that point to the list delete the proxy entries and the resolve process is restarted.
  4. If there is an existing configured dynamic entry and the router receives a GARP, ARP, or NA for the IP with a MAC that is not contained in the MAC list, the message is discarded and the proxy-ARP or proxy-ND entry is deleted. The resolve process is restarted.
  5. If there is an existing configured dynamic entry and the router receives a GARP, ARP, or NA for the IP with a MAC contained in the MAC list, the existing entry is overridden by the IP and new MAC, assuming the confirm procedure passes.
  6. The dup-detect and confirm procedures work for the configured dynamic entries when the MAC changes are between MACs in the MAC list. Changes to an off-list MAC cause the entry to be deleted and the resolve process is restarted.

5.4.2. BGP-EVPN MAC-Mobility

EVPN defines a mechanism to allow the smooth mobility of MAC addresses from an NVE to another NVE. The 7750 SR, 7450 ESS, and 7950 XRS support this procedure as well as the MAC-mobility extended community in MAC advertisement routes as follows:

  1. The router honors and generates the SEQ (Sequence) number in the mac mobility extended community for mac moves.
  2. When a MAC is EVPN-learned and it is attempted to be learned locally, a BGP update is sent with SEQ number changed to "previous SEQ"+1 (exception: mac duplication num-moves value is reached).
  3. SEQ number = zero or no mac mobility ext-community are interpreted as sequence zero.
  4. In case of mobility, the following MAC selection procedure is followed:
    1. If a PE has two or more active remote EVPN routes for the same MAC (VNI can be the same or different), the highest SEQ number is selected. The tie-breaker is the lowest IP (BGP NH IP).
    2. If a PE has two or more active EVPN routes and it is the originator of one of them, the highest SEQ number is selected. The tie-breaker is the lowest IP (BGP NH IP of the remote route is compared to the local system address).
Note:

When EVPN multi-homing is used in EVPN-MPLS, the ESI is compared to determine whether a MAC received from two different PEs has to be processed within the context of MAC mobility or multi-homing. Two MAC routes that are associated with the same remote or local ESI but different PEs are considered reachable through all those PEs. Mobility procedures are not triggered as long as the MAC route still belongs to the same ESI.

5.4.3. BGP-EVPN MAC-Duplication

EVPN defines a mechanism to protect the EVPN service from control plane churn as a result of loops or accidental duplicated MAC addresses. The 7750 SR, 7450 ESS, and 7950 XRS support an enhanced version of this procedure as described in this section.

A situation may arise where the same MAC address is learned by different PEs in the same VPLS because of two (or more hosts) being misconfigured with the same (duplicate) MAC address. In such situation, the traffic originating from these hosts would trigger continuous MAC moves among the PEs attached to these hosts. It is important to recognize such situation and avoid incrementing the sequence number (in the MAC Mobility attribute) to infinity.

To remedy such situation, a router that detects a MAC mobility event by way of local learning starts a window <in-minutes> timer (default value of window = 3) and if it detects num-moves <num> before the timer expires (default value of num-moves = 5), it concludes that a duplicate MAC situation has occurred. The router then alerts the operator with a trap message. The offending MAC address can be shown using the show service id x bgp-evpn command:

10 2014/01/14 01:00:22.91 UTC MINOR: SVCMGR #2331 Base 
"VPLS Service 1 has MAC(s) detected as duplicates by EVPN mac-
duplication detection."
# show service id 1 bgp-evpn 
===============================================================================
BGP EVPN Table
===============================================================================
MAC Advertisement  : Enabled            Unknown MAC Route  : Disabled
VXLAN Admin Status : Enabled            Creation Origin    : manual
MAC Dup Detn Moves : 5                  MAC Dup Detn Window: 3
MAC Dup Detn Retry : 9                  Number of Dup MACs : 1
-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses             Time Detected
-------------------------------------------------------------------------------
00:00:00:00:00:12                            01/14/2014 01:00:23
-------------------------------------------------------------------------------
===============================================================================

After detecting the duplicate, the router stops sending and processing any BGP MAC advertisement routes for that MAC address until one of the following occurs:

  1. The MAC is flushed due to a local event (SAP or SDP-binding associated with the MAC fails) or the reception of a remote update with better SEQ number (due to a mac flush at the remote router).
  2. The retry <in-minutes> timer expires, which will flush the MAC and restart the process.
Note:

The other routers in the VPLS instance will forward the traffic for the duplicate MAC address to the router advertising the best route for the MAC.

The values of num-moves and window are configurable to allow for the required flexibility in different environments. In scenarios where BGP rapid-update evpn is configured, the operator might want to configure a shorter window timer than in scenarios where BGP updates are sent every (default) min-route-advertisement interval.

Mac-duplication is always enabled in EVPN-VXLAN VPLS services, and the preceding described mac duplication parameters can be configured per VPLS service under the bgp-evpn mac-duplication context:

*A:DGW1>config>service>vpls>bgp-evpn# info 
----------------------------------------------
       mac-advertisement 
       unknown-mac-route
       mac-duplication
         detect num-moves num window in_mins
         [no] retry in_mins
       vxlan bgp 1 vxlan-instance 1
         no shutdown
       exit

5.4.4. Conditional Static MAC and Protection

RFC 7432 defines the use of the sticky bit in the mac-mobility extended community to signal static mac addresses. These addresses must be protected in case there is an attempt to dynamically learn them in a different place in the EVPN-VXLAN VPLS service.

In the 7750 SR, 7450 ESS, and 7950 XRS, any conditional static mac defined in an EVPN-VXLAN VPLS service will be advertised by BGP-EVPN as a static address, that is, with the sticky bit set. An example of the configuration of a conditional static mac is shown below:

*A:PE63>config>service>vpls# info 
----------------------------------------------
            description "vxlan-service"
...     
            sap 1/1/1:1000 create
            exit
            static-mac                
                mac 00:ca:ca:ca:ca:00 create sap 1/1/1:1000 monitor fwd-status
            exit
            no shutdown
 
*A:PE64# show router bgp routes evpn mac hunt mac-address 00:ca:ca:ca:ca:00 
...
===============================================================================
BGP EVPN Mac Routes
===============================================================================
Network        : 0.0.0.0/0
Nexthop        : 192.0.2.63
From           : 192.0.2.63
Res. Nexthop   : 192.168.19.1
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                   
Connector      : None
Community      : target:65000:1000     mac-mobility:Seq: 0/Static
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.63
Flags          : Used  Valid  Best  IGP  
Route Source   : Internal               
AS-Path        : No As-Path
EVPN type      : MAC                    
ESI            : 0:0:0:0:0:0:0:0:0:0    Tag            : 1063
IP Address     : ::                     RD             : 65063:1000
Mac Address    : 00:ca:ca:ca:ca:00      Mac Mobility   : Seq:0
Neighbor-AS    : N/A
Source Class   : 0                      Dest Class     : 0
-------------------------------------------------------------------------------
Routes : 1                            
===============================================================================

Local static MACs or remote MACs with sticky bit are considered as 'protected'. A packet entering a SAP / SDP-binding will be discarded if its source MAC address matches one of these 'protected' MACs.

5.4.5. Auto-Learn MAC Protect and Restricting Protected Source MACs

Auto-learn MAC protect, together with the ability to restrict where the protected source MACs are allowed to enter the service, can be enabled within an EVPN-MPLS and EVPN-VXLAN VPLS and routed VPLS services, but not in PBB-EVPN services. The protection, using the auto-learn-mac-protect command (described in Auto-Learn MAC Protect), and the restrictions, using the restrict-protected-src [discard-frame] command, operate in the same way as in a non-EVPN VPLS service.

  1. When auto-learn-mac-protect is enabled on an object, source MAC addresses learned on that object are marked as protected within the FDB.
  2. When restrict-protected-src is enabled on an object and a protected source MAC is received on that object, the object is automatically shutdown (requiring the operator to shutdown then no shutdown the object in order to make it operational again).
  3. When restrict-protected-src discard-frame is enabled on an object and a frame with a protected source MAC is received on that object, that frame is discarded.

In addition, the following behavioral differences are specific to EVPN services:

  1. An implicit restrict-protected-src discard-frame command is enabled by default on SAPs, mesh-SDPs and Spoke-SDPs. As this is the default, it is not possible to configure this command in an EVPN service. This default state can be seen in the show output for these objects, for example on a SAP:
    *A:PE# show service id 1 sap 1/1/9:1 detail
    ===============================================================================
    Service Access Points(SAP)
    ===============================================================================
    Service Id         : 1
    SAP                : 1/1/9:1                  Encap             : q-tag
    ...
    RestMacProtSrc Act : none (oper: Discard-frame)
  2. A restrict-protected-src discard-frame can be optionally enabled on EVPN-MPLS/VXLAN destinations within EVPN services. When enabled, frames that have a protected source MAC address are discarded if received on any EVPN-MPLS/VXLAN destination in this service, unless the MAC address is learned and protected on an EVPN-MPLS/VXLAN destination in this service. This is enabled as follows:
            configure
               service
                  vpls <service id>
                       bgp-evpn
                           mpls bgp <instance>
                               [no] restrict-protected-src discard-frame
                       vxlan instance <instance> vni <vni-id>
                           [no] restrict-protected-src discard-frame
  3. Auto-learned protected MACs are advertised to remote PEs in an EVPN MAC/IP advertisement route with the sticky bit set.
  4. The source MAC protection action relating to the restrict-protected-src [discard-frame] commands also applies to MAC addresses learned by receiving an EVPN MAC/IP advertisement route with the sticky bit set from remote PEs. This causes remotely configured conditional static MACs and auto-learned protected MACs to be protected locally.
  5. In all-active multi-homing scenarios, if auto-learn-mac-protect is configured on all-active SAPs and restrict-protected-src discard-frame is enabled on EVPN-MPLS/VXLAN destinations, traffic from the CE that enters one multi-homing PE and needs to be switched through the other multi-homing PE will be discarded on the second multi-homing PE. Each multi-homing PE will protect the CE's MAC on its local all-active SAP, which results in any frames with the CE's MAC address as the source MAC being discarded as they are received on the EVPN-MPLS/VXLAN destination from the other multi-homing PE.

Conditional static MACs, EVPN static MACs and locally protected MACs are marked as protected within the FDB, as shown in the example output.

*A:PE# show service fdb-mac
===============================================================================
Service Forwarding Database
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age
-------------------------------------------------------------------------------
1         00:00:00:00:00:01 sap:1/1/9:1              LP/30    01/05/16 11:58:22
1         00:00:00:00:00:02 vxlan-1:                 EvpnS:P  01/05/16 11:58:23
                            10.1.1.2:1
1         00:00:00:00:01:01 sap:1/1/9:1              CStatic: 01/04/16 20:05:02
                                                     P
1         00:00:00:00:01:02 vxlan-1:                 EvpnS:P  01/04/16 20:18:02
                            10.1.1.2:1
-------------------------------------------------------------------------------
No. of Entries: 4
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

In this output:

  1. the first MAC is locally protected using the auto-learn-mac-protect command
  2. the second MAC has been protected using the auto-learn-mac-protect command on a remote PE
  3. the third MAC is a locally configured conditional static MAC
  4. the fourth MAC is a remotely configured conditional static MAC

5.4.6. Blackhole MAC and its Application to Proxy-ARP/Proxy-ND Duplicate Detection

A blackhole MAC is a local FDB record. It is similar to a conditional static MAC; it is associated with a black-hole (similar to a VPRN blackhole static-route in VPRNs) instead of a SAP or SDP-binding. A blackhole MAC can be added by using the following command:

config>service>vpls# static-mac mac
mac <ieee-address> [create] black-hole

The static blackhole MAC can have security applications (for example, replacement of MAC filters) for certain MACs. When used in combination with restrict-protected-src, the static blackhole MAC provides a simple and scalable way to filter MAC DA or SA in the data plane, regardless of how the frame arrived at the system (using SAP or SDP-bindings or EVPN endpoints).

For example, when a specified static-mac mac 00:00:ca:fe:ca:fe create black-hole is added to a service, the following behavior occurs:

  1. The configured MAC is created as a static MAC with a black-hole source identifier.
    *A:PE1# show service id 1 fdb detail                  
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId    MAC               Source-Identifier        Type     Last Change
                                                         Age      
    -------------------------------------------------------------------------------
    1         00:ca:ca:ba:ca:01 eES:                     Evpn     06/29/15 23:21:34
                                01:00:00:00:00:71:00:00:00:01
    1         00:ca:ca:ba:ca:06 eES:                     Evpn     06/29/15 23:21:34
                                01:74:13:00:74:13:00:00:74:13
    1         00:ca:00:00:00:00 sap:1/1/1:2              CStatic:P 06/29/15 23:20:58
    1         00:ca:fe:ca:fe:00 black-hole               CStatic:P  06/29/15 23:20:00
    1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS:P    06/29/15 20:40:13
                                192.0.2.69:262133
    -------------------------------------------------------------------------------
    No. of MAC Entries: 5
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
    ===============================================================================
  2. After it has been successfully added to the FDB, the blackhole MAC will be treated like any other protected MAC, as follows:
    1. The blackhole MAC will be added as protected (CStatic:P) and advertised in EVPN as static.
    2. SAP or SDP-bindings or EVPN endpoints where the restrict-protected-src discard-frame is enabled will discard frames where MAC SA is equal to blackhole MAC.
    3. SAP or SDP-bindings where restrict-protected-src (no discard-frame) is enabled will go operationally down if a frame with MAC SA is equal to blackhole MAC is received.
  3. After the blackhole MAC has been successfully added to the FDB, any frame arriving at any SAP or SDP-binding or EVPN endpoint with MAC DA is equal to blackhole MAC will be discarded.

Blackhole MACs can also be used in services with proxy-ARP/proxy-ND enabled to filter traffic with destination to anti-spoof-macs. The anti-spoof-mac provides a way to attract traffic to a specified IP when a duplicate condition is detected for that IP address (see section ARP/ND Snooping and Proxy Support for more information); however, the system still needs to drop the traffic addressed to the anti-spoof-mac by using either a MAC filter or a blackhole MAC.

The user does not need to configure MAC filters when configuring a static-black-hole MAC address for the anti-spoof-mac function. To use a blackhole MAC entry for the anti-spoof-mac function in a proxy-ARP/proxy-ND service, the user needs to configure:

  1. the static-black-hole option for the anti-spoof-mac
    *A:PE1# config>service>vpls>proxy-arp# 
    dup-detect window 3 num-moves 5 hold-down max anti-spoof-
    mac 00:66:66:66:66:00 static-black-hole
  2. a static blackhole MAC using the same MAC address used for the anti-spoof-mac
    *A:PE1# config>service>vpls# 
    static-mac mac 00:66:66:66:66:00 create black-hole

When this configuration is complete, the behavior of the anti-spoof-mac function changes as follows:

  1. In the EVPN, the MAC is advertised as Static. Locally, the MAC will be shown in the FDB as “CStatic” and associated with a black-hole.
  2. The combination of the anti-spoof-mac and the static-black-hole ensures that any frame that arrives at the system with MAC DA = anti-spoof-mac is discarded, regardless of the ingress endpoint type (SAP or SDP-binding or EVPN) and without the need for a filter.
  3. If, instead of discarding traffic, the user wants to redirect it using MAC DA as the anti-spoof-mac, then redirect filters should be configured on SAPs or SDP-bindings (instead of the static-black-hole option).

When the static-black-hole option is not configured with the anti-spoof-mac, the behavior of the anti-spoof-mac function, as described in ARP/ND Snooping and Proxy Support, remains unchanged. In particular:

  1. the anti-spoof-mac is not programmed in the FDB
  2. any attempt to add a Static MAC (or any other MAC) with the anti-spoof-mac value will be rejected by the system
  3. a MAC filter is needed to discard traffic with MAC DA = anti-spoof-mac.

5.4.7. Blackhole MAC for EVPN Loop Detection

SR OS can combine a blackhole MAC address concept and the EVPN MAC duplication procedures to provide loop protection in EVPN networks. The feature is compliant with the MAC mobility and multi-homing functionality in RFC 7432. The config>service>vpls>bgp-evpn>mac-duplication>black-hole-dup-mac CLI command enables the feature.

If enabled, there are no apparent changes in the MAC duplication; however, if a duplicated MAC is detected (for example, M1), then the router performs the following:

  1. adds M1 to the duplicate MAC list
  2. programs M1 in the FDB as a “Protected” MAC associated with a blackhole endpoint (where “type” is set to EvpnD:P and Source-Identifier is black-hole)

While the MAC type value remains EvpnD:P, the following additional operational details apply.

  1. Incoming frames with MAC DA = M1 are discarded by the ingress IOM, regardless of the ingress endpoint type (SAP, SDP, or EVPN), based on an FDB MAC lookup.
  2. Incoming frames with MAC SA = M1 are discarded by the ingress IOM or cause the router to bring down the SAP or SDP-binding, depending on the restrict-protected-src setting on the SAP, SDP, or EVPN endpoint.

The following sample CLI shows an example EVPN-MPLS service where black-hole-dup-mac is enabled and MAC duplication programs the duplicate MAC as a blackhole.

19 2016/12/20 19:45:59.69 UTC MINOR: SVCMGR #2331 Base 
"VPLS Service 30 has MAC(s) detected as duplicates by EVPN mac-duplication detec
tion."
*A:PE-5# configure service vpls 30 
*A:PE-5>config>service>vpls# info 
----------------------------------------------
            bgp
            exit
            bgp-evpn
                evi 30
                mac-duplication
                    detect num-moves 3 window 3
                    retry 6
                    black-hole-dup-mac
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:30 create
                no shutdown
            exit
            spoke-sdp 56:30 leaf-ac create
                no shutdown
            exit
            no shutdown
----------------------------------------------
*A:PE-5# show service id 30 bgp-evpn 
===============================================================================
BGP EVPN Table
===============================================================================
MAC Advertisement  : Enabled            Unknown MAC Route  : Disabled
CFM MAC Advertise  : Disabled           
VXLAN Admin Status : Disabled           Creation Origin    : manual
MAC Dup Detn Moves : 3                  MAC Dup Detn Window: 3
MAC Dup Detn Retry : 6                  Number of Dup MACs : 1
MAC Dup Detn BH    : Enabled            
IP Route Advert    : Disabled           
 
EVI                : 30                 
Ing Rep Inc McastAd: Enabled            
Accept IVPLS Flush : Disabled           
Send EVPN Encap    : Enabled            
-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses             Time Detected
-------------------------------------------------------------------------------
00:11:00:00:00:01                            12/20/2016 19:46:00
-------------------------------------------------------------------------------
<snip>
...
*A:PE-5# show service id 30 fdb detail 
===============================================================================
Forwarding Database, Service 30
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
30        00:11:00:00:00:01 black-hole               EvpnD:P  12/20/16 19:46:00
-------------------------------------------------------------------------------
No. of MAC Entries: 1
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================

If the retry time expires, the MAC is flushed from the FDB and the process starts again. The clear service id 30 evpn mac-dup-detect {ieee-address | all} command clears the duplicate blackhole MAC address.

Note:

The clear service id 30 fdb command clears learned MAC addresses; blackhole MAC addresses are not cleared.

Support for the black-hole-dup-mac command and the preceding associated loop detection procedures is as follows:

  1. not supported on B-VPLS, I-VPLS, M-VPLS, or R-VPLS services
  2. fully supported on EVPN-VXLAN and EVPN-MPLS VPLS services (including EVPN E-Tree)
  3. fully supported with EVPN MAC mobility and EVPN-MPLS multi-homing

5.4.8. CFM Interaction with EVPN Services

Ethernet Connectivity and Fault Management (ETH-CFM) allows the operator to validate and measure Ethernet Layer 2 services using standard IEEE 802.1ag and ITU-T Y.1731 protocols. Each tool performs a unique function and adheres to that tool's specific PDU and frame format and the associate rules governing the transmission, interception, and process of the PDU. Detailed information describing the ETH-CFM architecture, the tools, and various functions is located in the various OAM and Diagnostics guides and is not repeated here.

EVPN provides powerful solution architectures. ETH-CFM is supported in the various Layer 2 EVPN architectures. Since the destination Layer 2 MAC address, unicast or multicast, is ETH-CFM tool dependent (i.e. ETH-CC is sent as an L2 multicast and ETH-DM is sent as an L2 unicast), the ETH-CFM function is allowed to multicast and broadcast to the virtual EVPN connections. The Maintenance Endpoint (MEP) and Maintenance Intermediate Point (MIP) do not populate the local Layer 2 MAC Address forwarding database (FDB) with the MAC related to the MEP and MIP. This means that the 48-bit IEEE MAC address is not exchanged with peers and all ETH-CFM frames are broadcast across all virtual connections. To prevent the flooding of unicast packets and allow the remote forwarding databases to learn the remote MEP and MIP Layer 2 MAC addresses, the command cfm-mac-advertisement must be configured under the config>service>vpls>bgp-evpn context. This allows the MEP and MIP Layer 2 IEEE MAC addresses to be exchanged with peers. This command will track configuration changes and send the required updates via the EVPN notification process related to a change.

Up MEP, Down MEP, and MIP creation is supported on the SAP, spoke, and mesh connections within the EVPN service. There is no support for the creation of ETH-CFM Management Points (MPs) on the virtual connection. VirtualMEP (vMEP) is supported with a VPLS context and the applicable EVPN Layer 2 VPLS solution architectures. The vMEP follows the same rules as the general MPs. When a vMEP is configured within the supported EVPN service, the ETH-CFM extraction routines are installed on the SAP, Binding, and EVPN connections within an EVPN VPLS Service. The vMEP extraction within the EVPN-PBB context requires the vmep-extensions parameter to install the extraction on the EVPN connections.

When MPs are used in combination with EVPN multi-homing, the following must be considered:

  1. Behavior of operationally down MEPs on SAPs/SDP-bindings with EVPN multi-homing:
    1. All-active multi-homing: no ETH-CFM is expected to be used in this case, since the two (or more) SAPs/SDP-bindings on the PEs will be oper-up and active; however, the CE will have a single LAG and will respond as though it is connected to a single system. In addition to that, cfm-mac-advertisement can lead to traffic loops in all-active multi-homing.
    2. Single-active multi-homing: operationally down MEPs defined on single-active ethernet-segment SAPs/SDP-bindings will not send any CCMs when the PE is non-DF for the ES and fault-propagation is configured. For single-active multi-homing, the behavior will be equivalent to MEPs defined on BGP-MH saps/binds.
  2. Behavior for operationally up MEPs on ES SAPs/SDP-bindings with EVPN multi-homing:
    1. All-active multi-homing: operationally up MEPs defined on non-DF ES SAPs can send CFM packets. However, they cannot receive CCMs (the SAP is removed from the default multicast list) or unicast CFM packets (because the MEP MAC is not installed locally in the FDB; unicast CFM packets will be treated as unknown, and not sent to the non-DF SAP MEP).
    2. Single-active multi-homing: operationally up MEPs should be able to send or receive CFM packets normally.
    3. operationally up MEPs defined on LAG SAPs require the command process_cpm_traffic_on_sap_down so that they can process CFM when the LAG is down and act as regular Ethernet ports.

Due to the above considerations, the use of ETH-CFM in EVPN multi-homed SAPs/SDP-bindings is only recommended on operationally down MEPs and single-active multi-homing. ETH-CFM is used in this case to notify the CE of the DF or non-DF status.

5.4.9. Configuring EVPN-VXLAN and EVPN-MPLS in the Same VPLS Service

When two BGP instances are added to a VPLS service, both BGP-EVPN MPLS and BGP-EVPN VXLAN can be configured at the same time in the service. A maximum of two BGP instances are supported in the same VPLS, where BGP-EVPN MPLS and BGP-EVPN VXLAN can both use BGP instance 1 or 2, as long as they use different instances.

For example, in a service where EVPN-VXLAN and EVPN-MPLS are configured together, the config>service>vpls>bgp-evpn>vxlan bgp 1 and config>service>vpls>bgp-evpn>mpls bgp 2 commands allow the user to associate the BGP-EVPN MPLS to a different instance from that associated with BGP-EVPN VXLAN, and have both encapsulations simultaneously enabled in the same service. At the control plane level, MAC or IP routes received in one instance are consumed and re-advertised in the other instance as long as the route is the best route for a specific MAC. Inclusive multicast routes are independently generated for each BGP instance. From a data plane perspective, the EVPN-MPLS and EVPN-VXLAN destinations are instantiated in different implicit Split Horizon Groups (SHGs) so that traffic can be forwarded between them.

The following example shows a VPLS service with two BGP instances and both VXLAN and MPLS encapsulations configured for the same BGP-EVPN service.

*A:PE-1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls and evpn-vxlan in the same service"
  vxlan instance 1 vni 7000 create
  exit
  bgp 
    route-distinguisher 10:2
    route-target target:64500:1
  exit
  bgp 2 
    route-distinguisher 10:1
    route-target target:64500:1
  exit
  bgp-evpn
    evi 7000
    incl-mcast-orig-ip 10.12.12.12
    vxlan bgp 1 vxlan-instance 1
      no shutdown 
    mpls bgp 2
      control-word
      auto-bind-tunnel 
        resolution any
      exit
      force-vlan-vc-forwarding
      no shutdown
    exit
  exit
  no shutdown

The following list describe the preceding example:

  1. bgp 1 or bgp is the default BGP instance
  2. bgp 2 is the additional instance required when both bgp-evpn vxlan and bgp-evpn mpls are enabled in the service
  3. The commands supported in instance 1 are also available for the second instance with the following considerations.
    1. pw-template-binding: the pw-template-binding can only exist in instance 1; it is not supported in instance 2.
    2. route-distinguisher: the operating route-distinguisher in both bgp instances must be different
    3. route-target: the route-target in both instances can be the same or different
    4. vsi-import and vsi-export: import and export policies can also be defined for either bgp instance
  4. MPLS and VXLAN can use either BGP instance and the instance is associated when bgp-evpn mpls or bgp-evpn vxlan is created. The bgp-evpn vxlan command must include not only the association to a BGP instance, but also to a vxlan-instance (since VPLS services support two VXLAN instances).
    Note:

    The bgp-evpn vxlan no shutdown command is only allowed if bgp-evpn mpls shutdown is configured, or if the BGP instance associated with the MPLS has a different route distinguisher than the VXLAN instance.

The following features are not supported when two BGP instances are enabled on the same VPLS service:

  1. SDP-bindings
  2. R-VPLS, M-VPLS, I-VPLS, B-VPLS, or E-Tree VPLS
  3. Proxy-ARP and proxy-ND
  4. BGP Multi homing
  5. IGMP, MLD, and PIM snooping

5.4.9.1. BGP-EVPN Routes in Services Configured with Two BGP Instances

From a BGP perspective, the two BGP instances configured in the service are independent of each other. The redistribution of routes between the BGP instances is resolved at the EVPN application layer.

By default, if BGP-EVPN VXLAN and BGP-EVPN MPLS are both enabled in the same service, BGP will send the generated EVPN routes twice: once with the RFC 5512 BGP encapsulation extended community set to VXLAN and a second time with the encapsulation type set to MPLS.

Usually, a DC gateway will peer a pair of Route-Reflectors (RRs) in the DC and a pair of RRs in the WAN. For this reason, the user needs to add router policies so that EVPN-MPLS routes are only sent to the WAN RRs and EVPN-VXLAN routes are only sent to the DC RRs. The following examples show how you can configure router policies.

config>router>bgp# 
 vpn-apply-import
 vpn-apply-export
 group "WAN"                
  family evpn                
  type internal                
  export "allow only mpls"                 
  neighbor 192.0.2.6           
 group "DC"               
  family evpn                
  type internal                
  export "allow only vxlan"                 
  neighbor 192.0.2.2
config>router>policy-options# info 
----------------------------------------------
         community "vxlan" members "bgp-tunnel-encap:VXLAN"
         community "mpls" members "bgp-tunnel-encap:MPLS"
            policy-statement "allow only mpls"
                entry 10
                  from
                    family evpn
                    community vxlan
                  action drop
                  exit
                exit
            exit
            policy-statement "allow only vxlan"
                entry 10
             from
              family evpn
              community mpls
                    action drop
                    exit
                exit
            exit

In a BGP instance, the EVPN routes are imported based on the route-targets and regular BGP selection procedures, regardless of their encapsulation.

The BGP-EVPN routes are generated and redistributed between BGP instances based on the following rules.

  1. Auto-discovery (AD) routes (type 1) are not generated by services with two BGP instances. However, AD routes are received from the EVPN-MPLS peers and processed for aliasing and backup functions as usual.
  2. MAC/IP routes (type 2) received in one of the two BGP instances are imported and the MACs added to the FDB according to the existing selection rules. If the MAC is installed in the FDB, it is re-advertised in the other BGP instance with the new BGP attributes corresponding to the BGP instance (route-target, route-distinguisher, and so on). The following considerations apply to these routes.
    1. The mac-advertisement command governs the advertisement of any MACs (even those learned from BGP).
    2. A MAC route is redistributed only if it is the best route based on the EVPN selection rules.
    3. If a MAC route is the best route and has to be redistributed, the MAC/IP information, along with the MAC Mobility Extended Community, is propagated in the redistribution.
    4. The router redistributes any MAC route update for which any attribute has changed. For example, a change in the SEQ or sticky bit in one instance is updated in the other instance for a route that is selected as the best MAC route.
  3. EVPN inclusive multicast routes are generated independently for each BGP instance with the corresponding BGP encapsulation extended community (VXLAN or MPLS). Also, the following considerations apply to these routes.
    1. Ingress Replication (IR) and Assisted Replication (AR) routes are supported in the EVPN-VXLAN BGP instance. If AR is configured, the AR IP address must be a loopback address different from the system-ip and the configured originating-ip address.
    2. The IR, P2MP mLDP, and composite inclusive multicast routes are supported in the EVPN-MPLS BGP instance.
    3. The modification of the incl-mcast-orig-ip command is supported, subject to the following considerations.
      1. The configured IP in the incl-mcast-orig-ip command is encoded in the originating-ip field of the inclusive multicast Routes for IR, P2MP, and composite routes.
      2. The originating-ip field of the AR routes is still derived from the service>system>vxlan>assisted-replication-ip configured value.
    4. EVPN handles the inclusive multicast routes in a service based on the following rules.
      1. For IR routes, the EVPN destination is set up based on the NLRI next hop.
      2. For P2MP mLDP routes, the PMSI Tunnel Attribute tunnel-id is used to join the mLDP tree.
      3. For composite P2MP-IR routes, the PMSI Tunnel Attribute tunnel-id is used to join the tree and create the P2MP bind. The NLRI next-hop is used to build the IR destination.
      4. For AR routes, the NLRI next-hop is used to build the destination.
      5. The following applies if a router receives two inclusive multicast routes in the same instance.
        - If the routes have the same originating-ip but different route-distinguishers and next-hops, the router processes both routes. In the case of IR routes, it sets up two destinations: one to each next-hop.
        - If the routes have the same originating-ip, different route distinguishers, but same next hops, the router sets up only one binding for IR routes.
        - The router ignores inclusive multicast routes received with its own originating-ip, regardless of the route-distinguisher.
  4. IP-Prefix routes (type 5) are not generated or imported by a service with two BGP instances.

5.4.9.2. Anycast Redundant Solution for Dual BGP Instance Services

Figure 192 shows the Anycast mechanism used to support gateway redundancy for dual BGP instance services. The example shows two redundant DC gateways (DC GWs) where the VPLS services contain two BGP instances: one each for EVPN-VXLAN and EVPN-MPLS.

Figure 192:  Multi-homed Anycast Solution 

The example shown in Figure 192 depends on the ability of the two DC GWs to send the same inclusive multicast route to the remote PE or NVEs, such that:

  1. The remote PE or NVEs create a single BUM destination to one of the DC GWs (because the BGP selects only the best route to the DC GWs).
  2. The DC GWs do not create a destination between each other.

This solution avoids loops for BUM traffic, and known unicast traffic can use either DC GW router, depending on the BGP selection. The following CLI example output shows the configuration of each DC GW.

/* bgp configuration on DCGW1 and DCGW2 */
config>router>bgp# 
 group ”WAN"                
  family evpn                
  type internal                
  neighbor 192.0.2.6          
 group ”DC"                
  family evpn                
  type internal                
  neighbor 192.0.2.2
/* vpls service configuration */
DCGW-1# config>service>vpls(1)#
-----------------------
bgp 
  route-distinguisher 64501:12
  route-target target:64500:1
exit
bgp 2
  route-distinguisher 64502:12
  route-target target:64500:1
exit
vxlan instance 1 vni 1 create
exit
bgp-evpn 
  evi 1
  incl-mcast-orig-ip 10.12.12.12
  vxlan bgp 1 vxlan-instance 1
   no shutdown 
  mpls bgp 2
   no shutdown
   auto-bind-tunnel 
     resolution any
   exit
<snip>
DCGW-2# config>service>vpls(1)#
-----------------------
bgp 
  route-distinguisher 64501:12
  route-target target:64500:1
exit
bgp 2
  route-distinguisher 64502:12
  route-target target:64500:1
exit
vxlan instance 1 vni 1 create
exit
bgp-evpn 
  evi 1
  incl-mcast-orig-ip 10.12.12.12
  vxlan bgp 1 vxlan-instance 1
   no shutdown 
  mpls bgp 2
   no shutdown
   auto-bind-tunnel 
     resolution any
<snip>

Based on the preceding configuration example, the DC GWs behavior in this scenario is as follows:

  1. DCGW-1 and DCGW-2 send inclusive multicast routes to the DC RR and WAN RR with the same route key. For example:
    1. DCGW-1 and DCGW-2 both send an IR route to DC RR with RD=64501:12, orig-ip=10.12.12.12, and a different next-hop and tunnel ID
    2. DCGW-1 and DCGW-2 both send an IR route to WAN RR with RD=64502:12, orig-ip=10.12.12.12, and different next-hop and tunnel ID
  2. DCGW-1 and DCGW-2 both receive MAC/IP routes from DC and WAN that will be redistributed to the other BGP instances, assuming that the route is selected as best route and the MAC is installed in the FDB
    1. As described in section BGP-EVPN Routes in Services Configured with Two BGP Instances, router peer policies are required so that only VXLAN or MPLS routes are sent or received for a specific peer
  3. Configuration of the same incl-mcast-orig-ip address in both DCGWs enables the Anycast solution for BUM traffic due to the following reasons.
    1. The configured originating-ip is not required to be a reachable IP address and this forces the remote PE or NVEs to select only one of the two DC GWs.
    2. The BGP next-hops are allowed to be the system-ip or even a loopback address. In both cases, the BGP next-hops are not required to be reachable in their respective networks.

In the example shown in Figure 192, PE-1 will pick up DC GW-1's inclusive multicast route (because of its lower BGP next-hop) and create a BUM destination to 192.0.2.4. When sending BUM traffic for VPLS-1, it will only send the traffic to DC GW-1. In the same way, the DC GWs will not set up BUM destinations between each other as they use the same originating-ip in their inclusive multicast routes.

The remote PE or NVEs perform a similar BGP selection for MAC or IP routes, as a specific MAC is sent by the two DC GWs with the same route-key. A PE or NVE will send known unicast traffic for a specific MAC to only one DC GW.

5.4.9.3. Using P2MP mLDP in Redundant Anycast DC GWs

Figure 193 shows an example of a common BGP EVPN service configured in redundant Anycast DC GWs and mLDP used in the MPLS instance.

Note:

Packet duplication may occur if the service configuration is not performed carefully.

Figure 193:  Anycast Multi-homing and mLDP 

When mLDP is used with multiple Anycast multi-homing DC GWs, the same originating IP address must be used by all the DC GWs. Failure to do so may result in packet duplication.

In the example shown in Figure 193, each pair of DC GWs (DCGW1/DCGW2 and DCGW3/DCGW4) is configured with a different originating IP address, which causes the following behavior.

  1. DCGW3 and DCGW4 receive the inclusive multicast routes with the same route key from DCGW1 and DCGW2.
  2. Both DC GWs (DCGW3 and DCGW4) select only one route, which is generally the same, for example, DCGW1's inclusive multicast route.
  3. As a result, DCGW3 and DCGW4 join the mLDP tree with root in DCGW1, creating packet duplication when DCGW1 sends BUM traffic.
  4. Remote PE nodes with a single BGP-EVPN instance join the mLDP tree without any problem.

To avoid the packet duplication shown in Figure 193, Nokia recommends to configure the same originating IP address in all four DC GWs (DCGW1/DCGW2 and DCGW3/DCGW4). However, the route-distinguishers can be different per pair.

The following behavior occurs if the same originating IP address is configured on the DC GW pairs shown in Figure 193.

Note:

This configuration allows the use of mLDP as long as BUM traffic is not required between the two DCs. Ingress Replication must be used if BUM traffic between the DCs is required.

  1. DCGW3 and DCGW4 do not join any mLDP tree sourced from DCGW1 or DCGW2, which prevents any packet duplication. This is because a router will ignore inclusive multicast routes received with its own originating-ip, regardless of the route-distinguisher.
  2. PE1 joins the mLDP trees from the two DCs.

5.4.9.4. Interconnect Ethernet-Segment Solution for Dual BGP Instance Services

SR OS supports Interconnect ESs (I-ES) for VXLAN as per IETF Draft draft-ietf-bess-dci-evpn-overlay. An I-ES is a virtual ES that allows DC GWs with two BGP instances to handle VXLAN access networks as any other type of ES. I-ESs support the RFC 7432 multi-homing functions, including single-active and all-active, ESI-based split-horizon filtering, DF election, and aliasing and backup on remote EVPN-MPLS PEs.

In addition to the EVPN multi-homing features, the main advantages of the I-ES redundant solution compared to the redundant solution described in Anycast Redundant Solution for Dual BGP Instance Services are as follows.

  1. The use of I-ES for redundancy in dual BGP-instance services allows local SAPs on the DCGWs.
  2. P2MP mLDP can be used to transport BUM traffic between DCs that use I-ES without any risk of packet duplication. As described in Using P2MP mLDP in Redundant Anycast DC GWs, packet duplication may occur in the Anycast DC GW solution when mLDP is used in the WAN.

Where EVPN-MPLS networks are interconnected to EVPN-VXLAN networks, the I-ES concept applies only to the access VXLAN network; the EVPN-MPLS network does not modify its existing behavior.

Figure 194 shows the use of I-ES for Layer 2 EVPN DCI between VXLAN and MPLS networks.

Figure 194:  The Interconnect ES Concept 

The following example shows how I-ES-1 would be provisioned on DC GW1 and the association between I-ES to a given VPLS service. A similar configuration would occur on DC GW2 in the I-ES.

New I-ES configuration:

DCGW1#config>service>system>bgp-evpn# 
ethernet-segment I-ES-1 virtual create
 esi 01:00:00:00:12:12:12:12:12:00
 service-carving
  mode auto
 multi-homing all-active
 network-interconnect-vxlan 1
 service-id
   service-range 1 to 1000
 no shutdown

Service configuration:

DCGW1#config>service>vpls(1)# 
 vxlan instance 1 vni 1 instance 1 create
 exit
bgp
 route-distinguisher 1:1 
bgp 2
 route-distinguisher 2:2               
bgp-evpn        
  evi 1
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any                     
   no shutdown
 
...
 
DCGW1#config>service>vpls(2)#  
 vxlan instance 1 vni 2 create
 exit
bgp
 route-distinguisher 3:3 
bgp 2
 route-distinguisher 4:4               
 bgp-evpn        
  evi 2
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any   
   no shutdown
 sap 1/1/1:1 create
 exit 

The above configuration associates I-ES-1 to the VXLAN instance in services VPLS1 and VPLS 2. The I-ES is modeled as a virtual ES, with the following considerations.

  1. The commands network-interconnect-vxlan and service-id service-range svc-id [to svc-id] are required within the Ethernet segment.
    1. The network-interconnect-vxlan parameter identifies the VXLAN instance associated with the virtual ES. The value of the parameter must be set to 1. This command is rejected in a non-virtual ES.
    2. The service -range parameter associates the specific service range to the ES. The ES must be configured as network-interconnect-vxlan before any service range can be added.
    3. The ES parameters port, lag, sdp, vc-id-range, dot1q, and qinq cannot be configured in the Ethernet segment when a network-interconnect-vxlan instance is configured. The source-bmac-lsb option is blocked, as the I-ES cannot be associated with an I-VPLS or PBB-Epipe service. The remaining ES configuration options are supported.
    4. All services with two BGP instances associate the VXLAN destinations and ingress VXLAN instances to the Ethernet segment.
  2. Multiple services can be associated with the same Ethernet segment, with the following considerations.
    1. In a DC with two DC GWs (as in Figure 194), only two I-ESs are needed to load-balance, where one half of the dual BGP-instance services would be associated with one I-ES (for example, I-ES-1, in the above configuration) and one half to another I-ES.
    1. Up to eight service ranges per VXLAN instance can be configured. Ranges may overlap within the same ES, but not between different ESs.
    1. The service range can be configured before the service.
  3. After the I-ES is configured using network-interconnect-vxlan, the ES operational state depends exclusively on the ES administrative state. Because the I-ES is not associated with a physical port or SDP, when testing the non-revertive service carving manual mode, an Ethernet segment shutdown and no shutdown event results in the node sending its own administrative preference and DP bit and taking control if the preference and DP bit are higher than the current DF. This is because the peer ES routes are not present at the EVPN application layer when the ES is configured for no shutdown; therefore, the PE sends its own administrative preference and DP values. Therefore, for I-ESs, the non-revertive mode works only for node failures.
  4. A VXLAN instance may be placed in MhStandby under any of the following situations:
    1. if the PE is single-active NDF for that I-ES
    2. if the VXLAN service is added to the I-ES and either the ES or BGP-EVPN MPLS is shut down in all the services included in the ES
      The following example shows the change of the MhStandby flag from false to true when BGP-EVPN MPLS is shut down for all the services in the I-ES.
      A:PE-4# show service id 500 vxlan instance 1 oper-flags 
      ===============================================================================
      VPLS VXLAN oper flags
      ===============================================================================
      MhStandby                              : false
      ===============================================================================
      A:PE-4# configure service vpls 500 bgp-evpn vxlan shutdown 
      *A:PE-4# show service id 500 vxlan instance 1 oper-flags    
      ===============================================================================
      VPLS VXLAN oper flags
      ===============================================================================
      MhStandby                              : true
      ===============================================================================

5.4.9.4.1. BGP-EVPN Routes on Dual BGP-instance Services with I-ES

The configuration of an I-ES on DC GWs with two BGP-instances has the following impact on the advertisement and processing of BGP-EVPN routes.

  1. For EVPN MAC/IP routes, the following considerations apply.
    1. MAC/IP routes received in the EVPN-MPLS BGP-instance are re-advertised in the EVPN-VXLAN BGP-instance with the ESI set to zero.
    2. EVPN-VXLAN PEs and NVEs in the DC receive the same MAC from two or more different MAC/IP routes from the DC GWs, which perform regular EVPN MAC/IP route selection.
    3. MAC/IP routes received in the EVPN-VXLAN BGP-instance are re-advertised in the EVPN-MPLS BGP-instance with the configured non-zero I-ESI value, assuming the VXLAN instance is not in an MhStandby operational state; otherwise the MAC/IP routes are dropped.
    4. EVPN-MPLS PEs in the WAN receive the same MAC from two or more DC GWs, set with the same ESI. In this case, regular aliasing and backup functions occur as usual.
  2. ES routes are exchanged for the I-ES. The routes should be sent only to the MPLS network and not to the VXLAN network. This can be achieved by using router policies.
  3. AD per-ES and AD per-EVI are also advertised for the I-ES, and should be sent only to the MPLS network and not to the VXLAN. As for ES routes, router polices can be used to prevent AD routes from being sent to VXLAN peers.

In general, when I-ESs are used for redundancy, the use of router policies is needed to avoid control plane loops with MAC/IP routes. Consider the following to avoid control plane loops:

  1. Loops created by remote MACs
    Remote EVPN-MPLS MAC/IP routes are re-advertised into EVPN-VXLAN routes with an SOO (Site Of Origin) EC added by a BGP peer or VSI export policy identifying the DC GW pair. The other DC GW in the pair drops EVPN-VXLAN MAC routes tagged with the pair SOO. Router policies are needed to add SOO and drop routes received with self SOO.
    When remote EVPN-VXLAN MAC/IP routes are re-advertised into EVPN-MPLS, the DC GWs automatically drop EVPN-MPLS MAC/IP routes received with their own non-zero I-ESI.
  2. Loops created by local SAP MACs
    Local SAP MACs are learned and MAC/IP routes are advertised into both BGP instances. The MAC/IP routes advertised in the EVPN-VXLAN instance are dropped by the peer based on the SOO router policies as described above for loops created by remote MACs. The DC GW local MACs are always learned over the EVPN-MPLS destinations between the DC GWs.

The following outlines the considerations for BGP peer policies on DC GW1 to avoid control plane loops. Similar policies would be configured on DC GW2.

  1. Avoid sending service VXLAN routes to MPLS peers and service MPLS routes to VXLAN peers.
  2. Avoid sending AD and ES routes to VXLAN peers.
  3. Add SOO to VXLAN routes sent to the ES peer.
  4. Drop VXLAN routes received from the ES peer.

The following shows the CLI configuration.

A:DCGW1# configure router bgp 
A:DCGW1>config>router>bgp# info 
----------------------------------------------
            family vpn-ipv4 evpn
            vpn-apply-import
            vpn-apply-export
            rapid-withdrawal
            rapid-update vpn-ipv4 evpn
            group "wan"
                type internal
                export "allow only mpls" 
                neighbor 192.0.2.4
                exit
                neighbor 192.0.2.5
                exit
            exit
            group "internal"
                type internal
                neighbor 192.0.2.1
                    export "allow only vxlan" 
                exit
                neighbor 192.0.2.3
                    import "drop SOO-DCGW-23" 
                    export "add SOO to vxlan routes" 
                exit                  
            exit
            no shutdown
----------------------------------------------
A:DCGW1>config>router>bgp# /configure router policy-options    
A:DCGW1>config>router>policy-options# info 
----------------------------------------------
            community "mpls" members "bgp-tunnel-encap:MPLS"
            community "vxlan" members "bgp-tunnel-encap:VXLAN"
            community "SOO-DCGW-23" members "origin:64500:23"
// This policy prevents the router from sending service VXLAN routes to MPLS peers. //
            policy-statement "allow only mpls"
                entry 10
                    from
                        community "vxlan"
                        family evpn
                    exit
                    action drop
                    exit
                exit
            exit
// This policy ensures the router only exports routes that include the VXLAN encapsulation. //
            policy-statement "allow only vxlan"
                entry 10
                    from
                        community "vxlan"
                        family evpn
                    exit
                    action accept
                    exit
                exit                  
                default-action drop
                exit
            exit
// This import policy avoids importing routes with a self SOO. //
            policy-statement "drop SOO-DCGW-23"
                entry 10
                    from
                        community "SOO-DCGW-23"
                        family evpn
                    exit
                    action drop
                    exit
                exit
            exit
// This import policy adds SOO only to VXLAN routes. This allows the peer to drop routes based on the SOO, without affecting the MPLS routes. //
            policy-statement "add SOO to vxlan routes"
                entry 10
                    from
                        community "vxlan"
                        family evpn
                    exit
                    action accept
                        community add "SOO-DCGW-23"
                    exit
                exit                  
                default-action accept
                exit
            exit
----------------------------------------------

5.4.9.4.2. Single-Active Multi-Homing on I-ES

When an I-ES is configured as single-active and is no shutdown with at least one associated service, the DC GWs send ES and AD routes as for any ES and runs DF election as normal, based on the ES routes, with the candidate list being pruned by the AD routes.

Figure 195 shows the expected behavior for a single-active I-ES.

Figure 195:  Interconnect ES — Single-Active 

As shown in Figure 195, the Non-Designated-Forwarder (NDF) for a specified service carries out the following tasks.

  1. From a data path perspective, the VXLAN instance on the NDF goes into an MhStandby operational state and blocks ingress and egress traffic on the VXLAN destinations associated with the I-ES.
  2. MAC/IP routes and the FDB process
    1. MAC/IP routes associated with the VXLAN instance and re-advertised to EVPN-MPLS peers are withdrawn.
    2. MAC/IP routes corresponding to local SAP MACs or EVPN-MPLS binding MACs are withdrawn if they were advertised to the EVPN-VXLAN instance.
    3. Received MAC/IP routes associated with the VXLAN instance are not installed in the FDB. MAC routes show as “used” in BGP, however, only the MAC route received from MPLS (from the ES peer) is programmed.
  3. Inclusive Multicast Ethernet Tag (IMET) routes process
    1. IMET-AR-R routes (IMET-AR with replicator role) must be withdrawn if the VXLAN instance goes into an MhStandby operational state. Only the DF advertises the IMET-AR-R routes.
    2. IMET-IR advertisements in the case of the NDF (or MhStandby) are controlled by the command config>service>vpls>bgp-evpn>vxlan [no] send-imet-ir-on-ndf.
      By default, the command is enabled and the router advertises IMET-IR routes, even if the PE is NDF (MhStandby). This attracts BUM traffic, but also speeds up convergence in the case of a DF switchover. The command is supported for single-active and all-active.
      If the command is disabled, the router withdraws the IMET-IR routes when the PE is NDF and will not attract BUM traffic.

The I-ES DF PE for the service continues advertising IMET and MAC/IP routes for the associated VXLAN instance as usual, as well as forwarding on the DF VXLAN bindings. When the DF DC GW receives BUM traffic, it sends the traffic with the egress ESI label if needed.

5.4.9.4.3. All-Active Multi-Homing on I-ES

The same considerations for ES and AD routes, and DF election apply for all-active multi-homing as for single-active multi-homing; the difference is in the behavior on the NDF DC GW. The NDF for a specified service performs the following tasks.

  1. From a data path perspective, the NDF blocks ingress and egress paths for broadcast and multicast traffic on the VXLAN instance bindings associated with the I-ES, while unknown and known unicast traffic is still allowed. The unknown unicast traffic is transmitted on the NDF if there is no risk of duplication. For example, unknown unicast packets are transmitted on the NDF if they do not have an ESI label, do not have an EVPN BUM label, and they pass a MAC SA suppression. In the example in Figure 196, the NDF transmits unknown unicast traffic. Regardless of whether DC GW1 is a DF or NDF, it accepts the unknown unicast packets and floods to local SAPs and EVPN destinations. When sending to DC GW2, the router sends the ESI-label identifying the I-ES. Due to the ESI-label suppression, DC GW2 does not send unknown traffic back to the DC.
    Figure 196:  All-active Multi-Homing and Unknown Unicast on the NDF 
  2. MAC/IP routes and the FDB process
    1. MAC/IP routes associated with the VXLAN instance are advertised normally.
    2. MACs are installed as normal in the FDB for received MAC/IP routes associated with the VXLAN instance.
  3. IMET routes process
    1. As is the case for single-active multi-homing, IMET-AR-R routes must be withdrawn on the NDF (MhStandby state). Only the DF advertises the IMET-AR-R routes.
    2. The IMET-IR advertisements in the case of the NDF (or MhStandby) are controlled by the command config>service>vpls>bgp-evpn>vxlan [no] send-imet-ir-on-ndf, as in single-active multi-homing.

The I-ES DF PE for the service continues advertising IMET and MAC/IP routes for the associated VXLAN instance as usual. When the DF DC GW receives BUM traffic, it sends the traffic with the egress ESI label if needed.

5.4.10. Configuring multi-instance EVPN-VXLAN in the Same VPLS Service

As described in Configuring EVPN-VXLAN and EVPN-MPLS in the Same VPLS Service, two BGP instances are supported in VPLS services, where one instance can be associated to EVPN-VXLAN and the other instance to EVPN-MPLS. In addition, both BGP instances in a VPLS service can also be associated to EVPN-VXLAN.

For example, a VPLS service can be configured with two VXLAN instances that use VNI 500 and 501 respectively, and those instances are associated to different BGP instances:

*A:PE-2# configure service vpls 500
*A:PE-2>config>service>vpls# info
----------------------------------------------
vxlan instance 1 vni 500 create
exit
vxlan instance 2 vni 501 create
exit
bgp
 route-distinguisher 192.0.2.2:500
 vsi-export "vsi-500-export"
 vsi-import "vsi-500-import"
exit
bgp 2
 route-distinguisher 192.0.2.2:501
 vsi-export "vsi-501-export"
 vsi-import "vsi-501-import"
exit
bgp-evpn
 incl-mcast-orig-ip 23.23.23.23
 evi 500
 vxlan bgp 1 vxlan-instance 1
 no shutdown
 exit
 vxlan bgp 2 vxlan-instance 2
 no shutdown
 exit
exit
stp
shutdown
exit
no shutdown
----------------------------------------------

From a data plane perspective, each VXLAN instance is instantiated in a different implicit SHG, so that traffic can be forwarded between them.

At a control plane level, the processing of MAC or IP routes and inclusive multicast routes is described in BGP-EVPN Routes in Services Configured with Two BGP Instances with the differences described in BGP-EVPN Routes in Multi-Instance EVPN-VXLAN Services.

The following features are not supported along with dual BGP instance EVPN-VXLAN services:

  1. I-VPLS, B-VPLS, M-VPLS, and E-tree service types
  2. BGP-VPLS
  3. R-VPLS
  4. EVPN ES association to VXLAN instance 1 when instance 2 is VXLAN and not MPLS
  5. SAPs and SDP-bindings when BGP instance 1 and BGP instance 2 are both associated to VXLAN
  6. ESI-based PBR on the VPLS service
  7. IGMP, MLD, and PIM-snooping

In addition, multi-instance EVPN-VXLAN services support:

  1. Assisted-replication for IPv4 VTEPs in both VXLAN instances, where a single assisted-replication IPv4 address can be used for both instances.
  2. Non-system IP and IPv6 termination, where a single vxlan-src-vtep ip-address can be configured for each service, and therefore used for the two instances.

5.4.10.1. BGP-EVPN Routes in Multi-Instance EVPN-VXLAN Services

If two BGP instances with the same encapsulation (VXLAN) are configured in the same VPLS service, different import route-targets in each BGP instance are mandatory (although this is not enforced).

BGP-EVPN Routes in Services Configured with Two BGP Instances describes the use of policies to avoid sending WAN routes (routes meant to be redistributed from DC to WAN) to the DC again and DC routes (routes meant to be redistributed from WAN to DC) to the WAN again. Those policies are based on export policy statements that match on the RFC 5512 BGP Encapsulation Extended Community (MPLS and VXLAN respectively).

When the two BGP instances are VXLAN based, the above policies matching on different BGP encapsulation extended community is not feasible since both instances advertise routes with VXLAN encapsulation. Because the export route targets in the two BGP instances must be different, the policies, to avoid sending WAN routes back to the WAN and DC routes back to the DC can be based on export policies that prevent routes with a DC route target from being sent to the WAN peers (and opposite for routes with WAN route target).

Other than the specifications described in this section, the processing of MAC or IP routes and inclusive multicast Ethernet tag routes in multi-instance EVPN-VXLAN services follow the rules described in BGP-EVPN Routes in Services Configured with Two BGP Instances.

5.4.10.2. Anycast Redundant Solution for Multi-Instance EVPN-VXLAN Services

The solution described in Anycast Redundant Solution for Dual BGP Instance Services is also supported in multi-instance EVPN-VXLAN VPLS services.

The following CLI sample output shows the configuration of DCGW-1 and DCGW-2 in Figure 192 where VPLS 500 is a multi-instance EVPN-VXLAN service and BGP instance 2 is associated to VXLAN instead of MPLS.

*A:DCGW-1(and DCGW-2)>config>service>vpls(500)# info
----------------------------------------------
vxlan instance 1 vni 500 create
exit
vxlan instance 2 vni 501 create
exit
bgp
 route-distinguisher 192.0.2.2:500
 vsi-export "vsi-500-export"
 vsi-import "vsi-500-import"
exit
bgp 2
 route-distinguisher 192.0.2.2:501
 vsi-export "vsi-501-export"
 vsi-import "vsi-501-import"
exit
bgp-evpn
 incl-mcast-orig-ip 23.23.23.23
 evi 500
 vxlan bgp 1 vxlan-instance 1
 no shutdown
 exit
 vxlan bgp 2 vxlan-instance 2
 no shutdown
 exit
exit
stp
shutdown
exit
no shutdown
----------------------------------------------
*A:DCGW-1(and DCGW-2)# show router policy "vsi-500-export"
entry 10
 action accept
 community add "RT64500:500" "SOO-23"
 exit
exit
*A:DCGW-1(and DCGW-2)# show router policy "vsi-500-import"
entry 10
 from
 community "SOO-23"
 family evpn
 exit
 action drop
 exit
exit
entry 20
 from
 community "RT64500:500"
 family evpn
 exit
 action accept
 exit
exit
*A:DCGW-1(and DCGW-2)# show router policy "vsi-501-export"
entry 10
 action accept
 community add "RT64500:501" "SOO-23"
 exit
exit
*A:PE-2>config>service>vpls# show router policy "vsi-501-import"
entry 10
 from
 community "SOO-23"
 family evpn
 exit
 action drop
 exit
exit
entry 20
 from
 community "RT64500:501"
 family evpn
 exit
 action accept
 exit
exit

Refer to Anycast Redundant Solution for Dual BGP Instance Services for a full description of this solution.

5.4.11. EVPN IP-Prefix Route Interoperability

SR OS supports the three IP-VRF-to-IP-VRF models defined in draft-ietf-bess-evpn-prefix-advertisement for EVPN-VXLAN and EVPN-MPLS R-VPLS services. Those three models are known as:

  1. interface-less IP-VRF-to-IP-VRF
  2. interface-ful IP-VRF-to-IP-VRF with SBD IRB (Supplementary Bridge Domain Integrated Routing Bridging)
  3. interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB

SR OS supports all three models for IPv4 prefixes and the interface-ful with SBD IRB model for IPv6 prefixes. The three models have pros and cons, and different vendors have chosen different models depending on the use-cases that they intend to address. When a third-party vendor is connected to an SR OS router, it is important to know which of the three models the third-party vendor implements. The following sections describe the models and the required configuration in each of them.

5.4.11.1. Interface-ful IP-VRF-to-IP-VRF with SBD IRB Model

The SBD is equivalent to an R-VPLS that connects all the PEs that are attached to the same tenant VPRN. Interface-ful refers to the fact that there is a full IRB interface between the VPRN and the SBD (an interface object with MAC and IP addresses, over which interface parameters can be configured).

Figure 197 illustrates this model.

Figure 197:  Interface-ful IP-VRF-to-IP-VRF with SBD IRB Model 

Figure 197 shows a 7750 SR and a third-party router using interface-ful IP-VRF-to-IP-VRF with SBD IRB model. The two routers are attached to a VPRN for the same tenant, and those VPRNs are connected by R-VPLS-2, or SBD. Both routers exchange IP prefix routes with a non-zero GW IP (this is the IP address of the SBD IRB). The SBD IRB MAC and IP are advertised in a MAC/IP route. On reception, the IP prefix route creates a route-table entry in the VPRN, where the GW IP must be recursively resolved to the information provided by the MAC/IP route and installed in the ARP and FDB tables.

This model is explained in detail in EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes. As an example, and based on Figure 197 above, the following CLI output shows the configuration of a 7750 SR SBD and VPRN, using on this interface-ful with SBD IRB mode:

7750SR#config>service#
 
vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
 evi 2
 ip-route-advertisement
 mpls bgp 1
 auto-bind-tunnel resolution any
 no shutdown
 
vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
 address 192.168.0.0/16
 vpls "sbd"

5.4.11.2. Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB Model

Interface-ful refers to the fact that there is a full IRB interface between the VPRN and the SBD. However, the SBD IRB is unnumbered in this model, which means no IP address is configured on it. In SR OS, an unnumbered SBD IRB is equivalent to an R-VPLS linked to a VPRN interface through an EVPN tunnel. See EVPN for VXLAN in EVPN Tunnel R-VPLS Services for more information.

Figure 198 illustrates this model.

Figure 198:  Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB Model 

Figure 198 shows a 7750 SR and a third-party router running interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model. The IP prefix routes are now expected to have a zero GW IP and the MAC in the router's MAC extended community used for the recursive resolution to a MAC/IP route.

The corresponding configuration of the 7750 SR VPRN and SBD in the example could be:

7750SR#config>service#
 
vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
evi 2
ip-route-advertisement
mpls bgp 1
auto-bind-tunnel resolution any
no shutdown
 
vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
vpls "sbd"
evpn-tunnel

5.4.11.3. Interface-less IP-VRF-to-IP-VRF Model

This model is called interface-less because it does not require an SBD connecting the VPRNs for the tenant, and no recursive resolution is required upon receiving an IP prefix route. In other words, the IP prefix route's next hop is directly resolved to an EVPN tunnel, without the need for any other route. The ingress PE uses the received router's MAC extended community address as the inner destination MAC address for the EVPN packets sent to the prefix.

Figure 199 illustrates this model.

Figure 199:  Interface-less IP-VRF-to-IP-VRF Model 

In SR OS, this model is implemented as follows:

  1. There is no data path difference between this model and the existing R-VPLS EVPN tunnel model or the model described in Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB Model.
  2. This model is enabled by configuring config>service>vprn>if>vpls>evpn-tunnel and bgp-evpn>ip-route-advertisement. In addition, since the SBD IRB MAC/IP route is no longer needed, the command bgp-evpn>no mac-advertisement prevents the MAC/IP route from being advertised.
  3. The IP prefix routes are processed as follows:
    1. On transmission, there is no change in the way IPv4 prefix routes are processed compared to the evpn-tunnel configuration:
      1. IPv4 prefix routes are advertised in the route-table, with GW-IP=0 and the corresponding MAC extended community.
      2. If bgp-evpn> no mac-advertisement is configured, no MAC/IP route will be sent for the R-VPLS.
    2. The received IPv4 prefix routes are processed as follows:
      1. Upon receiving an IPv4 prefix route with a router's MAC extended community, an internal MAC/IP route is generated with the encoded MAC and the RD, Ethernet tag, ESI, Label/VNI and next hop from the IP prefix route itself.
      2. If there is no other competing received MAC/IP route for the same MAC, this IP prefix-derived MAC/IP route is selected and the MAC installed in the R-VPLS FDB with type "Evpn".
      3. Once the MAC is installed in FDB, there are no differences between this interface-less model and the interface-ful with unnumbered SBD IRB model. Therefore, SR OS is compatible with the received IP prefix routes for both models.

A typical configuration of a PE's SBD and VPRN that work in interface-less model follows:

7750SR#config>service#
 
vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
evi 2
no mac-advertisement
ip-route-advertisement
mpls bgp 1
auto-bind-tunnel resolution any
no shutdown
vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
vpls "sbd"
evpn-tunnel

5.4.12. ARP-ND Host Routes for Extended Layer-2 Data Centers

SR OS supports the creation of host routes for IP addresses that are present in the ARP or neighbor tables of a routing context. These host routes are referred to as ARP-ND routes and can be advertised using EVPN or IP-VPN families. A typical use-case where ARP-ND routes are needed is the extension of Layer-2 Data Centers (DCs). Figure 200 illustrates this use-case.

Figure 200:  Extended Layer-2 Data Centers 

Subnet 10.0.0.0/16 in Figure 200 is extended throughout two DCs. The DC gateways are connected to the users of subnet 20.0.0.0/24 on PE1 using IP-VPN (or EVPN). If the virtual machine VM 10.0.0.1 is connected to DC1, when PE1 needs to send traffic to host 10.0.0.1, it performs a Longest Prefix Match (LPM) lookup on the VPRN’s route table. If the only IP prefix advertised by the four DC GWs was 10.0.0.0/16, PE1 could send the packets to the DC where the VM is not present.

To provide efficient downstream routing to the DC where the VM is located, DCGW1 and DCGW2 must generate host routes for the VMs to which they connect. When the VM moves to the other DC, DCGW3 and DCGW4 must be able to learn the VM’s host route and advertise it to PE1. DCGW1 and DCGW2 must withdraw the route for 10.0.0.1, since the VM is no longer in the local DC.

In this case, the SR OS is able to learn the VM’s host route from the generated ARP or ND messages when the VM boots or when the VM moves.

A route owner type called “ARP-ND” is supported in the base or VPRN route table. The ARP-ND host routes have a preference of 1 in the route table and are automatically created out of the ARP or ND neighbor entries in the router instance.

When the config>service>vprn/ies>interface>arp-populate-host-route command is enabled, the static, dynamic, and EVPN ARP entries of the routing context create ARP-ND host routes in the route table. Similarly, ARP-ND host routes are created in the IPv6 route table out of static, dynamic, and EVPN neighbor entries if the config>service>vprn/ies>interface>ipv6>nd-populate-host-route command is enabled.

The arp and nd-populate-host-route commands are used with the following features:

  1. adding ARP-ND hosts — A route tag can be added to ARP-ND hosts using the arp-route-tag and nd-route-tag commands. This tag can be matched on BGP VRF export and peer export policies.
  2. keeping entries active — The ARP-ND host routes are kept in the route table as long as the corresponding ARP or neighbor entry is active. Even if there is no traffic destined to them, the arp-proactive-refresh and nd-proactive-refresh commands configure the node to keep the entries active by sending an ARP refresh message 30 seconds before the arp-timeout or starting NUD when the stale time expires.
  3. speeding up learning — To speed up the learning of the ARP-ND host routes, the arp-learn-unsolicited and nd-learn-unsolicited commands can be configured. When arp-learn-unsolicited is enabled, received unsolicited ARP messages (typically GARPs) create an ARP entry, and consequently, an ARP-ND route if arp-populate-host-route is enabled. Similarly, unsolicited Neighbor Advertisement messages create a stale neighbor. If nd-populate-host-route is enabled, a confirmation message (NUD) is sent for all the neighbor entries created as stale, and if confirmed, the corresponding ARP-ND routes are added to the route table.
Note:

The ARP-ND host routes are created in the route table but not in the routing context FIB. This helps preserve the FIB scale in the router.

In Figure 200, enabling arp-populate-host-route on the DCGWs allows them to learn or advertise the ARP-ND host route 10.0.0.1/32 when the VM is locally connected and to remove or withdraw the host routes when the VM is no longer present in the local DC.

5.4.13. BGP and EVPN Route Selection for EVPN Routes

When two or more EVPN routes are received at a PE, BGP route selection typically takes place when the route key or the routes are equal. When the route key is different, but the PE has to make a selection (for instance, the same MAC is advertised in two routes with different RDs), BGP will hand over the routes to EVPN and the EVPN application will perform the selection.

EVPN and BGP selection criteria are described below.

  1. EVPN route selection for MAC routes: when two or more routes with the same mac-length/mac but different route key are received, BGP will hand the routes over to EVPN. EVPN will select the route based on the following tiebreaking order:
    1. Conditional static MACs (local protected MACs).
    2. Auto-learned protected MACs (locally learned MACs on SAPs or mesh/Spoke-SDPs due to the configuration of auto-learn-mac-protect).
    3. EVPN ES PBR MACs (see ES PBR MAC routes below).
    4. EVPN static MACs (remote protected MACs).
    5. Data plane learned MACs (regular learning on saps/sdp-bindings).
    6. EVPN MACs with higher SEQ number.
    7. EVPN E-Tree root MACs.
    8. EVPN non-RT-5 MACs (this tie-breaking rule is only observed if the selection algorithm is comparing received MAC routes and internal MAC routes derived from the MACs in IP-Prefix routes, i.e. RT-5 MACs).
    9. Lowest IP (next-hop IP of the EVPN NLRI).
    10. Lowest Ethernet tag (that will be zero for MPLS and might be different from zero for VXLAN).
    11. Lowest RD.
    12. Lowest BGP instance (this tie-breaking rule is only considered if the above rules fail to select a unique MAC and the service has two BGP instances of the same encapsulation).
  2. ES PBR MAC routes: when a PBR filter with a forward action to an ESI and SF-IP (Service Function IP) exists, a MAC route is created by the system. This MAC route will be compared to other MAC routes received from BGP.
    1. When ARP resolves (it can be static, EVPN, or dynamic) for a SF-IP and the system has an AD EVI route for the ESI, a "MAC route" is created by ES PBR with the <MAC Address = ARPed MAC Address, VTEP = AD EVI VTEP, VNI = AD EVI VNI, RD = ES PBR RD (special RD), Static = 1> and installed in EVPN.
    2. This MAC route doesn't add anything (back) to ARP; however, it goes through the MAC route selection in EVPN and triggers the FDB addition if it is the best route.
    3. In terms of priority, this route's priority is lower than local static but higher than remote EVPN static (number 2 in the tiebreaking order above).
    4. If there are two competing ES PBR MAC routes, then the selection goes through the rest of checks (Lowest IP > Lowest RD).
  3. The BGP route selection for MAC routes with the same route-key follows the following priority order:
    1. EVPN static MACs (remote protected MACs).
    2. EVPN MACs with higher sequence number.
    3. Regular BGP selection (local-pref, aigp metric, shortest as-path, …, lowest IP).
  4. The BGP route selection for the rest of the EVPN routes: regular BGP selection is followed.
Note:

In case BGP has to run an actual selection and a specified (otherwise valid) EVPN route 'loses' to another EVPN route, the non-selected route will be displayed by the show router BGP routes evpn x detail command with a tie-breaker reason.

Note:

Protected MACs do not overwrite EVPN static MACs; in other words, if a MAC is in the FDB and protected due to it being received with the sticky/static bit set in a BGP EVPN update and a frame is received with the source MAC on an object configured with auto-learn-mac-protect, that frame will be dropped due to the implicit restrict-protected-src discard-frame. The reverse is not true; when a MAC is learned and protected using auto-learn-mac-protect, its information is not overwritten with the contents of a BGP update containing the same MAC address.

5.4.14. LSP Tagging for BGP Next-Hops or Prefixes and BGP-LU

It is possible to constrain the tunnels used by the system for resolution of BGP next-hops or prefixes and BGP labeled unicast routes using LSP administrative tags. Refer to LSP Tagging and Auto-Bind Using Tag Information in the 7450 ESS, 7750 SR, 7950 XRS, and VSR MPLS Guide for further details.

5.4.15. Interaction of EVPN and Other Features

This section contains information about EVPN and how it interacts with other features.

5.4.15.1. Interaction of EVPN-VXLAN and EVPN-MPLS with Existing VPLS Features

When enabling existing VPLS features in an EVPN-VXLAN or an EVPN-MPLS enabled service, the following must be considered:

  1. EVPN-VXLAN services are not supported on I-VPLS/B-VPLS. VXLAN cannot be enabled on those services. EVPN-MPLS is only supported in regular VPLS and B-VPLS. Other VPLS types, such as etree or m-vpls, are not supported with either EVPN-VXLAN or EVPN-MPLS.
  2. In general, no router-generated control packets will be sent to the EVPN destination bindings, except for ARP, VRRP, ping, BFD and Eth-CFM for EVPN-VXLAN, and proxy-ARP/proxy-ND confirm messages and Eth-CFM for EVPN-MPLS.
  3. xSTP and M-VPLS services:
    1. xSTP can be configured in bgp-evpn services. BPDUs will not be sent over the EVPN bindings.
    2. bgp-evpn is blocked in m-vpls services; however, a different m-vpls service can manage a SAP or spoke-sdp in a bgp-evpn enabled service.
  4. mac-move—in bgp-evpn enabled VPLS services, mac-move can be used in saps/sdp-bindings; however, the MACs being learned through BGP-EVPN will not be considered.
    Note:

    The mac duplication already provides a protection against mac-moves between EVPN and saps/sdp-bindings.

  5. disable-learning and other fdb-related tools—these will only work for data plane learned mac addresses.
  6. mac-protect—mac-protect cannot be used in conjunction with EVPN.
    Note:

    EVPN provides its own protection mechanism for static mac addresses.

  7. MAC OAM—MAC OAM tools are not supported for bgp-evpn services, that is: mac-ping, mac-trace, mac-populate, mac-purge, and cpe-ping.
  8. EVPN multi-homing and BGP-MH can be enabled in the same VPLS service, as long as they are not enabled in the same SAP-SDP or Spoke-SDP. There is no limitation on the number of BGP-MH sites supported per EVPN-MPLS service.
    Note:

    The number of BGP-MH sites per EVPN-VXLAN service is limited to 1.

  9. SAPs/SDP-bindings that belong to a specified ES but are configured on non-bgp-evpn-mpls-enabled VPLS or Epipe services will be kept down with the StandByForMHProtocol flag.
  10. CPE-ping is not supported on EVPN services but it is in PBB-EVPN services (including I-VPLS and PBB-Epipe). CPE-ping packets will not be sent over EVPN destinations. CPE-ping will only work on local active SAP or SDP-bindings in I-VPLS and PBB-Epipe services.
  11. Other commands not supported in conjunction with bgp-evpn:
    1. bgp-vpls
    2. Endpoints and attributes
    3. Subscriber management commands under service, SAP, and sdp-binding interfaces
    4. MLD-snooping and attributes
    5. BPDU translation
    6. L2PT termination
    7. MAC-pinning
  12. Other commands not supported in conjunction with bgp-evpn mpls are:
    1. VSD-domains
    2. SPB configuration and attributes

5.4.15.2. Interaction of PBB-EVPN with Existing VPLS Features

In addition to the B-VPLS considerations described in section Interaction of EVPN-VXLAN and EVPN-MPLS with Existing VPLS Features, the following specific interactions for PBB-EVPN should also be considered.

  1. When bgp-evpn mpls is enabled in a b-vpls service, an i-vpls service linked to that b-vpls cannot be an R-VPLS (the allow-ip-int-bind command is not supported).
  2. The ISID value of 0 is not allowed for PBB-EVPN services (I-VPLS and Epipes).
  3. The ethernet-segments can be associated with b-vpls SAPs/SDP-bindings and i-vpls/epipe SAPs/SDP-bindings,; however, the same ES cannot be associated with b-vpls and i-vpls/epipe SAP or SDP-bindings at the same time.
  4. When PBB-epipes are used with PBB-EVPN multi-homing, Spoke-SDPs are not supported on ethernet-segments.
  5. When bgp-evpn mpls is enabled, eth-tunnels are not supported in the b-vpls instance.

5.4.15.3. Interaction of EVPN-VXLAN and EVPN-MPLS with Existing VPRN or IES Features

When enabling existing VPRN features on interfaces linked to EVPN-VXLAN R-VPLS or EVPN-MPLS R-VPLS interfaces, consider that the following are not supported:

  1. the commands arp-populate and authentication-policy
  2. dynamic routing protocols such as IS-IS, RIP, and OSPF
  3. BFD on EVPN tunnel interfaces

When enabling existing IES features on interfaces linked to EVPN-VXLAN RVPLS or EVPN-MPLS R-VPLS interfaces, the following are not supported:

  1. the following commands:
    1. if>vpls>evpn-tunnel
    2. bgp-evpn>ip-route-advertisement
    3. arp-populate
    4. authentication-policy
  2. dynamic routing protocols such as IS-IS, RIP, and OSPF

5.4.15.4. Routing Policies for BGP EVPN IP Prefixes

BGP routing policies are supported for IP prefixes imported or exported through BGP-EVPN.

When applying routing policies to control the distribution of prefixes between EVPN and IP-VPN, the user must consider that both families are completely separate as far as BGP is concerned and that when prefixes are imported in the VPRN routing table, the BGP attributes are lost to the other family. The use of route tags allows the controlled distribution of prefixes across the two families.

Figure 201 shows an example of how VPN-IPv4 routes are imported into the RTM (Routing Table Manager), and then passed to EVPN for its own process.

Note:

VPN-IPv4 routes can be tagged at ingress and that tag is preserved throughout the RTM and EVPN processing, so that the tag can be matched at the egress BGP routing policy.

Figure 201:  IP-VPN Import and EVPN Export BGP Workflow 

Policy tags can be used to match EVPN IP prefixes that were learned not only from BGP VPN-IPv4 but also from other routing protocols. The tag range supported for each protocol is different:

<tag>  : accepts in decimal or hex
        [0x1..0xFFFFFFFF]H (for OSPF and IS-IS)
        [0x1..0xFFFF]H (for RIP)
        [0x1..0xFF]H (for BGP)

Figure 202 shows an example of the reverse workflow: routes imported from EVPN and exported from RTM to BGP VPN-IPv4.

Figure 202:  EVPN Import and I-VPN Export BGP Workflow  

The preceding described behavior and the use of tags is also valid for vsi-import and vsi-export policies in the R-VPLS.

The policy behavior for EVPN ip-prefixes can then be summarized in the following statements.

  1. For EVPN prefix routes received and imported in RTM:
    1. Policy entries can match on communities and add tags. This works at the peer level or at the vsi-import level.
    2. Policy entries can match on family evpn.
  2. For exporting RTM to EVPN prefix routes:
    1. Policy entries can match on tags and based on that, add communities, accept, or reject. This works at the peer level or the vsi-export level.

Policy entries can add tags for static routes, RIP, OSPF, IS-IS, BGP, and ARP-ND routes that can then be matched on the BGP peer export policy or VSI export policy for EVPN prefix routes.