5.2.1.1. VXLAN ECMP and LAG

The DC GW supports ECMP load balancing to reach the destination VTEP. Also, any intermediate core node in the Data Center should be able to provide further load balancing across ECMP paths because the source UDP port of each tunneled packet is derived from a hash of the customer inner packet. The following must be considered:

5.2.1.2. VXLAN VPLS Tag Handling

The following describes the behavior on the 7750 SR, 7450 ESS, and 7950 XRS with respect to VLAN tag handling for VXLAN VPLS services:

5.2.1.3. VXLAN MTU Considerations

For VXLAN VPLS services, the network port MTU must be at least 50 Bytes (54 Bytes if dot1q) greater than the Service-MTU to allow enough room for the VXLAN encapsulation.

The Service-MTU is only enforced on SAPs, (any SAP ingress packet with MTU greater than the service-mtu is discarded) and not on VXLAN termination (any VXLAN ingress packet makes it to the egress SAP regardless of the configured service-mtu).

Note: The router never fragments or reassemble VXLAN packets. In addition, the router always sets the DF (Do not Fragment) flag in the VXLAN outer IP header.

5.2.1.4. VXLAN QoS

VXLAN is a network port encapsulation; therefore, the QoS settings for VXLAN are controlled from the network QoS policies.

5.2.1.5. VXLAN Ping

A new VXLAN troubleshooting tool, VXLAN Ping, is available to verify VXLAN VTEP connectivity. The VXLAN Ping command is available from interactive CLI and SNMP.

This tool allows the operator to specify a wide range of variables to influence how the packet is forwarded from the VTEP source to VTEP termination. The ping function requires the operator to specify a different test-id (equates to originator handle) for each active and outstanding test. The required local service identifier from which the test is launched determines the source IP (the system IP address) to use in the outer IP header of the packet. This IP address is encoded into the VXLAN header Source IP TLV. The service identifier also encodes the local VNI. The outer-ip-destination must equal the VTEP termination point on the remote node, and the dest-vni must be a valid VNI within the associated service on the remote node. The outer source IP address is automatically detected and inserted in the IP header of the packet. The outer source IP address uses the IPv4 system address by default.

If the VTEP is created using a non-system source IP address via the vxlan-src-vtep command, the outer source IP address uses the address specified by vxlan-src-vtep. The remainder of the variables are optional.

The VXLAN PDU is encapsulated in the appropriate transport header and forwarded within the overlay to the appropriate VTEP termination. The VXLAN router alert (RA) bit is set to prevent forwarding OAM PDU beyond the terminating VTEP. Since handling of the router alert bit was not defined in some early releases of VXLAN implementations, the VNI Informational bit (I-bit) is set to “0” for OAM packets. This indicates that the VNI is invalid, and the packet should not be forwarded. This safeguard can be overridden by including the i-flag-on option that sets the bit to “1”, valid VNI. Ensure that OAM frames meant to be contained to the VTEP are not forwarded beyond its endpoints.

The supporting VXLAN OAM ping draft includes a requirement to encode a reserved IEEE MAC address as the inner destination value. However, at the time of implementation, that IEEE MAC address had not been assigned. The inner IEEE MAC address defaults to 00:00:00:00:00:00, but may be changed using the inner-l2 option. Inner IEEE MAC addresses that are included with OAM packets are not learned in the local Layer 2 forwarding databases.

The echo responder terminates the VXLAN OAM frame, and takes the appropriate response action, and include relevant return codes. By default, the response is sent back using the IP network as an IPv4 UDP response. The operator can choose to override this default by changing the reply-mode to overlay. The overlay return mode forces the responder to use the VTEP connection representing the source IP and source VTEP. If a return overlay is not available, the echo response is dropped by the responder.

Support is included for:

The VXLAN OAM PDU includes two timestamps. These timestamps are used to report forward direction delay. Unidirectional delay metrics require accurate time of day clock synchronization. Negative unidirectional delay values are reported as “0.000”. The round trip value includes the entire round trip time including the time that the remote peer takes to process that packet. These reported values may not be representative of network delay.

The following example commands and outputs show how the VXLAN Ping function can be used to validate connectivity. The echo output includes a new header to better describe the VXLAN ping packet headers and the various levels.

5.2.1.6. EVPN-VXLAN Routed VPLS Multicast Routing Support

IPv4 and IPv6 multicast routing is supported in an EVPN-VXLAN VPRN and IES routed VPLS service through its IP interface when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side, sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 Multicast Routing Support for more details.

5.2.1.7. IGMP and MLD Snooping on VXLAN

The delivery of IP multicast in VXLAN services can be optimized with IGMP and MLD snooping. IGMP and MLD snooping are supported in EVPN-VXLAN VPLS services and in EVPN-VXLAN VPRN/IES R-VPLS services. When enabled, IGMP and MLD reports are snooped on SAPs or SDP bindings, but also on VXLAN bindings, to create or modify entries in the MFIB for the VPLS service.

When configuring IGMP and MLD snooping in EVPN-VXLAN VPLS services, consider the following:

5.2.1.8. PIM Snooping on VXLAN

PIM snooping for IPv4 and IPv6 are supported in an EVPN-EVPN-VXLAN VPLS or R-VPLS service (with the R-VPLS attached to a VPRN or IES service). The snooping operation is similar to that within a VPLS service (see PIM Snooping for VPLS) and supports both PIM snooping and PIM proxy modes.

PIM snooping for IPv4 is enabled using the config>service>vpls>pim-snooping command.

PIM snooping for IPv6 is enabled using the config>service>vpls>pim-snooping no ipv6-multicast-disable command.

When using PIM snooping for IPv6, the default forwarding is MAC-based with optional support for SG-based (see IPv6 Multicast Forwarding). SG-based forwarding requires FP3- or higher-based hardware.

It is not possible to configure max-num-groups for VXLAN bindings.

5.2.1.9. Static VXLAN Termination in Epipe Services

By default, the system IP address is used to terminate and generate VXLAN traffic. The following configuration example shows an Epipe service that supports static VXLAN termination:

Where:

The following features are not supported by Epipe services with VXLAN destinations.

5.2.1.10. Static VXLAN Termination in VPLS/R-VPLS Services

VXLAN instances in VPLS and R-VPLS can be configured with egress VTEPs. This is referred as static vxlan-instances. The following configuration example shows a VPLS service that supports a static vxlan-instance:

Specifically the following can be stated:

Static VXLAN instances can use non-system IPv4/IPv6 termination.

5.2.1.11. Non-System IPv4 and IPv6 VXLAN Termination in VPLS, R-VPLS, and Epipe Services

By default, only VXLAN packets with the same IP destination address as the system IPv4 address of the router can be terminated and processed for a subsequent MAC lookup. A router can simultaneously terminate VXLAN tunnels destined for its system IP address and three additional non-system IPv4 or IPv6 addresses, which can be on the base router or VPRN instances. This section describes the configuration requirements for services to terminate VXLAN packets destined for a non-system loopback IPv4 or IPv6 address on the base router or VPRN.

Perform the following steps to configure a service with non-system IPv4 or IPv6 VXLAN termination:

FPE Creation

A Forwarding Path Extension (FPE) is required to terminate non-system IPv4 or IPv6 VXLAN tunnels.

In a non-system IPv4 VXLAN termination, the FPE function is used for additional processing required at ingress (VXLAN tunnel termination) only, and not at egress (VXLAN tunnel origination).

If the IPv6 VXLAN terminates on a VPLS or Epipe service, the FPE function is used at ingress only, and not at egress.

For R-VPLS services terminating IPv6 VXLAN tunnels and also for VPRN VTEPs, the FPE is used for the egress as well as the VXLAN termination function. In the case of R-VPLS, an internal static SDP is created to allow the required extra processing.

See the “Forwarding Path Extension” section of the 7450 ESS, 7750 SR, 7950 XRS, and VSR Interface Configuration Guide for information about FPE configuration and functions.

FPE Association with VXLAN Termination

The FPE must be associated with the VXLAN termination application. The following sample configuration shows two FPEs and their corresponding association. FPE 1 uses the base router and FPE 2 is configured for VXLAN termination on VPRN 10.

VXLAN Router Loopback Interface

Create the interface that terminates and originates the VXLAN packets. The interface is created as a router interface, which is added to the Interior Gateway Protocol (IGP) and used by the BGP as the EVPN NLRI next hop.

Because the system cannot terminate the VXLAN on a local interface address, a subnet must be assigned to the loopback interface and not a host IP address that is /32 or /128. In the following example, all the addresses in subnet 11.11.11.0/24 (except 11.11.11.1, which is the interface IP) and subnet 10.1.1.0/24 (except 10.1.1.1) can be used for tunnel termination. The subnet is advertised using the IGP and is configured on either the base router or a VPRN. In the example, two subnets are assigned, in the base router and VPRN 10 respectively.

A local interface address cannot be configured as a VXLAN tunnel-termination IP address in the CLI, as shown in the following example.

The subnet can be up to 31 bits. For example, to use 10.11.11.1 as the VXLAN termination address, the subnet should be configured and advertised as shown in the following sample configuration.

It is not a requirement for the remote PEs and NVEs to have the specific /32 or /128 IP address in their RTM to resolve the BGP EVPN NLRI next hop or forward the VXLAN packets. An RTM with a subnet that contains the remote VTEP can also perform these tasks.

Note: The system does not check for a pre-existing local base router loopback interface with a subnet corresponding to the VXLAN tunnel termination address. If a tunnel termination address is configured and the FPE is operationally up, the system starts terminating VXLAN traffic and responding ICMP messages for that address. The following conditions are ignored in this scenario: the presence of a loopback interface in the base router the presence of an interface with the address contained in the configured subnet, and no loopback

The following sample output includes an IPv6 address in the base router. It could also be configured in a VPRN instance.

VXLAN Termination VTEP Addresses

The service>system>vxlan>tunnel-termination context allows the user to configure non-system IP addresses that can terminate the VXLAN and their corresponding FPEs.

As shown in the following example, an IP address may be associated with a new or existing FPE already terminating the VXLAN. The list of addresses that can terminate the VXLAN can include IPv4 and IPv6 addresses.

The tunnel-termination command creates internal loopback interfaces that can respond to ICMP requests. In the following sample output, an internal loopback is created when the tunnel termination address is added (for 10.11.11.1 and 2001:db8:1000::1). The internal FPE router interfaces created by the VXLAN termination function are also shown in the output. Similar loopback and interfaces are created for tunnel termination addresses in a VPRN (not shown).

VXLAN Services

By default, the VXLAN services use the system IP address as the source VTEP of the VXLAN encapsulated frames. The vxlan-src-vtep command in the config>service>vpls or config>service>epipe context enables the system to use a non-system IPv4 or IPv6 address as the source VTEP for the VXLAN tunnels in that service.

A different vxlan-src-vtep can be used for different services, as shown in the following example where two different services use different non-system IP addresses as source VTEPs.

In addition, if a vxlan-src-vtep is configured and the service uses EVPN, the IP address is also used to set the BGP NLRI next hop in EVPN route advertisements for the service.

Note: The BGP EVPN next hop can be overridden by the use of export policies based on the following rules. A BGP peer policy can override a next hop pushed by the vxlan-src-vtep configuration. If the VPLS service is IPv6 (that is, the vxlan-src-vtep is IPv6) and a BGP peer export policy is configured with next-hop-self, the BGP next-hop is overridden with an IPv6 address auto-derived from the IP address of the system. The auto-derivation is based on RFC 4291. For example, ::ffff:10.20.1.3 is auto-derived from system IP 10.20.1.3. The policy checks the address type of the next hop provided by the vxlan-src-vtep command. If the command provides an IPv6 next hop, the policy is unable use an IPv4 address to override the IPv6 address provided by the vxlan-src-vtep command.

After the preceding steps are performed to configure a VXLAN termination, the VPLS, R-VPLS, or Epipe service can be used normally, except that the service terminates VXLAN tunnels with a non-system IPv4 or IPv6 destination address (in the base router or a VPRN instance) instead of the system IP address only.

The FPE vxlan-termination function creates internal router interfaces and loopbacks that are displayed by the show commands. When configuring IPv6 VXLAN termination on an R-VPLS service, as well as the internal router interfaces and loopbacks, the system creates internal SDP bindings for the required egress processing. The following output shows an example of an internal FPE-type SDP binding created for IPv6 R-VPLS egress processing.

When BGP EVPN is used, the BGP peer over which the EVPN-VXLAN updates are received can be an IPv4 or IPv6 peer, regardless of whether the next-hop is an IPv4 or IPv6 address.

The same VXLAN tunnel termination address cannot be configured on different router instances; that is, on two different VPRN instances or on a VPRN and the base router.

5.2.2.1. BGP-EVPN Control Plane for VXLAN Overlay Tunnels

RFC 8365 describes EVPN as the control plane for overlay-based networks. The 7750 SR, 7450 ESS, and 7950 XRS support all routes and features described in RFC 7432 that are required for the DC GW function. EVPN multihoming and BGP multihoming based on the L2VPN BGP address family are both supported if redundancy is needed.

Figure 145 shows the EVPN MP-BGP NLRI, required attributes and extended communities, and two route types supported for the DC GW Layer 2 applications:

EVPN Route Type 3 – Inclusive Multicast Ethernet Tag Route

Route type 3 is used to set up the flooding tree (BUM flooding) for a specified VPLS service in the data center. The received inclusive multicast routes add entries to the VPLS flood list in the 7750 SR, 7450 ESS, and 7950 XRS. The tunnel types supported in an EVPN route type 3 when BGP-EVPN MPLS is enabled are ingress replication, P2MP MLDP, and composite tunnels.

Ingress Replication (IR) and Assisted Replication (AR) are supported for VXLAN tunnels. See Layer 2 Multicast Optimization for VXLAN (Assisted-Replication) for more information about the AR.

If ingress-repl-inc-mcast-advertisement is enabled, a route type 3 is generated by the router per VPLS service as soon as the service is in an operationally up state. The following fields and values are used:

EVPN Route Type 2 – MAC/IP Advertisement Route

The 7750 SR, 7450 ESS, and 7950 XRS generates this route type for advertising MAC addresses. The router generates MAC advertisement routes for the following:

The route type 2 generated by a router uses the following fields and values:

When EVPN-VXLAN multihoming is enabled, type 1 routes (Auto-Discovery per-ES and per-EVI routes) and type 4 routes (ES routes) are also generated and processed. See BGP-EVPN Control Plane for MPLS Tunnels for more information about route types 1 and 4.

EVPN Route Type 5 – IP Prefix Route

Figure 147 shows the IP prefix route or route-type 5.

Figure 147:  EVPN Route-Type 5  

The router generates this route type for advertising IP prefixes in EVPN. The router generates IP prefix advertisement routes for:

The route-type 5 generated by a router uses the following fields and values:

All the routes in EVPN-VXLAN is sent with the RFC 5512 tunnel encapsulation extended community, with the tunnel type value set to VXLAN.

5.2.2.2. EVPN for VXLAN in VPLS Services

The EVPN-VXLAN service is designed around the current VPLS objects and the additional VXLAN construct.

Figure 138 shows a DC with a Layer 2 service that carries the traffic for a tenant who wants to extend a subnet beyond the DC. The DC PE function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where a VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the Network Virtualization Edge (NVE) devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The VPLS instances in the redundant DC GW and the DC NVEs are connected by VXLAN bindings. BGP-EVPN provides the required control plane for such VXLAN connectivity.

The DC GW routers are configured with a VPLS per tenant that will provide the VXLAN connectivity to the Nuage VPLS instances. On the router, each tenant VPLS instance is configured with:

The bgp-evpn context specifies the encapsulation type (only vxlan is supported) to be used by EVPN and other parameters like the unknown-mac-route and mac-advertisement commands. These commands are typically configured in three different ways:

Other parameters related to EVPN or VXLAN are:

After the VPLS is configured and operationally up, the router will send/receive inclusive multicast Ethernet Tag routes, and a full-mesh of VXLAN connections will be automatically created. These VXLAN “auto-bindings” can be characterized as follows:

After the flooding domain is setup, the routers and DC NVEs start advertising MAC addresses, and the routers can learn MACs and install them in the FDB. Some considerations are the following:

5.2.2.2.3. Use of the unknown-mac-route

This section describes the behavior of the EVPN-VXLAN service in the router when the unknown-mac-route and BGP-MH are configured at the same time.

The use of EVPN, as the control plane of NVO networks in the DC, provides a significant number of benefits as described in IETF Draft draft-ietf-bess-evpn-overlay.

However, there is a potential issue that must be addressed when a VPLS DCI is used for an NVO3-based DC: all the MAC addresses learned from the WAN side of the VPLS must be advertised by BGP EVPN updates. Even if optimized BGP techniques like RT-constraint are used, the number of MAC addresses to advertise or withdraw (in case of failure) from the DC GWs can be difficult to control and overwhelming for the DC network, especially when the NVEs reside in the hypervisors.

The 7750 SR, 7450 ESS, and 7950 XRS solution to this issue is based on the use of an unknown-mac-route address that is advertised by the DC PEs. By using this unknown-mac-route advertisement, the DC tenant may decide to optionally turn off the advertisement of WAN MAC addresses in the DC GW, therefore, reducing the control plane overhead and the size of the FDB tables in the NVEs.

The use of the unknown-mac-route is optional and helps to reduce the amount of unknown-unicast traffic within the data center. All the receiving NVEs supporting this concept will send any unknown-unicast packet to the owner of the unknown-mac-route, as opposed to flooding the unknown-unicast traffic to all other NVEs that are part of the same VPLS.

Note: Although the router can be configured to generate and advertise the unknown-mac-route, the router will never honor the unknown-mac-route and will flood to the TLS-flood list when an unknown-unicast packet arrives at an ingress SAP or SDP binding.

The use of the unknown-mac-route assumes the following:

1. A fully virtualized DC where all the MACs are control-plane learned, and learned previous to any communication (no legacy TORs or VLAN connected servers).
2. The only exception is MACs learned over the SAPs/SDP bindings that are part of the BGP-MH WAN site-id. Only one site-id is supported in this case.
3. No other SAPs/SDP bindings out of the WAN site-id are supported, unless only static MACs are used on those SAPs/SDP bindings.

Therefore, when unknown-mac-route is configured, it will only be generated when one of the following applies:

1. No site is configured and the service is operationally up.
2. A BGP-MH site is configured AND the DC GW is Designated Forwarder (DF) for the site. In case of BGP-MH failover, the unknown-mac-route will be withdrawn by the former DF and advertised by the new DF.

5.2.2.3. EVPN for VXLAN in R-VPLS Services

Figure 139 shows a DC with a Layer 2 service that carries the traffic for a tenant who extends a subnet within the DC, while the DC GW is the default gateway for all the hosts in the subnet. The DC GW function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where an R-VPLS instance exists for that particular tenant. Within the DC, the tenant will have VPLS instances in all the NVE devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The WAN connectivity will be based on existing IP-VPN features.

In this model, the DC GW routers are configured with a R-VPLS (bound to the VPRN that provides the WAN connectivity) per tenant that will provide the VXLAN connectivity to the Nuage VPLS instances. This model provides inter-subnet forwarding for L2-only TORs and other L2 DC NVEs.

On the router:

On the Nuage VSGs and NVEs:

Other considerations:

EVPN-enabled R-VPLS services are also supported on IES interfaces.

5.2.2.3.1. EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes

Figure 140 shows a Layer 3 DC model, where a VPRN is defined in the DC GWs, connecting the tenant to the WAN. That VPRN instance will be connected to the VPRNs in the NVEs by means of an IRB backhaul R-VPLS. Since the IRB backhaul R-VPLS provides connectivity only to all the IRB interfaces and the DC GW VPRN is not directly connected to all the tenant subnets, the WAN ip-prefixes in the VPRN routing table must be advertised in EVPN. In the same way, the NVEs will send IP prefixes in EVPN that will be received by the DC GW and imported in the VPRN routing table.

Note: To generate or process IP prefixes sent or received in EVPN route type 5, support for IP route advertisement must be enabled in BGP-EVPN using the bgp-evpn>ip-route-advertisement command. This command is disabled by default and must be explicitly enabled. The command is tied to the allow-ip-int-bind command required for R-VPLS, and it is not supported on an R-VPLS linked to IES services.

Local router interface host addresses are not advertised in EVPN by default. To advertise them, the ip-route-advertisement incl-host command must be enabled. For example:

===============================================================================

Route Table (Service: 2)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                         Active     Metric

-------------------------------------------------------------------------------

10.1.1.0/24                                   Local   Local     00h00m11s  0

       if                                              Y            0

10.1.1.100/32                                 Local   Host      00h00m11s  0

       if                                              Y            0

==============================================================================

For the case displayed by the output above, the behavior is the following:

1. ip-route-advertisement only local subnet (default) - 10.1.1.0/24 is advertised
2. ip-route-advertisement incl-host local subnet, host - 10.1.1.0/24 and 10.1.1.100/32 are advertised

Below is an example of VPRN (500) with two IRB interfaces connected to backhaul R-VPLS services 501 and 502 where EVPN-VXLAN runs:

vprn 500 customer 1 create            

            ecmp 4

            route-distinguisher 65072:500

            vrf-target target:65000:500

            interface "evi-502" create

                address 10.20.20.72/24

                vpls "evpn-vxlan-502"

                exit

            exit

            interface "evi-501" create

                address 10.10.10.72/24

                vpls "evpn-vxlan-501"

                exit

            exit

            no shutdown

vpls 501 name “evpn-vxlan-501” customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 501 create

            exit

            bgp

                route-distinguisher 65072:501

                route-target export target:65000:501 import target:65000:501

            exit

            bgp-evpn

                ip-route-advertisement incl-host

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit                  

            exit

            no shutdown

        exit

vpls 502 name “evpn-xvlan-502” customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 502 create

            exit

            bgp

                route-distinguisher 65072:502

                route-target export target:65000:502 import target:65000:502

            exit

            bgp-evpn

                ip-route-advertisement incl-host

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit

            exit

            no shutdown

        exit

 

When the above commands are enabled, the router will:

1. Receive route-type 5 routes and import the IP prefixes and associated IP next-hops into the VPRN routing table.
   1. If the route-type 5 is successfully imported by the router, the prefix included in the route-type 5 (for example, 10.0.0.0/24), will be added to the VPRN routing table with a next-hop equal to the GW IP included in the route (for example, 192.0.0.1. that refers to the IRB IP address of the remote VPRN behind which the IP prefix sits).
   2. When the router receives a packet from the WAN to the 10.0.0.0/24 subnet, the IP lookup on the VPRN routing table will yield 192.0.0.1 as the next-hop. That next-hop will be resolved to a MAC in the ARP table and the MAC resolved to a VXLAN tunnel in the FDB table

      Note: IRB MAC and IP addresses are advertised in the IRB backhaul R-VPLS in routes type 2.

2. Generate route-type 5 routes for the IP prefixes in the associated VPRN routing table.
   1. For example, if VPRN-1 is attached to EVPN R-VPLS 1 and EVPN R-VPLS 2, and R-VPLS 2 has bgp-evpn ip-route-advertisement configured, the 7750 SR will advertise the R-VPLS 1 interface subnet in one route-type 5.
3. Routing policies can filter the imported and exported IP prefix routes accordingly.

The VPRN routing table can receive routes from all the supported protocols (BGP-VPN, OSPF, IS-IS, RIP, static routing) as well as from IP prefixes from EVPN, as shown below:

*A:PE72# show router 500 route-table                      

===============================================================================

Route Table (Service: 500)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

10.20.20.0/24                                 Local   Local     01d11h10m  0

       evi-502                                                      0

10.20.20.71/32                                Remote  BGP EVPN  00h02m26s  169

       10.10.10.71                                                  0

10.10.10.0/24                                Remote  Static    00h00m05s  5

       10.10.10.71                                                  1

10.16.0.1/32                                 Remote  BGP EVPN  00h02m26s  169

       10.10.10.71                                                  0

-------------------------------------------------------------------------------

No. of Routes: 4

The following considerations apply:

1. The route Preference for EVPN IP prefixes is 169.
   1. BGP IP-VPN routes have a preference of 170 by default, therefore, if the same route is received from the WAN over BGP-VPRN and from BGP-EVPN, then the EVPN route will be preferred.
2. When the same route-type 5 prefix is received from different GW IPs, ECMP is supported if configured in the VPRN.
3. All routes in the VPRN routing table (as long as they do not point back to the EVPN R-VPLS interface) are advertised via EVPN.

Although the description above is focused on IPv4 interfaces and prefixes, it applies to IPv6 interfaces too. The following considerations are specific to IPv6 VPRN R-VPLS interfaces:

1. IPv4 and IPv6 interfaces can be defined on R-VPLS IP interfaces at the same time (dual-stack).
2. The user may configure specific IPv6 Global Addresses on the VPRN R-VPLS interfaces. If a specific Global IPv6 Address is not configured on the interface, the Link Local Address interface MAC/IP will be advertised in a route type 2 as soon as IPv6 is enabled on the VPRN R-VPLS interface.
3. Routes type 5 for IPv6 prefixes will be advertised using either the configured Global Address or the implicit Link Local Address (if no Global Address is configured).
   If more than one Global Address is configured, normally the first IPv6 address will be used as GW IP. The “first IPv6 address” refers to the first one on the list of IPv6 addresses shown via show>router id>interface interface ipv6 or via SNMP.
   The rest of the addresses will be advertised only in MAC-IP routes (Route Type 2) but not used as GW IP for IPv6 prefix routes.

5.2.2.3.2. EVPN for VXLAN in EVPN Tunnel R-VPLS Services

Figure 141 shows an L3 connectivity model that optimizes the solution described in EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes. Instead of regular IRB backhaul R-VPLS services for the connectivity of all the VPRN IRB interfaces, EVPN tunnels can be configured. The main advantage of using EVPN tunnels is that they don't need the configuration of IP addresses, as regular IRB R-VPLS interfaces do.

In addition to the ip-route-advertisement command, this model requires the configuration of the config>service>vprn>if>vpls <name> evpn-tunnel.

Note: EVPN tunnels can be enabled independently of the ip-route-advertisement command, however, no route-type 5 advertisements will be sent or processed. Neither command, evpn-tunnel and ip-route-advertisement, is supported on R-VPLS services linked to IES interfaces.

The example below shows a VPRN (500) with an EVPN-tunnel R-VPLS (504):

        vprn 500 name "vprn500" customer 1 create

            ecmp 4

            route-distinguisher 65071:500

            vrf-target target:65000:500

            interface "evi-504" create

                vpls "evpn-vxlan-504"

                    evpn-tunnel

                exit

            exit

            no shutdown

        exit

        vpls 504 name "evpn-vxlan-504" customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 504 create

            exit

            bgp

                route-distinguisher 65071:504

                route-target export target:65000:504 import target:65000:504

            exit

            bgp-evpn

                ip-route-advertisement

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit

            exit

            no shutdown

        exit

A specified VPRN supports regular IRB backhaul R-VPLS services as well as EVPN tunnel R-VPLS services.

Note: EVPN tunnel R-VPLS services do not support SAPs or SDP-binds.

The process followed upon receiving a route-type 5 on a regular IRB R-VPLS interface differs from the one for an EVPN-tunnel type:

1. IRB backhaul R-VPLS VPRN interface:
   1. When a route-type 2 that includes an IP prefix is received and it becomes active, the MAC/IP information is added to the FDB and ARP tables. This can be checked with the show>router>arp command and the show>service>id>fdb detail command.
   2. When route -type 5 is received and becomes active for the R-VPLS service, the IP prefix is added to the VPRN routing table, regardless of the existence of a route-type 2 that can resolve the GW IP address. If a packet is received from the WAN side and the IP lookup hits an entry for which the GW IP (IP next-hop) does not have an active ARP entry, the system will use ARP to get a MAC. If ARP is resolved but the MAC is unknown in the FDB table, the system will flood into the TLS multicast list. Routes type 5 can be checked in the routing table with the show>router>route-table command and the show>router>fib command.
2. EVPN tunnel R-VPLS VPRN interface:
   1. When route -type 2 is received and becomes active, the MAC address is added to the FDB (only).
   2. When a route-type 5 is received and active, the IP prefix is added to the VPRN routing table with next-hop equal to EVPN tunnel: GW-MAC.
      For example, ET-d8:45:ff:00:01:35, where the GW-MAC is added from the GW-MAC extended community sent along with the route-type 5.
      If a packet is received from the WAN side, and the IP lookup hits an entry for which the next-hop is a EVPN tunnel: GW-MAC, the system will look up the GW-MAC in the FDB. Usually a route-type 2 with the GW-MAC is previously received so that the GW-MAC can be added to the FDB. If the GW-MAC is not present in the FDB, the packet will be dropped.
   3. IP prefixes with GW-MACs as next-hops are displayed by the show router command, as shown below:

 

*A:PE71# show router 500 route-table 

===============================================================================

Route Table (Service: 500)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

10.20.20.72/32                                Remote  BGP EVPN  00h23m50s  169

       10.10.10.72                                                  0

10.30.30.0/24                                 Remote  BGP EVPN  01d11h30m  169

       evi-504 (ET-d8:45:ff:00:01:35)                               0

10.10.10.0/24                                Remote  BGP VPN   00h20m52s  170

       192.0.2.69 (tunneled)                                        0

10.1.0.0/16                                  Remote  BGP EVPN  00h22m33s  169

       evi-504 (ET-d8:45:ff:00:01:35)                               0

-------------------------------------------------------------------------------

No. of Routes: 4

 

The GW-MAC as well as the rest of the IP prefix BGP attributes are displayed by the show>router>bgp>routes>evpn>ip-prefix command.

*A:Dut-A# show router bgp routes evpn ip-prefix prefix 3.0.1.6/32 detail

===============================================================================

BGP Router ID:10.20.1.1        AS:100         Local AS:100       

===============================================================================

Legend -

Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid

Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

 

===============================================================================

BGP EVPN IP-Prefix Routes

===============================================================================

-------------------------------------------------------------------------------

Original Attributes

 

Network        : N/A

Nexthop        : 10.20.1.2

From           : 10.20.1.2

Res. Nexthop   : 192.168.19.1

Local Pref.    : 100                    Interface Name : NotAvailable

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                  

Connector      : None

Community      : target:100:1 mac-nh:00:00:01:00:01:02

                 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 10.20.1.2

Flags          : Used  Valid  Best  IGP 

Route Source   : Internal              

AS-Path        : No As-Path

EVPN type      : IP-PREFIX             

ESI            : N/A                    Tag            : 1

Gateway Address: 00:00:01:00:01:02     

Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1

MPLS Label    : 262140

Route Tag      : 0xb

Neighbor-AS    : N/A

Orig Validation: N/A                   

Source Class   : 0                      Dest Class     : 0

 

Modified Attributes

 

Network        : N/A                 

Nexthop        : 10.20.1.2

From           : 10.20.1.2

Res. Nexthop   : 192.168.19.1

Local Pref.    : 100                    Interface Name : NotAvailable

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                  

Connector      : None

Community      : target:100:1 mac-nh:00:00:01:00:01:02

                 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 10.20.1.2

Flags          : Used  Valid  Best  IGP 

Route Source   : Internal              

AS-Path        : 111

EVPN type      : IP-PREFIX             

ESI            : N/A                    Tag            : 1

Gateway Address: 00:00:01:00:01:02     

Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1

MPLS Label    : 262140

Route Tag      : 0xb

Neighbor-AS    : 111

Orig Validation: N/A                   

Source Class   : 0                      Dest Class     : 0

 

-------------------------------------------------------------------------------

Routes : 1

===============================================================================

 

EVPN tunneling is also supported on IPv6 VPRN interfaces. When sending IPv6 prefixes from IPv6 interfaces, the GW-MAC in the route type 5 (IP-prefix route) is always zero. If no specific Global Address is configured on the IPv6 interface, the routes type 5 for IPv6 prefixes will always be sent using the Link Local Address as GW-IP. The following example output shows an IPv6 prefix received via BGP EVPN.

*A:PE71# show router 30 route-table ipv6 

 

===============================================================================

IPv6 Route Table (Service: 30)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

2001:db8:1000::/64                            Local   Local     00h01m19s  0

       int-PE-71-CE-1                                               0

2001:db8:2000::1/128                          Remote  BGP EVPN  00h01m20s  169

       fe80::da45:ffff:fe00:6a-"int-evi-301"                        0

-------------------------------------------------------------------------------

No. of Routes: 2

Flags: n = Number of times nexthop is repeated

       B = BGP backup route available

       L = LFA nexthop available

       S = Sticky ECMP requested

===============================================================================

 

*A:PE71# show router bgp routes evpn ipv6-prefix prefix 2001:db8:2000::1/128 hunt 

===============================================================================

 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      

===============================================================================

 Legend -

 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid

                 l - leaked

 Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

 

===============================================================================

BGP EVPN IP-Prefix Routes

===============================================================================

-------------------------------------------------------------------------------

RIB In Entries

-------------------------------------------------------------------------------

Network        : N/A

Nexthop        : 192.0.2.69

From           : 192.0.2.69

Res. Nexthop   : 192.168.19.2

Local Pref.    : 100                    Interface Name : int-71-69

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                   

Connector      : None

Community      : target:64500:301 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 192.0.2.69

Flags          : Used  Valid  Best  IGP  

Route Source   : Internal

AS-Path        : No As-Path

EVPN type      : IP-PREFIX              

ESI            : N/A                    Tag            : 301

Gateway Address: fe80::da45:ffff:fe00:* 

Prefix         : 2001:db8:2000::1/128   Route Dist.    : 192.0.2.69:301

MPLS Label     : 0                      

Route Tag      : 0                      

Neighbor-AS    : N/A

Orig Validation: N/A                    

Source Class   : 0                      Dest Class     : 0

Add Paths Send : Default                

Last Modified  : 00h41m17s              

 

-------------------------------------------------------------------------------

RIB Out Entries

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Routes : 1

=============================================================================== 

5.2.2.4. EVPN-VPWS for VXLAN Tunnels

BGP-EVPN Control Plane for EVPN-VPWS

EVPN-VPWS uses route-type 1 and route-type 4; it does not use route-types 2, 3 or 5. Figure 148 shows the encoding of the required extensions for the Ethernet A-D per-EVI routes. The encoding follows the guidelines described in RFC 8214.

Figure 148:  EVPN VPWS BGP Extensions 

If the advertising PE has an access SAP-SDP or spoke SDP that is not part of an Ethernet Segment (ES), then the PE populates the fields of the AD per-EVI route with the following values:

If the advertising PE has an access SAP-SDP or spoke SDP that is part of an ES, the AD per-EVI route is sent with the information described above, with the following minor differences:

Also, ES and AD per-ES routes are advertised and processed for the Ethernet-Segment, as described in RFC 7432 ESs. The ESI label sent with the AD per-ES route is used by BUM traffic on VPLS services; it is not used for Epipe traffic.

EVPN-VPWS for VXLAN Tunnels in Epipe Services

BGP-EVPN can be enabled in Epipe services with either SAPs or spoke SDPs at the access, as shown in Figure 149.

Figure 149:  EVPN-MPLS VPWS 

EVPN-VPWS is supported in VXLAN networks that also run EVPN-VXLAN in VPLS services. From a control plane perspective, EVPN-VPWS is a simplified point-to-point version of RFC 7432 for E-Line services for the following reasons:

In the following configuration example, Epipe 2 is an EVPN-VPWS service between PE2 and PE4 (as shown in Figure 149).

The following considerations apply for the above example configuration:

EVPN-VPWS Epipes can also be configured with the following characteristics:

Using A/S PW and MC-LAG with EVPN-VPWS Epipes

The use of A/S PW (for access spoke SDP) and MC-LAG (for access SAPs) provides an alternative redundant solution for EVPN-VPWS that do not use the EVPN multi homing procedures described in RFC 8214. Figure 150 shows the use of both mechanisms in a single Epipe.

Figure 150:  A/S PW and MC-LAG Support on EVPN-VPWS 

In Figure 150, an A/S PW connects the CE to PE1 and PE2 (left-hand side of the diagram), and an MC-LAG connects the CE to PE3 and PE4 (right-hand side of the diagram). As EVPN multi homing is not used, there are no AD per-ES routes or ES routes. The redundancy is handled as follows:

EVPN Multi-homing for EVPN-VPWS Services

EVPN multi homing is supported for EVPN-VPWS Epipe services with the following considerations:

The DF election for Epipes that is defined in an all-active multi homing ES is not relevant because all PEs in the ES behave in the same way as follows:

Aliasing is supported for traffic sent to an ES destination. If ECMP is enabled on the ingress PE, per-flow load balancing is performed to all PEs that advertise P=1. The PEs that advertise P=0, are not considered as next hops for an ES destination.

Note: The ingress PE will load balance the traffic if shared queuing or ingress policing is enabled on the access SAPs.

Although DF election is not relevant for Epipes in an all-active multi homing ES, it is essential for the following forwarding and backup functions in a single-active multihoming ES.

Non-system IPv4/IPv6 VXLAN Termination for EVPN-VPWS Services

EVPN-VPWS services support non-system IPv4/IPv6 VXLAN termination. For system configuration information, see Non-System IPv4 and IPv6 VXLAN Termination in VPLS, R-VPLS, and Epipe Services.

EVPN multi-homing is supported when the PEs use non-system IP termination, however some extra configuration steps are needed in this case.

5.2.2.4.1. EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes

Figure 140 shows a Layer 3 DC model, where a VPRN is defined in the DC GWs, connecting the tenant to the WAN. That VPRN instance will be connected to the VPRNs in the NVEs by means of an IRB backhaul R-VPLS. Since the IRB backhaul R-VPLS provides connectivity only to all the IRB interfaces and the DC GW VPRN is not directly connected to all the tenant subnets, the WAN ip-prefixes in the VPRN routing table must be advertised in EVPN. In the same way, the NVEs will send IP prefixes in EVPN that will be received by the DC GW and imported in the VPRN routing table.

Note: To generate or process IP prefixes sent or received in EVPN route type 5, the support for IP route advertisement must be enabled in BGP-EVPN. This is performed through the bgp-evpn>ip-route-advertisement command. This command s disabled by default and must be explicitly enabled. The command is tied to the allow-ip-int-bind command required for R-VPLS, and it is not supported on R-VPLS linked to IES services.

Local router interface host addresses are not advertised in EVPN by default. To advertise them, the ip-route-advertisement incl-host command must be enabled. For example:

===============================================================================

Route Table (Service: 2)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                         Active     Metric

-------------------------------------------------------------------------------

10.1.1.0/24                                   Local   Local     00h00m11s  0

       if                                              Y            0

10.1.1.100/32                                 Local   Host      00h00m11s  0

       if                                              Y            0

==============================================================================

For the case displayed by the output above, the behavior is the following:

1. ip-route-advertisement only local subnet (default) - 10.1.1.0/24 is advertised
2. ip-route-advertisement incl-host local subnet, host - 10.1.1.0/24 and 10.1.1.100/32 are advertised

Below is an example of VPRN (500) with two IRB interfaces connected to backhaul R-VPLS services 501 and 502 where EVPN-VXLAN runs:

vprn 500 customer 1 create            

            ecmp 4

            route-distinguisher 65072:500

            vrf-target target:65000:500

            interface "evi-502" create

                address 10.20.20.72/24

                vpls "evpn-vxlan-502"

                exit

            exit

            interface "evi-501" create

                address 10.10.10.72/24

                vpls "evpn-vxlan-501"

                exit

            exit

            no shutdown

vpls 501 name "evpn-vxlan-501" customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 501 create

            exit

            bgp

                route-distinguisher 65072:501

                route-target export target:65000:501 import target:65000:501

            exit

            bgp-evpn

                ip-route-advertisement incl-host

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit                  

            exit

            no shutdown

        exit

vpls 502 name "evpn-vxlan-502" customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 502 create

            exit

            bgp

                route-distinguisher 65072:502

                route-target export target:65000:502 import target:65000:502

            exit

            bgp-evpn

                ip-route-advertisement incl-host

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit

            exit

            no shutdown

        exit

 

When the above commands are enabled, the router will:

1. Receive route-type 5 routes and import the IP prefixes and associated IP next-hops into the VPRN routing table.
   1. If the route-type 5 is successfully imported by the router, the prefix included in the route-type 5 (for example, 10.0.0.0/24), will be added to the VPRN routing table with a next-hop equal to the GW IP included in the route (for example, 192.0.0.1. that refers to the IRB IP address of the remote VPRN behind which the IP prefix sits).
   2. When the router receives a packet from the WAN to the 10.0.0.0/24 subnet, the IP lookup on the VPRN routing table will yield 192.0.0.1 as the next-hop. That next-hop will be resolved to a MAC in the ARP table and the MAC resolved to a VXLAN tunnel in the FDB table

      Note: IRB MAC and IP addresses are advertised in the IRB backhaul R-VPLS in routes type 2.

2. Generate route-type 5 routes for the IP prefixes in the associated VPRN routing table.
   1. For example, if VPRN-1 is attached to EVPN R-VPLS 1 and EVPN R-VPLS 2, and R-VPLS 2 has bgp-evpn ip-route-advertisement configured, the 7750 SR will advertise the R-VPLS 1 interface subnet in one route-type 5.
3. Routing policies can filter the imported and exported IP prefix routes accordingly.

The VPRN routing table can receive routes from all the supported protocols (BGP-VPN, OSPF, IS-IS, RIP, static routing) as well as from IP prefixes from EVPN, as shown below:

*A:PE72# show router 500 route-table                      

===============================================================================

Route Table (Service: 500)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

10.20.20.0/24                                 Local   Local     01d11h10m  0

       evi-502                                                      0

10.20.20.71/32                                Remote  BGP EVPN  00h02m26s  169

       10.10.10.71                                                  0

10.10.10.0/24                                Remote  Static    00h00m05s  5

       10.10.10.71                                                  1

10.16.0.1/32                                 Remote  BGP EVPN  00h02m26s  169

       10.10.10.71                                                  0

-------------------------------------------------------------------------------

No. of Routes: 4

The following considerations apply:

1. The route Preference for EVPN IP prefixes is 169.
   1. BGP IP-VPN routes have a preference of 170 by default, therefore, if the same route is received from the WAN over BGP-VPRN and from BGP-EVPN, then the EVPN route will be preferred.
2. When the same route-type 5 prefix is received from different GW IPs, ECMP is supported if configured in the VPRN.
3. All routes in the VPRN routing table (as long as they do not point back to the EVPN R-VPLS interface) are advertised via EVPN.

Although the description above is focused on IPv4 interfaces and prefixes, it applies to IPv6 interfaces too. The following considerations are specific to IPv6 VPRN R-VPLS interfaces:

1. IPv4 and IPv6 interfaces can be defined on R-VPLS IP interfaces at the same time (dual-stack).
2. The user may configure specific IPv6 Global Addresses on the VPRN R-VPLS interfaces. If a specific Global IPv6 Address is not configured on the interface, the Link Local Address interface MAC/IP will be advertised in a route type 2 as soon as IPv6 is enabled on the VPRN R-VPLS interface.
3. Routes type 5 for IPv6 prefixes will be advertised using either the configured Global Address or the implicit Link Local Address (if no Global Address is configured).
   If more than one Global Address is configured, normally the first IPv6 address will be used as GW IP. The “first IPv6 address” refers to the first one on the list of IPv6 addresses shown via show router <id> interface <interface> IPv6 or via SNMP.
   The rest of the addresses will be advertised only in MAC-IP routes (Route Type 2) but not used as GW IP for IPv6 prefix routes.

5.2.2.4.2. EVPN for VXLAN in EVPN Tunnel R-VPLS Services

Figure 141 shows an L3 connectivity model that optimizes the solution described in EVPN for VXLAN in IRB Backhaul R-VPLS Services and IP Prefixes. Instead of regular IRB backhaul R-VPLS services for the connectivity of all the VPRN IRB interfaces, EVPN tunnels can be configured. The main advantage of using EVPN tunnels is that they don't need the configuration of IP addresses, as regular IRB R-VPLS interfaces do.

In addition to the ip-route-advertisement command, this model requires the configuration of the config>service>vprn>if>vpls <name> evpn-tunnel.

Note: evpn-tunnel can be enabled independently of ip-route-advertisement, however, no route-type 5 advertisements will be sent or processed in that case. Neither command, evpn-tunnel and ip-route-advertisement, is supported on R-VPLS services linked to IES interfaces.

The example below shows a VPRN (500) with an EVPN-tunnel R-VPLS (504):

        vprn 500 customer 1 create

            ecmp 4

            route-distinguisher 65071:500

            vrf-target target:65000:500

            interface "evi-504" create

                vpls "evpn-vxlan-504"

                    evpn-tunnel

                exit

            exit

            no shutdown

        exit

        vpls 504 name “evpn-vxlan-504” customer 1 create

            allow-ip-int-bind

            vxlan instance 1 vni 504 create

            exit

            bgp

                route-distinguisher 65071:504

                route-target export target:65000:504 import target:65000:504

            exit

            bgp-evpn

                ip-route-advertisement

                vxlan bgp 1 vxlan-instance 1

                    no shutdown

                exit

            exit

            no shutdown

        exit

A specified VPRN supports regular IRB backhaul R-VPLS services as well as EVPN tunnel R-VPLS services.

Note: EVPN tunnel R-VPLS services do not support SAPs or SDP-binds.

The process followed upon receiving a route-type 5 on a regular IRB R-VPLS interface differs from the one for an EVPN-tunnel type:

1. IRB backhaul R-VPLS VPRN interface:
   1. When a route-type 2 that includes an IP prefix is received and it becomes active, the MAC/IP information is added to the FDB and ARP tables. This can be checked with the show>router>arp command and the show>service>id>fdb detail command.
   2. When route-type 5 is received and becomes active for the R-VPLS service, the IP prefix is added to the VPRN routing table, regardless of the existence of a route-type 2 that can resolve the GW IP address. If a packet is received from the WAN side and the IP lookup hits an entry for which the GW IP (IP next-hop) does not have an active ARP entry, the system will use ARP to get a MAC. If ARP is resolved but the MAC is unknown in the FDB table, the system will flood into the TLS multicast list. Routes type 5 can be checked in the routing table with the show>router>route-table command and the show>router>fib command.
2. EVPN tunnel R-VPLS VPRN interface:
   1. When route-type 2 is received and becomes active, the MAC address is added to the FDB (only).
   2. When a route-type 5 is received and active, the IP prefix is added to the VPRN routing table with next-hop equal to EVPN tunnel: GW-MAC.
      For example, ET-d8:45:ff:00:01:35, where the GW-MAC is added from the GW-MAC extended community sent along with the route-type 5.
      If a packet is received from the WAN side, and the IP lookup hits an entry for which the next-hop is a EVPN tunnel: GW-MAC, the system will look up the GW-MAC in the FDB. Usually a route-type 2 with the GW-MAC is previously received so that the GW-MAC can be added to the FDB. If the GW-MAC is not present in the FDB, the packet will be dropped.
   3. IP prefixes with GW-MACs as next-hops are displayed by the show router command, as shown below:

 

*A:PE71# show router 500 route-table 

===============================================================================

Route Table (Service: 500)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

10.20.20.72/32                                Remote  BGP EVPN  00h23m50s  169

       10.10.10.72                                                  0

10.30.30.0/24                                 Remote  BGP EVPN  01d11h30m  169

       evi-504 (ET-d8:45:ff:00:01:35)                               0

10.10.10.0/24                                Remote  BGP VPN   00h20m52s  170

       192.0.2.69 (tunneled)                                        0

10.1.0.0/16                                  Remote  BGP EVPN  00h22m33s  169

       evi-504 (ET-d8:45:ff:00:01:35)                               0

-------------------------------------------------------------------------------

No. of Routes: 4

 

The GW-MAC as well as the rest of the IP prefix BGP attributes are displayed by the show>router>bgp>routes>evpn>ip-prefix command.

*A:Dut-A# show router bgp routes evpn ip-prefix prefix 3.0.1.6/32 detail

===============================================================================

BGP Router ID:10.20.1.1        AS:100         Local AS:100       

===============================================================================

Legend -

Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid

Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

 

===============================================================================

BGP EVPN IP-Prefix Routes

===============================================================================

-------------------------------------------------------------------------------

Original Attributes

 

Network        : N/A

Nexthop        : 10.20.1.2

From           : 10.20.1.2

Res. Nexthop   : 192.168.19.1

Local Pref.    : 100                    Interface Name : NotAvailable

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                  

Connector      : None

Community      : target:100:1 mac-nh:00:00:01:00:01:02

                 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 10.20.1.2

Flags          : Used  Valid  Best  IGP 

Route Source   : Internal              

AS-Path        : No As-Path

EVPN type      : IP-PREFIX             

ESI            : N/A                    Tag            : 1

Gateway Address: 00:00:01:00:01:02     

Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1

MPLS Label    : 262140

Route Tag      : 0xb

Neighbor-AS    : N/A

Orig Validation: N/A                   

Source Class   : 0                      Dest Class     : 0

 

Modified Attributes

 

Network        : N/A                 

Nexthop        : 10.20.1.2

From           : 10.20.1.2

Res. Nexthop   : 192.168.19.1

Local Pref.    : 100                    Interface Name : NotAvailable

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                  

Connector      : None

Community      : target:100:1 mac-nh:00:00:01:00:01:02

                 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 10.20.1.2

Flags          : Used  Valid  Best  IGP 

Route Source   : Internal              

AS-Path        : 111

EVPN type      : IP-PREFIX             

ESI            : N/A                    Tag            : 1

Gateway Address: 00:00:01:00:01:02     

Prefix         : 3.0.1.6/32             Route Dist.    : 10.20.1.2:1

MPLS Label    : 262140

Route Tag      : 0xb

Neighbor-AS    : 111

Orig Validation: N/A                   

Source Class   : 0                      Dest Class     : 0

 

-------------------------------------------------------------------------------

Routes : 1

===============================================================================

 

EVPN tunneling is also supported on IPv6 VPRN interfaces. When sending IPv6 prefixes from IPv6 interfaces, the GW-MAC in the route type 5 (IP-prefix route) is always zero. If no specific Global Address is configured on the IPv6 interface, the routes type 5 for IPv6 prefixes will always be sent using the Link Local Address as GW-IP. The following example output shows an IPv6 prefix received via BGP EVPN.

*A:PE71# show router 30 route-table ipv6 

 

===============================================================================

IPv6 Route Table (Service: 30)

===============================================================================

Dest Prefix[Flags]                            Type    Proto     Age        Pref

      Next Hop[Interface Name]                                    Metric   

-------------------------------------------------------------------------------

2001:db8:1000::/64                            Local   Local     00h01m19s  0

       int-PE-71-CE-1                                               0

2001:db8:2000::1/128                          Remote  BGP EVPN  00h01m20s  169

       fe80::da45:ffff:fe00:6a-"int-evi-301"                        0

-------------------------------------------------------------------------------

No. of Routes: 2

Flags: n = Number of times nexthop is repeated

       B = BGP backup route available

       L = LFA nexthop available

       S = Sticky ECMP requested

===============================================================================

 

*A:PE71# show router bgp routes evpn ipv6-prefix prefix 2001:db8:2000::1/128 hunt 

===============================================================================

 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      

===============================================================================

 Legend -

 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid

                 l - leaked

 Origin codes  : i - IGP, e - EGP, ? - incomplete, > - best, b - backup

 

===============================================================================

BGP EVPN IP-Prefix Routes

===============================================================================

-------------------------------------------------------------------------------

RIB In Entries

-------------------------------------------------------------------------------

Network        : N/A

Nexthop        : 192.0.2.69

From           : 192.0.2.69

Res. Nexthop   : 192.168.19.2

Local Pref.    : 100                    Interface Name : int-71-69

Aggregator AS  : None                   Aggregator     : None

Atomic Aggr.   : Not Atomic             MED            : 0

AIGP Metric    : None                   

Connector      : None

Community      : target:64500:301 bgp-tunnel-encap:VXLAN

Cluster        : No Cluster Members

Originator Id  : None                   Peer Router Id : 192.0.2.69

Flags          : Used  Valid  Best  IGP  

Route Source   : Internal

AS-Path        : No As-Path

EVPN type      : IP-PREFIX              

ESI            : N/A                    Tag            : 301

Gateway Address: fe80::da45:ffff:fe00:* 

Prefix         : 2001:db8:2000::1/128   Route Dist.    : 192.0.2.69:301

MPLS Label     : 0                      

Route Tag      : 0                      

Neighbor-AS    : N/A

Orig Validation: N/A                    

Source Class   : 0                      Dest Class     : 0

Add Paths Send : Default                

Last Modified  : 00h41m17s              

 

-------------------------------------------------------------------------------

RIB Out Entries

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Routes : 1

=============================================================================== 

5.2.3.1. XMPP Interface on the DC GW

The Extensible Messaging and Presence Protocol (XMPP) is an open technology for real-time communication using XML (Extensible Markup Language) as the base format for exchanging information. The XMPP provides a way to send small pieces of XML from one entity to another in close to real time.

In a Nuage DC, an XMPP ejabberd server will have an interface to the Nuage VSD as well as the Nuage VSC/VSG and the 7750 SR, 7450 ESS, or 7950 XRS DC GW.

Figure 152 shows the basic XMPP architecture in the data center. While a single XMPP server is represented in the diagram, XMPP allows for easy server clustering and performs message replication to the cluster. It is similar to how BGP can scale and replicate the messages through the use of route reflectors.

Also the VSD is represented as a single server, but a cluster of VSD servers (using the same data base) will be a very common configuration in a DC.

Figure 152:  Basic XMPP Architecture 

In the Nuage solution, each XMPP client, including the 7750 SR, 7450 ESS, and 7950 XRS, is referred to with a JID (JabberID) in the following format: username@xmppserver.domain. The xmppserver.domain points to the XMPP Server.

To enable the XMPP interface on the 7750 SR, 7450 ESS, or 7950 XRS, the following command must be added to indicate to which XMPP server address the DC GW has to register, as well as the router’s JID:

Where:

Note: The DNS must be configured on the router so that the XMPP server name can be resolved. XMPP relies on the Domain Name System (DNS) to provide the underlying structure for addressing, instead of using raw IP addresses. The DNS is configured using the following BOF commands: bof primary-dns, bof secondary-dns, bof dns-domain.

After the XMPP server is properly configured, the router can generate or receive XMPP stanza elements, such as presence and IQ (Information/Query) messages. IQ messages are used between the VSD and the router to request and receive configuration parameters. The status of the XMPP communication channel can be checked with the following command:

In addition to the XMPP server, the router must be configured with a VSD system-id that uniquely identifies the router in the VSD:

After the above configuration is complete, the router will subscribe to a VSD XMPP PubSub node to discover the available VSD servers. Then, the router will be discovered in the VSD UIs. On the router, the available VSD servers can be shown with the following command.

5.2.3.2. Overview of the Static-Dynamic VSD Integration Model

In the Static-Dynamic integration model, the DC and DC GW management entities can be the same or different. The DC GW operator will provision the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN. VSD will only push the required parameters so that those WAN services can be attached to the existing DC domains.

Figure 153 shows the workflow for the attachment of the WAN services defined on the DC GW to the DC domains.

Figure 153:  WAN Services Attachment Workflow 

The Static-Dynamic VSD integration model can be summarized in the steps shown in Figure 153 and described in the following procedure.

5.2.3.3. VSD-Domains and Association to Static-Dynamic Services

In the Static-Dynamic integration model, VSD can only provision certain parameters in VPLS and/or VPRN services. When VSD and the DC GW exchange XMPP messages for a specified service, they use vsd-domains to identify those services. A vsd-domain is a tag that will be used by the 7750 SR, 7450 ESS, or 7950 XRS router and the VSD to correlate a configuration to a service. When redundant DC GWs are available, the vsd-domain for the same service can have the same or a different name in the two redundant DC GWs.

There are four different types of vsd-domains that can be configured in the router:

The domains will be configured in the config>service# context and assigned to each service.

5.2.3.4. Fully-Dynamic VSD Integration Model

In the Static-Dynamic VSD integration model, the VPLS/VPRN service, as well as most of the parameters, must be provisioned “statically” through usual procedures (CLI, SNMP, and so on). VSD will “dynamically” send the parameters that are required for the attachment of the VPLS/VPRN service to the L2/L3 domain in the Data Center. In the Fully-Dynamic VSD integration model, the entire VPLS/VPRN service configuration will be dynamically driven from VSD and no static configuration is required. Through the existing XMPP interface, the VSD will provide the 7750 SR, 7450 ESS, and 7950 XRS routers with a handful of parameters that will be translated into a service configuration by a python-script. This python-script provides an intermediate abstraction layer between VSD and the router, translating the VSD data model into a 7750 SR, 7450 ESS, or 7950 XRS CLI data model.

In this Fully-Dynamic integration model, the DC and DC GW management entities are usually the same. The DC GW operator will provision the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN and the DC. VSD will push the required parameters so that the router can create the service completely and get it attached to an existing DC domain.

The workflow of the Fully-Dynamic integration model is shown in Figure 154.

Figure 154:  Fully-Dynamic VSD Integration Model Workflow 

The Fully-Dynamic VSD integration model can be summarized in the steps shown in Figure 154 and described in the following procedure.

5.2.3.4.1. Python Script Implementation Details

A python-script provides an intermediate abstraction layer between VSD and the router, translating the VSD data model into the 7750 SR, 7450 ESS, or 7950 XRS router CLI data model. VSD will use metadata key=value parameters to provision service-specific attributes on the 7750. The XMPP messages get to the 7750 and are passed transparently to the Python module so that the CLI is generated and the service created.

Note: The CLI generated by the python-script is not saved in the configuration file; it can be displayed by running the info include-dynamic command on the service contexts. The configuration generated by the python-script is protected and cannot be changed by a CLI user.

The following example shows how the python-script works for a VSD generated configuration:

1. The following configuration is added to the 7750 SR. In this case, py-l2 is the python-policy received from VSD that will call the l2domain_service.py python script:

*A:PE1>config>python# info 

----------------------------------------------

        python-script "l2-domain_services" create

            primary-url "ftp://1.1.1.1/cf2/l2domain_service.py"

            no shutdown

        exit

        python-policy "py-l2" create

            description "Python script to create L2 domains"

            vsd script "l2-domain_services"

        exit

----------------------------------------------

*A:PE1>config>service>vsd# info 

----------------------------------------------

  service-range 64000 to 65000

----------------------------------------------

1. VSD will send metadata containing the service parameters. This opaque parameter string will be passed to the python script and is composed of tag=value pairs, with the following format:
2. In addition, other information provided by the VSD, (domain, vni, rt-i, rt-e, and service type) is bundled with the metadata string and passed to the python script. For example:

{'rt': 'target:64000:64000', 'rte': 'target:64000:64000', 'domain': 'L2-DOMAIN-

1', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1, sap=1/1/

10:3000'}

The user should consider the following:

1. The python script is solely responsible for generating the configuration; no configuration aspects of the Static-Dynamic model are used. The python script must be written in a manner similar to those used by RADIUS Dynamic Business Services. Currently, RADIUS Dynamic Business Services and the Fully-Dynamic VSD model are mutually exclusive, one or the other can operate on the same system, but not both at the same time.
2. The following scripts must be defined in order to set up, modify, revert, and tear down the configuration for a service: setup_script(), modify_script(), revert_script(), and teardown_script(). These names must always be the same in all scripts. The revert_script() is only required if the modify_script() is defined, the latter being optional.
3. When the configuration for a new domain name is received from the VSD, the metadata and the VSD parameters are concatenated into a single dictionary and setup_script() is called. Within the python script:
   1. The VSD UI parameters are referenced as vsdParams['rt'], vsdParams ['domain'], and so on.
   2. The metadata parameters are referenced as vsdParams['metadata'].
4. When the startup script is executed, the config>service>vsd>domain is created outside the script context before running the actual script. The teardown script will remove the vsd domain.
   1. If a specified setup_script() fails, the teardown_script() is invoked.
5. When subsequent configuration messages are received from the VSD, the new parameter list is generated again from the VSD message and compared to the last parameter list that was successfully executed.
   1. If the two strings are identical, no action is taken.
   2. If there is a difference between the strings, the modify_script() function is called.
   3. If the modify_script() fails, the revert_script() is invoked. The teardown_script() is invoked if the revert_script() fails or does not exist.
6. The python-policy is always present in the attributes received from VSD; if the VSD user does not include a policy name, VSD will include 'default' as the python-policy. Hence, care must be taken to ensure that the 'default' policy is always configured in the 7750.
7. If the scripts are incorrect, teardown and modify procedures could leave orphaned objects. An admin mode (enable-vsd-config) is available to enable an administrator to clean up these objects; it is strictly meant for cleaning orphaned objects only.

   Note: The CLI configured services cannot be modified when the user is in enable-vsd-config mode.

8. Unless the CLI user enters the enable-vsd-config mode, changes of the dynamic created services are not allowed in the CLI. However, changes through SNMP are allowed.
9. The command tools perform service vsd evaluate-script is introduced to allow the user to test their setup and to modify and tear down python scripts in a lab environment without the need to be connected to a VSD. The successful execution of the command for action setup will create a vsd domain and the corresponding configuration, just as the system would do when the parameters are received from VSD.

The following example shows the use of the tools perform service vsd evaluate-script command:

*A:PE1# tools perform service vsd evaluate-script domain-name "L2-DOMAIN-5" type l2-

domain action setup policy "py-l2" vni 64000 rt-i target:64000:64000 rt-

e target:64000:64000 metadata "rd=1:1, sap=1/1/10:3000"   

 

Success

 

The following example output shows a python-script that can set up or tear down L2-DOMAINs.

*A:PE1# show python python-script "l2-domain_services" source-in-use 

 

===============================================================================

Python script "l2-domain_services"

===============================================================================

Admin state   : inService

Oper state    : inService

Primary URL   : ftp://1.1.1.1/timos86/cses-V71/cf2/l2domain_service.py

Secondary URL : (Not Specified)

Tertiary URL  : (Not Specified)

Active URL    : primary

 

-------------------------------------------------------------------------------

Source (dumped from memory)

-------------------------------------------------------------------------------

      1 from alc import dyn

      2 

      3 def setup_script(vsdParams):

      4     

      5     print ("These are the VSD params: " + str(vsdParams))

      6     servicetype = vsdParams.get('servicetype')

      7     vni = vsdParams.get('vni')

      8     metadata = vsdParams['metadata']

      9     print ("VSD metadata: " + str(metadata))

     10     metadata = dict(e.split('=') for e in metadata.split(','))

     11     print ("Modified metadata: " + str(metadata))

     12     vplsSvc_id = dyn.select_free_id("service-id")

     13     print ("this is the free svc id picked up by the system: " + vplsSvc_id)

     14      

     15     if servicetype == "L2DOMAIN":

     16         rd = metadata['rd']

     17         sap_id = metadata[' sap']

     18         print ('servicetype, VPLS id, rd, sap:', servicetype, vplsSvc_id, rd

, sap_id)

     19         dyn.add_cli("""

     20           configure service

     21               vpls %(vplsSvc_id)s name evi%(vplsSvc_id)s customer 1 create

     22                   description vpls%(vplsSvc_id)s

     23                   bgp

     24                       route-distinguisher %(rd)s

     25                       route-target %(rt)s

     26                   exit

     27                   vxlan instance 1 vni %(vni)s create

     28                   exit

     29                   bgp-evpn 

     30                       evi %(vplsSvc_id)s

     31                       vxlan bgp 1 vxlan-instance 1

     32                           no shutdown

     33                       exit

     34                   exit

     35                   sap %(sap_id)s create

     36                   exit

     37                   no shutdown

     38                   exit

     39               exit

     40           exit

     41         """ % {'vplsSvc_id' : vplsSvc_id, 'vni' : vsdParams['vni'], 'rt' : v

sdParams['rt'], 'rd' : metadata['rd'], 'sap_id' : sap_id})

     42         # L2DOMAIN returns setupParams: vplsSvc_id, servicetype, vni, sap

     43         return {'vplsSvc_id' : vplsSvc_id, 'servicetype' : servicetype, 'vni

' : vni, 'sap_id' : sap_id}

     44 

     45 

     46 #---------------------------------------------------------------------------

     47 

     48 def teardown_script(setupParams):

     49     print ("These are the teardown_script setupParams: " + str(setupParams))

     50     servicetype = setupParams.get('servicetype')   

     51     if servicetype == "L2DOMAIN":

     52       dyn.add_cli("""

     53         configure service

     54             vpls %(vplsSvc_id)s

     55                 no description 

     56                 bgp-evpn 

     57                     vxlan 

     58                         shutdown

     59                     exit

     60                     no evi

     61                     exit

     62                 no vxlan instance 1 vni %(vni)s

     63                 bgp

     64                    no route-distinguisher 

     65                    no route-target 

     66                 exit

     67                 no bgp

     68                 no bgp-evpn

     69                 sap %(sap_id)s

     70                    shutdown

     71                    exit

     72                 no sap %(sap_id)s

     73                 shutdown

     74                 exit

     75                 no vpls %(vplsSvc_id)s

     76             exit

     77         exit                  

     78       """ % {'vplsSvc_id' : setupParams['vplsSvc_id'], 'vni' : setupParams['

vni'], 'sap_id' : setupParams['sap_id']})

     79       return setupParams

     80 

     81 

     82 d = { "script" : (setup_script, None, None, teardown_script) }

     83 

     84 dyn.action(d)

===============================================================================

For instance, assuming that the VSD sends the following:

{'rt': 'target:64000:64000', 'rte': 'target:64000:64000', 'domain': 'L2-DOMAIN-

5', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1, sap=1/1/

10:3000 '}

The system will execute the setup script as follows:

123 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup

"dyn-script req=setup: L2-DOMAIN-5

  state=init->waiting-for-setup

"

 

124 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup

"dyn-script req=setup: L2-DOMAIN-5

  state=waiting-for-setup->generating-setup

"

 

125 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base Python Output

"Python Output: l2-domain_services

These are the VSD params: {'rt': 'target:64000:64000', 'rte': 'target:64000:6400

0', 'domain': '', 'servicetype': 'L2DOMAIN', 'vni': '64000', 'metadata': 'rd=1:1

, sap=1/1/10:3000 '}

VSD metadata: rd=1:1, sap=1/1/10:3000 

Modified metadata: {'rd': '1:1', ' sap': '1/1/10:3000 '}

this is the free svc id picked up by the system: 64000

('servicetype, VPLS id, rd, sap:', 'L2DOMAIN', '64000', '1:1', '1/1/10:3000 ')

"

Success

 

126 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base Python Result

"Python Result: l2-domain_services

"

 

127 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script req=setup

"dyn-script req=setup: L2-DOMAIN-5

  state=generating-setup->executing-setup

"

 

128 2015/06/16 23:35:40.22 UTC MINOR: DEBUG #2001 Base dyn-script cli 1/1

"dyn-script cli 1/1: script:L2-DOMAIN-5(cli 698 dict 0->126)

 

          configure service

              vpls 64000 name “evi64000” customer 1 create

                  description vpls64000

                  bgp

                      route-distinguisher 1:1

                      route-target target:64000:64000

                  exit

                  vxlan instance 1 vni 64000 create

                  exit

                  bgp-evpn 

                      evi 64000

                      vxlan bgp 1 vxlan-instance 1

                          no shutdown

                      exit

                  exit

                  sap 1/1/10:3000  create

                  exit

                  no shutdown

                  exit

              exit

          exit

        "

 

143 2015/06/16 23:35:40.23 UTC MINOR: DEBUG #2001 Base dyn-script req=setup

"dyn-script req=setup: L2-DOMAIN-5

  state=executing-setup->established"

5.2.4.1. Replicator (AR-R) Procedures

An AR-R configuration is shown in the following example.

In this example configuration, the BGP advertises a new inclusive multicast route with tunnel-type = AR, type (T) = AR-R, and tunnel-id = originating-ip = next-hop = assisted-replication-ip (IP address 10.2.2.2 in the preceding example). In addition to the AR route, the AR-R sends a regular IR route if ingress-repl-inc-mcast-advertisement is enabled.

Note: You should disable the ingress-repl-inc-mcast-advertisement command if the AR-R does not have any SAP or SDP bindings and is used solely for Assisted-Replication functions.

The AR-R builds a flooding list composed of ACs (SAPs and SDP bindings) and VXLAN tunnels to remote nodes in the VPLS. All objects in the flooding list are broadcast/multicast (BM) and unknown unicast (U) capable. The following sample output of the show service id vxlan command shows that the VXLAN destinations in the flooding list are tagged as “BUM”.

When the AR-R receives a BUM packet on an AC, the AR-R forwards the packet to its flooding list (including the local ACs and remote VTEPs).

When the AR-R receives a BM packet on a VXLAN tunnel, it checks the IP DA of the underlay IP header and performs the following BM packet processing.

Note: The AR-R function is only relevant to BM packets; it does not apply to unknown unicast packets. If the AR-R receives unknown unicast packets, it sends them to the flooding list, skipping the VXLAN tunnels.

5.2.4.2. Leaf (AR-L) procedures

An AR-L is configured as shown in the following example.

In this example configuration, the BGP advertises a new inclusive multicast route with a tunnel-type = IR, type (T) = AR-L and tunnel-id = originating-ip = next-hop = IR-IP (IP address terminating VXLAN normally, either system-ip or vxlan-src-vtep address).

The AR-L builds a single flooding list per service but controlled by the BM and U flags. These flags are displayed in the following show service id vxlan command sample output.

The AR-L creates the following VXLAN destinations when it receives and selects a Replicator-AR route or the Regular-IR routes.

The BM traffic is only sent to the selected AR-R, whereas the U (unknown unicast) traffic is sent to all the destinations with the U flag.

The AR-L performs per-service load-balancing of the BM traffic when two or more AR-Rs exist in the same service. The AR Leaf creates a list of candidate PEs for each AR-R (ordered by IP and VNI; candidate 0 being the lowest IP and VNI). The replicator is selected out of a modulo function of the service-id and the number of replicators, as shown in the following sample output.

A change in the number of Replicator-AR routes (for example, if a route is withdrawn or a new route appears) affects the result of the hashing, which may cause a different AR-R to be selected.

Note: An AR-L waits for the configured replicator-activation-time before sending the BM packets to the AR-R. In the interim, the AR-L uses regular ingress replication procedures. This activation time allows the AR-R to program the Leaf VTEP. If the timer is zero, the AR-R may receive packets from a not-yet-programmed source VTEP, in which case it will discard the packets.

The following list summarizes other aspects of the AR-L behavior.

Figure 155 shows the expected replication behavior for BM traffic when received at the access on an AR-R, AR-L, or RNVE router. Unknown unicast follows regular ingress replication behavior regardless of the role of the ingress node for the specific service.

Figure 155:  AR BM Replication Behavior for a BM Packet 

5.2.4.3. Assisted-Replication Interaction with Other VPLS Features

The Assisted-Replication feature has the following limitations.

5.2.5.1. Policy Based Forwarding in VPLS Services for Nuage Service Chaining Integration in L2-Domains

Figure 156 shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on VPLS services. In this example, the DC GW, PE1, is connected to an L2-DOMAIN that exists in the DC and must redirect the traffic to the Service Function SF-1. The regular Layer 2 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

Figure 156:  PBF to ESI Function 

An operator must configure a PBF match/action filter policy entry in an IPv4 or MAC ingress access or network filter deployed on a VPLS interface using CLI/SNMP/NETCONF management interfaces. The PBF target is the first service function in the chain (SF-1) that is identified by an ESI.

In the example shown in Figure 156, the PBF filter will redirect the matching packets to ESI 0x01 in VPLS-1.

Note: Figure 156 represents ESI as “0x01” for simplicity; in reality, the ESI is a 10-byte number.

As soon as the redirection target is configured and associated with the vport connected to SF-1, the Nuage VSC (Virtual Services Controller, or the remote PE3 in the example) advertises the location of SF-1 via an Auto-Discovery Ethernet Tag route (route type 1) per-EVI. In this AD route, the ESI associated with SF-1 (ESI 0x01) is advertised along with the VTEP (PE3's IP) and VNI (VNI-1) identifying the vport where SF-1 is connected. PE1 will send all the frames matching the ingress filter to PE3's VTEP and VNI-1.

Note: When packets get to PE3, VNI-1 (the VNI advertised in the AD route) will indicate that a cut-through switching operation is needed to deliver the packets straight to the SF-1 vport, without the need for a regular MAC lookup.

The following filter configuration shows an example of PBF rule redirecting all the frames to an ESI.

When the filter is properly applied to the VPLS service (VPLS-301 in this example), it will show 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported.

Details of the received AD route that resolves the filter forwarding are shown in the following 'show router bgp routes' command.

This AD route, when used for PBF redirection, is added to the list of EVPN-VXLAN bindings for the VPLS service and shown as 'L2 PBR' type:

If the AD route is withdrawn, the binding will disappear and the filter will be inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

Note: ES-based PBF filters can be applied only on services with the default bgp (vxlan) instance (instance 1).

5.2.5.2. Policy Based Routing in VPRN Services for Nuage Service Chaining Integration in L2-DOMAIN-IRB Domains

Figure 157 shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on L2-DOMAIN-IRB domains. In this example, the DC GW, PE1, is connected to an L2-DOMAIN-IRB that exists in the DC and must redirect the traffic to the Service Function SF-1 with IP address 10.10.10.1. The regular Layer 3 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

Figure 157:  PBR to ESI Function 

In this case, an operator must configure a PBR match/action filter policy entry in an IPv4 ingress access or network filter deployed on IES/VPRN interface using CLI, SNMP or NETCONF management interfaces. The PBR target identifies first service function in the chain (ESI 0x01 in Figure 157, identifying where the Service Function is connected and the IPv4 address of the SF) and EVPN VXLAN egress interface on the PE (VPRN routing instance and R-VPLS interface name). The BGP control plane together with ESI PBR configuration are used to forward the matching packets to the next-hop in the EVPN-VXLAN data center chain (through resolution to a VNI and VTEP). If the BGP control plane information is not available, the packets matching the ESI PBR entry will be, by default, forwarded using regular routing. Optionally, an operator can select to drop the packets when the ESI PBR target is not reachable.

The following filter configuration shows an example of a PBR rule redirecting all the matching packets to an ESI.

In this use case, the following are required in addition to the ESI: the sf-ip (10.10.10.1 in the example above), router instance (300), and vas-interface.

The sf-ip is used by the system to know which inner MAC DA it has to use when sending the redirected packets to the SF. The SF-IP will be resolved to the SF MAC following regular ARP procedures in EVPN-VXLAN.

The router instance may be the same as the one where the ingress filter is configured or may be different: for instance, the ingress PBR filter can be applied on an IES interface pointing at a VPRN router instances that is connected to the DC fabric.

The vas-interface refers to the R-VPLS interface name through which the SF can be found. The VPRN instance may have more than one R-VPLS interface, therefore, it is required to specify which R-VPLS interface to use.

When the filter is properly applied to the VPRN or IES service (VPRN-300 in this example), it will show 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported and the SF-IP resolved to a MAC address.

In the FDB for the R-VPLS 301, the MAC address is associated with the VTEP and VNI specified by the AD route, and not by the MAC/IP route anymore. When a PBR filter with a forward action to an ESI and SF-IP (Service Function IP) exists, a MAC route is auto-created by the system and this route has higher priority that the remote MAC, or IP routes for the MAC (see BGP and EVPN Route Selection for EVPN Routes).

The following shows that the AD route creates a new EVPN-VXLAN binding and the MAC address associated with the SF-IP uses that 'binding':

For Layer 2, if the AD route is withdrawn or the SF-IP ARP not resolved, the filter will be inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

5.2.6.1. Local Bias for EVPN VXLAN Multihoming

EVPN MPLS, as described in EVPN for MPLS Tunnels, uses ESI-labels to identify the BUM traffic sourced from a given ES. The egress PE performs a label lookup to find the ESI label below the EVI label and to determine if a frame can be forwarded to a local ES. Because VXLAN does not support ESI-labels, or any MPLS label for that matter, the split-horizon filtering must be based on the tunnel source IP address. This also implies that the SAP-to-SAP forwarding rules must be changed when the SAPs belong to local ESs, irrespective of the DF state. This new forwarding is what RFC 8365 refers to as local bias. Figure 159 illustrates the local bias forwarding behavior.

Figure 159:  EVPN-VXLAN Multihoming with Local Bias 

Local bias is based on the following principles.

With this approach, the ingress PE must perform replication locally to all directly-attached ESs (regardless of the DF Election state) for all flooded traffic coming from the access interfaces. BUM frames received on any SAP are flooded to:

As an example, in Figure 159, PE2 receives BUM traffic from Host-3 and it forwards it to the remote PEs and the local ES SAP, even though the SAP is in NDF state.

The following rules apply to egress PE forwarding for EVPN-VXLAN services.

For example, in Figure 159, PE3 receives BUM traffic on VXLAN. PE3 identifies the source VTEP as a PE with which two ESs are shared, hence it does not forward the BUM frames to the two shared ESs. It forwards to the non-shared ES (Host-5) because it is in DF state. PE4 receives BUM traffic and forwards it based on normal rules because it does not share any ESs with PE2.

The following command can be used to check whether the local PE has enabled the local bias procedures for a given ES:

5.2.6.2. Known Limitations for Local Bias

In EVPN MPLS networks, an ingress PE that uses ingress replication to flood unknown unicast traffic pushes a BUM MPLS label that is different from a unicast label. The egress PEs use this BUM label to identify such BUM traffic and thus apply DF filtering for All-Active multi-homed sites. In PBB-EVPN, in addition to the multicast label, the egress PE can also rely on the multicast B-MAC DA to identify customer BUM traffic.

In VXLAN there are no BUM labels or any tunnel indication that can assist the egress PE in identifying the BUM traffic. As such, the egress PE must solely rely on the C-MAC destination address, which may create some transient issues that are depicted in Figure 160.

Figure 160:  EVPN-VXLAN Multihoming and Unknown Unicast Issues 

As shown in Figure 160, top diagram, in absence of the mentioned unknown unicast traffic indication there can be transient duplicate traffic to All-Active multi-homed sites under the following condition: CE1’s MAC address is learned by the egress PEs (PE1 and PE2) and advertised to the ingress PE3; however, the MAC advertisement has not been received or processed by the ingress PE, resulting in the host MAC address to be unknown on the ingress PE3 but known on the egress PEs. Therefore, when a packet destined to CE1 address arrives on PE3, it floods it via ingress replication to PE1 or PE2 and, because CE1’s MAC is known to PE1 and PE2, multiple copies are sent to CE1.

Another issue is shown at the bottom of Figure 160. In this case, CE1’s MAC address is known on the ingress PE3 but unknown on PE1 and PE2. If PE3’s aliasing hashing picks up the path to the ES’ NDF, a black-hole occurs.

The above two issues are solved in MPLS, as unicast known and unknown frames are identified with different labels.

Finally, another issue is outlined in Figure 161. Under normal circumstances, when CE3 sends BUM traffic to PE3, the traffic is “local-biased” to PE3’s SAP3 even though it is NDF for the ES. The flooded traffic to PE2 is forwarded to CE2, but not to SAP2 because the local bias split-horizon filtering takes place.

Figure 161:  Blackhole Created by a Remote SAP Shutdown 

The right side of the diagram in Figure 161 shows an issue when SAP3 is manually shutdown. In this case, PE3 withdraws the AD per-EVI route corresponding to SAP3; however, this does not change the local bias filtering for SAP2 in PE2. Therefore, when CE3 sends BUM traffic, it can neither be forwarded to CE23 via local SAP3 nor can it be forwarded by PE2.

5.2.6.3. Non-system IPv4 and IPv6 VXLAN Termination for EVPN VXLAN Multihoming

EVPN VXLAN multi-homing is supported on VPLS and R-VPLS services when the PEs use non-system IPv4 or IPv6 termination, however, as with EVPN VPWS services, additional configuration steps are required.

5.3.2.1. EVPN and VPLS Integration

The 7750 SR, 7450 ESS, or 7950 XRS router SR OS EVPN implementation supports RFC 8560 so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service. Since EVPN will not be deployed in green-field networks, this feature is useful for the integration between both technologies and even for the migration of VPLS services to EVPN-MPLS.

The following behavior enables the integration of EVPN and SDP bindings in the same VPLS network:

a) Systems with EVPN endpoints and SDP bindings to the same far-end bring down the SDP bindings.

b) The user can add spoke SDPs and all the EVPN-MPLS endpoints in the same split horizon group (SHG).

c) The system disables the advertisement of MACs learned on spoke SDPs and SAPs that are part of an EVPN split horizon group.

The operation in services with BGP-VPLS and BGP-EVPN is equivalent to the one outlined above for BGP-AD and BGP-EVPN.

5.3.2.2. EVPN Single-Active Multi-Homing and BGP-VPLS Integration

In a VPLS service to which multiple EVPN PEs and BGP-VPLS PEs are attached, single-active multi-homing is supported on two or more of the EVPN PEs with no special considerations. All-active multi-homing is not supported, since the traffic from the all-active multi-homed CE could cause a MAC flip-flop effect on remote BGP-VPLS PEs, asymmetric flows, or other issues.

Figure 165 illustrates a scenario with a single-active ethernet-segment used in a service where EVPN PEs and BGP-VPLS are integrated.

Figure 165:  BGP-VPLS to EVPN Integration and Single-Active MH 

Although other single-active examples are supported, in Figure 165, CE1 is connected to the EVPN PEs via a single LAG (lag-1). The LAG is associated to ethernet-segment 1 on PE1 and PE2, which is configured as single-active and with oper-group 1. PE1 and PE2 make use of lag>monitor-oper-group 1 so that the non-DF PE can signal the non-DF state to CE1 (in the form of LACP out-of-synch or power-off).

In addition to the BGP-VPLS routes sent for the service ve-id, the multi-homing PEs in this case need to generate additional BGP-VPLS routes per Ethernet Segment (per VPLS service) for the purpose of MAC flush on the remote BGP-VPLS PEs in case of failure.

The sap>bgp-vpls-mh-veid number command should be configured on the SAPs that are part of an EVPN single-active Ethernet Segment, and allows the advertisement of L2VPN routes that indicate the state of the multi-homed SAPs to the remote BGP-VPLS PEs. Upon a Designated Forwarder (DF) switchover, the F and D bits of the generated L2VPN routes for the SAP ve-id are updated so that the remote BGP-VPLS PEs can perform a mac-flush operation on the service and avoid blackholes.

As an example, in case of a failure on the ethernet-segment sap on PE1, PE1 must indicate PE3 and PE4 the need to flush MAC addresses learned from PE1 (flush-all-from-me message). Otherwise, for example, PE3 continues sending traffic with MAC DA = CE1 to PE1, and PE1 blackholes the traffic.

In the Figure 165 example:

Other considerations:

5.3.2.3. Auto-Derived Route-Distinguisher (RD) in Services with Multiple BGP Families

In a VPLS service, multiple BGP families and protocols can be enabled at the same time. When bgp-evpn is enabled, bgp-ad and bgp-mh are supported as well. A single RD is used per service and not per BGP family/protocol.

The following rules apply:

5.3.2.4. EVPN Multi-Homing in VPLS Services

EVPN multi-homing implementation is based on the concept of the ethernet-segment. An ethernet-segment is a logical structure that can be defined in one or more PEs and identifies the CE (or access network) multi-homed to the EVPN PEs. An ethernet-segment is associated with port, LAG, PW port, or SDP objects and is shared by all the services defined on those objects. In the case of virtual ESs, individual VID or VC-ID ranges can be associated to the port, LAG, or PW port, SDP objects defined in the ethernet-segment.

Each ethernet-segment has a unique Ethernet Segment Identifier (esi) that is 10 bytes long and is manually configured in the router.

Note: The esi is advertised in the control plane to all the PEs in an EVPN network; therefore, it is very important to ensure that the 10-byte esi value is unique throughout the entire network. Single-homed CEs are assumed to be connected to an Ethernet-Segment with esi = 0 (single-homed Ethernet-Segments are not explicitly configured).

This section describes the behavior of the EVPN multi-homing implementation in an EVPN-MPLS service.

5.3.2.4.1. EVPN All-Active Multi-Homing

As described in RFC 7432, all-active multi-homing is only supported on access LAG SAPs and it is mandatory that the CE is configured with a LAG to avoid duplicated packets to the network. LACP is optional. SR OS, also, supports the association of a PW port to an all-active multi-homing ES.

Three different procedures are implemented in 7750 SR, 7450 ESS, and 7950 XRS SR OS to provide all-active multi-homing for a specified Ethernet-Segment:

1. DF (Designated Forwarder) election
2. Split-horizon
3. Aliasing

Figure 166 shows the need for DF election in all-active multi-homing.

Figure 166:  DF Election 

The DF election in EVPN all-active multi-homing avoids duplicate packets on the multi-homed CE. The DF election procedure is responsible for electing one DF PE per ESI per service; the rest of the PEs being non-DF for the ESI and service. Only the DF will forward BUM traffic from the EVPN network toward the ES SAPs (the multi-homed CE). The non-DF PEs will not forward BUM traffic to the local Ethernet-Segment SAPs.

Note: BUM traffic from the CE to the network and known unicast traffic in any direction is allowed on both the DF and non-DF PEs.

Figure 167 shows the EVPN split-horizon concept for all-active multi-homing.

Figure 167:  Split-Horizon 

The EVPN split-horizon procedure ensures that the BUM traffic originated by the multi-homed PE and sent from the non-DF to the DF, is not replicated back to the CE (echoed packets on the CE). To avoid these echoed packets, the non-DF (PE1) will send all the BUM packets to the DF (PE2) with an indication of the source Ethernet-Segment. That indication is the ESI Label (ESI2 in the example), previously signaled by PE2 in the AD per-ESI route for the Ethernet-Segment. When PE2 receives an EVPN packet (after the EVPN label lookup), the PE2 will find the ESI label that will identify its local Ethernet-Segment ESI2. The BUM packet will be replicated to other local CEs but not to the ESI2 SAP.

Figure 168 shows the EVPN aliasing concept for all-active multi-homing.

Figure 168:  Aliasing  

Because CE2 is multi-homed to PE1 and PE2 using an all-active Ethernet-Segment, 'aliasing' is the procedure by which PE3 can load-balance the known unicast traffic between PE1 and PE2, even if the destination MAC address was only advertised by PE1 as in the example. When PE3 installs MAC1 in the FDB, it will associate MAC1 not only with the advertising PE (PE1) but also with all the PEs advertising the same esi (ESI2) for the service. In this example, PE1 and PE2 advertise an AD per-EVI route for ESI2, therefore, the PE3 installs the two next-hops associated with MAC1.

Aliasing is enabled by configuring ECMP greater than 1 in the bgp-evpn mpls context.

5.3.2.4.1.1. All-Active Multi-Homing Service Model

The following shows an example PE1 configuration that provides all-active multi-homing to the CE2 shown in Figure 168.

*A:PE1>config>lag(1)# info 

----------------------------------------------

  mode access

  encap-type dot1q

  port 1/1/2

  lacp active administrative-key 1 system-id 00:00:00:00:00:22

  no shutdown

 

*A:PE1>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.1.1.1:0

  ethernet-segment "ESI2" create

    esi 01:12:12:12:12:12:12:12:12:12

    multi-homing all-active

    service-carving

    lag 1 

    no shutdown

 

*A:PE1>config>redundancy>evpn-multi-homing# info 

----------------------------------------------

    boot-timer 120

    es-activation-timer 10

 

*A:PE1>config>service>vpls# info 

----------------------------------------------

  description "evpn-mpls-service with all-active multihoming"

  bgp

  bgp-evpn

    evi 10

    mpls bgp 1

      no shutdown

      auto-bind-tunnel resolution any

  sap lag-1:1 create 

  exit

In the same way, PE2 is configured as follows:

*A:PE1>config>lag(1)# info 

----------------------------------------------

  mode access

  encap-type dot1q

  port 1/1/1

  lacp active administrative-key 1 system-id 00:00:00:00:00:22

  no shutdown

 

*A:PE1>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.1.1.1:0

  ethernet-segment "ESI12" create

    esi 01:12:12:12:12:12:12:12:12:12

    multi-homing all-active

    service-carving

    lag 1 

    no shutdown

 

*A:PE1>config>redundancy>evpn-multi-homing# info 

----------------------------------------------

    boot-timer 120

    es-activation-timer 10

 

*A:PE1>config>service>vpls# info 

----------------------------------------------

  description "evpn-mpls-service with all-active multihoming"

  bgp

    route-distinguisher 65001:60

    route-target target:65000:60

  bgp-evpn

    evi 10

    mpls bgp 1

      no shutdown

      auto-bind-tunnel resolution any

  sap lag-1:1 create 

  exit

The preceding configuration will enable the all-active multi-homing procedures. The following must be considered:

1. The ethernet-segment must be configured with a name and a 10-byte esi:
   1. config>service>system>bgp-evpn#ethernet-segment <es_name> create
   2. config>service> system>bgp-evpn>ethernet-segment# esi <value>
2. When configuring the esi, the system enforces the 6 high-order octets after the type to be different from zero (so that the auto-derived route-target for the ES route is different from zero). Other than that, the entire esi value must be unique in the system.
3. Only a LAG or a PW port can be associated with the all-active ethernet-segment. This LAG will be exclusively used for EVPN multi-homing. Other LAG ports in the system can be still used for MC-LAG and other services.
4. When the LAG is configured on PE1 and PE2, the same admin-key, system-priority, and system-id must be configured on both PEs, so that CE2 responds as though it is connected to the same system.
5. The same ethernet-segment may be used for EVPN-MPLS and PBB-EVPN services.

   Note: The source-bmac-lsb attribute must be defined for PBB-EVPN (so that it will only be used in PBB-EVPN, and ignored by EVPN). Other than EVPN-MPLS and PBB-EVPN I-VPLS/Epipe services, no other Layer 2 services are allowed in the same ethernet-segment (regular VPLS or EVPN-VXLAN SAPs defined on the ethernet-segment will be kept operationally down).

6. Only one SAP per service can be part of the same ethernet-segment.

5.3.2.4.1.2. ES Discovery and DF Election Procedures

The ES discovery and DF election is implemented in three logical steps, as shown in Figure 169.

Figure 169:  ES Discovery and DF Election 

Step 1 - ES Advertisement and Discovery

Ethernet-segment ESI-1 is configured as per the previous section, with all the required parameters. When ethernet-segment no shutdown is executed, PE1 and PE2 will advertise an ES route for ESI-1. They will both include the route-target auto-derived from the MAC portion of the configured ESI. If the route-target address family is configured in the network, this will allow the RR to keep the dissemination of the ES routes under control.

In addition to the ES route, PE1 and PE2 will advertise AD per-ESI routes and AD per-EVI routes.

1. AD per-ESI routes will announce the Ethernet-Segment capabilities, including the mode (single-active or all-active) as well as the ESI label for split-horizon.
2. AD per-EVI routes are advertised so that PE3 knows what services (EVIs) are associated with the ESI. These routes are used by PE3 for its aliasing and backup procedures.

Step 2 - DF Election

When ES routes exchange between PE1 and PE2 is complete, both run the DF election for all the services in the ethernet-segment.

PE1 and PE2 elect a Designated Forwarder (DF) per <ESI, service>. The default DF election mechanism in 7750 SR, 7450 ESS, and 7950 XRS SR OS is service-carving (as per RFC 7432). The following applies when enabled on a specified PE:

1. An ordered list of PE IPs where ESI-1 resides is built. The IPs are gotten from the Origin IP fields of all the ES routes received for ESI-1, as well as the local system address. The lowest IP will be considered ordinal '0' in the list.
2. The local IP can only be considered a “candidate” after successful ethernet-segment no shutdown for a specified service.

   Note: The remote PE IPs must be present in the local PE's RTM so that they can participate in the DF election.

3. A PE will only consider a specified remote IP address as candidate for the DF election algorithm for a specified service if, as well as the ES route, the corresponding AD routes per-ESI and per-EVI for that PE have been received and properly activated.
4. All the remote PEs receiving the AD per-ES routes (for example, PE3), will interpret that ESI-1 is all-active if all the PEs send their AD per-ES routes with the single-active bit = 0. Otherwise, if at least one PE sends an AD route per-ESI with the single-active flag set or the local ESI configuration is single-active, the ESI will behave as single-active.
5. An es-activation-timer can be configured at the redundancy>bgp-evpn-multi-homing>es-activation-timer level or at the service>system>bgp-evpn>eth-seg>es-activation-timer level. This timer, which is 3 seconds by default, delays the transition from non-DF to DF for a specified service, after the DF election has run.
   1. This use of the es-activation-timer is different from zero and minimizes the risks of loops and packet duplication due to “transient” multiple DFs.
   2. The same es-activation-timer should be configured in all the PEs that are part of the same ESI. It is up to the user to configure either a long timer to minimize the risks of loops/duplication or even es-activation-timer=0 to speed up the convergence for non-DF to DF transitions. When the user configures a specific value, the value configured at ES level supersedes the configured global value.
6. The DF election is triggered by the following events:
   1. config>service>system>bgp-evpn>eth-seg# no shutdown triggers the DF election for all the services in the ESI.
   2. Reception of a new update/withdrawal of an ES route (containing an ESI configured locally) triggers the DF election for all the services in the ESI.
   3. Reception of a new update/withdrawal of an AD per-ES route (containing an ESI configured locally) triggers the DF election for all the services associated with the list of route-targets received along with the route.
   4. Reception of a new update of an AD per-ES route with a change in the ESI-label extended community (single-active bit or MPLS label) triggers the DF election for all the services associated with the list of route-targets received along with the route.
   5. Reception of a new update/withdrawal of an AD route per-EVI (containing an ESI configured locally) triggers the DF election for that service.

1. When the PE boots up, the boot-timer will allow the necessary time for the control plane protocols to come up before bringing up the Ethernet-Segment and running the DF algorithm. The boot-timer is configured at system level - config>redundancy>bgp-evpn-multi-homing# boot-timer - and should use a value long enough to allow the IOMs and BGP sessions to come up before exchanging ES routes and running the DF election for each EVI/ISID.
   1. The system will not advertise ES routes until the boot timer expires. This will guarantee that the peer ES PEs don't run the DF election either until the PE is ready to become the DF if it needs to.
   2. The following show command displays the configured boot-timer as well as the remaining timer if the system is still in boot-stage.

A:PE1# show redundancy bgp-evpn-multi-homing 

===============================================================================

Redundancy BGP EVPN Multi-homing Information

===============================================================================

Boot-Timer              : 10 secs                 

Boot-Timer Remaining    : 0 secs                  

ES Activation Timer     : 3 secs                  

===============================================================================

1. When service-carving mode auto is configured (default mode), the DF election algorithm will run the function [V(evi) mod N(peers) = i(ordinal)] to identify the DF for a specified service and ESI, as described in the following example:
   1. As shown in Figure 169, PE1 and PE2 are configured with ESI-1. Given that V(10) mod N(2) = 0, PE1 will be elected DF for VPLS-10 (because its IP address is lower than PE2's and it is the first PE in the candidate list).

      Note: The algorithm takes the configured evi in the service as opposed to the service-id itself. The evi for a service must match in all the PEs that are part of the ESI. This guarantees that the election algorithm is consistent across all the PEs of the ESI. The evi must be always configured in a service with SAPs/SDP bindings that are created in an ES.

2. A manual service-carving option is allowed so that the user can manually configure for which evi identifiers the PE is primary: service-carving mode manual / manual evi <start-evi> to <end-evi>
   1. The system will be the PE forwarding/multicasting traffic for the evi identifiers included in the configuration. The PE will be secondary (non-DF) for the non-specified evi identifiers.
   2. If a range is configured but the service-carving is not mode manual, then the range has no effect.
   3. Only two PEs are supported when service-carving mode manual is configured. If a third PE is configured with service-carving mode manual for an ESI, the two non-primary PEs will remain non-DF regardless of the primary status.
   4. For example, as shown in Figure 169: if PE1 is configured with service-carving manual evi 1 to 100 and PE2 with service-carving manual evi 101 to 200, then PE1 will be the primary PE for service VPLS 10 and PE2 the secondary PE.
3. When service-carving is disabled, the lowest originator IP will win the election for a specified service and ESI:

config>service>system>bgp-evpn>eth-seg>service-carving> mode off

The following show command displays the ethernet-segment configuration and DF status for all the EVIs and ISIDs (if PBB-EVPN is enabled) configured in the ethernet-segment.

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-1" all 

===============================================================================

Service Ethernet Segment

===============================================================================

Name                    : ESI-1

Admin State             : Up                 Oper State         : Up

ESI                     : 01:00:00:00:00:71:00:00:00:01

Multi-homing            : allActive          Oper Multi-homing  : allActive

Source BMAC LSB         : 71-71              

ES BMac Tbl Size        : 8                  ES BMac Entries    : 1

Lag Id                  : 1                  

ES Activation Timer     : 0 secs             

Exp/Imp Route-Target    : target:00:00:00:00:71:00

 

Svc Carving             : auto               

ES SHG Label            : 262142             

===============================================================================

===============================================================================

EVI Information 

===============================================================================

EVI                 SvcId               Actv Timer Rem      DF

-------------------------------------------------------------------------------

1                   1                   0                   no

-------------------------------------------------------------------------------

Number of entries: 1

===============================================================================

-------------------------------------------------------------------------------

DF Candidate list

-------------------------------------------------------------------------------

EVI                                     DF Address

-------------------------------------------------------------------------------

1                                       192.0.2.69

1                                       192.0.2.72

-------------------------------------------------------------------------------

Number of entries: 2

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

===============================================================================

ISID Information 

===============================================================================

ISID                SvcId               Actv Timer Rem      DF

-------------------------------------------------------------------------------

20001               20001               0                   no

-------------------------------------------------------------------------------

Number of entries: 1

===============================================================================                                 

-------------------------------------------------------------------------------

DF Candidate list

-------------------------------------------------------------------------------

ISID                                    DF Address

-------------------------------------------------------------------------------

20001                                   192.0.2.69

20001                                   192.0.2.72

-------------------------------------------------------------------------------

Number of entries: 2

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

===============================================================================

BMAC Information 

===============================================================================

SvcId                                   BMacAddress

-------------------------------------------------------------------------------

20000                                   00:00:00:00:71:71

-------------------------------------------------------------------------------

Number of entries: 1

===============================================================================

Step 3 - DF and Non-DF Service Behavior

Based on the result of the DF election or the manual service-carving, the control plane on the non-DF (PE1) will instruct the data path to remove the LAG SAP (associated with the ESI) from the default flooding list for BM traffic (unknown unicast traffic may still be sent if the EVI label is a unicast label and the source MAC address is not associated to the ESI). On PE1 and PE2, both LAG SAPs will learn the same MAC address (coming from the CE). For instance, in the following show commands, 00:ca:ca:ba:ce:03 is learned on both PE1 and PE2 access LAG (on ESI-1). However, PE1 learns the MAC as 'Learned' whereas PE2 learns it as 'Evpn'. This is due to the CE2 hashing the traffic for that source MAC to PE1. PE2 learns the MAC through EVPN but it associates the MAC to the ESI SAP, because the MAC belongs to the ESI.

*A:PE1# show service id 1 fdb detail 

===============================================================================

Forwarding Database, Service 1

===============================================================================

ServId    MAC               Source-Identifier        Type     Last Change

                                                     Age      

-------------------------------------------------------------------------------

1         00:ca:ca:ba:ce:03 sap:lag-1:1              L/0      06/11/15 00:14:47

1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:06

                            192.0.2.70:262140

1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 00:09:39

                            192.0.2.72:262141

-------------------------------------------------------------------------------

No. of MAC Entries: 3

-------------------------------------------------------------------------------

Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static

===============================================================================

 

*A:PE2# show service id 1 fdb detail 

===============================================================================

Forwarding Database, Service 1

===============================================================================

ServId    MAC               Source-Identifier        Type     Last Change

                                                     Age      

-------------------------------------------------------------------------------

1         00:ca:ca:ba:ce:03 sap:lag-1:1              Evpn     06/11/15 00:14:47

1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 00:09:40

                            192.0.2.69:262141

1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:40

                            192.0.2.70:262140

-------------------------------------------------------------------------------

No. of MAC Entries: 3

-------------------------------------------------------------------------------

Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static

===============================================================================

When PE1 (non-DF) and PE2 (DF) exchange BUM packets for evi 1, all those packets will be sent including the ESI label at the bottom of the stack (in both directions). The ESI label advertised by each PE for ESI-1 can be displayed by the following command:

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-1" 

===============================================================================

Service Ethernet Segment

===============================================================================

Name                    : ESI-1

Admin State             : Up                 Oper State         : Up

ESI                     : 01:00:00:00:00:71:00:00:00:01

Multi-homing            : allActive          Oper Multi-homing  : allActive

Source BMAC LSB         : 71-71              

ES BMac Tbl Size        : 8                  ES BMac Entries    : 1

Lag Id                  : 1                  

ES Activation Timer     : 0 secs             

Exp/Imp Route-Target    : target:00:00:00:00:71:00

 

Svc Carving             : auto               

ES SHG Label            : 262142             

===============================================================================

*A:PE2# show service system bgp-evpn ethernet-segment name "ESI-1" 

 

===============================================================================

Service Ethernet Segment

===============================================================================

Name                    : ESI-1

Admin State             : Up                 Oper State         : Up

ESI                     : 01:00:00:00:00:71:00:00:00:01

Multi-homing            : allActive          Oper Multi-homing  : allActive

Source BMAC LSB         : 71-71              

ES BMac Tbl Size        : 8                  ES BMac Entries    : 0

Lag Id                  : 1                  

ES Activation Timer     : 20 secs            

Exp/Imp Route-Target    : target:00:00:00:00:71:00

 

Svc Carving             : auto               

ES SHG Label            : 262142             

===============================================================================

5.3.2.4.1.3. Aliasing

Following the example in Figure 169, if the service configuration on PE3 has ECMP > 1, PE3 will add PE1 and PE2 to the list of next-hops for ESI-1. As soon as PE3 receives a MAC for ESI-1, it will start load-balancing between PE1 and PE2 the flows to the remote ESI CE. The following command shows the FDB in PE3.

Note: mac 00:ca:ca:ba:ce:03 is associated with the Ethernet-Segment eES:01:00:00:00:00:71:00:00:00:01 (esi configured on PE1 and PE2 for ESI-1).

*A:PE3# show service id 1 fdb detail 

===============================================================================

Forwarding Database, Service 1

===============================================================================

ServId    MAC               Source-Identifier        Type     Last Change

                                                     Age      

-------------------------------------------------------------------------------

1         00:ca:ca:ba:ce:03 eES:                     Evpn     06/11/15 00:14:47

                            01:00:00:00:00:71:00:00:00:01

1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 00:09:18

                            192.0.2.69:262141

1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 00:09:18

                            192.0.2.70:262140

1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 00:09:39

                            192.0.2.72:262141

-------------------------------------------------------------------------------

No. of MAC Entries: 4

-------------------------------------------------------------------------------

Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static

===============================================================================

The following command shows all the EVPN-MPLS destination bindings on PE3, including the ES destination bindings.

The Ethernet-Segment eES:01:00:00:00:00:71:00:00:00:01 is resolved to PE1 and PE2 addresses:

*A:PE3# show service id 1 evpn-mpls 

===============================================================================

BGP EVPN-MPLS Dest

===============================================================================

TEP Address     Egr Label     Num. MACs   Mcast           Last Change

                 Transport                                

-------------------------------------------------------------------------------

192.0.2.69      262140        0           Yes             06/10/2015 14:33:30

                ldp                                        

192.0.2.69      262141        1           No              06/10/2015 14:33:30

                ldp                                        

192.0.2.70      262139        0           Yes             06/10/2015 14:33:30

                ldp                                        

192.0.2.70      262140        1           No              06/10/2015 14:33:30

                ldp                                        

192.0.2.72      262140        0           Yes             06/10/2015 14:33:30

                ldp                                        

192.0.2.72      262141        1           No              06/10/2015 14:33:30

                ldp                                        

192.0.2.73      262139        0           Yes             06/10/2015 14:33:30

                ldp                                        

192.0.2.254     262142        0           Yes             06/10/2015 14:33:30

                bgp                                        

-------------------------------------------------------------------------------

Number of entries : 8

-------------------------------------------------------------------------------

===============================================================================

===============================================================================

BGP EVPN-MPLS Ethernet Segment Dest

===============================================================================

Eth SegId                     TEP Address     Egr Label   Last Change

                                               Transport  

-------------------------------------------------------------------------------

01:00:00:00:00:71:00:00:00:01 192.0.2.69      262141      06/10/2015 14:33:30

                                              ldp          

01:00:00:00:00:71:00:00:00:01 192.0.2.72      262141      06/10/2015 14:33:30

                                              ldp          

01:74:13:00:74:13:00:00:74:13 192.0.2.73      262140      06/10/2015 14:33:30

                                              ldp          

-------------------------------------------------------------------------------

Number of entries : 3

-------------------------------------------------------------------------------

===============================================================================

PE3 will perform aliasing for all the MACs associated with that ESI. This is possible because PE1 is configured with ECMP parameter >1:

*A:PE3>config>service>vpls# info 

----------------------------------------------

            bgp

            exit

            bgp-evpn

                evi 1

                mpls bgp 1

                    ecmp 4

                    auto-bind-tunnel

                        resolution any

                    exit

                    no shutdown

                exit

            exit

            proxy-arp

                shutdown

            exit

            stp

                shutdown

            exit

            sap 1/1/1:2 create

            exit

            no shutdown

5.3.2.4.1.4. Network Failures and Convergence for All-Active Multi-Homing

Figure 170 shows the behavior on the remote PEs (PE3) when there is an ethernet-segment failure.

Figure 170:  All-Active Multi-Homing ES Failure 

The unicast traffic behavior on PE3 is as follows:

1. PE3 can only forward MAC DA = CE2 to both PE1 and PE2 when the MAC advertisement route from PE1 (or PE2) and the set of Ethernet AD per-ES routes and Ethernet AD per-EVI routes from PE1 and PE2 are active at PE3.
2. If there was a failure between CE2 and PE2, PE2 would withdraw its set of Ethernet AD and ES routes, then PE3 would forward traffic destined to CE2 to PE1 only. PE3 does not need to wait for the withdrawal of the individual MAC.
3. The same behavior would be followed if the failure had been at PE1.
4. If after (2), PE2 withdraws its MAC advertisement route, then PE3 treats traffic to MAC DA = CE2 as unknown unicast, unless the MAC had been previously advertised by PE1.

For BUM traffic, the following events would trigger a DF election on a PE and only the DF would forward BUM traffic after the esi-activation-timer expiration (if there was a transition from non-DF to DF).

1. Reception of ES route update (local ES shutdown/no shutdown or remote route)
2. New AD-ES route update/withdraw
3. New AD-EVI route update/withdraw
4. Local ES port/SAP/service shutdown
5. Service carving range change (affecting the evi)
6. Multi-homing mode change (single/all active to all/single-active)

5.3.2.4.1.4.1. Logical Failures on ESs and Blackholes

Be aware of the effects triggered by certain 'failure scenarios'; some of these scenarios are shown in Figure 171:

Figure 171:  Blackhole Caused by SAP/SVC Shutdown 

If an individual VPLS service is shutdown in PE1 (the example is also valid for PE2), the corresponding LAG SAP will go operationally down. This event will trigger the withdrawal of the AD per-EVI route for that particular SAP. PE3 will remove PE1 of its list of aliased next-hops and PE2 will take over as DF (if it was not the DF already). However, this will not prevent the network from black-holing the traffic that CE2 'hashes' to the link to PE1. Traffic sent from CE2 to PE2 or traffic from the rest of the CEs to CE2 will be unaffected, so this situation is not easily detected on the CE.

The same result occurs if the ES SAP is administratively shutdown instead of the service.

Note: When bgp-evpn mpls shutdown is executed, the SAP associated with the ES will be brought operationally down (StandbyforMHprotocol) and so will the entire service if there are no other SAPs or SDP bindings in the service. However, if there are other SAPs/SDP bindings, the service will remain operationally up.

5.3.2.4.1.4.2. Transient Issues Due to MAC Route Delays

Some situations may cause potential transient issues to occur. These are shown in Figure 172 and explained below.

Figure 172:  Transient Issues Caused by “slow” MAC Learning 

Transient packet duplication caused by delay in PE3 to learn MAC1:

This scenario is illustrated by the diagram on the left in Figure 172. In an all-active multi-homing scenario, if a specified MAC address is not yet learned in a remote PE, but is known in the two PEs of the ES, for example, PE1 and PE2, the latter PEs might send duplicated packets to the CE.

In an all-active multi-homing scenario, if a specified MAC address (for example, MAC1), is not learned yet in a remote PE (for example, PE3), but it is known in the two PEs of the ES (for example, PE1 and PE2), the latter PEs might send duplicated packets to the CE.

This issue is solved by the use of ingress-replication-bum-label in PE1 and PE2. If configured, PE1/PE2 will know that the received packet is an unknown unicast packet, therefore, the NDF (PE1) will not send the packets to the CE and there will not be duplication.

Note: Even without the ingress-replication-bum-label, this is only a transient situation that would be solved as soon as MAC1 is learned in PE3.

Transient blackhole caused by delay in PE1 to learn MAC1:

This case is illustrated by the diagram on the right in Figure 172. In an all-active multi-homing scenario, MAC1 is known in PE3 and aliasing is applied to MAC1. However, MAC1 is not known yet in PE1, the NDF for the ES. If PE3 hashing picks up PE1 as the destination of the aliased MAC1, the packets will be blackholed. This case is solved on the NDF by not blocking unknown unicast traffic that arrives with a unicast label. If PE1 and PE2 are configured using ingress-replication-bum-label, PE3 will send unknown unicast with a BUM label and known unicast with a unicast label. In the latter case, PE1 considers it is safe to forward the frame to the CE, even if it is unknown unicast. It is important to note that this is a transient issue and as soon as PE1 learns MAC1 the frames are forwarded as known unicast.

5.3.2.4.2. EVPN Single-Active Multi-Homing

The 7750 SR, 7450 ESS, and 7950 XRS SR OS supports single-active multi-homing on access LAG SAPs, regular SAPs, and spoke SDPs for a specified VPLS service. For LAG SAPs, the CE will be configured with a different LAG to each PE in the Ethernet-Segment (as opposed to a single LAG in an all-active multi-homing).

The following SR OS procedures support EVPN single-active multi-homing for a specified Ethernet-Segment:

1. DF (Designated Forwarder) election
   As in all-active multi-homing, DF election in single-active multi-homing determines the forwarding for BUM traffic from the EVPN network to the Ethernet-Segment CE. Also, in single-active multi-homing, DF election also determines the forwarding of any traffic (unicast/BUM) and in any direction (to/from the CE).
2. Backup PE
   In single-active multi-homing, the remote PEs do not perform aliasing to the PEs in the Ethernet-Segment. The remote PEs identify the DF based on the MAC routes and send the unicast flows for the Ethernet-Segment to the PE in the DF and program a backup PE as an alternative next-hop for the remote ESI in case of failure.
   This RFC 7432 procedure is known as 'Backup PE' and is shown in Figure 173 for PE3.

   Figure 173:  Backup PE 

   

5.3.2.4.2.1. Single-Active Multi-Homing Service Model

The following shows an example of PE1 configuration that provides single-active multi-homing to CE2, as shown in Figure 173.

*A:PE1>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.1.1.1:0

  ethernet-segment "ESI2" create

    esi 01:12:12:12:12:12:12:12:12:12

    multi-homing single-active

    service-carving

    sdp 1 

    no shutdown

 

*A:PE1>config>redundancy>evpn-multi-homing# info 

----------------------------------------------

    boot-timer 120

    es-activation-timer 10

 

*A:PE1>config>service>vpls# info 

----------------------------------------------

  description "evpn-mpls-service with single-active multihoming"

  bgp

  bgp-evpn

    evi 10

    mpls bgp 1

      no shutdown

      auto-bind-tunnel resolution any

  spoke-sdp 1:1 create 

  exit

 

The PE2 example configuration for this scenario is as follows:

*A:PE1>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.1.1.1:0

  ethernet-segment "ESI2" create

    esi 01:12:12:12:12:12:12:12:12:12

    multi-homing single-active

    service-carving

    sdp 2 

    no shutdown

 

*A:PE1>config>redundancy>evpn-multi-homing# info 

----------------------------------------------

    boot-timer 120

    es-activation-timer 10

 

*A:PE1>config>service>vpls# info 

----------------------------------------------

  description "evpn-mpls-service with single-active multihoming"

  bgp

  bgp-evpn

    evi 10

    mpls bgp 1

      no shutdown

      auto-bind-tunnel resolution any

  spoke-sdp 2:1 create 

  exit

 

In single-active multi-homing, the non-DF PEs for a specified ESI will block unicast and BUM traffic in both directions (upstream and downstream) on the object associated with the ESI. Other than that, single-active multi-homing is similar to all-active multi-homing with the following differences:

1. The ethernet-segment will be configured for single-active: service>system>bgp-evpn>eth-seg>multi-homing single-active.
2. The advertisement of the ESI-label in an AD per-ESI is optional for single-active Ethernet-Segments. The user can control the no advertisement of the ESI label by using the following command: service>system>bgp-evpn>eth-seg>multi-homing single-active no-esi-label. By default, the ESI label is used for single-active ESs too.
3. For single-active multi-homing, the Ethernet-Segment can be associated with a port and sdp, as well as a lag-id, as shown in Figure 173, where:
   1. port would be used for single-active SAP redundancy without the need for lag.
   2. sdp would be used for single-active spoke SDP redundancy.
   3. lag would be used for single-active LAG redundancy

      Note: In this case, key, system-id, and system-priority must be different on the PEs that are part of the Ethernet-Segment).

4. For single-active multi-homing, when the PE is non-DF for the service, the SAPs/spoke SDPs on the Ethernet-Segment will be down and show StandByForMHProtocol as the reason.
5. From a service perspective, single-active multi-homing can provide redundancy to CEs (MHD, Multi-Homed Devices) or networks (MHN, Multi-Homed Networks) with the following setup:
   1. LAG with or without LACP
      In this case, the multi-homed ports on the CE will be part of the different LAGs (a LAG per multi-homed PE will be used in the CE). The non-DF PE for each service can signal that the SAP is operationally down if eth-cfm fault-propagation-enable {use-if-tlv | suspend-ccm} is configured.
   2. Regular Ethernet 802.1q/ad ports
      In this case, the multi-homed ports on the CE/network will not be part of any LAG. Eth-cfm can also be used for non-DF indication to the multi-homed device/network.
   3. Active-standby PWs
      In this case, the multi-homed CE/network is connected to the PEs through an MPLS network and an active/standby spoke SDP per service. The non-DF PE for each service will make use of the LDP PW status bits to signal that the spoke SDP is operationally down on the PE side.

5.3.2.4.2.2. ES and DF Election Procedures

In all-active multi-homing, the non-DF keeps the SAP up, although it removes it from the default flooding list. In the single-active multi-homing implementation the non-DF will bring the SAP or SDP binding operationally down. Refer to the ES Discovery and DF Election Procedures for more information.

The following show commands display the status of the single-active ESI-7413 in the non-DF. The associated spoke SDP is operationally down and it signals PW Status standby to the multi-homed CE:

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-7413"       

 

===============================================================================

Service Ethernet Segment

===============================================================================

Name                    : ESI-7413

Admin State             : Up                 Oper State         : Up

ESI                     : 01:74:13:00:74:13:00:00:74:13

Multi-homing            : singleActive       Oper Multi-homing  : singleActive

Source BMAC LSB         : <none>             

Sdp Id                  : 4                  

ES Activation Timer     : 0 secs             

Exp/Imp Route-Target    : target:74:13:00:74:13:00

 

Svc Carving             : auto               

ES SHG Label            : 262141             

===============================================================================

*A:PE1# show service system bgp-evpn ethernet-segment name "ESI-7413" evi 1 

===============================================================================

EVI DF and Candidate List

===============================================================================

EVI           SvcId         Actv Timer Rem      DF  DF Last Change

-------------------------------------------------------------------------------

1             1             0                   no  06/11/2015 20:05:32

===============================================================================

===============================================================================

DF Candidates                           Time Added

-------------------------------------------------------------------------------

192.0.2.70                              06/11/2015 20:05:20

192.0.2.73                              06/11/2015 20:05:32

-------------------------------------------------------------------------------

Number of entries: 2

===============================================================================

*A:PE1# show service id 1 base 

===============================================================================

Service Basic Information

===============================================================================

Service Id        : 1                   Vpn Id            : 0

Service Type      : VPLS                

Name              : (Not Specified)

Description       : (Not Specified)

 

<snip>

-------------------------------------------------------------------------------

Service Access & Destination Points

-------------------------------------------------------------------------------

Identifier                               Type         AdmMTU  OprMTU  Adm  Opr

-------------------------------------------------------------------------------

sap:1/1/1:1                              q-tag        9000    9000    Up   Up

sdp:4:13 S(192.0.2.74)                   Spok         0       8978    Up   Down

===============================================================================

* indicates that the corresponding row element may have been truncated.

 

 

*A:PE1# show service id 1 all | match Pw 

Local Pw Bits      : pwFwdingStandby

Peer Pw Bits       : None

 

*A:PE1# show service id 1 all | match Flag 

Flags              : StandbyForMHProtocol

Flags              : None

 

5.3.2.4.2.3. Backup PE Function

A remote PE (PE3 in Figure 173) will import the AD routes per ESI, where the single-active flag is set. PE3 will interpret that the Ethernet-Segment is single-active if at least one PE sends an AD route per-ESI with the single-active flag set. MACs for a specified service and ESI will be learned from a single PE, that is, the DF for that <ESI, EVI>.

The remote PE will install a single EVPN-MPLS destination (TEP, label) for a received MAC address and a backup next-hop to the PE for which the AD routes per-ESI and per-EVI are received. For instance, in the following command, 00:ca:ca:ba:ca:06 is associated with the remote ethernet-segment eES 01:74:13:00:74:13:00:00:74:13. That eES is resolved to PE(192.0.2.73), which is the DF on the ES.

*A:PE3# show service id 1 fdb detail 

===============================================================================

Forwarding Database, Service 1

===============================================================================

ServId    MAC               Source-Identifier        Type     Last Change

                                                     Age      

-------------------------------------------------------------------------------

1         00:ca:ca:ba:ca:02 sap:1/1/1:2              L/0      06/12/15 00:33:39

1         00:ca:ca:ba:ca:06 eES:                     Evpn     06/12/15 00:33:39

                            01:74:13:00:74:13:00:00:74:13

1         00:ca:fe:ca:fe:69 eMpls:                   EvpnS    06/11/15 21:53:47

                            192.0.2.69:262118

1         00:ca:fe:ca:fe:70 eMpls:                   EvpnS    06/11/15 19:59:57

                            192.0.2.70:262140

1         00:ca:fe:ca:fe:72 eMpls:                   EvpnS    06/11/15 19:59:57

                            192.0.2.72:262141

-------------------------------------------------------------------------------

No. of MAC Entries: 5

-------------------------------------------------------------------------------

Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static

===============================================================================

 

*A:PE3# show service id 1 evpn-mpls 

===============================================================================

BGP EVPN-MPLS Dest

===============================================================================

TEP Address     Egr Label     Num. MACs   Mcast           Last Change

                 Transport                                

-------------------------------------------------------------------------------

192.0.2.69      262118        1           Yes             06/11/2015 19:59:03

                ldp                                        

192.0.2.70      262139        0           Yes             06/11/2015 19:59:03

                ldp                                        

192.0.2.70      262140        1           No              06/11/2015 19:59:03

                ldp                                        

192.0.2.72      262140        0           Yes             06/11/2015 19:59:03

                ldp                                        

192.0.2.72      262141        1           No              06/11/2015 19:59:03

                ldp                                        

192.0.2.73      262139        0           Yes             06/11/2015 19:59:03

                ldp                                        

192.0.2.254     262142        0           Yes             06/11/2015 19:59:03

                bgp                                        

-------------------------------------------------------------------------------

Number of entries : 7

-------------------------------------------------------------------------------

===============================================================================

===============================================================================

BGP EVPN-MPLS Ethernet Segment Dest

===============================================================================

Eth SegId                     TEP Address     Egr Label   Last Change

                                               Transport  

-------------------------------------------------------------------------------

01:74:13:00:74:13:00:00:74:13 192.0.2.73      262140      06/11/2015 19:59:03

                                              ldp          

-------------------------------------------------------------------------------

Number of entries : 1

-------------------------------------------------------------------------------

===============================================================================

If PE3 sees only two single-active PEs in the same ESI, the second PE will be the backup PE. Upon receiving an AD per-ES/per-EVI route withdrawal for the ESI from the primary PE, the PE3 will start sending the unicast traffic to the backup PE immediately.

If PE3 receives AD routes for the same ESI and EVI from more than two PEs, the PE will not install any backup route in the data path. Upon receiving an AD per-ES/per-EVI route withdrawal for the ESI, it will flush the MACs associated with the ESI.

5.3.2.4.2.4. Network Failures and Convergence for Single-Active Multi-Homing

Figure 174 shows the remote PE (PE3) behavior when there is an Ethernet-Segment failure.

Figure 174:  Single-Active Multi-Homing ES Failure 

The PE3 behavior for unicast traffic is as follows:

1. PE3 forwards MAC DA = CE2 to PE2 when the MAC Advertisement Route came from PE2 and the set of Ethernet AD per-ES routes and Ethernet AD per-EVI routes from PE1 and PE2 are active at PE3.
2. If there was a failure between CE2 and PE2, PE2 would withdraw its set of Ethernet AD and ES routes, then PE3 would immediately forward the traffic destined for CE2 to PE1 only (the backup PE). PE3 does not need to wait for the withdrawal of the individual MAC.
3. After the (2) PE2 withdraws its MAC advertisement route, PE3 will treat traffic to MAC DA = CE2 as unknown unicast, unless the MAC has been previously advertised by PE1.

Also, a DF election on PE1 is triggered. In general, a DF election is triggered by the same events as for all-active multi-homing. In this case, the DF will forward traffic to CE2 when the esi-activation-timer expiration occurs (the timer kicks in when there is a transition from non-DF to DF).

5.3.4.1. BGP-EVPN Control Plane for EVPN-VPWS

EVPN-VPWS for MPLS tunnels uses the RFC 8214 BGP extensions described in EVPN-VPWS for VXLAN Tunnels, with the following differences for the Ethernet AD per-EVI routes:

5.3.4.2. EVPN for MPLS Tunnels in Epipe Services (EVPN-VPWS)

The use and configuration of EVPN-VPWS services is described in EVPN-VPWS for VXLAN Tunnels with the following differences when the EVPN-VPWS services use MPLS tunnels instead of VXLAN.

When MPLS tunnels are used, the bgp-evpn>mpls context must be configured in the Epipe. As an example, if Epipe 2 is an EVPN-VPWS service that uses MPLS tunnels between PE2 and PE4, this would be its configuration:

Where the following BGP-EVPN commands, specific to MPLS tunnels, are supported in the same way as in VPLS services:

EVPN-VPWS Epipes with MPLS tunnels can also be configured with the following characteristics:

The following features, described in EVPN-VPWS for VXLAN Tunnels, are also supported for MPLS tunnels:

5.3.4.3. PW Ports-based ESs for EVPN-VPWS

PW ports in ESs and virtual ESs (vESs) are supported for EVPN-VPWS MPLS services. In addition to lag, port and sdp objects, pw-port pw-port-id can be configured. PW-port based ESs support all-active mode only, and not single-active mode. The following statements apply:

For all the above scenarios, fault-propagation to the access CE only works in case of physical failures. Administrative shutdown of individual Epipes, PW-SAPs, Ethernet-segments or BGP-EVPN may result in traffic black-holes.

Figure 176 illustrates the use of pw-ports in ESs. In this example, an FPE-based pw-port is associated with the ES, where the stitching service itself uses EVPN-VPWS as well.

Figure 176:  ES FPE-based pw-port - access EVPN-VPWS 

In this example, the following applies:

5.3.5.1. EVPN-MPLS Multi-Homing and Passive VRRP

SAP and spoke SDP based ESs are supported on R-VPLS services where bgp-evpn mpls is enabled.

Figure 177 shows an example of EVPN-MPLS multi-homing in R-VPLS services, with the following assumptions.

Figure 177:  EVPN-MPLS Multi-Homing in R-VPLS Services 

In the example in Figure 177, the hosts connected to CE1 and CE4 could use regular VRRP for default gateway redundancy; however, this may not be the most efficient way to provide upstream routing.

For example, if PE1 and PE2 are using regular VRRP, the upstream traffic from CE1 may be hashed to the backup IRB VRRP interface, instead of being hashed to the master. The same thing may occur for single-active multi-homing and regular VRRP for PE3 and PE4. The traffic from CE4 will be sent to PE3, while PE4 may be the VRRP master. In that case, PE3 will have to send the traffic to PE4, instead of route it directly.

In both cases, unnecessary bandwidth between the PEs is used to get to the master IRB interface. In addition, VRRP scaling is limited if aggressive keepalive timers are used.

Because of these issues, passive VRRP is recommended as the best method when EVPN-MPLS multi-homing is used in combination with R-VPLS redundant interfaces.

Passive VRRP is a VRRP setting in which the transmission and reception of keepalive messages is completely suppressed, and therefore the VPRN interface always behaves as the master. Passive VRRP is enabled by adding the passive keyword to the VRRP instance at creation, as shown in the following examples:

config service vprn 1 interface int-1 vrrp 1 passive

config service vprn 1 interface int-1 ipv6 vrrp 1 passive

For example, if PE1, PE2, and PE5 in Figure 177 use passive VRRP, even if each individual R-VPLS interface has a different MAC/IP address, because they share the same VRRP instance 1 and the same backup IP, the three PEs will own the same virtual MAC and virtual IP address (for example, 00-00-5E-00-00-01 and 10.0.0.254). The virtual MAC is auto-derived from 00-00-5E-00-00-VRID per RFC 3768. The following is the expected behavior when passive VRRP is used in this example.

The following list summarizes the advantages of using passive VRRP mode versus regular VRRP for EVPN-MPLS multi-homing in R-VPLS services.

5.3.6.1. BGP-EVPN Control Plane for PBB-EVPN

PBB-EVPN uses a reduced subset of the routes and procedures described in RFC 7432. The supported routes are:

5.3.6.1.3. EVPN Route Type 4 - ES Route

This route type is used for DF election as described in section BGP-EVPN Control Plane for MPLS Tunnels.

Note: The EVPN route type 1—Ethernet Auto Discovery route is not used in PBB-EVPN.

5.3.6.2. PBB-EVPN for I-VPLS and PBB Epipe Services

The 7750 SR, 7450 ESS, and 7950 XRS SR OS implementation of PBB-EVPN reuses the existing PBB-VPLS model, where N I-VPLS (or Epipe) services can be linked to a B-VPLS service. BGP-EVPN is enabled in the B-VPLS and the B-VPLS becomes an EVI (EVPN Instance). Figure 178 shows the PBB-EVPN model in the SR OS.

Figure 178:  PBB-EVPN for I-VPLS and PFF Epipe Services 

Each PE in the B-VPLS domain advertises its source-bmac as either configured in (b)vpls>pbb>source-bmac or auto-derived from the chassis mac. The remote PEs will install the advertised B-MACs in the B-VPLS FDB. If a specified PE is configured with an ethernet-segment associated with an I-VPLS or PBB Epipe, it may also advertise an es-bmac for the Ethernet-Segment.

In the example shown in Figure 178, when a frame with MAC DA = AA gets to PE1, a mac lookup is performed on the I-VPLS FDB and B-MAC-34 is found. A B-MAC lookup on the B-VPLS FDB yields the next-hop (or next-hops if the destination is in an all-active Ethernet-Segment) to which the frame is sent. As in PBB-VPLS, the frame will be encapsulated with the corresponding PBB header. A label specified by EVPN for the B-VPLS and the MPLS transport label are also added.

If the lookup on the I-VPLS FDB fails, the system sends the frame encapsulated into a PBB packet with B-MAC DA = Group B-MAC for the ISID. That packet will be distributed to all the PEs where the ISID is defined and contains the EVPN label distributed by the Inclusive Multicast routes for that ISID, as well as the transport label.

For PBB-Epipes, all the traffic is sent in a unicast PBB packet to the B-MAC configured in the pbb-tunnel.

The following CLI output shows an example of the configuration of an I-VPLS, PBB-Epipe, and their corresponding B-VPLS.

Configure the bgp-evpn context as described in section EVPN for MPLS Tunnels in VPLS Services (EVPN-MPLS).

Some EVPN configuration options are not relevant to PBB-EVPN and are not supported when BGP-EVPN is configured in a B-VPLS; these are as follows:

When bgp-evpn>mpls no shutdown is added to a specified B-VPLS instance, the following considerations apply:

5.3.6.2.1. Flood Containment for I-VPLS Services

In general, PBB technologies in the 7750 SR, 7450 ESS, or 7950 XRS SR OS support a way to contain the flooding for a specified I-VPLS ISID, so that BUM traffic for that ISID only reaches the PEs where the ISID is locally defined. Each PE will create an MFIB per I-VPLS ISID on the B-VPLS instance. That MFIB supports SAP or SDP bindings endpoints that can be populated by:

1. MMRP in regular PBB-VPLS
2. IS-IS in SPBM

In PBB-EVPN, B-VPLS EVPN endpoints can be added to the MFIBs using EVPN Inclusive Multicast Ethernet Tag routes.

The example in Figure 179 shows how the MFIBs are populated in PBB-EVPN.

Figure 179:  PBB-EVPN and I-VPLS Flooding Containment 

When the B-VPLS 10 is enabled, PE1 will advertise as follows:

1. A B-MAC route containing PE1's system B-MAC (00:01 as configured in pbb>source-bmac) along with an MPLS label.
2. An Inclusive Multicast Ethernet Tag route (IMET route) with Ethernet-tag = 0 that will allow the remote B-VPLS 10 instances to add an entry for PE1 in the default multicast list.

Note: The MPLS label that will be advertised for the MAC routes and the inclusive multicast routes for a specified B-VPLS can be the same label or a different label. As in regular EVPN-MPLS, this will depend on the [no] ingress-replication-bum-label command.

When I-VPLS 2001 (ISID 2001) is enabled as per the CLI in the preceding section, PE1 will advertise as follows:

1. An additional inclusive multicast route with Ethernet-tag = 2001. This will allow the remote PEs to create an MFIB for the corresponding ISID 2001 and add the corresponding EVPN binding entry to the MFIB.
   This default behavior can be modified by the configured isid-policy. For instance, for ISIDs 1-2000, configure as follows:

isid-policy

    entry 10 create

      no advertise-local 

      range 1 to 2000

      use-def-mcast

This configuration has the following effect for the ISID range:

1. no advertise-local instructs the system to not advertise the local active ISIDs contained in the 1 to 2001 range.
2. use-def-mcast instructs the system to use the default flooding list as opposed to the MFIB.

The ISID flooding behavior on B-VPLS SAPs and SDP bindings is as follows:

1. B-VPLS SAPs and SDP bindings are only added to the TLS-multicast list and not to the MFIB list (unless static-isids are configured, which is only possible for SAPs/SDP bindings and not BGP-AD spoke SDPs).
   As a result, if the system needs to flood ISID BUM traffic and the ISID is also defined in remote PEs connected through SAPs or spoke SDPs without static-isids, then an isid-policy must be configured for the ISID so that the ISID uses the default multicast list.
2. When an isid-policy is configured and a range of ISIDs use the default multicast list, the remote PBB-EVPN PEs will be added to the default multicast list as long as they advertise an IMET route with an ISID included in the policy's ISID range. PEs advertising IMET routes with Ethernet-tag = 0 are also added to the default multicast list (7750 SR, 7450 ESS, or 7950 XRS SR OS behavior).
3. The B-VPLS 10 also allows the ISID flooding to legacy PBB networks via B-SAPs or B-SDPs. The legacy PBB network B-MACs will be dynamically learned on those SAPs/binds or statically configured through the use of conditional static-macs. The use of static-isids is required so that non-local ISIDs are advertised.

sap 1/1/1:1 create 

exit

spoke-sdp 1:1 create

  static-mac

      mac 00:fe:ca:fe:ca:fe create sap 1/1/1:1 monitor fwd-status

  static-isid

      range 1 isid 3000 to 5000 create

Note: The configuration of PBB-Epipes does not trigger any IMET advertisement.

5.3.6.2.3. PBB-EVPN Multi-Homing in I-VPLS and PBB Epipe Services

The 7750 SR, 7450 ESS, and 7950 XRS SR OS PBB-EVPN implementation supports all-active and single-active multi-homing for I-VPLS and PBB Epipe services.

PBB-EVPN multi-homing reuses the ethernet-segment concept described in section EVPN Multi-Homing in VPLS Services. However, unlike EVPN-MPLS, PBB-EVPN does not use AD routes; it uses B-MACs for split-horizon checks and aliasing.

5.3.6.2.3.1. System B-MAC Assignment in PBB-EVPN

RFC 7623 describes two types of B-MAC assignments that a PE can implement:

1. Shared B-MAC addresses that can be used for single-homed CEs and a number of multi-homed CEs connected to Ethernet-Segments.
2. Dedicated B-MAC addresses per Ethernet-Segment.

In this document and in 7750 SR, 7450 ESS, and 7950 XRS SR OS terminology:

1. A shared-bmac (in IETF) is a source-bmac as configured in service>(b)vpls>pbb>source-bmac
2. A dedicated-bmac per ES (in IETF) is an es-bmac as configured in service>pbb>use-es-bmac

B-MAC selection and use depends on the multi-homing model; for single-active mode, the type of B-MAC will impact the flooding in the network as follows:

1. All-active multi-homing requires es-bmacs.
2. Single-active multi-homing can use es-bmacs or source-bmacs.
   1. The use of source-bmacs minimizes the number of B-MACs being advertised but has a larger impact on C-MAC flush upon ES failures.
   2. The use of es-bmacs optimizes the C-MAC flush upon ES failures at the expense of advertising more B-MACs.

5.3.6.2.3.2. PBB-EVPN All-Active Multi-Homing Service Model

Figure 180 shows the use of all-active multi-homing in the 7750 SR, 7450 ESS, and 7950 XRS SR OS PBB-EVPN implementation.

Figure 180:  PBB-EVPN All-Active Multi-Homing 

For example, the following shows the ESI-1 and all-active configuration in PE3 and PE4. As in EVPN-MPLS, all-active multi-homing is only possible if a LAG is used at the CE. All-active multi-homing uses es-bmacs, that is, each ESI will be assigned a dedicated B-MAC. All the PEs part of the ES will source traffic using the same es-bmac.

In Figure 180 and the following configuration, the es-bmac used by PE3 and PE4 will be B-MAC-34, i.e. 00:00:00:00:00:34. The es-bmac for a specified ethernet-segment is configured by the source-bmac-lsb along with the (b-)vpls>pbb>use-es-bmac command.

Configuration in PE3:

*A:PE3>config>lag(1)# info 

----------------------------------------------

  mode access

  encap-type dot1q

  port 1/1/1

  lacp active administrative-key 32768

  no shutdown

 

*A:PE3>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.3.3.3:0

  ethernet-segment ESI-1 create

    esi 00:34:34:34:34:34:34:34:34:34

    multi-homing all-active

    service-carving auto

    lag 1

    source-bmac-lsb 00:34 es-bmac-table-size 8

    no shutdown 

 

*A:PE3>config>service>vpls 1(b-vpls)# info 

----------------------------------------------

  bgp    

  exit    

  bgp-evpn

    evi 1

    mpls bgp 1

      no shutdown

      ecmp 2

      auto-bind-tunnel resolution any

  exit    

  pbb   

    source-bmac 00:00:00:00:00:03

    use-es-bmac

 

*A:PE3>config>service>vpls (i-vpls)# info 

----------------------------------------------

  pbb

    backbone-vpls 1

  sap lag-1:101 create

 

*A:PE1>config>service>epipe (pbb)# info 

----------------------------------------------

  pbb

    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102

  sap lag-1:102 create

 

Configuration in PE4:

*A:PE4>config>lag(1)# info 

----------------------------------------------

  mode access

  encap-type dot1q

  port 1/1/1

  lacp active administrative-key 32768

  no shutdown

 

*A:PE4>config>service>system>bgp-evpn# info 

----------------------------------------------

  route-distinguisher 10.4.4.4:0

  ethernet-segment ESI-1 create

    esi 00:34:34:34:34:34:34:34:34:34

    multi-homing all-active

    service-carving auto

    lag 1

    source-bmac-lsb 00:34 es-bmac-table-size 8

    no shutdown 

 

 

*A:PE4>config>service>vpls 1(b-vpls)# info 

----------------------------------------------

  bgp    

  exit    

  bgp-evpn

    evi 1

    mpls bgp 1

      no shutdown

      ecmp 2

      auto-bind-tunnel resolution any

  exit    

  pbb   

    source-bmac 00:00:00:00:00:04

    use-es-bmac

 

*A:PE4>config>service>vpls (i-vpls)# info 

----------------------------------------------

  pbb

    backbone-vpls 1

  sap lag-1:101 create

 

*A:PE4>config>service>epipe (pbb)# info 

----------------------------------------------

  pbb

    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102

  sap lag-1:102 create

The above configuration will enable the all-active multi-homing procedures for PBB-EVPN.

Note: The ethernet-segment ESI-1 can also be used for regular VPLS services.

The following considerations apply when the ESI is used for PBB-EVPN.

1. ESI association: Only LAG is supported for all-active multi-homing. The following commands are used for the LAG to ESI association:
   1. config>service>system>bgp-evpn>ethernet-segment# lag <id>
   2. config>service>system>bgp-evpn>ethernet-segment# source-bmac-lsb <MAC-lsb> [es-bmac-table-size <size>]
   3. Where:
      1. The same ESI may be used for EVPN and PBB-EVPN services.
      2. For PBB-EVPN services, the source-bmac-lsb attribute is mandatory and ignored for EVPN-MPLS services.
      3. The source-bmac-lsb attribute must be set to a specific 2-byte value. The value must match on all the PEs part of the same ESI, for example, PE3 and PE4 for ESI-1. This means that the configured pbb>source-bmac on the two PEs for B-VPLS 1 must have the same 4 most significant bytes.
      4. The es-bmac-table-size parameter modifies the default value (8) for the maximum number of virtual B-MACs that can be associated with the ethernet-segment, i.e. es-bmacs. When the source-bmac-lsb is configured, the associated es-bmac-table-size is reserved out of the total FDB space.
      5. When multi-homing all-active is configured within the ethernet-segment, only a LAG can be associated with it. The association of a port or an sdp will be restricted by the CLI.
2. If service-carving is configured in the ESI, the DF election algorithm will be a modulo function of the ISID and the number of PEs part of the ESI, as opposed to a modulo function of evi and number of PEs (used for EVPN-MPLS).
3. A service-carving mode manual option is added so that the user can control what PE is DF for a specified ISID. The PE will be DF for the configured ISIDs and non-DF for the non-configured ISIDs.
4. DF election: An all-active Designated Forwarder (DF) election is also carried out for PBB-EVPN. In this case, the DF election defines which of the PEs of the ESI for a specified I-VPLS is the one able to send the downstream BUM traffic to the CE. Only one DF per ESI is allowed in the I-VPLS service, and the non-DF will only block BUM traffic and in the downstream direction.
5. Split-horizon function: In PBB-EVPN, the split-horizon function to avoid echoed packets on the CE is based on an ingress lookup of the ES B-MAC (as opposed to the ESI label in EVPN-MPLS). In Figure 180 PE3 sends packets using B-MAC SA = BMAC-34. PE4 does not send those packets back to the CE because BMAC-34 is identified as the es-bmac for ESI-1.
6. Aliasing: In PBB-EVPN, aliasing is based on the ES B-MAC sent by all the PEs part of the same ESI. See the following section for more information. In Figure 180 PE1 performs load balancing between PE3 and PE4 when sending unicast flows to BMAC-34 (es-bmac for ESI-1).

In the configuration above, a PBB-Epipe is configured in PE3 and PE4, both pointing at the same remote pbb tunnel backbone-dest-mac. On the remote PE, i.e. PE1, the configuration of the PBB-Epipe will point at the es-bmac:

*A:PE1>config>service>epipe (pbb)# info 

----------------------------------------------

  pbb

    tunnel 1 backbone-dest-mac 00:00:00:00:00:34 isid 102

  sap 1/1/1:102 create

When PBB-Epipes are used in combination with all-active multi-homing, Nokia recommends using bgp-evpn mpls ingress-replication-bum-label in the PEs where the ethernet-segment is created, that is in PE3 and PE4. This guarantees that in case of flooding in the B-VPLS service for the PBB Epipe, only the DF will forward the traffic to the CE.

Note: PBB-Epipe traffic always uses B-MAC DA = unicast; therefore, the DF cannot check whether the inner frame is unknown unicast or not based on the group B-MAC. Therefore, the use of an EVPN BUM label is highly recommended.

Aliasing for PBB-epipes with all-active multi-homing only works if shared-queuing or ingress policing is enabled on the ingress PE epipe. In any other case, the IOM will send the traffic to a single destination (no ECMP will be used in spite of the bgp-evpn mpls ecmp setting).

All-active multi-homed es-bmacs are treated by the remote PEs as eES:MAX-ESI BMACs. The following example shows the FDB in B-VPLS 1 in PE1 as shown in Figure 180:

*A:PE1# show service id 1 fdb detail 

 

===============================================================================

Forwarding Database, Service 1

===============================================================================

ServId    MAC               Source-Identifier        Type     Last Change

                                                     Age      

-------------------------------------------------------------------------------

1         00:00:00:00:00:03 eMpls:                   EvpnS    06/12/15 15:35:39

                            192.0.2.3:262138

1         00:00:00:00:00:04 eMpls:                   EvpnS    06/12/15 15:42:52

                            192.0.2.4:262130

1         00:00:00:00:00:34 eES:                     EvpnS    06/12/15 15:35:57

                            MAX-ESI

-------------------------------------------------------------------------------

No. of MAC Entries: 3

-------------------------------------------------------------------------------

Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static

===============================================================================

 

The show service id evpn-mpls on PE1 shows that the remote es-bmac, i.e. 00:00:00:00:00:34, has two associated next-hops, i.e. PE3 and PE4:

*A:PE1# show service id 1 evpn-mpls 

 

===============================================================================

BGP EVPN-MPLS Dest

===============================================================================

TEP Address     Egr Label     Num. MACs   Mcast           Last Change

                 Transport                                

-------------------------------------------------------------------------------

192.0.2.3       262138        1           Yes             06/12/2015 15:34:48

                ldp                                        

192.0.2.4       262130        1           Yes             06/12/2015 15:34:48

                ldp                                        

-------------------------------------------------------------------------------

Number of entries : 2

-------------------------------------------------------------------------------

===============================================================================

 

===============================================================================

BGP EVPN-MPLS Ethernet Segment Dest

===============================================================================

Eth SegId                     TEP Address     Egr Label   Last Change

                                               Transport  

-------------------------------------------------------------------------------

No Matching Entries

===============================================================================

 

===============================================================================

BGP EVPN-MPLS ES BMAC Dest

===============================================================================

VBMacAddr                   TEP Address     Egr Label     Last Change

                                             Transport    

-------------------------------------------------------------------------------

00:00:00:00:00:34           192.0.2.3       262138        06/12/2015 15:34:48

                                            ldp            

00:00:00:00:00:34           192.0.2.4       262130        06/12/2015 15:34:48

                                            ldp            

-------------------------------------------------------------------------------

Number of entries : 2

-------------------------------------------------------------------------------

=============================================================================== 

5.3.6.2.3.3. Network failures and convergence for all-active multi-homing

ES failures are resolved by the PEs withdrawing the es-bmac. The remote PEs will withdraw the route and update their list of next-hops for a specified es-bmac.

No mac-flush of the I-VPLS FDB tables is required as long as the es-bmac is still in the FDB.

When the route corresponding to the last next-hop for a specified es-bmac is withdrawn, the es-bmac will be flushed from the B-VPLS FDB and all the C-MACs associated with it will be flushed too.

The following events will trigger a withdrawal of the es-bmac and the corresponding next-hop update in the remote PEs:

1. B-VPLS transition to operationally down status.
2. Change of pbb>source-bmac.
3. Change of es-bmac (or removal of pbb use-es-bmac).
4. Ethernet-segment transition to operationally down status.

Note: Individual SAPs going operationally down in an ES will not generate any BGP withdrawal or indication so that the remote nodes can flush their C-MACs. This is solved in EVPN-MPLS by the use of AD routes per EVI; however, there is nothing similar in PBB-EVPN for indicating a partial failure in an ESI.

5.3.6.2.3.4. PBB-EVPN Single-Active Multi-Homing Service Model

In single-active multi-homing, the non-DF PEs for a specified ESI will block unicast and BUM traffic in both directions (upstream and downstream) on the object associated with the ESI. Other than that, single-active multi-homing will follow the same service model defined in the PBB-EVPN All-Active Multi-Homing Service Model section with the following differences:

1. The ethernet-segment will be configured for single-active: service>system>bgp-evpn>eth-seg>multi-homing single-active.
2. For single-active multi-homing, the ethernet-segment can be associated with a port and sdp, as well as a lag.
3. From a service perspective, single-active multi-homing can provide redundancy to the following services and access types:
   1. I-VPLS LAG and regular SAPs
   2. I-VPLS active/standby spoke SDPs
   3. EVPN single-active multi-homing is supported for PBB-Epipes only in two-node scenarios with local switching.
4. While all-active multi-homing only uses es-bmac assignment to the ES, single-active multi-homing can use source-bmac or es-bmac assignment. The system allows the following user choices per B-VPLS and ES:
   1. A dedicated es-bmac per ES can be used. In that case, the pbb>use-es-bmac command will be configured in the B-VPLS and the same procedures explained in PBB-EVPN All-Active Multi-Homing Service Model will follow with one difference. While in all-active multi-homing all the PEs part of the ESI will source the PBB packets with the same source es-bmac, single-active multi-homing requires the use of a different es-bmac per PE.
   2. A non-dedicated source-bmac can be used. In this case, the user will not configure pbb>use-es-bmac and the regular source-bmac will be used for the traffic. A different source-bmac has to be advertised per PE.
   3. The use of source-bmacs or es-bmacs for single-active multi-homed ESIs has a different impact on C-MAC flushing, as shown in Figure 181.

   Figure 181:  Source-Bmac Versus Es-Bmac C-MAC Flushing  

   
   1. If es-bmacs are used as shown in the representation on the right in Figure 181, a less-impacting C-MAC flush is achieved, therefore, minimizing the flooding after ESI failures. In case of ESI failure, PE1 will withdraw the es-bmac 00:12 and the remote PE3 will only flush the C-MACs associated with that es-bmac (only the C-MACs behind the CE are flushed).
   2. If source-bmacs are used, as shown on the left-hand side of Figure 181, in case of ES failure, a BGP update with higher sequence number will be issued by PE1 and the remote PE3 will flush all the C-MACs associated with the source-bmac. Therefore, all the C-MACs behind the PE's B-VPLS will be flushed, as opposed to only the C-MACs behind the ESI's CE.
5. As in EVPN-MPLS, the non-DF status can be notified to the access CE or network:
   1. LAG with or without LACP: In this case, the multi-homed ports on the CE will not be part of the same LAG. The non-DF PE for each service may signal that the LAG SAP is operationally down by using eth-cfm fault-propagation-enable {use-if-tlv|suspend-ccm}.
   2. Regular Ethernet 802.1q/ad ports: In this case, the multi-homed ports on the CE/network will not be part of any LAG. The non-DF PE for each service will signal that the SAP is operationally down by using eth-cfm fault-propagation-enable {use-if-tlv|suspend-ccm}.
   3. Active-standby PWs: in this case, the multihomed CE/network is connected to the PEs through an MPLS network and an active/standby spoke SDP per service. The non-DF PE for each service will make use of the LDP PW status bits to signal that the spoke SDP is standby at the PE side. Nokia recommends that the CE suppresses the signaling of PW status standby.

5.3.6.2.3.5. Network Failures and Convergence for Single-Active Multihoming

ESI failures are resolved depending on the B-MAC address assignment chosen by the user:

1. If the B-MAC address assignment is based on the use of es-bmacs, DF and non-DFs will send the es-bmac/ESI=0 for a specified ESI. Each PE will have a different es-bmac for the same ESI (as opposed to the same es-bmac on all the PEs for all-active). In case of an ESI failure on a PE:
   1. The PE will withdraw its es-bmac route triggering a mac-flush of all the C-MACs associated with it in the remote PEs.
2. If the B-MAC address assignment is based on the use of source-bmac, DF and non-DFs will advertise their respective source-bmacs. In case of an ES failure:
   1. The PE will re-advertise its source-bmac with a higher sequence number (the new DF will not re-advertise its source-bmac).
   2. The far-end PEs will interpret a source-bmac advertisement with a different sequence number as a flush-all-from-me message from the PE detecting the failure. They will flush all the C-MACs associated with that B-MAC in all the ISID services.

The following events will trigger a C-MAC flush notification. A 'C-MAC flush notification' means the withdrawal of a specified B-MAC or the update of B-MAC with a higher sequence number (SQN). Both BGP messages will make the remote PEs flush all the C-MACs associated with the indicated B-MAC:

1. B-VPLS transition to operationally down status. This will trigger the withdrawal of the associated B-MACs, regardless of the use-es-bmac setting.
2. Change of pbb>source-bmac. This will trigger the withdrawal and re-advertisement of the source-bmac, causing the corresponding C-MAC flush in the remote PEs.
3. Change of es-bmac (removal of pbb use-es-bmac). This will trigger the withdrawal of the es-bmac and re-advertisement of the new es-bmac.
4. Ethernet-Segment (ES) transition to operationally down or admin-down status. This will trigger an es-bmac withdrawal (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
5. Service Carving Range change for the ES. This will trigger an es-bmac update with higher SQN (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
6. Change in the number of candidate PEs for the ES. This will trigger an es-bmac update with higher SQN (if use-es-bmac is used) or an update of the source-bmac with a higher SQN (if no use-es-bmac is used).
7. In an ESI, individual SAPs/SDP bindings or individual I-VPLS going operationally down will not generate any BGP withdrawal or indication so that the remote nodes can flush their C-MACs. This is solved in EVPN-MPLS by the use of AD routes per EVI; however, there is nothing similar in PBB-EVPN for indicating a partial failure in an ESI.

5.3.6.2.3.6. PBB-Epipes and EVPN Multi-Homing

EVPN multi-homing is supported with PBB-EVPN Epipes, but only in a limited number of scenarios. In general, the following applies to PBB-EVPN Epipes:

1. PBB-EVPN Epipes don't support spoke SDPs that are associated with EVPN ESs.
2. PBB-EVPN Epipes support all-active EVPN multi-homing as long as no local-switching is required in the Epipe instance where the ES is defined.
3. PBB-EVPN Epipes support single-active EVPN multi-homing only in a two-node case scenario.

Figure 182 shows the EVPN MH support in a three-node scenario.

Figure 182:  PBB-EVPN MH in a Three-Node Scenario 

EVPN MH support in a three-node scenario has the following characteristics:

1. All-active EVPN multi-homing is fully supported (diagram on the left in Figure 182). CE1 might also be multi-homed to other PEs, as long as those PEs are not PE2 or PE3. In this case, PE1 Epipe's pbb-tunnel would be configured with the remote ES B-MAC.
2. Single-active EVPN multi-homing is not supported in a three (or more)-node scenario (diagram on the right in Figure 182). Since PE1's Epipe pbb-tunnel can only point at a single remote B-MAC and single-active multi-homing requires the use of separate B-MACs on PE2 and PE3, the scenario is not possible and not supported regardless of the ES association to port/LAG/sdps.
3. Regardless of the EVPN multi-homing type, the CLI prevents the user from adding a spoke SDP to an Epipe, if the corresponding SDP is part of an ES.

Figure 183 shows the EVPN MH support in a two-node scenario.

Figure 183:  PBB-EVPN MH in a Two-Node Scenario 

EVPN MH support in a two-node scenario has the following characteristics, as shown in Figure 183:

1. All-active multi-homing is not supported for redundancy in this scenario because PE1's pbb-tunnel cannot point at a locally defined ES B-MAC. This is represented in the left-most scenario in Figure 183.
2. Single-active multi-homing is supported for redundancy in a two-node three or four SAP scenario, as displayed by the two right-most scenarios in Figure 183).
   In these two cases, the Epipe pbb-tunnel will be configured with the source B-MAC of the remote PE node.
   When two SAPs are active in the same Epipe, local-switching is used to exchange frames between the CEs.

5.3.6.2.5. PBB-EVPN ISID-Based C-MAC Flush

SR OS supports ISID-based C-MAC flush procedures for PBB-EVPN I-VPLS services where no single-active ESs are used. SR OS also supports C-MAC flush procedure where other redundancy mechanisms, such as BGP-MH, need these procedures to avoid blackholes caused by a SAP or spoke SDP failure.

The C-MAC flush procedures are enabled on the I-VPLS service using the config>service>vpls>pbb>send-bvpls-evpn-flush CLI command. The feature can be disabled on a per-SAP or per-spoke SDP basis by using the disable-send-bvpls-evpn-flush command in the config>service>vpls>sap or config>service>vpls>spoke-sdp context.

With the feature enabled on an I-VPLS service and a SAP or spoke SDP, if there is a SAP or spoke SDP failure, the router sends a C-MAC flush notification for the corresponding B-MAC and ISID. The router receiving the notification flushes all the C-MACs associated with the indicated B-MAC and ISID when the config>service>vpls>bgp-evpn>accept-ivpls-evpn-flush command is enabled for the B-VPLS service.

The C-MAC flush notification consists of an EVPN B-MAC route that is encoded as follows: the ISID to be flushed is encoded in the Ethernet Tag field and the sequence number is incremented with respect to the previously advertised route.

If send-bvpls-evpn-flush is configured on an I-VPLS with SAPs or spoke SDPs, one of the following rules must be observed.

1. The disable-send-bvpls-evpn-flush option is configured on the SAPs or spoke SDPs.
2. The SAPs or spoke SDPs are not on an ES.
3. The SAPs or spoke SDPs are on an ES or vES with no src-bmac-lsb enabled.
4. The no use-es-bmac is enabled on the B-VPLS.

ISID-based C-MAC flush can be enabled in I-VPLS services with ES or vES. If enabled, the expected interaction between the RFC 7623-based C-MAC flush and the ISID-based C-MAC flush is as follows.

1. If send-bvpls-evpn-flush is enabled in an I-VPLS service, the ISID-based C-MAC flush overrides (replaces) the RFC 7623-based C-MAC flushing.
2. For each ES, vES, or B-VPLS, the system checks for at least one I-VPLS service that does not have send-bvpls-evpn-flush enabled.
   1. If ISID-based C-MAC flush is enabled for all I-VPLS services, RFC 7623-based C-MAC flushing is not triggered; only ISID-based C-MAC flush notifications are generated.
   2. If at least one I-VPLS service is found with no ISID-based C-MAC flush enabled, then RFC 7623-based C-MAC flushing notifications are triggered based on ES events.
      ISID-based C-MAC flush notifications are also generated for I-VPLS services that have send-bvpls-evpn-flush enabled.

Figure 184 shows an example where the ISID-based C-MAC flush prevents blackhole situations for a CE that is using BGP-MH as the redundancy mechanism in the I-VPLS with an ISID of 3000.

Figure 184:  Per-ISID C-MAC Flush Following a SAP Failure 

When send-bvpls-evpn-flush is enabled, the I-VPLS service is ready to send per-ISID C-MAC flush messages in the form of B-MAC/ISID routes. The first B-MAC/ISID route for an I-VPLS service is sent with sequence number zero; subsequent updates for the same route increment the sequence number. A B-MAC/ISID route for the I-VPLS is advertised or withdrawn in the following cases:

1. during I-VPLS send-bvpls-evpn-flush configuration and deconfiguration
2. during I-VPLS association and disassociation from the B-VPLS service
3. during I-VPLS operational status change (up/down)
4. during B-VPLS operational status change (up/down)
5. during B-VPLS bgp-evpn mpls status change (no shutdown/shutdown)
6. during B-VPLS operational source B-MAC change
7. if no disable-send-bvpls-evpn-flush is configured for a SAP or spoke SDP, upon a failure on that SAP or spoke SDP, the system will send a per-ISID C-MAC flush message; that is, a B-MAC/ISID route update with an incremented sequence number

If the user explicitly configures disable-send-bvpls-evpn-flush for a SAP or spoke SDP, the system will not send per-ISID C-MAC flush messages for failures on that SAP or spoke SDP.

The B-VPLS on the receiving node must be configured with bgp-evpn>accept-ivpls-evpn-flush to accept and process C-MAC flush non-zero Ethernet-tag MAC routes. If the accept-ivpls-evpn-flush command is enabled (the command is disabled by default), the node accepts non-zero Ethernet-tag MAC routes (B-MAC/ISID routes) and processes them. When a new B-MAC/ISID update (with an incremented sequence number) for an existing route is received, the router will flush all the C-MACs associated with that B-MAC and ISID. The B-MAC/ISID route withdrawals will also cause a C-MAC flush.

Note: Only B-MAC routes with the Ethernet Tag field set to zero are considered for B-MAC installation in the FDB.

The following CLI example shows the commands that enable the C-MAC flush feature on PE1 and PE3.

*A:PE-1>config>service>vpls(i-vpls)# info 

----------------------------------------------

  pbb

    backbone-vpls 10

    send-bvpls-evpn-flush

    exit

  exit

  bgp

    route-distinguisher 65000:1

    vsi-export “vsi_export”

    vsi-import “vsi_import”

  exit

  site “CE-1” create

    site-id 1

    sap lag-1:1

    site-activation-timer 3

    no shutdown

  exit

  sap lag-1:1 create

    no disable-send-bvpls-evpn-flush

    no shutdown

    exit

<snip>

*A:PE-3>config>service>vpls(b-vpls 10)# info 

----------------------------------------------

<snip>

  bgp-evpn

    accept-ivpls-evpn-flush

In the preceding example, with send-bvpls-evpn-flush enabled on the I-VPLS service of PE1, a B-MAC/ISID route (for pbb source-bmac address B-MAC 00:..:01 and ISID 3000) is advertised. If the SAP goes operationally down, PE1 will send an update of the source B-MAC address (00:..:01) for ISID 3000 with a higher sequence number.

With accept-ivpls-evpn-flush enabled on PE3’s B-VPLS service, PE3 flushes all C-MACs associated with B-MAC 00:01 and ISID 3000. The C-MACs associated with other B-MACs or ISIDs are retained in PE3’s FDB.

5.3.6.2.6. PBB-EVPN ISID-based Route Targets

Routers with PBB-EVPN services use the following route types to advertise the ISID of a specific service.

1. Inclusive Multicast Ethernet Tag routes (IMET-ISID routes) are used to auto-discover ISIDs in the PBB-EVPN network. The routes encode the service ISID in the Ethernet Tag field.
2. BMAC-ISID routes are only used when ISID-based C-MAC flush is configured. The routes encode the ISID in the Ethernet Tag field.

Although the preceding routes are only relevant for routers where the advertised ISID is configured, they are sent with the B-VPLS route-target by default. As a result, the routes are unnecessarily disseminated to all the routers in the B-VPLS network.

SR OS supports the use of per-ISID or group of ISID route-targets, which limits the dissemination of IMET-ISID or BMAC-ISID routes for a specific ISID to the PEs where the ISID is configured.

The config>service>(b-)vpls>isid-route-target>isid-range from [to to] [auto-rt | route-target rt] command allows the user to determine whether the IMET-ISID and BMAC-ISID routes are sent with the B-VPLS route-target (default option, no command), or a route-target specific to the ISID or range of ISIDs.

The following configuration example shows how to configure ISID ranges as auto-rt or with a specific route-target.

*A:PE-3>config>service>(b-)vpls>bgp-evpn# 

isid-route-target 

[no] isid-range <from> [to <to>] {auto-rt|route-target <rt>}

/* For example:

*A:PE-3>config>service>(b-)vpls>bgp-evpn# 

isid-route-target

 isid-range 1000 to 1999 auto-rt

 isid-range 2000 route-target target:65000:2000

The auto-rt option auto-derives a route-target per ISID in the following format:

<2-byte-as-number>:<4-byte-value>

Where: 4-byte-value = 0x30+ISID, as described in RFC 7623. Figure 185 shows the format of the auto-rt option.

Figure 185:  PBB-EVPN auto-rt ISID-based Route Target Format  

Where:

1. If it is 2 bytes, then the AS number is obtained from the config>router>autonomous-system command. If the AS number exceeds the 2 byte limit, then the low order 16-bit value is used.
2. A = 0 for auto-derivation
3. Type = 3, which corresponds to an ISID-type route-target
4. ISID is the 24-bit ISID
5. The type and sub-type are 0x00 and 0x02.

If isid-route-target is enabled, the export and import directions for IMET-ISID and BMAC-ISID route processing are modified as follows.

1. Exported IMET-ISID and BMAC-ISID routes
   1. For local I-VPLS ISIDs and static ISIDs, IMET-ISID routes are sent individually with an ISID-based route-target (and without a B-VPLS route-target) unless the ISID is contained in an ISID policy for which no advertise-local is configured.
   2. If both isid-route-target and send-bvpls-evpn-flush options are enabled for an I-VPLS, the BMAC-ISID route is also sent with the ISID-based route-target and no B-VPLS route-target.
   3. The isid-route-target command affects the IMET-ISID and BMAC-ISID routes only. The BMAC-0, IMET-0 (B-MAC and IMET routes with Ethernet Tag == 0), and ES routes are not impacted by the command.
2. Imported IMET-ISID and BMAC-ISID routes
   1. Upon enabling isid-route-target for a specific I-VPLS, the BGP starts importing IMET-ISID routes with ISID-based route-targets, and (assuming the bgp-evpn accept-ivpls-evpn-flush option is enabled) BMAC-ISID routes with ISID-based route-targets.
   2. The new ISID-based RTs are added for import operations when the I-VPLS is associated to the B-VPLS service (and not based on the I-VPLS operational status), or when the static-isid is added.
   3. The system does not maintain a mapping of the route-targets and ISIDs for the imported routes. For example, if I-VPLS 1 and 2 are configured with the isid-route-target option and IMET-ISID=2 route is received with a route-target corresponding to ISID=1, then BGP imports the route and the router processes it.
   4. The router does not check the format of the received auto-derived route-targets. The route is imported as long as the route-target is on the list of RTs for the B-VPLS.
3. If the isid-route-target option is configured for one or more I-VPLS services, the vsi-import and vsi-export policies are blocked in the B-VPLS. BGP peer import and export policies are still allowed. Matching on the export ISID-based route-target is supported.