5. EVPN-VXLAN for layer 3

The EVPN Layer 3 configuration model builds on the model for EVPN routes described in Chapter 4, EVPN-VXLAN for layer-2 and multi-homing. Understanding the concepts in the EVPN-VXLAN Layer-2 chapter is required to understand this chapter.

This chapter addresses connectivity between subnets across multiple Broadcast Domains (BDs) of the same tenant as defined in the EVPN Interface-less (IFL) model [draft-ietf-bess-evpn-prefix-advertisement]. It is based on the advertisement and processing of IP prefixes using EVPN type 5 routes. This chapter defines how CEs or servers can be multi-homed to multiple leaf nodes in an EVPN IFL network. It also describes other EVPN L3 topics such as:

  1. IRB subinterface extensions for EVPN-VXLAN Layer 3
  2. unicast routing PE-CE
  3. layer 3 host mobility

5.1. Applicability

The information and configuration in this chapter are based on SR Linux Release 21.6.

5.2. Overview

Figure 8 shows four leaf routers attached to the same tenant or IP-VRF domain. Servers are connected to different subnets and, therefore, different BDs. The leaf routers provide inter-subnet forwarding by using the EVPN IFL model as defined in [draft-ietf-bess-evpn-prefix-advertisement]. The SR Linux implementation is fully standard and third-party routers, such as LEAF-4, can be connected to the same IP-VRF domain as SR Linux routers.

Figure 8:  EVPN IP-VRF domain in a multi-tenant DC 

The procedures in this chapter define the configuration and operation for:

  1. the EVPN IFL model for EVPN-VXLAN Layer 3 and the EVPN IP prefix routes
  2. multi-homing in an EVPN-VXLAN Layer 3 solution
  3. host route mobility aspects
  4. PE-CE unicast routing on EVPN-VXLAN Layer 3 networks
  5. EVPN-VXLAN Layer 3 feature parity for IPv6 prefixes

5.3. Configuring EVPN-VXLAN IP-VRF domains

Figure 9 shows the configuration of an EVPN-VXLAN IP-VRF-10 distributed in three leaf routers, with different subnets, and multi-homing for SERVER-1.

Figure 9:  Example of EVPN-VXLAN IP-VRF domain 

5.3.1. Preconfiguring the underlay network

Prior to configuring the overlay BD, the underlay connectivity must be configured.

This chapter uses the same underlay configuration defined for EVPN-VXLAN Layer-2 and Multi-Homing. Refer to section  4.3.1 Configuring the underlay network in chapter  4 and review this section to understand the underlay configuration before proceeding.

5.3.2. Configuring the LEAF-3 IP-VRF domain

Once LEAF-3 is pre-configured as defined in Preconfiguring the underlay network, use the following steps to enable EVPN-VXLAN on LEAF-3.

As shown in Figure 9, LEAF-3 is attached to IP-VRF-10 and HOST-3 is connected to BD3. BD3 is mapped to subnet 103.1.1.0/24 and its IRB sub-interface is the default-gateway to all hosts in BD3.

  1. In candidate mode, create the interfaces and bridged sub-interfaces that will connect LEAF-3 BD3 to HOST-3. In this example:
    1. Ethernet-1/2 connects HOST-3 to LEAF-3. Although this interface could be defined untagged, this example configures the interface as tagged (vlan-tagging true).
    2. A sub-interface with index 3 is created under the interface and must be configured as type bridged. Bridged sub-interfaces can be associated to MAC-VRF instances as described in Chapter 4.
    3. The sub-interface uses vlan-id any, which captures any traffic that is not specified in other sub-interfaces of the same interface.
    Note:

    The IRB sub-interface will expect no vlan tags so that traffic forwarded from HOST-3 can be routed. If HOST-3 sends frames tagged with a vlan-id, the frames would be classified in the BD3 context, but the sub-interface will not strip off the vlan tag, and frames will not be routed.

    --{ [FACTORY] + candidate shared default }--[ interface ethernet-1/2 ]-- 
    # info
        admin-state enable
        vlan-tagging true
        subinterface 3 {
            type bridged
            admin-state enable
            vlan {
                encap {
                    single-tagged {
                        vlan-id any
                    }
                }
            }
        }
  2. After creating the access sub-interfaces, create the vxlan-interfaces. This allows MAC-VRFs of the same BD to be connected throughout the IP fabric.
    In this example, no other leaf router is attached to BD3, so no vxlan-interface is needed in BD3. The configuration of the vxlan-interface is only shown for completeness. Reference Chapter 4 for details on vxlan-interfaces and their characteristics.
    Example: vxlan1 vxlan-interface 3 configuration
    --{ [FACTORY] + candidate shared default }--[ tunnel-interface vxlan1 ]--
    # info 
        vxlan-interface 3 {
            type bridged
            ingress {
                vni 3
            }
        }
  3. IP-VRF instances in the leaf routers are also connected by VXLAN tunnels, therefore, vxlan-interfaces of type routed must be created.
    In this example, tunnel-interface vxlan2 vxlan-interface 10 is configured. While the configuration of the routed vxlan-interface is similar to the bridged vxlan-interface, this type ensures a routed vxlan-interface only attaches to an ip-vrf, and a bridged vxlan-interface only attaches to a mac-vrf.
    Example: VXLAN tunnel configuration
    --{ [FACTORY] + candidate shared default }--[ tunnel-interface vxlan2 ]--
    # info
        vxlan-interface 10 {
            type routed
            ingress {
                vni 10
            }
        }
  4. Configure the IRB interface and sub-interface that links BD3 and IP-VRF-10 together to allow packets from/to subnet 103.1.1.0/24 to route to/from remote subnets in the local or remote leaf routers of the same tenant.
    Note that since BD3 is not present in another leaf, the IRB sub-interface is not configured as an anycast-gw sub-interface. However, an operator may want to configure all IRB sub-interfaces as anycast-gw in case the BD is extended later. Reference section 5.3.3, Configuring the IP-VRF Domain on LEAF-2 and LEAF-4 for anycast-gw configuration details.
    Example: IRB configuration
    --{ [FACTORY] + candidate shared default }--[ interface irb0 ]--
    # info
        subinterface 3 {
            ipv4 {
                address 103.1.1.254/24 {
                }
            }
        }
  5. Configure the network-instance type mac-vrf and associate it with the bridged IRB interfaces and the vxlan-interface. In the example that follows, BD3 is connected to HOST-3 and to the IRB sub-interface that is also attached to IP-VRF-10.
    Although the B is not needed, in this example, the bgp-evpn and vxlan configuration is shown for completeness. For details about bgp-vpn, bgp-evpn, and vxlan-interface configuration, refer to chapter 4.
    Example: mac-vrf configuration and bridged interface association
    --{ [FACTORY] + candidate shared default }--[ network-instance BD3 ]--
    # info
        type mac-vrf
        interface ethernet-1/2.3 {
        }
        interface irb0.3 {
        }
        vxlan-interface vxlan1.3 {
        }
        protocols {
            bgp-evpn {
                bgp-instance 1 {
                    admin-state enable
                    vxlan-interface vxlan1.3
                    evi 3
                    ecmp 2
                }
            }
            bgp-vpn {
                bgp-instance 1 {
                    route-target {
                        export-rt target:64500:3
                        import-rt target:64500:3
                    }
                }
            }
        }
  6. Configure IP-VRF-10 with bgp-evpn enabled using the EVPN IFL model. In this example, IP-VRF is configured with the irb0.3 interface for connectivity to BD3 and vxlan2.10. This allows the extension of IP-VRF-10 to remote leaf routers. A loopback interface is configured in the IP-VRF to test connectivity among IP-VRFs of the same tenant.
    Once configured, all local routes learned on IP-VRF-10 route-table will be advertised in IP Prefix routes (or IPv6 Prefix routes for local IPv6 routes).
    Example: Enable EVPN IFL model on IP-VRF-10
    --{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF* ]--
    # info
        network-instance IP-VRF-10 {
            type ip-vrf
            interface irb0.3 {
            }
            interface lo10.2 {
            }
            vxlan-interface vxlan2.10 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        vxlan-interface vxlan2.10
                        evi 10
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:10
                            import-rt target:64500:10
                        }
                    }
                }
            }
        }
    The bgp-vpn instance is configured as described in Chapter 4 (with a network-instance of type ip-vrf). Likewise, the bgp-evpn container enables EVPN in the ip-vrf and associates it to the routed vxlan-interface. The RD and RT can be auto-derived from the evi just like they can in mac-vrfs. Explicitly configured RD/RTs can override the auto-configured ones. Import and export policies are supported.
  7. Review the changes. If correct, commit the changes.
    # commit stay 
    --{ candidate shared default }--[  ]-- 

5.3.3. Configuring the IP-VRF Domain on LEAF-2 and LEAF-4

LEAF-2 and LEAF-4 are configured in the same way as LEAF-3, but with the addition of multi-homing, anycast-gw interfaces, and related configurations.

Use the following procedure to enable EVPN-VXLAN Layer 3 on LEAF-2 and LEAF-4. Considerations for configuring the IRB sub-interfaces (Step 4) are provided in section 5.3.3.1 IRB sub-interface considerations, if needed.

  1. In candidate mode, create the interfaces, and bridged sub-interfaces (including LAG) to connect HOST-2, HOST-4, SERVER-1, and CE-3.
    In this example, Ethernet Segment ES-1 is associated to lag1 and is multi-homed to SERVER-1 lag. LACP is enabled on lag1 (but can be disabled) and the admin-key, system-id-mac, and system-priority match on both LEAF-2 and LEAF-4.
    Example (1 of 2): Interfaces and bridged sub-interfaces configuration (LEAF-2)
    // interfaces in Leaf-2
    --{ [FACTORY] + candidate shared default }--[ interface * ]--
    # info        
        interface ethernet-1/11 {
            description ES-1
            ethernet {
                aggregate-id lag1
            }
        }
        interface ethernet-1/12 {
            description to-host-2
            admin-state enable
            vlan-tagging true
            subinterface 24 {
                type bridged
                vlan {
                    encap {
                        single-tagged {
                            vlan-id any
                        }
                    }
                }
            }
        }
        interface ethernet-1/22 {
            description to-CE-3
            vlan-tagging true
            subinterface 2 {
                type bridged
                admin-state enable
                vlan {
                    encap {
                        single-tagged {
                            vlan-id 2
                        }
                    }
                }
            }    
        }
        interface lag1 {
            admin-state enable
            vlan-tagging true
            subinterface 24 {
                type bridged
                vlan {
                    encap {
                        single-tagged {
                            vlan-id 24
                        }
                    }
                }
            }
            lag {
                lag-type lacp
                member-speed 100G
                lacp {
                    interval FAST
                    lacp-mode ACTIVE
                    admin-key 24
                    system-id-mac 00:00:00:00:00:24
                    system-priority 24
                }
            }
        }
    Example (2 of 2): Interfaces and bridged sub-interfaces configuration (LEAF-4)
    // Interfaces in Leaf-4
    --{ [FACTORY] + candidate shared default }--[ interface * ]--
    # info
        interface ethernet-1/4 {
            description ES-1
            ethernet {
                aggregate-id lag1
            }
        }
        interface ethernet-1/13 {
            description to-host-4
            admin-state enable
            vlan-tagging true
            subinterface 4 {
                type bridged
                admin-state enable
                vlan {
                    encap {
                        single-tagged {
                            vlan-id any
                        }
                    }
                }
            }
        }
        interface lag1 {
            admin-state enable
            vlan-tagging true
            subinterface 24 {
                type bridged
                vlan {
                    encap {
                        single-tagged {
                            vlan-id 24
                        }
                    }
                }
            }
            lag {
                lag-type lacp
                member-speed 100G
                lacp {
                    interval FAST
                    lacp-mode ACTIVE
                    admin-key 24
                    system-id-mac 00:00:00:00:00:24
                    system-priority 24
                }
            }
        }
  2. Create the all-active Ethernet Segment ES-1 that is attached to LEAF-2 and LEAF-4. Details about the ES configuration and operation can be found in Chapter 4. Note that the following example only shows the Ethernet Segment configuration. The bgp-vpn>bgp-instance 1 must also be configured as described in Chapter 4.
    Example: ES configuration
    // ES-1 on Leaf-2
    --{ [FACTORY] + candidate shared default }--[ system network-instance protocols
     evpn ethernet-segments ]--
    # info
        bgp-instance 1 {
            ethernet-segment ES-1 {
                admin-state enable
                esi 01:24:24:24:24:24:24:00:00:01
                interface lag1
                multi-homing-mode all-active
            }
        }
    // ES-1 on Leaf-4
    --{ [FACTORY] + candidate shared default }--[ system network-instance protocols
     evpn ethernet-segments ]--
    A:dut4# info
        bgp-instance 1 {
            ethernet-segment ES-1 {
                admin-state enable
                esi 01:24:24:24:24:24:24:00:00:01
                interface lag1
                multi-homing-mode all-active
            }
        }
  3. Once the access bridged sub-interfaces are created, create the vxlan-interfaces to facilitate connectivity between network-instances across the IP fabric. The mac-vrf network-instances require both vxlan-interfaces of type bridged and ip-vrfs of type routed.
    In the following example, all the vxlan-interfaces for all network-instances on LEAF-2 and LEAF-4 nodes are configured as follows:
    Example: vxlan-interface configuration
    // vxlan-interfaces on Leaf-2
    --{ [FACTORY] + candidate shared default }--[ tunnel-interface * ]--
    # info
        tunnel-interface vxlan1 {
            vxlan-interface 2 {
                type bridged
                ingress {
                    vni 2
                }
            }
            vxlan-interface 24 {
                type bridged
                ingress {
                    vni 24
                }
            }
        }
        tunnel-interface vxlan2 {
            vxlan-interface 10 {
                type routed
                ingress {
                    vni 10
                }
            }
        }
    // vxlan-interfaces on Leaf-4
    --{ [FACTORY] + candidate shared default }--[ tunnel-interface * ]--
    # info
        tunnel-interface vxlan1 {
            vxlan-interface 4 {
                type bridged
                ingress {
                    vni 4
                }
            }
            vxlan-interface 24 {
                type bridged
                ingress {
                    vni 24
                }
            }
        }
        tunnel-interface vxlan2 {
            vxlan-interface 10 {
                type routed
                ingress {
                    vni 10
                }
            }
        }
  4. Configure the IRB sub-interfaces that will link the mac-vrf and ip-vrf network-instances for inter-subnet-forwarding. Refer to section 5.3.3.1 IRB sub-interface considerations, for details on configuring IRB sub-interfaces.
    Example: IRB sub-interface configuration
    // IRB interfaces in Leaf-2
    --{ [FACTORY] + candidate shared default }--[ interface irb* ]--
    # info 
        interface irb0 {
            subinterface 2 {
                ipv4 {
                    address 20.1.1.2/24 {
                    }
                    arp {
                        learn-unsolicited true
                        ]
                    }
                }
                anycast-gw {
                }
            }
            subinterface 24 {
                ipv4 {
                    address 101.1.1.2/24 {
                    }
                    address 101.1.1.254/24 {
                        anycast-gw true
                        primary
                    }
                    address 102.1.1.254/24 {
                    }
                    arp {
                        learn-unsolicited true
                        debug [
                            messages
                        ]
                        host-route {
                            populate dynamic {
                            }
                        }
                        evpn {
                            advertise dynamic {
                            }
                        }
                    }
                }
                anycast-gw {
                }
            }
        }
    // IRB interfaces in Leaf-4
    --{ [FACTORY] + candidate shared default }--[ interface irb* ]--
    # info
        interface irb0 {
            subinterface 4 {
                ipv4 {
                    address 104.1.1.4/24 {
                    }
                    address 104.1.1.254/24 {
                        anycast-gw true
                        primary
                    }
                    arp {
                        learn-unsolicited true
                        debug [
                            messages
                        ]
                    }
                }
                anycast-gw {
                }
            }
            subinterface 24 {
                ipv4 {
                    address 101.1.1.4/24 {
                    }
                    address 101.1.1.254/24 {
                        anycast-gw true
                        primary
                    }
                    arp {
                        learn-unsolicited true
                        debug [
                            messages
                        ]
                        host-route {
                            populate dynamic {
                            }
                        }
                        evpn {
                            advertise dynamic {
                            }
                        }
                    }
                }
                anycast-gw {
                }
            }
  5. Configure the network-instances of type mac-vrf and associate the bridged sub-interfaces, irb sub-interfaces, and vxlan-interfaces. Then enable bgp-evpn.
    Example: network instance configuration and association
    // MAC-VRFs in Leaf-2
    --{ [FACTORY] + candidate shared default }--[ network-instance BD* ]--
    # info
        network-instance BD2 {
            type mac-vrf
            interface ethernet-1/22.2 {
            }
            interface irb0.2 {
            }
            vxlan-interface vxlan1.2 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        admin-state enable
                        vxlan-interface vxlan1.2
                        evi 2
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:2
                            import-rt target:64500:2
                        }
                    }
                }
            }
        }
        network-instance BD24 {
            type mac-vrf
            interface ethernet-1/12.24 {
            }
            interface irb0.24 {
            }
            interface lag1.24 {
            }
            vxlan-interface vxlan1.24 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        admin-state enable
                        vxlan-interface vxlan1.24
                        evi 24
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:24
                            import-rt target:64500:24
                        }
                    }
                }
            }
        }
    // MAC-VRFs in Leaf-4
    --{ [FACTORY] + candidate shared default }--[ network-instance BD* ]--
    # info
        network-instance BD24 {
            type mac-vrf
            interface irb0.24 {
            }
            interface lag1.24 {
            }
            vxlan-interface vxlan1.24 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        admin-state enable
                        vxlan-interface vxlan1.24
                        evi 24
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:24
                            import-rt target:64500:24
                        }
                    }
                }
            }
        }
        network-instance BD4 {
            type mac-vrf
            interface ethernet-1/13.4 {
            }
            interface irb0.4 {
            }
            vxlan-interface vxlan1.4 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        admin-state enable
                        vxlan-interface vxlan1.4
                        evi 4
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:4
                            import-rt target:64500:4
                        }
                    }
                }
            }
        }
  6. Configure IP-VRF-10 with bgp-evpn enabled with the EVPN IFL model. The IP-VRF is configured with the IRB sub-interfaces previously created for BD2, BD4, and BD24. The IP-VRF instances are connected VXLAN tunnels, so routed vxlan-interfaces need to be associated to IP-VRF-10. A loopback interface is configured in the IP-VRF to test connectivity among IP-VRFs of the same tenant.
    In the following example, LEAF-2's IP-VRF is configured with a BGP PE-CE neighbor to CE-3.
    Example: IP-VRF-10 configuration
    // IP-VRF-10 in Leaf-2
    --{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF* ]--
    # info
        network-instance IP-VRF-10 {
            type ip-vrf
            interface irb0.2 {
            }
            interface irb0.24 {
            }
            interface lo10.2 {
            }
            vxlan-interface vxlan2.10 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        vxlan-interface vxlan2.10
                        evi 10
                        ecmp 2
                    }
                }
                bgp {
                    admin-state enable
                    autonomous-system 645002
                    router-id 2.2.2.2
                    group eBGP-PE-CE {
                        admin-state enable
                        export-policy export-all
                        import-policy import-all
                        ipv4-unicast {
                            admin-state enable
                        }
                    }
                    neighbor 20.1.1.3 {
                        peer-as 645003
                        peer-group eBGP-PE-CE
                        local-as 645002 {
                        }
                        transport {
                            local-address 20.1.1.2
                        }
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:10
                            import-rt target:64500:10
                        }
                    }
                }
            }
        }
    // IP-VRF-10 in Leaf-4
    --{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF* ]--
    # info
        network-instance IP-VRF-10 {
            type ip-vrf
            interface irb0.4 {
            }
            interface irb0.24 {
            }
            interface lo10.2 {
            }
            vxlan-interface vxlan2.10 {
            }
            protocols {
                bgp-evpn {
                    bgp-instance 1 {
                        vxlan-interface vxlan2.10
                        evi 10
                        ecmp 2
                    }
                }
                bgp-vpn {
                    bgp-instance 1 {
                        route-target {
                            export-rt target:64500:10
                            import-rt target:64500:10
                        }
                    }
                }
            }
        }
  7. Review the changes. If correct, commit the changes.
    A:dut2# commit stay 
    --{ candidate shared default }--[  ]--                                                                                                                 

5.3.3.1. IRB sub-interface considerations

The following are considerations for configuring IRB sub-interfaces (performed in Step 4).

IRB sub-interfaces on BDs that are distributed to multiple leaf nodes must be configured with at least one anycast-gw IP address and an anycast-gw MAC address. When the anycast-gw container is configured, the anycast-gw MAC address is auto-derived as 00:00:5E:00:01:01 in all the leaf nodes. The MAC address can also be explicitly configured if desired. In addition:

  1. The same anycast-gw IP and MAC address must be configured on all IRBs of the same BD.
  2. All the hosts attached to the same BD use the same default-gateway (that is, the anycast-gw IP) irrespective of the leaf they are connected to. For example, irb0 sub-interfaces 24 are configured with the same anycast-gw IP on LEAF-2 and LEAF-4 (101.1.1.254) in the configuration example.
  3. Default gateway redundancy in DCs is realized using anycast-gws, and not VRRP. Anycast-gws avoid upstream tromboning for hosts that are multi-homed to multiple leaf nodes or for single-homed hosts that move to other leaf nodes of the same BD.

IRB sub-interfaces on BDs that are distributed may also be configured with non-anycast-gw IP addresses. This is only done when separate IPs are needed to check connectivity per leaf. For example, when LEAF-2 is configured with non-anycast-gw IPs 101.1.1.2 and 102.1.1.254, and LEAF-4 is configured with 104.1.1.4. and anycast-gw IPs exist in multiple nodes, the anycast-gw IPs should not be used in ICMP tools to check the availability of a leaf. Non-anycast-gw IPs should be used instead.

IRB sub-interfaces on distributed BDs should be configured with the following commands, as shown for subinterface 24 in LEAF-2 and LEAF-4 in the configuration example:

  1. arp learn-unsolicited true - Triggers the learning of ARP entries out of any ARP packet arriving at the IRB sub-interface, regardless of whether there was an ARP-Request issued from the IRB.
  2. arp host-route populate dynamic - Creates host routes (arp-nd) in the IP-VRF route-table from learned dynamic ARP entries in the IRB. The arp-nd host routes are then advertised to the remote leaf nodes. This assists the routing of downstream traffic to a given host without hair-pinning traffic via another leaf connected to the same BD of the host, but not connected to the host directly.
  3. arp evpn advertise dynamic - Advertises EVPN MAC/IP routes that include the MAC and the IP of the dynamic ARP entries. The advertisement of these routes synchronizes the ARP caches in all the IRB sub-interfaces of the same BD.

IRB sub-interfaces on BDs that are not distributed (that is, BDs attached to only one Leaf node) do not need to be configured with the following:

  1. arp host-route-populate dynamic as downstream routing is always direct to the connected leaf
  2. arp evpn advertise dynamic as ARP entries do not need to be synchronized with any other node

Examples of non-distributed BDs are BD2, BD4, and BD3 as shown in Figure 9. Their corresponding IRB sub-interfaces do not create host-routes or advertise EVPN MAC/IP routes for the ARP entries.

5.3.4. Configuring EVPN IFL interoperability to EVPN IFF unnumbered model

While EVPN IFL for VXLAN is supported by most DC vendors, Nuage WBX or VSC/VRS use the EVPN IFF Unnumbered model. By default, the SR Linux EVPN IFL (interface-less) model does not inter-operate with the EVPN IFF (interface-full) model. However, it is possible to configure the SR Linux EVPN IFL model to inter-operate with the EVPN IFF model.

For more information about EVPN IFL vs EVPN IFF models, see the SR Linux EVPN-VXLAN User Guide and draft-ietf-bess-evpn-prefix-advertisement.

To configure inter-operability in IP-VRF-10, configure the advertise-gateway-mac command as shown in the following example.

Example: EVPN IFF inter-operability in IP-VRF-10 configuration

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# protocols bgp-evpn bgp-instance 1 routes route-table mac-ip 
advertise-gateway-mac <value> ?
usage: advertise-gateway-mac <true|false>
 
If set to true in an ip-vrf where bgp-evpn is enabled, a MAC/IP route
containing the gateway-MAC is advertised.
 
This gateway-MAC matches the MAC advertised along with the EVPN IFL routes
type 5 for the ip-vrf network-instance. This advertisement is needed so that
the EVPN IFL (Interface-Less) model in the ip-vrf can interoperate with a remote
system working in EVPN IFF (Interface-ful) Unnumbered mode.
 
Positional arguments:
  value             [true|false, default false]

When set to true, the node advertises a MAC/IP route using all of the following:

  1. Gateway-mac for IP-VRF-10 (that is, the system-mac)
  2. RD/RT, next-hop, and VNI of IP-VRF-10, and
  3. Null IP address, ESI or Ethernet Tag ID

Example: MAC/IP route advertisement

In this example, the MAC/IP route advertised is from LEAF-3. The MAC address matches the system-mac advertised in any local RT5s.

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# protocols bgp-evpn bgp-instance 1 routes route-table mac-ip advertise-gateway-mac true        
                                               
--{ [FACTORY] +* candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# commit stay
All changes have been committed. Starting new transaction.
2021-04-15T01:18:24.688185-07:00 dut3 local6|DEBU sr_bgp_mgr: bgp|4942|5135|1393922|D:
 VR default (1) Peer 1: 1.1.1.1 UPDATE: Peer 1: 1.1.1.1 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPNandidate shared default
           root (8)  Thu 01:18AM
        NextHop len 4 NextHop 3.3.3.3
        Type: EVPN-MAC Len: 33 RD: 3.3.3.3:10 ESI: ESI-0, tag: 0, mac len: 48 mac:
              00:01:03:ff:00:00, IP len: 0, IP: NULL, label1: 10
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:10
        bgp-tunnel-encap:VXLAN

For IPv6, Nuage WBX devices support two EVPN L3 IPv6 modes: IFF unnumbered and IFF numbered. The SR Linux interoperability mode enabled by the advertise-gateway-mac command only works with devices that use EVPN IFF unnumbered. This is because EVPN IFL and EVPN IFF unnumbered models both use the same format in the IP prefix route, and differ only in the additional MAC/IP route for the gateway-mac. EVPN IFL and EVPN IFF numbered models have different IP prefix route formats, and cannot inter-operate.

5.3.5. Checking the EVPN IFL model in IP-VRFs

Once configured, the state of the IP-VRF-10 on the three leaf nodes and basic connectivity should be checked.

When the leaf nodes attached to IP-VRF-10 exchange at least one EVPN IP-Prefix route on all leaf nodes of the tenant, the bgp_mgr will request the fib_mgr to create a VXLAN tunnel to each next-hop of the received EVPN routes type 5 (RT5s). This assumes the tunnel had not already been created.

When a VXLAN tunnel to the remote VTEP exists, the bgp_mgr requests the next-hop resolution to the fib_mgr, and if it resolves, the RT5 is installed in the IP-VRF route-table. Using LEAF-3 as a reference, you can check that RT5s are received from the two remote leaf nodes, and then verify that VXLAN tunnels exist to their VTEPs and the RT5s are installed in the route-table. Loopbacks are configured on each IP-VRF-10 instance to verify reachability.

Example: Check IP-VRF-10 state and connectivity

The following can check RT5s for the loopbacks 22.22.22.22 and 44.44.44.44 advertised by the remote leaf nodes. You can check that the routes contain the expected IP-VRF-10 VNI, route-target, and the mac-nh which is used as the inner destination MAC when sending VXLAN packets to the prefix.

--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default protocols bgp routes evpn route-type 5 prefix 22.22.22.22/32 detail
--------------------------------------------------------------------------------------------------
Show report for the EVPN routes in network-instance  "default"
-------------------------------------------------------------------------------------------------
Route Distinguisher: 2.2.2.2:10
Tag-ID             : 0
ip-prefix-len      : 32
ip-prefix          : 22.22.22.22/32
neighbor           : 2.2.2.2
Gateway IP         : 0.0.0.0
Received paths     : 1
  Path 1: <Best,Valid,Used,>
    ESI             : 00:00:00:00:00:00:00:00:00:00
    VNI             : 10
    Route source    : neighbor 2.2.2.2 (last modified 46m16s ago)
    Route preference: No MED, LocalPref is 100
    Atomic Aggr     : false
    BGP next-hop    : 2.2.2.2
    AS Path         :  i
    Communities     : [target:64500:10, mac-nh:00:01:02:ff:00:00, bgp-tunnel-encap:VXLAN]
    RR Attributes   : No Originator-ID, Cluster-List is []
    Aggregation     : None
    Unknown Attr    : None
    Invalid Reason  : None
    Tie Break Reason: none
---------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default protocols bgp routes evpn route-type 5 prefix 44.44.44.44/32 summary
---------------------------------------------------------------------------------------------------
Show report for the BGP route table of network-instance "default"
--------------------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
-------------------------------------------------------------------------------------------------
BGP Router ID: 3.3.3.3      AS: 3333      Local AS: 3333
------------------------------------------------------------------------------------------------
Type 5 IP Prefix Routes
+--------+---------------+-----+----------------+----------+----------+-----+---------+
| Status |   Route-      | Tag | IP-address     | neighbor | Next-Hop | VNI | Gateway |
|        | distinguisher | -ID |                |          |          |     |         |
+========+===============+=====+================+==========+==========+=====+=========+
| u*>    | 4.4.4.4:10    | 0   | 44.44.44.44/32 | 4.4.4.4  | 4.4.4.4  | 10  | 0.0.0.0 |
+--------+---------------+-----+----------------+----------+----------+-----+---------+
---------------------------------------------------------------------------------------
1 IP Prefix routes 1 used, 1 valid
---------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Example: Check for VXLAN tunnel creation

Once the routes are correct, the VXLAN tunnels are created.

--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default tunnel-table all
------------------------------------------------------------------------------------------------------
Show report for network instance "default" tunnel table
-------------------------------------------------------------------------------------------------------
+-------------+-----------+-------+-------+--------+------------+----------+--------------------------+
| IPv4 Prefix |   Owner   | Type  | Index | Metric | Preference | Fib-prog |       Last Update        |
+=============+===========+=======+=======+========+============+==========+==========================+
| 2.2.2.2/32  | vxlan_mgr | vxlan | 7     | 0      | 0          | Y        | 2021-04-13T10:43:09.483Z |
| 4.4.4.4/32  | vxlan_mgr | vxlan | 6     | 0      | 0          | Y        | 2021-04-13T10:43:09.144Z |
+-------------+-----------+-------+-------+--------+------------+----------+--------------------------+
-------------------------------------------------------------------------------------------------------
2 VXLAN tunnels, 2 active, 0 inactive
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
Show report for network instance "default" tunnel table
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Example: Check for remote VTEPs and associated destinations

The following commands show the remote VTEPs and the associated destinations. A destination is the combination of the VTEP and VNI that is created when the EVPN routes are received and the VXLAN tunnel is created. IP destinations are created from RT5s.

# show tunnel vxlan-tunnel vtep 2.2.2.2 
----------------------------------------------------------------------------
Show report for vxlan-tunnels vtep
---------------------------------------------------------------------------
VTEP Address: 2.2.2.2
Index       : 202418627561
Last Change : 2021-04-13T11:47:09.000Z
---------------------------------------------------------------------------
Destinations
---------------------------------------------------------------------------
+------------------+-----------------+------------+-----------------------+
| Tunnel Interface | VXLAN Interface | Egress VNI |         Type          |
+==================+=================+============+=======================+
| vxlan1           | 1               | 1          | multicast-destination |
| vxlan2           | 10              | 10         | ip-destination        |
+------------------+-----------------+------------+-----------------------+
---------------------------------------------------------------------------
1 bridged destinations, 1 multicast, 0 unicast, 0 es
1 routed destinations
--------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show tunnel vxlan-tunnel vtep 4.4.4.4
---------------------------------------------------------------------------
Show report for vxlan-tunnels vtep
---------------------------------------------------------------------------
VTEP Address: 4.4.4.4
Index       : 202418627553
Last Change : 2021-04-14T15:40:23.000Z
--------------------------------------------------------------------------
Destinations
---------------------------------------------------------------------------
+------------------+-----------------+------------+-----------------------+
| Tunnel Interface | VXLAN Interface | Egress VNI |         Type          |
+==================+=================+============+=======================+
| vxlan1           | 1               | 1          | multicast-destination |
| vxlan1           | 1               | 1          | unicast-destination   |
| vxlan2           | 10              | 10         | ip-destination        |
+------------------+-----------------+------------+-----------------------+
---------------------------------------------------------------------------
2 bridged destinations, 1 multicast, 1 unicast, 0 es
1 routed destinations
--------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Example: Check IP-VRF-10 route table

The following command checks the IP-VRF-10 route table to ensure all the remote subnets and hosts are received and installed. All interface and local routes are automatically advertised in RT5s. Since ECMP=2 is configured in the IP-VRF-10, there are two ECMP paths for the 101.1.1.0/24 subnet, which is attached to both LEAF-2 and LEAF-4.

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show route-table ipv4-unicast summary
------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10
------------------------------------------------------------------------------------------------
+-----------------+-----+--------+------------+--------+------+--------------------+-----------+
|     Prefix      | ID  | Active | Route Type | Metric | Pref |  Next-hop (Type)   | Next-hop  |
|                 |     |        |            |        |      |                    | Interface |
+=================+=====+========+============+========+======+====================+===========+
| 31.31.31.31/32  | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 20.1.1.0/24     | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 20.1.1.3/32     | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 22.22.22.22/32  | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 33.33.33.33/32  | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 44.44.44.44/32  | 0   | true   | bgp-evpn   | 0      | 170  | 4.4.4.4 (indirect) | None      |
| 101.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
|                 |     |        |            |        |      | 4.4.4.4 (indirect) | None      |
| 101.1.1.1/32    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 102.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 103.1.1.0/24    | 0   | true   | local      | 0      | 0    | 103.1.1.3 (direct) | irb0.3    |
| 103.1.1.1/32    | 0   | true   | arp-nd     | 0      | 1    | 103.1.1.1 (direct) | irb0.3    |
| 103.1.1.3/32    | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 103.1.1.254/32  | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 103.1.1.255/32  | 0   | true   | host       | 0      | 0    | None (broadcast)   | None      |
| 104.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 4.4.4.4 (indirect) | None      |
+-----------------+-----+--------+------------+--------+------+--------------------+-----------+
------------------------------------------------------------------------------------------------
15 IPv4 routes total
15 IPv4 prefixes with active routes
1 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--

Example: Check route-table state for a RT5

Use the following command to check the route-table state for a RT5. This can be useful to understand how the RT5 is resolved to a vxlan tunnel and what vxlan VNI, inner source, and destination MACs are be used when sending VXLAN packets to that route. The following uses ECMP route 101.1.1.0/24 on LEAF-3. The route's next-hop group has two separate next-hops pointing at the remote LEAF-2 and LEAF-4 VTEPs:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# info from state route-table ipv4-unicast route 101.1.1.0/24 id 0 
    route-table {
        ipv4-unicast {
            route 101.1.1.0/24 id 0 {
                route-type bgp-evpn
                route-owner bgp_evpn_mgr
                metric 0
                preference 170
                active true
                last-app-update "a day ago"
                next-hop-group 202418627581
                resilient-hash false
                fib-programming {
                    status success
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# info from state route-table next-hop-group 202418627581 next-hop *
    route-table {
        next-hop-group 202418627581 {
            next-hop 0 {
                next-hop 202418627569
                resolved true
            }
            next-hop 1 {
                next-hop 202418627565
                resolved true
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# info from state route-table next-hop 202418627569 
    route-table {
        next-hop 202418627569 {
            type indirect
            ip-address 2.2.2.2
            resolving-tunnel {
                ip-prefix 2.2.2.2/32
                tunnel-type vxlan
                tunnel-owner vxlan_mgr
            }
            vxlan {
                vni 10
                source-mac 00:01:03:FF:00:00
                destination-mac 00:01:02:FF:00:00
            }
        }
    }

Example: Monitor pings

The following command monitors pings between the local LEAF-3 loopback and LEAF-2's loopback (22.22.22.22), the inner source, and destination MACs that are associated to the RT5's next-hop that are used in the actual packets. Note that the source-mac is the chassis MAC advertised in the mac-nh of the local RT5s:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
// a ping is sent from Leaf-3 to 22.22.22.22 (the loopback on Leaf-2’s IP-VRF-10)
# network-instance IP-VRF-10
 
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# ping 22.22.22.22 
Using network instance IP-VRF-10
PING 22.22.22.22 (22.22.22.22) 56(84) bytes of data.
64 bytes from 22.22.22.22: icmp_seq=1 ttl=64 time=4.88 ms
64 bytes from 22.22.22.22: icmp_seq=2 ttl=64 time=4.76 ms
^C
--- 22.22.22.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 4.767/4.827/4.888/0.092 ms
// the monitor command catches the inner packet sent on VXLAN
--{ [FACTORY] +! candidate shared default }--[ network-instance IP-VRF-10 ]--
# /tools system traffic-monitor destination-address 33.33.33.33 protocol icmp
Running as user "root" and group "root". This could be dangerous.
Capturing on 'monit'
1       0.000000000     ethernet-1/1    00:01:02:ff:00:00       00:01:03:ff:00:00
         22.22.22.22     33.33.33.33    ICMP    146     Echo (ping) reply    id=0x580c,
         seq=1/256, ttl=64
 
// the source MAC is the local chassis mac:
 
--{ [FACTORY] + candidate shared default }--[  ]-- 
# info from state platform chassis mac-address
    platform {
        chassis {
            mac-address 00:01:03:FF:00:00
        }
    }
 
// that source MAC is also advertised in the RT5’s mac-nh
 
--{ [FACTORY] + candidate shared default }--[  ]--
# info from state network-instance default bgp-rib evpn rib-in-out rib-out-post
  ip-prefix-routes 3.3.3.3:10 ethernet-tag-id 0 ip-prefix-length 32 ip-prefix
  33.33.33.33/32
  neighbor 2.2.2.2
    network-instance default {
        bgp-rib {
            evpn {
                rib-in-out {
                    rib-out-post {
                        ip-prefix-routes 3.3.3.3:10 ethernet-tag-id 0 ip-prefix-length 32
                        ip-prefix 33.33.33.33/32 neighbor 2.2.2.2 {
                            esi 00:00:00:00:00:00:00:00:00:00
                            gateway-ip 0.0.0.0
                            vni 10
                            attr-id 132
                            next-hop 3.3.3.3
                        }
                    }
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[  ]--
# info from state network-instance default bgp-rib attr-sets attr-set rib-out index 132
    network-instance default {
        bgp-rib {
            attr-sets {
                attr-set rib-out index 132 {
                    origin igp
                    atomic-aggregate false
                    next-hop 3.3.3.3
                    med 0
                    local-pref 100
                    aggregator {
                    }
                    pmsi-tunnel {
                    }
                    communities {
                        ext-community [
                            target:64500:10
                            mac-nh:00:01:03:ff:00:00
                            bgp-tunnel-encap:VXLAN
                        ]
                    }
                    unknown-attributes {
                    }
                }
            }
        }
    }
r
            }
            vxlan {
                vni 10
                source-mac 00:01:03:FF:00:00
                destination-mac 00:01:02:FF:00:00
            }
        }
    }

The received EVPN IFL IP Prefix routes are only installed in the IP-VRF-10 route-table if:

  1. the import route-target matches the RT5 route-target
  2. the RT5's VNI matches the VNI of the vxlan-interface in the IP-VRF-10
  3. the RT5's gateway-ip is zero

Additional guidelines:

  1. Importing an RT5 into multiple ip-vrf network-instances is not supported due to the VNI restriction: an ip-vrf can only use a single VNI for ingress and egress VXLAN packets. This is a TD3 limitation.
  2. The route next-hop cannot be changed in ip-vrf network-instances. It is always the system-ip in this release.
  3. The ip-vrf bgp-evpn bgp instance can be oper-down for the same reasons as the bgp-evpn bgp instance can be down in mac-vrfs. Refer to Chapter 4 for details.
  4. VXLAN statistics are also accounted for when EVPN-IFL is used.
  5. No MTU checks are done for VXLAN in EVPN-IFL. If the routed packet plus the VXLAN overhead exceeds the underlay interface MTU of the egress interface in the default network-instance, the packet is still encapsulated and sent to the remote leaf. No statistics increment or drops occur.
  6. Outer TTL for VXLAN packets is always initialized to 255 and not copied or propagated from or to the inner IP packet.

5.3.6. Checking PE-CE routing on an IP-VRF with EVPN-IFL

In an EVPN-VXLAN Layer 3 network, PE-CE routing refers to the unicast routing between a CE connected to a BD in a leaf node and the IRB sub-interface of the IP-VRF connected to the same BD. Static or BGP routing is supported in SR Linux. BFD can also be used between the IRB and the CE.

Example: Check PE-CE routing on IP-VRF

Figure 9 depicts a PE-CE BGP session between CE-3 and IP-VRF-10 in LEAF-2. This configuration is needed in IP-VRF-10 to enable a PE-CE BGP session to CE-3.

// IP-VRF-10 in Leaf-2
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF* ]--
# info 
    network-instance IP-VRF-10 {
        type ip-vrf
        interface irb0.2 {
        }
        interface irb0.24 {
        }
        interface lo10.2 {
        }
        vxlan-interface vxlan2.10 {
        }
        protocols {
            bgp-evpn {
                bgp-instance 1 {
                    vxlan-interface vxlan2.10
                    evi 10
                    ecmp 2
                }
            }
            bgp {
                admin-state enable
                autonomous-system 645002
                router-id 2.2.2.2
                group eBGP-PE-CE {
                    admin-state enable
                    export-policy export-all
                    import-policy import-all
                    ipv4-unicast {
                        admin-state enable
                    }
                }
                neighbor 20.1.1.3 {
                    peer-as 645003
                    peer-group eBGP-PE-CE
                    local-as 645002 {
                    }
                    transport {
                        local-address 20.1.1.2
                    }
                }
            }
            bgp-vpn {
                bgp-instance 1 {
                    route-target {
                        export-rt target:64500:10
                        import-rt target:64500:10
                    }
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show protocols bgp neighbor 20.1.1.3 
---------------------------------------------------------------------------------------------------
BGP neighbor summary for network-instance "IP-VRF-10"
Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow
--------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------
+-----------+----------+------------+--------+---------+------------+---------+---------+------------+
|  Net-Inst |    Peer  |   Group    |  Flags | Peer-AS |   State    |  Uptime | AFI/    | [Rx/Active |
|           |          |            |        |         |            |         | SAFI    |    /Tx]    |
+===========+============+==========+========+=========+============+=========+======================+
| IP-VRF-10 | 20.1.1.3 | eBGP-PE-CE | S      | 645003  | established| 1d:21h: | ipv4-   | [2/1/11]   |
|           |          |            |        |         |            | 46m:56s | unicast |            |
+------------+----------+-----------+--------+---------+------------+---------+---------+------------+
-----------------------------------------------------------------------------------------------------
1 configured neighbors, 1 configured sessions are established,0 disabled peers
0 dynamic peers

Example: PE-CE EBGP session: import and export policies

By default, all local routes to the IP-VRF route-table are automatically advertised in EVPN-IFL routes. This includes static routes, local routes, IGP routes, arp-nd host routes, etc. Consider the following for routes coming from or going to a PE-CE EBGP session.

  1. EVPN-IFL to PE-CE EBGP: An export policy must be configured so that EVPN IFL routes can be re-advertised to a CE on the PE-CE BGP session.
  2. PE-CE EBGP to EVPN-IFL: Either an import policy to accept the routes or ebgp-default-policy import-reject-all false must be configured so that the BGP routes are re-advertised to EVPN-IFL.

For example, the following two policies are configured to import and export all routes:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# /info routing-policy policy import-all
    routing-policy {
        policy import-all {
            statement 1 {
                action {
                    accept {
                    }
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# /info routing-policy policy export-all 
    routing-policy {
        policy export-all {
            statement 1 {
                action {
                    accept {
                    }
                }
            }
        }
    }

5.3.6.1. Additional PE-CE considerations

BGP PE-CE sessions can only be established with primary IP addresses. Therefore, in an IRB with both an anycast-gw-ip and a non-anycast-gw-ip, the BGP session can be setup against the non-anycast-gw-ip only if it is configured as primary.

A BGP session is not established if the configured BGP local-address for that session is a non-primary address. Adding a secondary address on an interface where the primary address has established a BGP session is supported.

Example: BGP PE-CE sessions and primary IP addresses

In the following, the local IP address is primary, but not an anycast-gw IP:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# /info from state interface irb0 subinterface 2
    interface irb0 {
        subinterface 2 {
            admin-state enable
            ip-mtu 1500
            name irb0.2
            ifindex 1082130435
            oper-state up
            last-change "a day ago"
            ipv4 {
                allow-directed-broadcast false
                address 20.1.1.2/24 {
                    anycast-gw false
                    origin static
                    primary
                    status preferred
                }
                arp {
                    duplicate-address-detection true
                    timeout 14400
                    learn-unsolicited true
                    debug [
                        messages
                    ]
                    neighbor 20.1.1.3 {
                        link-layer-address 00:01:03:FF:00:0B
                        origin dynamic
                        expiration-time "an hour from now"
                    }
                    host-route {
                        populate dynamic {
                        }
                    }
                    evpn {
                    }
                }
            }
            anycast-gw {
                virtual-router-id 1
                anycast-gw-mac 00:00:5E:00:01:01
                anycast-gw-mac-origin vrid-auto-derived
            }

In SR Linux, the route selection across BGP families (EVPN-IFL vs PE-CE IPv4/v6) occurs based on the route-table preference. For example, if the same prefix 31.31.31.31/32 is received on the IP-VRF-10's route-table via BGP PE-CE (ipv4 family) and via EVPN-IFL, the route with lowest route-table preference will win. By default, the preference for both EVPN-IFL and BGP PE-CE is set to 170. Therefore, for the PE-CE route to be selected, change the preference for the PE-CE routes to a value lower than 170 as shown in the following example.

Example: Changing the route preference

--{ [FACTORY] +* candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# diff             
      protocols {
          bgp {
              preference {
+                 ebgp 160
              }
          }
      }
--{ [FACTORY] +* candidate shared default }--[ network-instance IP-VRF-10 ]--
# commit stay
All changes have been committed. Starting new transaction.
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show route-table ipv4-unicast prefix 31.31.31.31/32 detail
---------------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10
---------------------------------------------------------------------------
Destination   : 31.31.31.31/32
ID            : 0
Route Type    : bgp
Metric        : 0
Preference    : 160
Active        : true
Last change   : 2021-04-15T09:05:53.745Z
Resilient hash: false
--------------------------------------------------------------------------
Next hops: 1 entries
20.1.1.3 (indirect) resolved by 20.1.1.3/32 (arp-nd)
  via 20.1.1.3 (direct) via [irb0.2] 
--------------------------------------------------------------------------
Destination   : 31.31.31.31/32
ID            : 1
Route Type    : bgp-evpn
Metric        : 0
Preference    : 170
Active        : false
Last change   : 2021-04-15T08:58:50.600Z
Resilient hash: false
-----------------------------------------------------------------------------
Next hops: 1 entries
3.3.3.3 (indirect) resolved by None (None)
-----------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 

In R21.6, there is no ECMP across different owners (for instance across EVPN-IFL and PE-CE BGP), only within the same routing owner.

Note the following is not supported:

  1. BGP PE-CE is not supported on CEs that make use of Ethernet Segments, i.e. EVPN Multi-Homing. This is because BGP PE-CE routes resolution using EVPN IFL routes, is blocked. The CE can still use non-EVPN multi-homing (L3 multi-homing). The CE can have a link connected to a sub-interface in LEAF-1/MAC-VRF-1 and another link to a sub-interface in LEAF-2/MAC-VRF-2, because MAC-VRF-1 and MAC-VRF-2 attached to different subnets.
  2. BGP PE-CE is not supported when the BGP session is established via VXLAN. That is, the BGP CE is connected to a leaf that is connected via vxlan to another leaf that hosts the IRB sub-interface where the BGP PE session is established.

5.3.7. Checking multi-homing in an EVPN-VXLAN Layer 3 network

A EVPN-VXLAN Layer 3 network needs to provide a multi-homing solution where upstream and downstream traffic is always routed efficiently, without hair-pinning. As shown in Figure 10, LEAF-2 and LEAF-4 are all-active multi-homed to SERVER-1. The use of IRB anycast-gw IP and MAC addresses, along with the synchronization of MACs and ARPs on the multi-homed leaf nodes, provides efficient routing.

Figure 10:  Example of EVPN-VXLAN layer 3 multi-homing 

5.3.7.1. anycast-gw IPs

The configuration of the anycast-gw must be consistent in the IRB sub-interfaces of LEAF-2 and LEAF-4. This can be checked on the state of the sub-interface:

Example: Check configuration consistency

// Leaf-2 irb0.24 sub-interface
--{ [FACTORY] + candidate shared default }--[  ]-- 
# info from state interface irb0 subinterface 24
    interface irb0 {
        subinterface 24 {
            admin-state enable
            ip-mtu 1500
            name irb0.24
            ifindex 1082130457
            oper-state up
            last-change "2 days ago"
            ipv4 {
                allow-directed-broadcast false
                address 101.1.1.254/24 {
                    anycast-gw true
                    origin static
                    primary
                    status preferred
                }
            anycast-gw {
                virtual-router-id 1
                anycast-gw-mac 00:00:5E:00:01:01
                anycast-gw-mac-origin vrid-auto-derived
            }
// Leaf-4 irb.024 sub-interface
--{ [FACTORY] + candidate shared default }--[  ]--
#  info from state interface irb0 subinterface 24
    interface irb0 {
        subinterface 24 {
            admin-state enable
            ip-mtu 1500
            name irb0.24
            ifindex 1082130457
            oper-state up
            last-change "2 days ago"
            ipv4 {
                allow-directed-broadcast false
                address 101.1.1.254/24 {
                    anycast-gw true
                    origin static
                    primary
                    status preferred
                }
            anycast-gw {
                virtual-router-id 1
                anycast-gw-mac 00:00:5E:00:01:01
                anycast-gw-mac-origin vrid-auto-derived
            }

The anycast-gw-mac address is automatically derived by default as 00:00:5e:00:01:VRID per draft-ietf-bess-evpn-inter-subnet-forwarding. It can also be manually configured. Either way, the anycast-gw IP and MAC must match in the two leaf nodes.

In the next example, HOST-12 is configured with default-gw 101.1.1.254 (the anycast-gw IP address of BD24). When HOST-12 ARPs for the default-gw IP, the ARP Request can be hashed to either leaf. Regardless which leaf gets the ARP Request, the ARP reply contains the anycast-gw MAC. Unicast traffic from HOST-12 can now be hashed to either leaf (for example, LEAF-2 in Figure 10) and the receiving leaf node will always route the traffic directly to LEAF-3 without sending it to the peer leaf first (LEAF-4 in the example). Using no anycast-gw IPs or MAC addresses causes hair-pinning and uses unnecessary spine bandwidth.

Example: anycast-gw IP address resolution

The following shows the resolution of the anycast-gw IP from HOST-12 and upstream routed traffic.

[:host-12]$ arp -n -I veth2
[:host-12]$ 
[:host-12]$ 
[:host-12]$ ping 33.33.33.33
PING 33.33.33.33 (33.33.33.33) 56(84) bytes of data.
02:58:15.782587 00:00:64:01:01:01 > Broadcast, ethertype ARP (0x0806), length 42:
  Request who-has 101.1.1.254 tell 101.1.1.1, length 28
 
02:58:15.787404 00:00:5e:00:01:01 > 00:00:64:01:01:01, ethertype ARP (0x0806),
  length 60: Reply 101.1.1.254 is-at 00:00:5e:00:01:01, length 46
 
02:58:15.787436 00:00:64:01:01:01 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800),
  length 98: 101.1.1.1 > 33.33.33.33: ICMP echo request, id 3140, seq 1, length 64
 
02:58:15.791393 00:00:5e:00:01:01 > 00:00:64:01:01:01, ethertype IPv4 (0x0800),
  length 98: 33.33.33.33 > 101.1.1.1: ICMP echo reply, id 3140, seq 1, length 64
 
64 bytes from 33.33.33.33: icmp_seq=1 ttl=63 time=8.96 ms
 
--- 33.33.33.33 ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8008ms
rtt min/avg/max/mdev = 2.362/3.792/8.962/1.907 ms
[:host-12]$ arp -n -i veth2
Address                  HWtype  HWaddress           Flags Mask            Iface
101.1.1.254              ether   00:00:5e:00:01:01   C                     veth2

As shown in Figure 10, downstream routed traffic from LEAF-3 to HOST-12 is routed directly by LEAF-2 or LEAF-4 without hair-pinning, regardless of who gets the packets. This occurs because HOST-12's ARP and the MAC entries are synchronized in both multi-homed leaf nodes. LEAF-2 learns 101.1.1.1->00:00:64:01:01:01 (host-12 ip and mac) as dynamic and advertises both in MAC/IP routes that are imported by LEAF-4. LEAF-4 installs the HOST-12 ARP and MAC entries as evpn. However, the MAC points at the local ES lag interface, and forwarding is direct to HOST-12.

Example: Synchronization in both multi-home leaf nodes

--{ [FACTORY] + candidate shared default }--[  ]-- 
# show arpnd arp-entries interface irb0 subinterface 24
+-----------+-----------+----------------+-----------+----------------------+------------------+
| Interface | Subinterf |    Neighbor    |  Origin   |  Link layer address  | Expiry           |
|           |    ace    |                |           |                      |                  |
+===========+===========+================+===========+======================+==================+
| irb0      |        24 |      101.1.1.1 |   dynamic | 00:00:64:01:01:01    | 3 hours from now |
| irb0      |        24 |      101.1.1.4 |      evpn | 00:01:04:FF:00:41    |                  |
+-----------+-----------+----------------+-----------+----------------------+------------------+
-----------------------------------------------------------------------------------------------
  Total entries : 2 (0 static, 2 dynamic)
-----------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table all
--------------------------------------------------------------------------------------------------
Mac-table of network instance BD24
---------------------------------------------------------------------------------------------------
+--------------+---------------------+-------+--------+--------+-------+--------------------------+
|   Address    | Destination         | Dest- |  Type  | Active | Aging |      Last Update         |
|              |                     | Index |        |        |       |                          |
+==============+=====================+=======+========+========+=======+==========================+
| 00:00:5E:00: | irb                 | 0     | irb-int| true   | N/A   | 2021-04-13T10:42:14.000Z |
|  01:01       |                     |       | erface-|        |       |                          |
|              |                     |       | anycast|        |       |                          |
| 00:00:64:01: | lag1.24             | 20    | learnt | true   | 285   | 2021-04-15T10:13:11.000Z |
|  01:01       |                     |       |        |        |       |                          |
| 00:00:66:01: | ethernet-1/12.24    | 17    | learnt | true   | 285   | 2021-04-15T10:13:11.000Z |
|  01:01       |                     |       |        |        |       |                          |
| 00:01:02:FF: | irb                 | 0     | irb-int| true   | N/A   | 2021-04-13T10:42:14.000Z |
|  00:41       |                     |       | erface |        |       |                          |
| 00:01:04:FF: | vxlan-interface:    | 202418| evpn-  | true   | N/A   | 2021-04-13T10:42:54.000Z |
|  00:41       |  vxlan1.24          |       |        |        |       |                          |
|              | vtep:4.4.4.4 vni:24 | 653897| static |        |       |                          |
+--------------+---------------------+-------+--------+--------+-------+--------------------------+
Total Irb Macs            :    1 Total    1 Active
Total Static Macs         :    0 Total    0 Active
Total Duplicate Macs      :    0 Total    0 Active
Total Learnt Macs         :    2 Total    2 Active
Total Evpn Macs           :    0 Total    0 Active
Total Evpn static Macs    :    1 Total    1 Active
Total Irb anycast Macs    :    1 Total    1 Active
Total Macs                :    5 Total    5 Active
-------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--   
 
// ARP and MAC entries for Leaf-4
 
--{ [FACTORY] + candidate shared default }--[  ]--
# show arpnd arp-entries interface irb0 subinterface 24
+-----------+-----------+----------------+-----------+----------------------+-------+
| Interface | Subinterf |    Neighbor    |  Origin   |  Link layer address  | Expiry|
|           |    ace    |                |           |                      |       |
+===========+===========+================+===========+======================+=======+
| irb0      |        24 |      101.1.1.1 |      evpn | 00:00:64:01:01:01    |       |
| irb0      |        24 |      101.1.1.2 |      evpn | 00:01:02:FF:00:41    |       |
+-----------+-----------+----------------+-----------+----------------------+-------+
------------------------------------------------------------------------------------
  Total entries : 2 (0 static, 2 dynamic)
------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table all
-------------------------------------------------------------------------------------
Mac-table of network instance BD24
------------------------------------------------------------------------------------
+-------------------+---------------------------+--------+---------+-------+-------+-------------------+
|      Address      |         Destination       |  Dest  |  Type   | Active| Aging |    Last Update    |
|                   |                           |  Index |         |       |       |                   |
+===================+===========================+========+=========+=======+=======+===================+
| 00:00:5E:00:01:01 | irb                       | 0      | irb-int | true  | N/A   | 2021-04-13T10:42: |
|                   |                           |        | erface- |       |       |   38.000Z         |
|                   |                           |        | anycast |       |       |                   |
| 00:00:64:01:01:01 | lag1.24                   | 18     | evpn    | true  | N/A   | 2021-04-15T10:04: |
|                   |                           |        |         |       |       |   12.000Z         |
| 00:00:66:01:01:01 | vxlan-interface:vxlan1.24 | 202418 | evpn    | true  | N/A   | 2021-04-15T10:13: |
|                   | vtep:2.2.2.2 vni:24       | 654989 |         |       |       |   12.000Z         |
| 00:01:02:FF:00:41 | vxlan-interface:vxlan1.24 | 202418 | evpn-   | true  | N/A   | 2021-04-13T10:42: |
|                   | vtep:2.2.2.2 vni:24       | 654989 | static  |       |       |   54.000Z         |
| 00:01:04:FF:00:41 | irb                       | 0      | irb-int | true  | N/A   | 2021-04-13T10:42: |
|                   |                           |        | erface  |       |       |  38.000Z          |
+-------------------+---------------------------+--------+---------+-------+-------+-------------------+
Total Irb Macs            :    1 Total    1 Active
Total Static Macs         :    0 Total    0 Active
Total Duplicate Macs      :    0 Total    0 Active
Total Learnt Macs         :    0 Total    0 Active
Total Evpn Macs           :    2 Total    2 Active
Total Evpn static Macs    :    1 Total    1 Active
Total Irb anycast Macs    :    1 Total    1 Active
Total Macs                :    5 Total    5 Active
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--

5.3.7.2. Non-anycast-gw IP addresses

In addition to using anycast-gw IPs for the routed traffic, the multi-homed leaf nodes also have non-anycast-gw IPs that can be used for ICMP. The examples that follow check the availability of each individual Leaf IRB (LEAF-2 and LEAF-4).

Example: Check LEAF-2 IRB availability

Leaf-2 has a non-anycast-gw IP 101.1.1.2:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# show interfaces irb0.24
====================================================================================
Net instance    : IP-VRF-10
Interface       : irb0.24
Oper state      : up
Ip mtu          : 1500
  Prefix                                    Origin       Status
  ==================================================================================
  101.1.1.2/24                              static       preferred
  101.1.1.254/24                            static       preferred, primary, anycast
  102.1.1.254/24                            static       preferred
  2001:db8:24::254/64                       static       preferred, primary, anycast
  fe80::200:5eff:fe00:101/64                link-layer   preferred, anycast
  fe80::201:2ff:feff:41/64                  link-layer   preferred
====================================================================================
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--

Example: Check LEAF-4 IRB availability

LEAF-4 has a non-anycast-gw IP 101.1.1.4 in the same IRB:

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# show interfaces irb0.24
=====================================================================================
Net instance    : IP-VRF-10
Interface       : irb0.24
Oper state      : up
Ip mtu          : 1500
  Prefix                                    Origin       Status
  ===================================================================================
  101.1.1.4/24                              static       preferred
  101.1.1.254/24                            static       preferred, primary, anycast
  fe80::200:5eff:fe00:101/64                link-layer   preferred, anycast
  fe80::201:4ff:feff:41/64                  link-layer   preferred
====================================================================================
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--

Both non-anycast-gw IPs are reachable from HOST-12. ARP Requests to non-anycast-gw IPs reply with the chassis mac of the leaf and not with the anycast-gw MAC of the IRB. This allows using the non-anycast-gw IPs for troubleshooting purposes when there are anycast-gw IPs on the same IRBs. The example output from HOST-12 demonstrates this:

Example: non-anycast-gw IPs reachability from host

[:host-12]$ arp -n -i veth2
Address                  HWtype  HWaddress           Flags Mask            Iface
101.1.1.254              ether   00:00:5e:00:01:01   C                     veth2
 
[:host-12]$ ping 101.1.1.2
PING 101.1.1.2 (101.1.1.2) 56(84) bytes of data.
 
03:25:41.291765 00:00:64:01:01:01 > Broadcast, ethertype ARP (0x0806), length 42: 
  Request who-has 101.1.1.2 tell 101.1.1.1, length 28
 
03:25:41.295105 00:01:02:ff:00:41 > 00:00:64:01:01:01, ethertype ARP (0x0806), 
   length 60: Reply 101.1.1.2 is-at 00:01:02:ff:00:41, length 46
 
03:25:41.295130 00:00:64:01:01:01 > 00:01:02:ff:00:41, ethertype IPv4 (0x0800), 
   length 98:101.1.1.1 > 101.1.1.2: ICMP echo request, id 3307, seq 1, length 64
 
03:25:41.299204 00:01:02:ff:00:41 > 00:00:64:01:01:01, ethertype IPv4 (0x0800), 
   length 98: 101.1.1.2 > 101.1.1.1: ICMP echo reply, id 3307, seq 1, length 64
 
64 bytes from 101.1.1.2: icmp_seq=1 ttl=64 time=7.59 ms
 
--- 101.1.1.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 2.073/3.684/7.596/2.269 ms
 
[:host-12]$ arp -n -i veth2
Address                  HWtype  HWaddress           Flags Mask            Iface
101.1.1.254              ether   00:00:5e:00:01:01   C                     veth2
101.1.1.2                ether   00:01:02:ff:00:41   C                     veth2
[:host-12]$ ping 101.1.1.4
PING 101.1.1.4 (101.1.1.4) 56(84) bytes of data.
 
03:25:52.696934 00:00:64:01:01:01 > Broadcast, ethertype ARP (0x0806), length 42:
   Request who-has 101.1.1.4 tell 101.1.1.1, length 28
 
03:25:52.700615 00:01:04:ff:00:41 > 00:00:64:01:01:01, ethertype ARP (0x0806), 
   length 60: Reply 101.1.1.4 is-at 00:01:04:ff:00:41, length 46
 
03:25:52.700649 00:00:64:01:01:01 > 00:01:04:ff:00:41, ethertype IPv4 (0x0800), 
   length 98: 101.1.1.1 > 101.1.1.4: ICMP echo request, id 3318, seq 1, length 64
 
03:25:52.703463 00:01:04:ff:00:41 > 00:00:64:01:01:01, ethertype IPv4 (0x0800), 
   length 98: 101.1.1.4 > 101.1.1.1: ICMP echo reply, id 3318, seq 1, length 64
 
64 bytes from 101.1.1.4: icmp_seq=1 ttl=64 time=6.64 ms
 
--- 101.1.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.200/3.821/6.648/2.006 ms
 
[:host-12]$ arp -n -i veth2
Address                  HWtype  HWaddress           Flags Mask            Iface
101.1.1.4                ether   00:01:04:ff:00:41   C                     veth2
101.1.1.254              ether   00:00:5e:00:01:01   C                     veth2
101.1.1.2                ether   00:01:02:ff:00:41   C                     veth2

5.3.7.3. Additional anycast gateway considerations

The following guidelines also apply for using anycast-gw in SR Linux.

In a bgp-evpn-enabled MAC-VRF with an IRB sub-interface, the following applies regardless if the IPs are configured as primary, anycast-gw, or neither of these.

  1. All IPv4 and IPv6 addresses associated to the IRB sub-interface are advertised in separate MAC/IP routes.
  2. The anycast-gw-mac and its corresponding anycast-gw IP address are advertised in a MAC/IP route.
  3. Any other existing non-anycast-gw IP will be advertised along with the interface MAC (hw-mac) in a MAC/IP route.
    For example, if irb0.24 is configured in LEAF-2 with anycast-gw (ip,mac)=(101.1.1.254/24, 00:5e:00:00:01:01), and 101.1.1.2/24 is also configured as non-anycast-gw IP, two MAC/IP routes are advertised in the context of BD24: MAC/IP route (101.1.1.254, 00:5e:00:00:01:01), and MAC/IP route (101.1.1.2, hw-mac-1).

For IPv6, Local Link Addresses (LLDs) are also advertised in addition to global addresses.

When the IRB sub-interface is admin disabled, the IRB MAC addresses are removed from the mac-table (and withdrawn from EVPN). ARP Requests and Neighbor Solicitation messages for the IRB sub-interface IP addresses from the hosts connected to the Broadcast Domain are only processed when coming from local sub-interfaces. These messages cannot be processed when received over VXLAN, so each of the Leaf routers attached to the same BD need to have their anycast IRB sub-interface operationally up to process the requests for the local hosts.

The IRB MAC addresses are protected in the mac-table if they are not anycast-gw-MACs. Protection means that received frames are dropped if their MAC SA match a protected MAC. The mac-table state shows the protection flag per MAC.

Example: mac-table state

--{ [FACTORY] + candidate shared default }--[  ]--
# info from state network-instance BD24 bridge-table mac-table mac *
    network-instance BD24 {
        bridge-table {
            mac-table {
                mac 00:00:5E:00:01:01 {
                    destination-type irb-interface
                    destination-index 0
                    type irb-interface-anycast
                    last-update "2 days ago"
                    destination irb
                    is-protected false
                }
                mac 00:01:02:FF:00:41 {
                    destination-type irb-interface
                    destination-index 0
                    type irb-interface
                    last-update "2 days ago"
                    destination irb
                    is-protected true
                }
                mac 00:01:04:FF:00:41 {
                    destination-type vxlan
                    destination-index 202418653897
                    type evpn-static
                    last-update "2 days ago"
                    destination "vxlan-interface:vxlan1.24 vtep:4.4.4.4 vni:24"
                    is-protected true
                }
            }
        }
    }

5.4. Testing and checking Layer 3 host mobility

In EVPN-VXLAN Layer 3 networks, multiple leaf nodes are attached to the same BD. Hosts of the same subnet can be connected to any of those leaf nodes. They can also move between leaf nodes of the same BD. In either case, the upstream and downstream traffic must be efficient and avoid hair-pinning. This is shown in Figure 11, where LEAF-2 and LEAF-4 configurations are modified (no ES) and HOST-12 was originally connected to LEAF-2.

Upstream traffic from HOST-12 to HOST-3 must be routed by LEAF-2 to LEAF-3 directly. If HOST-12 later moves to LEAF-4, upstream traffic to HOST-3 must be routed by LEAF-4 to LEAF-3 directly. This is accomplished using anycast-gw IPs and MACs on the IRB interfaces.

When HOST-12 is attached to LEAF-2, downstream traffic from HOST-3 must be sent from LEAF-3 to LEAF-2 directly. If HOST-12 later moves to LEAF-4, the routers need to update their tables quickly so that LEAF-3 routes the traffic to LEAF-4 directly, and no bandwidth is wasted on the spines due to unnecessary hair-pinning. This is achieved by learning HOST-12's IP address in the route-table of the connected leaf as a /32 route and advertising that host route in an EVPN IFL route.

Upon a mobility event to LEAF-4, LEAF-2 will withdraw the host route as fast as possible and LEAF-4 will then advertise the HOST-12 host route in an EVPN IFL route.

Figure 11:  Example of L3 host mobility 

5.4.1. Initial configuration - efficient host routing

In the initial configuration, HOST-12 is connected to LEAF-2. For LEAF-3 to route traffic (to HOST-12) directly to LEAF-2, LEAF-2 needs to learn HOST-12's IP and advertise its host route in an EVPN-IFL route.

In next example, the following parameter definitions apply.

  1. learn-unsolicited true - Triggers the node to snoop/process all solicited and unsolicited ARP messages received on sub-interfaces (no vxlan) and learns the corresponding ARP/ND entries as 'dynamic'. By default, the command is false and only solicited entries are learned, which does not guarantee host mobility.
    When enabled, dynamic ARP/ND entries are learned from the following messages received on the sub-interfaces (if the IPs fall into the local subnets):
    1. ARP and Neighbor Solicitation requests
    2. Gratuitous ARP requests and unsolicited neighbor advertisements
  2. host-route populate dynamic -Triggers the creation of arp-nd host routes in the IP-VRF-10 route-table out of dynamic ARP entries. These are disabled by default. The arp-nd host routes are not installed in the FIB, They are only used in the control plane and advertised to the EVPN-IFL network to attract traffic from LEAF-3. The arp-nd host routes can be exported in any routing protocol, such as EVPN-IFL routes, BGP IPv4/IPv6 routes, OSPF, and ISIS. They are supported in network-instances ip-vrf and default.
  3. evpn advertise dynamic - Triggers the advertisement of EVPN MAC/IP routes for the dynamic learned ARP entries and allows the synchronization of the ARP entries in all IRB sub-interfaces of the same BD. This is only supported on IRB sub-interfaces.
    The MAC/IP routes that are advertised for ARP/ND entries will contain the S bit set if the corresponding MAC entry in the mac-table is static.

Note that an equivalent command can be used for ND entries.

Example: Efficient host routing model

# subinterface 24 
--{ [FACTORY] + candidate shared default }--[ interface irb0 subinterface 24 ]--
# info
    ipv4 {
        address 101.1.1.2/24 {
        }
        address 101.1.1.254/24 {
            anycast-gw true
            primary
        }
        arp {
            learn-unsolicited true
            debug [
                messages
            ]
            host-route {
                populate dynamic {
                }
            }
            evpn {
                advertise dynamic {
                }
            }
        }
    }
    anycast-gw {
    }

The next examples show how an ARP Request from HOST-12 to a random IP in the subnet is enough for the irb0.24 to learn the dynamic ARP. It can then create a host route that is advertised as an EVPN IFL route, and imported by LEAF-3.

Example: LEAF-2 - HOST-12 unsolicited ARP Request

--{ [FACTORY] + candidate shared default }--[  ]--
# show arpnd arp-entries interface irb0 subinterface 24
+-----------+--------------+----------+-------+--------------------+------+
| Interface | Subinterface | Neighbor | Origin| Link layer address |Expiry|
+===========+==============+==========+=======+====================+======+
| irb0      |      24      |101.1.1.4 |  evpn | 00:01:04:FF:00:41  |      |
+-----------+--------------+----------+-------+--------------------+------+
--------------------------------------------------------------------------
  Total entries : 1 (0 static, 1 dynamic)
--------------------------------------------------------------------------

Example: Debug messages - ARP request received

2021-04-15T04:58:05.651861-07:00 dut2 local6|INFO sr_arp_nd_mgr: arpnd|1773|1773|19334|I:
  Received ARP request on interface irb0.24 (10247 - 22) from datapath. Source Mac :
  00:00:64:01:01:01 Source IP : 101.1.1.1 Target Mac : 00:00:00:00:00:00 Target IP :
  101.1.1.200

Example: Triggered learning of RT5 and RT2 advertisements

2021-04-15T04:58:06.019955-07:00 dut2 local6|DEBU sr_bgp_mgr: bgp|4933|5176|2128215|D:
  VR default (1) Peer 1: 3.3.3.3 UPDATE: Peer 1: 3.3.3.3 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 85
    Flag: 0x90 Type: 14 Len: 48 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 2.2.2.2
        Type: EVPN-IP-PREFIX Len: 34 RD: 2.2.2.2:10, tag: 0, ip_prefix: 101.1.1.1/32 
          gw_ip 0.0.0.0 Label: 10 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:10
        mac-nh:00:01:02:ff:00:00
        bgp-tunnel-encap:VXLAN
2021-04-15T04:58:06.020003-07:00 dut2 local6|DEBU sr_bgp_mgr: bgp|4933|5176|2128216|D:
  VR default (1) Peer 1: 3.3.3.3 UPDATE: Peer 1: 3.3.3.3 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 97
    Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 2.2.2.2
        Type: EVPN-MAC Len: 37 RD: 2.2.2.2:24 ESI: ESI-0, tag: 0, mac len: 48 mac:
        00:00:64:01:01:01, IP len: 4, IP: 101.1.1.1, label1: 24 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:24
        bgp-tunnel-encap:VXLAN
--{ [FACTORY] + candidate shared default }--[  ]--
# show arpnd arp-entries interface irb0 subinterface 24
+---------+----------+----------+---------+-------------------+------------------+
|Interface| Sub-     | Neighbor | Origin  | Link layer        |    Expiry        |
|         |interface |          |         |  address          |                  |
+=========+==========+==========+=========+===================+==================+
| irb0    |   24     |101.1.1.1 | dynamic | 00:00:64:01:01:01 | 3 hours from now |
| irb0    |   24     |101.1.1.4 |  evpn   | 00:01:04:FF:00:41 |                  |
+---------+----------+----------+---------+-------------------+------------------+
---------------------------------------------------------------------------------
  Total entries : 2 (0 static, 2 dynamic)
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show route-table ipv4-unicast prefix 101.1.1.1/32 detail 
--------------------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10
-----------------------------------------------------------------------------
Destination   : 101.1.1.1/32
ID            : 0
Route Type    : arp-nd
Metric        : 0
Preference    : 1
Active        : true
Last change   : 2021-04-15T11:58:05.653Z
Resilient hash: false
---------------------------------------------------------------------------------
Next hops: 1 entries
101.1.1.1 (direct) via [irb0.24] 
----------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]—

Example: LEAF-3 imports routes as bgp-evpn host route

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# show route-table ipv4-unicast summary
------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10
------------------------------------------------------------------------------------------
+----------------+------+-------+----------+-------+-----+--------------------+----------+
|     Prefix     |  ID  |Active | Route    | Metric| Pref|  Next-hop          | Next-hop |
|                |      |       |  Type    |       |     |   (Type)           |Interface |
+================+======+=======+==========+=======+=====+====================+==========+
| 20.1.1.0/24    | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 20.1.1.3/32    | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 22.22.22.22/32 | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 31.31.31.31/32 | 0    | true  | host     | 0     | 0   | None (extract)     | None     |
| 31.31.31.31/32 | 1    | false | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 33.33.33.33/32 | 0    | true  | host     | 0     | 0   | None (extract)     | None     |
| 44.44.44.44/32 | 0    | true  | bgp-evpn | 0     | 170 | 4.4.4.4 (indirect) | None     |
| 101.1.1.0/24   | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
|                |      |       |          |       |     | 4.4.4.4 (indirect) | None     |
| 101.1.1.1/32   | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 102.1.1.0/24   | 0    | true  | bgp-evpn | 0     | 170 | 2.2.2.2 (indirect) | None     |
| 103.1.1.0/24   | 0    | true  | local    | 0     | 0   | 103.1.1.3 (direct) | irb0.3   |
| 103.1.1.1/32   | 0    | true  | arp-nd   | 0     | 1   | 103.1.1.1 (direct) | irb0.3   |
| 103.1.1.3/32   | 0    | true  | host     | 0     | 0   | None (extract)     | None     |
| 103.1.1.254/32 | 0    | true  | host      | 0    | 0   | None (extract)     | None     |
| 103.1.1.255/32 | 0    | true  | host      | 0    | 0   | None (broadcast)   | None     |
| 104.1.1.0/24   | 0    | true  | bgp-evpn  | 0    | 170 | 4.4.4.4 (indirect) | None     |
+----------------+------+-------+-----------+------+-----+--------------------+----------+
-----------------------------------------------------------------------------------------
16 IPv4 routes total
15 IPv4 prefixes with active routes
1 IPv4 prefixes with active ECMP routes
----------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--

5.4.2. Mobility event - efficient host routing

When HOST-12 is attached to LEAF-2, the ARP entry must be maintained even if HOST-12 does not send any traffic. If the entry is removed or ages out, the associated arp-nd host route in IP-VRF-10 is removed and the EVPN-IFL route withdrawn. This can cause hair-pinning for traffic routed from LEAF-3. To maintain the HOST-12 ARP entry (and other dynamic ARP/ND entries), the system supports timer-based ARP/ND refreshes (ARP-Request for the host IP).

Timer-based refreshes are triggered 30 seconds before the ARP age-out timer expires, and irrespective of the arrival of packets requiring resolution for the entry. Note that in SROS, the arp-proactive-refresh command is needed so that entries are always refreshed irrespective of the arrival of packets that hit the entry. In SR Linux, this is the default behavior, so there is no command to enable the timer-based refreshes.

When HOST-12 moves from LEAF-2 to LEAF-4, LEAF-4 must advertise the host route for 101.1.1.1/32 in EVPN-IFL as fast as possible and LEAF-2 withdraws its EVPN-IFL route for it. The process used by LEAF-2 and LEAF-4 to update their ARP/route-tables once HOST-12 moves between them is called "EVPN Layer 3 host mobility". SR Linux provides this support per section 4 of draft-ietf-bess-evpn-inter-subnet-forwarding. EVPN Layer 3 host mobility supports the three cases specified in the draft:

  1. HOST-12 moves to LEAF-4 and generates a GARP
  2. HOST-12 moves to LEAF-4 and generates traffic, but not ARP
  3. HOST-12 moves to LEAF-4 and remains silent

To support fast mobility, SR Linux supports triggered refreshes. Triggered refreshes (ARP-Requests on events and not based on timer expiration) are issued from irb0.24 leaf nodes, for the existing dynamic ARP entry 101.1.1.1>00:00:64:01:01:01. The following events apply:

  1. an EVPN MAC/IP route for 101.1.1.1-> 00:00:64:01:01:01 is received
  2. an EVPN route for 00:00:64:01:01:01 (no IP) is received
  3. 00:00:64:01:01:01 ages out in the mac-table (or the entry in the MAC table is cleared manually)

As shown in Figure 11, when HOST-12 moves to LEAF-4, and if it issues a GARP or ethernet traffic, the advertised routes immediately updates the ARP/route tables on both leaf nodes. LEAF-3 then changes its next-hop for HOST-12 from LEAF-2 to LEAF-4.

Example: Silent move - HOST-2 initially attached to LEAF-2

--{ [FACTORY] + candidate shared default }—[  ]--
# show arpnd arp-entries interface irb0 subinterface 24 ipv4-address 101.1.1.1
       
+---------+----------+-----------+---------+-------------------+-----------------+
|Interface| Sub-     |  Neighbor | Origin  | Link layer        |     Expiry      |
|         |interface |           |         |  address          |                 |
+=========+==========+===========+=========+===================+=================+
| irb0    |  24      | 101.1.1.1 | dynamic | 00:00:64:01:01:01 | 3 hours from now|
+---------+----------+-----------+---------+-------------------+-----------------+
---------------------------------------------------------------------------------
  Total entries : 1 (0 static, 1 dynamic)
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table mac 00:00:64:01:01:01
---------------------------------------------------------------------------------
Mac-table of network instance BD24
---------------------------------------------------------------------------------
Mac                     : 00:00:64:01:01:01
Destination             : lag1.24
Dest Index              : 20
Type                    : learnt
Programming Status      : Success
Aging                   : 2680
Last Update             : 2021-04-15T14:32:45.000Z
Duplicate Detect time   : N/A
Hold down time remaining: N/A
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--

Example: Silent move - initial LEAF-4

 --{ [FACTORY] + candidate shared default }--[  ]-- 
# show arpnd arp-entries interface irb0 subinterface 24 ipv4-address 101.1.1.1
+---------+----------+-----------+---------+-------------------+-----------------+
|Interface| Sub-     |  Neighbor | Origin  | Link layer        |     Expiry      |
|         |interface |           |         |  address          |                 |
+=========+==========+===========+=========+===================+=================+
| irb0    |  24      | 101.1.1.1 | evpn    | 00:00:64:01:01:01 |                 |
+---------+----------+-----------+---------+-------------------+-----------------+
---------------------------------------------------------------------------------
  Total entries : 1 (0 static, 1 dynamic)
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table mac 00:00:64:01:01:01
---------------------------------------------------------------------------------
Mac-table of network instance BD24
---------------------------------------------------------------------------------
Mac                     : 00:00:64:01:01:01
Destination             : vxlan-interface:vxlan1.24 vtep:2.2.2.2 vni:24
Dest Index              : 202418654989
Type                    : evpn
Programming Status      : Success
Aging                   : N/A
Last Update             : 2021-04-15T14:15:00.000Z
Duplicate Detect time   : N/A
Hold down time remaining: N/A
-----------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]—

Example: Silent move - watch command output for LEAF-3

Every 2.0s: show route-table ipv4-unicast prefix 101.1.1.1/32 detail 
 (Executions 903, Thu 07:41:40AM)
                                                              
--------------------------------------------------------------------- 
IPv4 Unicast route table of network instance IP-VRF-10
--------------------------------------------------------------------- 
Destination   : 101.1.1.1/32                            
ID            : 0 
Route Type    : bgp-evpn 
Metric        : 0    
Preference    : 170  
Active        : true  
Last change   : 2021-04-15T14:15:00.450Z 
Resilient hash: false 
------------------------------------------------------------------
Next hops: 1 entries
2.2.2.2 (indirect) resolved by None (None)
------------------------------------------------------------------

Example: Silent move - move HOST-12 to LEAF-2

In this example, HOST-12 is moved to LEAF-4 to simulate a silent move. Immediately after flushing MAC 00:00:64:01:01:01 in LEAF-2, the MAC/IP routes are withdrawn and LEAF-2 issues three triggered refreshes.

--{ [FACTORY] + candidate shared default }--[  ]--
#  2021-04-15T07:43:16.422816-07:00 dut2 local6|INFO sr_arp_nd_mgr: arpnd|1773|1773|20438|I:
  Sending ARP request on interface irb0.24 (10247 - 22). Source Mac : 00:00:5E:00:01:01
  Source IP : 101.1.1.254 Target Mac : 00:00:00:00:00:00 Target IP : 101.1.1.1 Ethernet
  SA 00:00:5E:00:01:01 Ethernet DA FF:FF:FF:FF:FF:FF
 
2021-04-15T07:43:16.422816-07:00 dut2 local6|INFO sr_arp_nd_mgr: arpnd|1773|1773|20438|I:
  Sending ARP request on interface irb0.24 (10247 - 22). Source Mac : 00:00:5E:00:01:01
  Source IP : 101.1.1.254 Target Mac : 00:00:00:00:00:00 Target IP : 101.1.1.1 Ethernet
  SA 00:00:5E:00:01:01 Ethernet DA FF:FF:FF:FF:FF:FF
 
2021-04-15T07:43:16.422816-07:00 dut2 local6|INFO sr_arp_nd_mgr: arpnd|1773|1773|20438|I:
  Sending ARP request on interface irb0.24 (10247 - 22). Source Mac : 00:00:5E:00:01:01
  Source IP : 101.1.1.254 Target Mac : 00:00:00:00:00:00 Target IP : 101.1.1.1 Ethernet
  SA 00:00:5E:00:01:01 Ethernet DA FF:FF:FF:FF:FF:FF
 
    Flag: 0x90 Type: 15 Len: 77 Multiprotocol Unreachable NLRI:
        Address Family EVPN
        Type: EVPN-MAC Len: 37 RD: 2.2.2.2:24 ESI: ESI-0, tag: 0, mac len: 48 mac:
           00:00:64:01:01:01, IP len: 4, IP: 101.1.1.1, label1: 0 
        Type: EVPN-MAC Len: 33 RD: 2.2.2.2:24 ESI: ESI-0, tag: 0, mac len: 48 mac:
           00:00:64:01:01:01, IP len: 0, IP: NULL, label1: 0 

Example: Silent move - LEAF-2 updates

When the refreshes arrive at HOST-12 in LEAF-4, the ARP reply is consumed by LEAF-4 (since the MAC destination address matches the anycast-gw MAC address). LEAF-4 then advertises the MAC/IP routes and IP Prefix route for HOST-12.

 Type: EVPN-IP-PREFIX Len: 34 RD: 4.4.4.4:10, tag: 0, ip_prefix: 101.1.1.1/32
          gw_ip 0.0.0.0 Label: 10 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:10
        mac-nh:00:01:04:ff:00:00
        bgp-tunnel-encap:VXLAN
2021-04-15T07:43:18.763551-07:00 dut2 local6|DEBU sr_bgp_mgr: bgp|4933|5176|2169872|D:
   VR default (1) Peer 1: 4.4.4.4 UPDATE: Peer 1: 4.4.4.4 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 97
    Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 4.4.4.4
        Type: EVPN-MAC Len: 37 RD: 4.4.4.4:24 ESI: ESI-0, tag: 0, mac len: 48 mac:
          00:00:64:01:01:01, IP len: 4, IP: 101.1.1.1, label1: 24 
        Type: EVPN-MAC Len: 33 RD: 4.4.4.4:24 ESI: ESI-0, tag: 0, mac len: 48 mac:
          00:00:64:01:01:01, IP len: 0, IP: NULL, label1: 24 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:24
        bgp-tunnel-encap:VXLAN
        Type: EVPN-IP-PREFIX Len: 34 RD: 4.4.4.4:10, tag: 0, ip_prefix:
          101.1.1.1/32 gw_ip 0.0.0.0 Label: 10 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:10
        mac-nh:00:01:04:ff:00:00
        bgp-tunnel-encap:VXLAN
2021-04-15T07:43:18.766923-07:00 dut2 local6|INFO sr_arp_nd_mgr:
   arpnd|1773|1773|20450|I: The ARP entry for 101.1.1.1 has been updated.

After the move, LEAF-2 and LEAF-4 tables are updated, and LEAF-3 points at LEAF-4 as the next-hop for the HOST-12 route.

Example: Silent move - LEAF-2 tables

 --{ [FACTORY] + candidate shared default }--[  ]--
# show arpnd arp-entries interface irb0 subinterface 24 ipv4-address 101.1.1.1
               
+---------+----------+-----------+---------+-------------------+-----------------+
|Interface| Sub-     |  Neighbor | Origin  | Link layer        |     Expiry      |
|         |interface |           |         |  address          |                 |
+=========+==========+===========+=========+===================+=================+
| irb0    |  24      | 101.1.1.1 | evpn    | 00:00:64:01:01:01 |                 |
+---------+----------+-----------+---------+-------------------+-----------------+
----------------------------------------------------------------------------------
  Total entries : 1 (0 static, 1 dynamic)
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table mac 00:00:64:01:01:01
                 
--------------------------------------------------------------------------------
Mac-table of network instance BD24
--------------------------------------------------------------------------------
Mac                     : 00:00:64:01:01:01
Destination             : vxlan-interface:vxlan1.24 vtep:4.4.4.4 vni:24
Dest Index              : 202418653897
Type                    : evpn
Programming Status      : Success
Aging                   : N/A
Last Update             : 2021-04-15T14:43:18.000Z
Duplicate Detect time   : N/A
Hold down time remaining: N/A
--------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--

Example: Silent move - LEAF-4 tables

--{ [FACTORY] + candidate shared default }--[  ]--
# show arpnd arp-entries interface irb0 subinterface 24 ipv4-address 101.1.1.1 
                           
+---------+----------+-----------+---------+------------------+------------------+
|Interface| Sub-     |  Neighbor | Origin  | Link layer       |     Expiry       |
|         |interface |           |         |  address         |                  |
+=========+==========+===========+=========+==================+==================+
| irb0    |  24      | 101.1.1.1 | dynamic | 00:00:64:01:01:01| 3 hrs from now   |
+---------+----------+-----------+---------+------------------+------------------+
----------------------------------------------------------------------------------
  Total entries : 1 (0 static, 1 dynamic)
---------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance BD24 bridge-table mac-table mac 00:00:64:01:01:01
                                                        
---------------------------------------------------------------------------------
Mac-table of network instance BD24
---------------------------------------------------------------------------------
Mac                     : 00:00:64:01:01:01
Destination             : lag1.24
Dest Index              : 18
Type                    : learnt
Programming Status      : Success
Aging                   : 2875
Last Update             : 2021-04-15T14:43:18.000Z
Duplicate Detect time   : N/A
Hold down time remaining: N/A
-----------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--

Example: Silent move - watch command output for LEAF-3

Every 2.0s: show route-table ipv4-unicast prefix 101.1.1.1/32 detail
  (Executions 976, Thu 07:44:30AM)
          
-------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10 
------------------------------------------------------------------
Destination   : 101.1.1.1/32                                               
ID            : 0 
Route Type    : bgp-evpn 
Metric        : 0   
Preference    : 170  
Active        : true  
Last change   : 2021-04-15T14:43:19.767Z
Resilient hash: false 
-------------------------------------------------------------------
Next hops: 1 entries 
4.4.4.4 (indirect) resolved by None (None)
-------------------------------------------------------------------

5.5. EVPN-VXLAN Layer 3 feature parity for IPv6 prefixes

All the features discussed in this chapter are supported for IPv6 prefixes and hosts. EVPN IFL works for Prefix IPv6 routes without enabling a separate BGP family. EVPN supports IPv4 and IPv6 routes. In addition, all IRB sub-interfaces must be configured with the IPv6 container using the same commands used earlier in this chapter, but performed under “neighbor-discovery”.

Example: IPv6 container configuration

--{ [FACTORY] + candidate shared default }--[ interface irb0 subinterface 24 ]--
# info                                                      
    ipv4 {
        address 101.1.1.2/24 {
        }
        address 101.1.1.254/24 {
            anycast-gw true
            primary
        }
        address 102.1.1.254/24 {
        }
        arp {
            learn-unsolicited true
            debug [
                messages
            ]
            host-route {
                populate dynamic {
                }
            }
            evpn {
                advertise dynamic {
                }
            }
        }
    }
    ipv6 {
        address 2001:db8:24::254/64 {
            anycast-gw true
        }
        neighbor-discovery {
            learn-unsolicited both
            host-route {
                populate dynamic {
                }
            }
            evpn {
                advertise dynamic {
                }
            }
        }
    anycast-gw {
    }

5.5.1. Additional feature parity considerations

The anycast-gw container is common for IPv4 and IPv6. Therefore, the anycast-gw mac is the same for both families. Only one anycast-gw MAC is programmed in the interface, and IPv4 and IPv6 packets will use this anycast-gw-mac as MAC SA when sourcing packets to the BD.

LLA and global addresses are advertised in EVPN. The command neighbor-discovery learn-unsolicited both includes global and link local addresses.

The following example shows that when anycast-gw is enabled, an anycast-gw LLA is automatically generated. The anycast-gw ipv6 link local address is based off the anycast-gw-mac when the anycast-gw and the ipv6 containers are present. The logic to compute this new anycast-gw ipv6 link local address is the same as is used for computing the regular ipv6 LLA except the anycast-gw-mac is used instead of the interface mac. This new ipv6 LLA appears in the list of ipv6 addresses associated with the sub-interface, but with the attribute anycast-gw true.

Multicast NS messages will use the anycast-gw LLA and anycast-gw MAC. Unicast NS will use the global IPv6 and hw-address.

Example: LLA generation

--{ [FACTORY] + candidate shared default }--[ interface irb0 subinterface 24 ipv6 ]--
#  info from state 
    address 2001:db8:24::254/64 {
        anycast-gw true
        origin static
        primary
        status preferred
    }
    address fe80::200:5eff:fe00:101/64 {
        anycast-gw true
        origin link-layer
        status preferred
    }
    address fe80::201:2ff:feff:41/64 {
        origin link-layer
        status preferred
    }
    neighbor-discovery {
        duplicate-address-detection true
        reachable-time 30
        stale-time 14400
        learn-unsolicited both
        neighbor fe80::201:4ff:feff:41 {
            link-layer-address 00:01:04:FF:00:41
            origin evpn
        }
        host-route {
            populate dynamic {
            }
        }
        evpn {
            advertise dynamic {
                admin-tag 0
            }
        }
    }
    router-advertisement {
        router-role {
            current-hop-limit 64
            managed-configuration-flag false
            other-configuration-flag false
            max-advertisement-interval 600
            min-advertisement-interval 200
            reachable-time 0
            retransmit-time 0
            router-lifetime 1800
        }
    }