3. MAC-VRF network-instances for server aggregation

MAC-VRF network-instances can provide aggregation for a group of servers into the same subnet. This chapter defines concepts and procedures for configuring MAC-VRF network-instances and IRB subinterfaces.

3.1. Applicability

The information and configuration in this chapter are based on SR Linux Release 20.6.

3.2. Overview

Data Center (DC) servers or hosts are connected to TOR routers so that they can be reached from other TOR routers in the same IP fabric. The TOR nodes run BGP to learn and propagate subnet reachability in the underlaying routing infrastructure. The server or hosts connected to these TOR BGP routers use routed subinterfaces on the TOR, and static routes or a PE-CE BGP session, to learn or advertise reachability to the rest of the DC.

Each server requires a separate routed subinterface and subnet on the TOR and the number of subinterfaces and local routes in the route-table will grow linearly as the number of servers increase. The use of a MAC-VRF network-instance provides aggregation for a group of servers into the same subnet. This saves routes and subinterfaces in the TOR. A MAC-VRF is attached to the default network-instance by a single Integrated Routing and Bridging (IRB) interface and subnet, instead of a separate subinterface and route per server. Figure 2 illustrates this concept.

Figure 2:  MAC-VRF network-instances for server aggregation  

Figure 2 shows Leaf-1 and Leaf-2 configured with MAC-VRF instances that aggregate a group of servers. These servers are assigned IP addresses on the same subnet and are connected to the Leaf default network-instance by a single IRB subinterface. The servers use a PE-CE BGP session with the IRB IP address to exchange reachability. The use of the MAC-VRF with an IRB subinterface saves routed subinterfaces on the default network-instance; only one routed subinterface is needed instead of one per server.

3.3. Configuring MAC-VRF network-instances and IRB subinterfaces

Figure 3 illustrates how to configure MAC-VRF network-instances and their IRB subinterfaces to the default network-instance, and how EBGP sessions are configured with the servers. In this example, DUT2 is the TOR being configured. DUT1 and DUT3 are servers that are running BGP against DUT2's IRB subinterface.

Figure 3:  MAC-VRF and IRB example in DUT2 

3.3.1. Example: Configure DUT2 with MAC-VRF, IRB and static BGP on IRB

This example shows how to configure the DUT2 with a MAC-VRF, bridged subinterfaces to DUT1 and DUT3, and an IRB subinterface (reference Figure 3).

Note:

This example assumes DUT2 is pre-configured with a default network-instance that runs BGP sessions to the spine routers, as defined in the section: Using BGP for underlay routing.

  1. In candidate mode, create the interfaces and bridged subinterfaces to connect to DUT1 and DUT3. In this example:
    1. Connect Ethernet-1/1 and Ethernet-1/2 to DUT1 and DUT3 respectively. Although these interfaces could be defined untagged, this example configures them as tagged (vlan-tagging true).
    2. Create a subinterface with index 1 under each interface. The subinterface must be configured as type bridged. Bridged subinterfaces can be associated to MAC-VRF instances so that MAC learning and layer-2 forwarding can be enabled on them.
    3. The subinterfaces use vlan-id 100 since this is the VLAN id used by the servers (DUT1 and DUT2) to send and receive frames.
    Example configuration details:
    --{ candidate shared default }--[ interface * ]--
    A:dut2# info
        interface ethernet-1/1 {
            description dut2-dut1
            vlan-tagging true
            subinterface 1 {
                type bridged
                vlan {
                    encap {
                        single-tagged {
                            vlan-id 100
                        }
                    }
                }
            }
        }
        interface ethernet-1/2 {
            description dut2-dut3
            vlan-tagging true
            subinterface 1 {
                type bridged
                vlan {
                    encap {
                        single-tagged {
                            vlan-id 100
                        }
                    }
                }
            }
        }
  2. Configure an IRB interface and subinterface to connect the MAC-VRF to the existing default network-instance. The IRB is configured in a similar way to a loopback interface and subinterfaces. The IRB subinterface must be type routed, but does not need to be explicitly configured as routed.
    Example configuration details:
    --{ candidate shared default }--[ interface irb* ]--
    A:dut2# info
        interface irb1 {
            subinterface 1 {
                ipv4 {
                    address 10.0.0.2/24 {
                    }
                }
                ipv6 {
                    address 2001:db8::2/64 {
                    }
                }
            }
        }
  3. Configure the network-instance type mac-vrf and associate it to the bridged and irb interfaces.
    Example configuration details:
    --{ candidate shared default }--[ network-instance MAC-VRF-1 ]--
    A:dut2# info                              
        type mac-vrf
        admin-state enable
        interface ethernet-1/1.1 {
        }
        interface ethernet-1/2.1 {
        }
        interface irb1.1 {
        }
  4. Associate the same IRB interface to the network-instance default and configure the BGP IPv4 and IPv6 neighbors to DUT1 and DUT3. See Using BGP for underlay routing for more information on how to configure BGP sessions.
    Example configuration details:
    --{ candidate shared default }--[ network-instance default ]--            
    A:dut2# info
        type default
        admin-state enable
        router-id 2.2.2.2
        interface irb1.1 {
        }
        interface lo0.1 {
        }
        protocols {
            bgp {
                admin-state enable
                autonomous-system 64502
                router-id 10.0.0.2
                ebgp-default-policy {
                    import-reject-all false
                }
                failure-detection {
                    enable-bfd true
                    fast-failover true
                }
                group tor {
                    admin-state enable
                    export-policy pass-all
                    ipv4-unicast {
                        admin-state enable
                    }
                    ipv6-unicast {
                        admin-state enable
                    }
                    local-as 64502 {
                    }
                    timers {
                        minimum-advertisement-interval 1
                    }
                    trace-options {
                        flag update {
                        }
                        flag graceful-restart {
                        }
                        flag events {
                        }
                        flag keepalive {
                        }
                        flag notification {
                        }
                        flag open {
                        }
                        flag packets {
                        }
                        flag route {
                        }
                        flag socket {
                        }
                        flag timers {
                        }
                    }
                }
                ipv4-unicast {
                    admin-state enable
                }
                ipv6-unicast {
                    admin-state enable
                }
                neighbor 10.0.0.1 {
                    peer-as 64501
                    peer-group tor
                    transport {
                        local-address 10.0.0.2
                    }
                }
                neighbor 10.0.0.3 {
                    peer-as 64503
                    peer-group tor
                    transport {
                        local-address 10.0.0.2
                    }
                }
                neighbor 2001:db8::1 {
                    peer-as 64501
                    peer-group tor
                    transport {
                        local-address 2001:db8::2
                    }
                }
                neighbor 2001:db8::3 {
                    peer-as 64503
                    peer-group tor
                    transport {
                        local-address 2001:db8::2
                    }
                }
            }
        }
  5. Review all the changes and commit if correct.
    Example configuration details:
    --{ candidate shared default }--[  ]--
    A:dut2# commit stay 
  6. Check the state of the MAC-VRF and the connectivity to DUT1 and DUT3 using the following commands:
    1. show network-instance MAC-VRF-1 interfaces
    2. show network-instance default interfaces
    3. show network-instance MAC-VRF-1 bridge-table mac-table all
    4. show arpnd arp-entries interface irb1
    5. show arpnd neighbors interface irb1
    6. show network-instance default protocols bgp neighbor
    Example show commands:
    A:dut2# show network-instance MAC-VRF-1 interfaces
    ====================================================================
    Net instance    : MAC-VRF-1
    Interface       : ethernet-1/1.1
    Type            : bridged
    Oper state      : up
    ====================================================================
    Net instance    : MAC-VRF-1
    Interface       : ethernet-1/2.1
    Type            : bridged
    Oper state      : up
    ====================================================================
    Net instance    : MAC-VRF-1
    Interface       : irb1.1
    Oper state      : up
    Ip mtu          : 1500
                      Prefix                      Origin        Status
      ==================================================================
      10.0.0.2/24                               static
      2001:db8::2/64                            static       preferred
      fe80::201:2ff:feff:41/64                  link-layer   preferred
    ====================================================================
    A:dut2# show network-instance default interfaces
    ====================================================================
    Net instance    : default
    Interface       : irb1.1
    Oper state      : up
    Ip mtu          : 1500
                      Prefix                      Origin        Status
      ====================================================================
      10.0.0.2/24                               static
      2001:db8::2/64                            static       preferred
      fe80::201:2ff:feff:41/64                  link-layer   preferred
    =====================================================================
    Net instance    : default
    Interface       : lo0.1
    Oper state      : up
                      Prefix                      Origin        Status
      ====================================================================
      2.2.2.2/32                                static
      2001:db8:1::2/128                         static       preferred
    =====================================================================
    A:dut2# show network-instance MAC-VRF-1 bridge-table mac-table all
    ------------------------------------------------------------------------------------------------------
    Mac-table of network instance MAC-VRF-1
    ------------------------------------------------------------------------------------------------------
    +-------------------+----------------+-------+-----------+--------+-------+--------------------------+
    |        address    |   Destination  | Dest  |   Type    | Active | Aging |  Last Update             |
    |                   |                | Index |           |        |       |                          |
    +===================+================+=======+===========+========+=======+==========================+
    | 00:01:01:FF:00:00 | ethernet-1/1.1 | 16    | learnt    | true   | 287   | 2020-06-03T13:40:25.000Z |
    | 00:01:02:FF:00:41 | irb-interface  | 0     | irb-      | true   | N/A   | 2020-06-02T13:53:50.000Z |
    |                   |                |       | interface |        |       |                          |
    | 00:01:03:FF:00:00 | ethernet-1/2.1 | 17    | learnt    | true   | 287   | 2020-06-03T13:40:25.000Z |
    +-------------------+----------------+-------+-----------+--------+-------+--------------------------+
    Total Irb Macs            :    1 Total    1 Active
    Total Static Macs         :    0 Total    0 Active
    Total Duplicate Macs      :    0 Total    0 Active
    Total Learnt Macs         :    2 Total    2 Active
    Total Macs                :    3 Total    3 Active
    ---------------------------------------------------------------------------------------------------
    A:dut2# show arpnd arp-entries interface irb1
    +-----------+--------------+----------+---------+----------------------+-------------------+
    | Interface | Subinterface | Neighbor | Origin  |  Link layer address  |    Expiry         |
    +===========+==============+=================+=========+===============+===================+
    | irb1      |      1       | 10.0.0.1 | dynamic | 00:01:01:FF:00:00    | 3 hours from now  |
    | irb1      |      1       | 10.0.0.3 | dynamic | 00:01:03:FF:00:00    | 3 hours from now  |
    +-----------+--------------+----------+---------+----------------------+-------------------+
    --------------------------------------------------------------------------------------------
      Total entries : 2 (0 static, 2 dynamic)
    --------------------------------------------------------------------------------------------
    A:dut2# show arpnd neighbors interface irb1
    +-----------+-----------+----------------+---------+--------------------+---------+------------+--------+
    | Interface |   Sub     | Neighbor       | Origin  | Link layer address | Current | Next state |  Is    |
    |           | interface |                |         |                    |  state  |  change    | Router |
    +===========+===========+================+=========+====================+=========+============+========+
    | irb1      |    1      | 2001:db8::1    | dynamic | 00:01:01:FF:00:00  | stale   | 2 hours    | true   |
    |           |           |                |         |                    |         | from now   |        |
    | irb1      |    1      | 2001:db8::3    | dynamic | 00:01:03:FF:00:00  | stale   | 2 hours    | true   |
    |           |           |                |         |                    |         | from now   |        |
    | irb1      |    1      | fe80::201:1ff: | dynamic | 00:01:01:FF:00:00  | stale   | 2 hours    | true   |
    |           |           |  feff:0        |         |                    |         | from now   |        |
    | irb1      |    1      | fe80::201:3ff: | dynamic | 00:01:03:FF:00:00  | stale   | 2 hours    | true   |
    |           |           | feff:0         |         |                    |         | from now   |        |
    +-----------+-----------+----------------+---------+--------------------+---------+------------+--------+
    ---------------------------------------------------------------------------------------------------------
      Total entries : 4 (0 static, 4 dynamic)
    ---------------------------------------------------------------------------------------------------------
    A:dut2# show network-instance default protocols bgp neighbor
    --------------------------------------------------------------------------------------------------------
    BGP neighbor summary for network-instance "default"
    Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow
    ---------------------------------------------------------------------------------------------------------
    +----------+-------------+-------+------+-------+-------------+---------------+------------+------------+
    | Net-Inst |   Peer      | Group | Flag | Peer  |    State    |    Uptime     |  AFI/SAFI  | [Rx/Active |
    |          |             |       |      |  -AS  |             |               |            |    /Tx]    |
    +==========+=============+=======+======+=======+=============+===============+============+============+
    | default  | 10.0.0.1    | tor   | SB   | 64501 | established | 0d:3h:12m:33s | ipv4-unicast | [3/2/4]  |
    |          |             |       |      |       |             |               | ipv6-unicast | [3/2/4]  |
    | default  | 10.0.0.3    | tor   | SB   | 64503 | established | 0d:3h:10m:55s | ipv4-unicast | [3/2/4]  |
    |          |             |       |      |       |             |               | ipv6-unicast | [3/2/4]  |
    | default  | 2001:db8::1 | tor   | SB   | 64501 | established | 0d:3h:12m:31s | ipv4-unicast | [0/0/0]  |
    |          |             |       |      |       |             |               | ipv6-unicast | [6/0/6]  |
    | default  | 2001:db8::3 | tor   | SB   | 64503 | established | 0d:3h:10m:52s | ipv4-unicast | [0/0/0]  |
    |          |             |       |      |       |             |               | ipv6-unicast | [6/0/6]  |
    +----------+-------------+-------+------+-------+-------------+---------------+--------------+----------+
    ---------------------------------------------------------------------------------------------------------
    Summary:
    4 configured neighbors, 4 configured sessions are established,0 disabled peers
    0 dynamic peers

3.4. Advanced configuration: bridge-table settings

A MAC-VRF network-instance uses a bridge-table to forward frames between its subinterfaces. Some bridge-table properties can be configured. For example:

--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]
A:dut2# tree detail
bridge-table!    net_inst_mgr
+-- discard-unknown-dest-mac?    net_inst_mgr
+-- mac-learning    net_inst_mgr
|   +-- admin-state?    net_inst_mgr
|   +-- aging    net_inst_mgr
|       +-- admin-state?    net_inst_mgr
|       +-- age-time?    net_inst_mgr
+-- mac-duplication    net_inst_mgr
|   +-- admin-state?    net_inst_mgr
|   +-- monitoring-window?    net_inst_mgr
|   +-- num-moves?    net_inst_mgr
|   +-- hold-down-time?    net_inst_mgr
|   +-- action?    net_inst_mgr
+-- mac-limit    net_inst_mgr
|   +-- maximum-entries?    net_inst_mgr
|   +-- warning-threshold-pct?    net_inst_mgr
+-- static-mac    l2_static_mac_mgr
    +-- mac* [address]    l2_static_mac_mgr
        +-- address    l2_static_mac_mgr
        +-- destination?M    l2_static_mac_mgr

Where:

  1. The mac-learning container provides control over how macs are dynamically learned on the subinterfaces, including whether learning is enabled (admin-state) or the aging timer for the mac-table.
  2. The mac-duplication container controls how the system behaves when duplicate macs are detected.
  3. The mac-limit container provides parameters for limiting the maximum number of macs installed for a given mac-vrf.
  4. The static-macs provides control to configure and associate to either a subinterface destination or to a blackhole. Incoming frames with the source or destination mac matching a configured "blackholed mac" are discarded by the system.

3.5. Advanced configuration: using mac-duplication for loop protection

SR Linux supports mac-duplication detection and associated procedures to protect the system against network loops. Figure 4 shows a simple loop and describes the associated configuration.

Figure 4:  MAC-Duplication for loop protection 

Figure 4 shows the MAC-VRF 2 connected using two bridged subinterfaces to a layer-2 switch creating a loop. When a host with mac M1 sends a broadcast frame, a loop is created. MAC-duplication is by default enabled in mac-vrf network-instances with the following parameters:

--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table mac-
duplication ]--
A:dut2# info detail                          
    admin-state enable
    monitoring-window 3
    num-moves 5
    hold-down-time 10
    action stop-learning

The loops shown in Figure 4 are resolved in the following sequence:

  1. 1. MAC Duplication Detection.
    1. A mac M1 is declared as 'duplicate' when the number of moves across two or more subinterfaces exceed the configured num-moves in the configured monitoring-window.
    2. When M1 is 'duplicate', it is kept into a duplicate-entry list and stays associated to the last subinterface where the mac was learned before the number or moves exceed num-moves.
  2. 2. MAC Duplication Action.
    1. Once the mac M1 is declared 'duplicate' in a subinterface, an action is taken in that subinterface.
    2. The action is configurable per network-instance and can be overridden on a per subinterface basis.
    3. Possible actions on the subinterface are oper-down, blackhole, and stop-learning.
      1. oper-down - Brings down the subinterface, breaks the loop, and discards all the frames arriving on the subinterface.
      2. blackhole - Discards frames with a source or destination mac that matches the duplicate mac but allows the remaining frames to forward normally on the subinterface.
      3. stop-learning - Does not discard any frame on the subinterface and keeps the existing macs learned against it. No new macs are learned on the subinterface until the action is cleared.
--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table mac-duplication ]--
A:dut2# action <value>                                       
usage: action <blackhole|oper-down|stop-learning>
Action to take on the subinterface whose action is use-net-instance-action,
 upon detecting one or more mac addresses as duplicate
In particular:
- Oper-down: if configured, upon detecting a duplicate mac on the subinterface, the subinterface
  will be brought oper-down.
- Blackhole: upon detecting a duplicate mac on the subinterface, the mac will be blackholed. Any
  frame received on this or any other subinterface with MAC SA matching a blackhole mac is discarded.
- Stop-learning: this is the default action, compliant with RFC7432. Upon detecting a duplicate mac
 on the subinterface, the mac will not be relearned anymore on this or any subinterface.
Positional arguments:
  value 
  1. 3. MAC Duplication hold-down-time and process restart.
    1. When the configured hold-down-time expires, the duplicate mac is flushed from the mac-table and the entire process restarts for the mac.
    2. The duplicate action on a subinterface clears when there are no longer duplicate mac addresses in the subinterface.

As a loop protection mechanism, MAC-duplication is self-contained and does not require a control plane protocol that runs network-wide among network devices.

3.5.1. Example: Configure MAC-Duplication and troubleshoot loops in DUT2

Use this example to assist in configuring MAC-duplication. Consider MAC-VRF 1 is connected to a layer-2 switch (not shown) using two bridge sub-interfaces (ethernet-1/1.2 and ethernet-1/1.3). This creates a loop. DUT2 is configured with the following MAC-duplication settings:

--{ candidate shared default }--[ network-instance MAC-VRF-1 ]-- 
A:dut2# info
    type mac-vrf
    admin-state enable
    interface ethernet-1/1.1 {
    }
    interface ethernet-1/1.2 {
    }
    interface ethernet-1/1.3 {
    }
    interface ethernet-1/2.1 {
    }
    interface irb1.1 {
    }
    bridge-table {
        mac-duplication {
            num-moves 3
            hold-down-time 300
            action stop-learning
        }
    }

Where the subinterfaces are also configured with the following actions:

--{ * candidate shared default }--[ interface * subinterface * bridge-table mac-
duplication ]-- 
A:dut2# info
    interface ethernet-1/1 {
        subinterface 1 {
            bridge-table {
                mac-duplication {
                }
            }
        }
        subinterface 2 {
            bridge-table {
                mac-duplication {
                    action oper-down
                }
            }
        }
        subinterface 3 {
            bridge-table {
                mac-duplication {
                    action oper-down
                }
            }
        }
    }

In this example, the mac-duplication action configured under the network-instance is overridden by the more specific action under the subinterfaces 2 and 3. When traffic is generated by the remote layer-2 switch, the same mac address moves between ethernet-1/1.2 and ethernet-1/1.3. After the third move, the mac is declared a duplicate mac and displays in the duplicate-entries list:

A:dut2# show network-instance MAC-VRF-1 bridge-table mac-table all
------------------------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
------------------------------------------------------------------------------------------------------
+-------------------+----------------+-------+-----------+--------+-------+--------------------------+
| address           |  Destination   | Dest  |   Type    | Active | Aging |  Last Update             |
|                   |                | Index |           |        |       |                          |
+===================+================+=======+===========+========+=======+==========================+
| 00:01:01:FF:00:00 | ethernet-1/1.1 | 16    | learnt    | true   | 287   | 2020-06-03T13:40:25.000Z |
| 00:01:01:FF:00:41 | ethernet-1/1.3 | 20    | duplicate | true   | N/A   | 2020-06-05T20:07:24.000Z |
| 00:01:02:FF:00:41 | irb-interface  | 0     | irb-      | true   | N/A   | 2020-06-02T13:53:50.000Z |
|                   |                |       | interface |        |       |                          |
| 00:01:03:FF:00:00 | ethernet-1/2.1 | 17    | learnt    | true   | 287   | 2020-06-03T13:40:25.000Z |
+-------------------+----------------+-------+-----------+--------+-------+--------------------------+
Total Irb Macs            :    1 Total    1 Active
Total Static Macs         :    0 Total    0 Active
Total Duplicate Macs      :    1 Total    1 Active
Total Learnt Macs         :    2 Total    2 Active
Total Macs                :    4 Total    4 Active
----------------------------------------------------------------------------------------------------
     
                         
A:dut2# show network-instance MAC-VRF-1 bridge-table mac-duplication duplicate-entries
----------------------------------------------------------------------------------------------------
Mac-Duplication in network instance MAC-VRF-1
----------------------------------------------------------------------------------------------------
Admin state            : enable
Monitoring window      : 3
Number of moves allowed: 3
Hold down time         : 300
Action                 : stop-learning
----------------------------------------------------------------------------------------------------
+-------------------+----------------+------------+--------------------------+---------------------+
| Duplicate MAC     | Destination    | Dest Index |      Detect Time         | Hold Time Remaining |
+===================+================+============+==========================+=====================+
| 00:01:01:FF:00:41 | ethernet-1/1.3 | 20         | 2020-06-05T20:07:24.000Z | 270                 |
+-------------------+----------------+------------+--------------------------+---------------------+
Total Duplicate Macs      :    1 Total    0 Active
---------------------------------------------------------------------------------------------------

3.5.2. Using logs to detect duplicate MACs

A log event can help to troubleshoot when MACs are detected as duplicate, or when they are deleted after the hold-down-timer. For example:

[root@dut2 srlinux]# tail -f /var/log/srlinux/debug/sr_l2_mac_mgr.log 
2020-06-06T06:22:48.679070+00:00 dut2 local6|NOTI sr_l2_mac_mgr: bridgetable|2608|2608|00035|N: 
    A duplicate MAC address 00:01:01:FF:00:00 was detected on MAC-VRF-1.
2020-06-06T06:27:48.933312+00:00 dut2 local6|NOTI sr_l2_mac_mgr: bridgetable|2608|2608|00036|N: 
    A duplicate MAC address 00:01:01:FF:00:00 detected on MAC-VRF-1 is now deleted.

The network-instance manager logs also show when the subinterfaces go down due to mac-duplication. For example:

[root@dut2 srlinux]# tail -f /var/log/srlinux/debug/sr_net_inst_mgr.log 
2020-06-06T06:22:48.680609+00:00 dut2 local6|WARN sr_net_inst_mgr: netinst|2663|2663|00080|W: 
    The interface ethernet-1/1.3 in network-instance MAC-VRF-1 is now down for reason: 
    A duplicate MAC address has been detected