10. Hardware-assisted Hierarchical QoS

Hardware-assisted HQoS is an alternative to traditional HQoS which is described in Scheduler QoS Policies.

This model improves reaction time and provides more accurate control of downstream bursts for aggregate rate enforcement. It is based on:

  1. hardware aggregate shapers (hw-agg-shaper)
  2. additional hardware scheduling priorities (6 scheduling classes)
  3. fair bandwidth distribution algorithm
  4. software-based algorithm for enforcing HQoS at the vport level

10.1. Hardware Aggregate Shapers

Hardware-assisted HQoS is based on hardware aggregate shapers, which provide enforcement of aggregate rates for a group of queues in the hardware. In the FP4-based chipset, hardware aggregate shapers are available at egress only. The queues are allocated in a block of eight to each hardware aggregate shaper.

Hardware aggregate shapers are controlled by three parameters:

  1. rate — sets the aggregate rate of the hardware shaper
  2. burst-limit — sets the aggregate burst for the given hardware aggregate shaper
  3. adaptation-rule — sets how the administrative rates are translated into hardware operational values

Traffic is scheduled from the queues in a strict priority order based on the hardware scheduling priority assigned to each queue. This can be controlled using the sched-class command. There are 6 scheduling-classes available.

Hardware aggregate shapers do not need software-based HQoS to modify the oper-PIR of the queues to enforce an aggregate rate. This results in a significantly faster reaction time compared to traditional HQoS based on virtual schedulers.

The following examples show the configuration of a hardware aggregate shaper on a subscriber profile. The configuration is similar to traditional QoS configuration.

Classic

*A:SR-1s-test>config>subscr-mgmt# info
----------------------------------------------
        sla-profile "trialSLA" create
            ingress
                qos 2
                exit
            exit
            egress
                qos 2
                exit
            exit
        exit
        sub-profile "trialSub" create
            egress
                agg-rate-limit 1500000
            exit
        exit
----------------------------------------------
*A:SR-1s-test>config>subscr-mgmt#

MD-CLI

[/configure subscriber-mgmt]
A:admin@SR-1s# info
    sub-profile "trialSub" {
        egress {
            qos {
                agg-rate {
                    rate 1500000
                    burst-limit auto
                }
            }
        }
    }
    sla-profile "trialSLA" {
        egress {
            qos {
                sap-egress {
                    policy-name "trial-egr"
                }
            }
        }
        ingress {
            qos {
                sap-ingress {
                    policy-name "trial-ingr"
                }
            }
        }

10.2. Fair Bandwidth-distribution Algorithm

Hardware-assisted HQoS introduces a new way to implement queue-weight based bandwidth distribution.

The queue hardware-enforced FIR rate is used to provide weighted fair access between queues assigned to the same hardware scheduling priority (scheduling class) within a hardware aggregate shaper, by dynamically changing the queue's hardware scheduling priority as shown in Figure 35.

Figure 35:  Hardware FIR Enforcement for Dynamic Fairness Control 

Every queue in a hardware aggregate shaper is described by the following parameters:

  1. scheduling class — sets assignment to hardware priorities
  2. aggregate shaper weight — sets the queue fair-share of bandwidth at the given scheduling class level
  3. burst limit — sets the individual queue burst limit at PIR
  4. FIR burst limit — sets the individual queue burst limit at FIR

The aggregate shaper weight algorithm operates similarly to the virtual-scheduler HQoS algorithm, meaning that the offered load of the queues at the same scheduling class is weighted against the aggregate shaper weight to determine the FIR, which then controls hardware-based scheduling.

A FIR-based algorithm is more efficient and more responsive to dynamic changes in offered load. The PIR is not adjusted, which prevents aggregate-rate underruns in case of a sudden change in an incoming load for some queues.

When hardware aggregate shapers are enabled, CIR as a queue parameter is ignored.The queue-frame-based-accounting setting is also ignored when hardware aggregate shaping is configured.

The following examples show a SAP egress policy configuration. Weight-based bandwidth distribution in this example is only between queue 1 and queue 2. These queues are both assigned to the same scheduling class (sched-class 3), while queue 3 has a higher priority scheduling-class (sched-class 5) and is served with strict priority.

Classic

*A:SR-1s-test>config>qos>sap-egress# info
----------------------------------------------
            queue 1 create
                mbs 100 kilobytes
                burst-limit 500 bytes
                agg-shaper-weight 5
                sched-class 3
            exit
            queue 2 create
                mbs 100 kilobytes
                burst-limit 500 bytes
                agg-shaper-weight 15
                sched-class 3
            exit
            queue 3 create
                mbs 100 kilobytes
                burst-limit 500 bytes
                sched-class 5
            exit
            fc af create
                queue 2
            exit
            fc be create
                queue 1
            exit
            fc ef create
                queue 3
            exit
----------------------------------------------
*A:SR-1s-test>config>qos>sap-egress#

MD-CLI

[/configure qos]
A:admin@SR-1s# info
    sap-egress "trial-egr" {
        queue 1 {
            agg-shaper-weight 5
            burst-limit 500
            mbs 102400
            sched-class 3
        }
        queue 2 {
            agg-shaper-weight 15
            burst-limit 500
            mbs 102400
            sched-class 3
        }
        queue 3 {
            burst-limit 500
            mbs 102400
            sched-class 5
        }
        fc be {
            queue 1
        }
        fc af {
            queue 2
        }
        fc ef {
            queue 3
        }

10.3. Hardware Aggregate Shaper Scheduler Policy

A hardware aggregate shaper scheduler policy is an alternative to a port scheduler. A hardware aggregate shaper policy can only be assigned at the vport level.

A virtual vport scheduler is added to implement a bandwidth-distribution software algorithm to handle traffic oversubscription by subscriber hardware aggregate shapers at a vport.

The vport represents a downstream bandwidth constraint and therefore the algorithm enforces a maximum rate, which is an Ethernet on-the-wire rate. This algorithm also implements the logic to distribute the available bandwidth at the vport to its child hardware aggregate shapers.

The algorithm performs a similar function to the existing HQoS port scheduler, however, instead of modifying the oper-PIRs of queues, policers, or schedulers, it modifies the oper-PIR of the hardware aggregate shapers connected to it, and consequently triggers recalculation of the individual queue’s FIR, if required.

Figure 36 shows the functionality of a hardware aggregate shaper scheduler, including all parameters which the user can set. This architecture allows the user full flexibility to define SLA models.

Figure 36:  Vport Bandwidth Distribution 

The generic design in Figure 36 can be mapped into two SLA models as shown in Figure 37. Both SLA models show two subscriber categories for simplicity, but this model can be extended to any number of subscriber categories.

Figure 37:  SLA Models  

The SLA model (a) in Figure 37 shows the overall vport bandwidth distribution between gold and white categories in a ratio of 8:1 regardless of the number of subscribers in either category, and assuming that traffic passing through queue 7 for both subscriber categories is delay-critical but not significant in terms of volume.

The SLA model (b) in Figure 37 guarantees that the ratio between any gold and any white subscriber is 8:1. The overall bandwidth distribution between gold and white categories, however, it depends on the number of subscribers in each category.

SLA models are a design goal and must be tested to guarantee the expected behavior. SR OS configuration allows a user to construct other models by combining concepts and parameters in other ways. The resulting behavior is subject to a specific configuration and offered load.

When alternative SLA models are configured, the following considerations apply.

  1. If a group is used to enforce weighting between different scheduling classes, all queues attached to a specific hardware aggregate shaper must have the same scheduling class. This is because the software algorithm enforcing the hardware aggregate scheduler policy only controls the rate of the hardware aggregate shaper and not the rate of individual queues. If this recommendation is not followed, weights enforced at the group level may contradict weights enforced between queues within a single hardware aggregate shaper.
  2. Elevation-weight and groups for any given set of queues are mutually exclusive.

The hardware aggregate scheduler policy implements a congestion threshold that is used to trigger software recalculation. If the overall offered load for a given vport is below this threshold (expressed as a percentage of the vport aggregate rate), the software recalculation of all parameters is not performed to save CPU resources because it is assumed that all offered traffic passes through.

Similarly to a port scheduler policy, a hardware aggregate rate scheduler policy supports a monitor threshold that detects congestion. The configuration and operation are identical as described in Scheduler QoS Policies.

The following example hardware aggregate shaper scheduler policy describes the SLA model from Figure 37 (a).

Classic

*A:SR-1s-test>config>qos>hw-agg-shap-sched-plcy# info
----------------------------------------------
            max-rate percent 80.00
            group "g1" create
            exit
            sched-class 3 group "g1" weight 1
            sched-class 4 group "g1" weight 8
----------------------------------------------
*A:SR-1s-test# /configure port 1/1/1/c1/1 ethernet access egress
*A:SR-1s-test>config>port>ethernet>access>egress# info
----------------------------------------------
                    vport "dslam-a" create
                        hw-agg-shaper-scheduler-policy "trial"
                    exit
----------------------------------------------

MD-CLI

*(ex)[configure qos hw-agg-shaper-scheduler-policy "trial"]
A:admin@SR1-s# info
    max-percent-rate 80.0
    group "g1" { }
    sched-class 3 {
        group "g1"
        weight 1
    }
    sched-class 4 {
        group "g1"
        weight 8
    }
 
*(ex)[configure port 1/1/1/c1/1 ethernet access egress]
A:admin@SR1-s# info
    virtual-port "dslam-a" {
        hw-agg-shaper-scheduler-policy "trial"
    }

10.4. Interactions with Other QoS Features

The FP4-based chipset supports all existing QoS features including software-driven HQoS and hardware-assisted HQoS. The following sections describe how these features interact.

10.4.1. Activation of Hardware Aggregate Shapers

Hardware aggregate shapers are applicable to subscriber management only. This means that for other objects, such as SAP and queue-groups, only software-driven HQoS can be used. Hardware aggregate shapers and software-driven HQoS can coexist on the same FP.

  1. Enable hardware aggregate shapers on a given FP, use the following classic CLI command:
    config>qos>fp-resource-policy>aggregate-shapers auto-creation
    Similarly, to enable hardware aggregate shapers on a given FP, use following MD-CLI command:
    configure qos fp-resource-policy name aggregate-shapers auto-creation true
    This causes a reboot of the corresponding FP and the allocation of queue-sets in blocks of eight. This occurs for all objects already configured on the FP, regardless of how many queues they effectively use. If there are not enough queues to perform the allocation, the commit command fails in MD-CLI.
    The auto-creation command does not force subscriber hosts to use a hardware aggregate shaper. Even if this command is enabled, subscriber hosts on the specified FP still uses traditional HQoS if it is configured. Any hardware aggregate shaper-related configuration in the SAP egress profile, SLA profile, and sub-profile is ignored except in the scheduler class configuration, where by default, expedited and non-expedited classes are mapped into scheduling classes as follows.
    1. Non-expedited class queues map to sched-class 4.
    2. Expedited class queues map to sched-class 6.
  2. Enable hardware aggregate shapers for subscribers, the following classic CLI command is required:
    config>qos>fp-resource-policy>aggregate-shapers hw-agg-shaping subscribers
    Similarly, for MD-CLI, the following command is required:
    configure qos fp-resource-policy name aggregate-shapers subscribers true
  3. Enable the use of hardware aggregate shaping on all ports of the FP.
    Classic CLI command
    config>qos>fp-resource-policy>ports hqos-mode hw-agg-shaping
    MD-CLI command
    configure qos fp-resource-policy name ports hqos-mode hw-agg-shaping true

Classic CLI

*A:SR-1s-test>config>qos>fp-resource-policy# info
----------------------------------------------
            aggregate-shapers
                auto-creation
                hw-agg-shapers subscribers
            exit
            ports
                hqos-mode hw-agg-shaping
            exit
----------------------------------------------
*A:SR-1s-test>config>qos>fp-resource-policy#

In MD-CLI, it is recommended (but not required) to execute these commands in a single commit.

(ex)[/configure qos fp-resource-policy "test"]
    aggregate-shapers {
        auto-creation true
        hw-agg-shapers {
            subscribers true
        }
    }
    ports {
        hqos-mode hw-agg-shaping
    }

After these commands are applied to a specific FP, all subscriber hosts created on this FP uses hardware aggregate shaping and any traditional HQoS-related configuration is ignored.

SAP and queue group objects always use traditional HQoS configuration whether or not they are created on the same or different ports as subscriber hosts. Any hardware aggregate shaping configuration associated with a SAP or queue group is ignored.

10.4.2. Hardware Aggregate Shapers and LAG

LAGs using link or port-fair adapt-qos modes support hardware aggregate shapers; LAGs in distribute mode do not.

When a LAG spans multiple FPs, only one set of subscriber queues is created per FP (on the elected port), consequently there is only one hardware aggregate shaper created per FP for subscribers on a LAG. Egress subscriber traffic is hashed only to the active FP for that subscriber.

In these cases, the vport bandwidth distribution algorithm only manages the hardware aggregate shaper associated with the elected LAG port on the active FP. All hardware aggregate shapers not associated with the active FP have their oper PIRs set to their admin PIRs. This eliminates the need for the vport bandwidth distribution algorithm to manage the non-active hardware aggregate shapers and reduces the processing required.

It is recommended that for LAGs spanning different FPs, all participating FPs are configured to use hardware aggregate shapers. However, there are no checks to flag a mismatch of configuration between all member FPs.

10.4.3. Encapsulation Offset

In traditional HQoS for subscribers it is possible to define the subscriber aggregate rates, which should be adjusted with respect to last-mile encapsulation using the following command:

Classic: config>subscriber-mgmt>sub-profile>egress>encap-offset type

MD-CLI: configure subscriber-mgmt sub-profile name egress qos encap-offset type

There are multiple types of encapsulations available.

This causes the offered-load measurements and operational-rate values to be normalized with respect to encap-offset and avg-frame-overhead values. Based on these values, the corresponding operational rates are calculated, and the adjustment is transparent to users.

For hardware aggregate shapers, the shapers and FIR distribution rely on hardware enforcement while the hardware aggregate shaper scheduler policy is software-based. Because of this, queue weights in FIR calculations are normalized with respect to the encap-offset value. Therefore, in show commands related to hardware aggregate shapers the queue weights may differ from the configured values due to such adjustments.

However, weights in hardware aggregate shaper scheduler policy remains as configured, because offered-load and rate calculations in software take into account last-mile overhead in the same way as in software-driven HQoS.

If no encap-offset is set for the given hardware aggregate shaper, the on-the-wire-rate is enforced. Considering that offered-rates do not account for on-the-wire Ethernet overhead, the QoS software adjusts the aggregate rate of the shaper based on the average frame overhead calculated during HQoS calculation.

10.4.4. Advanced QoS Configuration Policy

The following configuration parameters are not applicable when hardware aggregate shaping is used:

  1. configure qos adv-config-policy name child-control bandwidth-distribution
    1. above-offered-allowance delta-consumed-agg-rate
    2. above-offered-allowance delta-consumed-agg-rate percent
    3. above-offered-allowance delta-consumed-higher-tier-rate
    4. above-offered-allowance delta-consumed-higher-tier-rate percent
    5. above-offered-allowance unconsumed-agg-rate
    6. above-offered-allowance unconsumed-agg-rate percent
    7. above-offered-allowance unconsumed-higher-tier-rate
    8. above-offered-allowance unconsumed-higher-tier-rate percent
    9. above-offered-cap percent
    10. above-offered-cap rate
    11. enqueue-on-pir-zero
    12. granularity percent
    13. granularity rate
    14. internal-scheduler-weight-mode
    15. limit-pir-zero-drain
    16. lub-init-min-pir
  2. configure card 1 virtual-scheduler-adjustment
    1. internal-scheduler-weight-mode