16. SR Linux applications

16.1. Overview

The SR Linux is a suite of modular, lightweight applications running like any others in a Linux environment. Each SR Linux application supports a different protocol or function, such as BGP, LLDP, AAA, and so on. These applications use gRPC and APIs to communicate with each other and external systems over TCP.

One SR Linux application, the application manager (app_mgr), is itself responsible for monitoring the health of the process IDs running each SR Linux applications, and restarting them if they fail. The application manager reads in application-specific YAML configuration and YANG models, and starts each application (or allows an application not to start if there no configuration exists for it). There is an instance of the app_mgr that handles applications running on the CPM, and an instance of the app_mgr on each IMM that handles applications running on the line card.

In addition to the Nokia-provided SR Linux applications, the SR Linux supports installation of user-defined applications, which are managed and configured in the same way as the default SR Linux applications.

This chapter presents examples of installing an application in SR Linux, managing installed SR Linux applications, and configuring settings for an SR Linux application by modifying its YAML configuration.

16.2. Installing an application

To install an application, copy the application files into the appropriate SR Linux directories, then reload the application manager and start the application.

The example in this section installs an application called fib_agent. The application consists of files named fib_agent.yml, fib_agent.sh, fib_agent.py, and fib_agent.yang. The fib_agent.yml file is installed in the /etc/opt/srlinux/appmgr/ directory. The .yml file for a user-defined application must reside in this directory in order for the app_mgr to read its YAML configuration.

The .yml file defines the locations of the other application files. The other application files can reside anywhere in the system other than in the /opt/srlinux/ directory or any tempfs file system.

In this example, the fib_agent.sh and fib_agent.py files are installed in the directory/user_agents/, and the fib_agent.yang file is installed in the directory/yang/. The locations for these files are defined in the fib_agent.yml file.

  1. Copy the application files into the SR Linux directories.
    # cp fib_agent.yml /etc/opt/srlinux/appmgr/.
    # cp fib_agent.sh /user_agents/.
    # cp fib_agent.py /user_agents/.
    # cp fib_agent.yang /yang/.
  2. From the SR Linux CLI, reload the application manager.
    --{ candidate }--[ ]--
    # tools system app-management application app_mgr reload
  3. Apply the changes to the configuration.
    --{ candidate }--[  ]--
    # fib-agent
    --{ candidate }--[ fib-agent ]--
    # commit stay
    All changes have been committed. Starting new transaction.
    --{ candidate }--[ fib-agent ]--
  4. Verify that the application is running.
    # show system application fib_agent
      +-----------+-----+---------+-----------------------+--------------------------+
      |  Name     | PID |  State  |        Version        |       Last Change        |
      +===========+=====+=========+=======================+==========================+
      | fib_agent | 227 | running | v21.3.0-61-gd19567393 | 2021-01-13T20:16:45.697Z |
      +-----------+-----+---------+-----------------------+--------------------------+

16.3. Managing applications

16.3.1. Starting an application

To start an SR Linux application instance, use the start option in the tools system app-management command. To terminate a running application instance and restart it, use the restart option.

Examples:

To start an SR Linux application instance:

# tools system app-management application mpls_mgr start
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was started

To restart an SR Linux application instance:

# tools system app-management application mpls_mgr restart
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 9
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was restarted

16.3.2. Terminating an application

You can use the stop, quit, or kill options in the tools system app-management command to terminate an SR Linux application.

  1. stop: Gracefully terminates the application, allowing it to clean up before exiting.
  2. quit: Terminates the application and generates a core dump. The core dump files are saved in the /var/log/srlinux/cores/ directory.
  3. kill: Terminates the application immediately, without allowing it to clean up before exiting.

Examples:

To terminate an application gracefully:

# tools system app-management application mpls_mgr stop
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 15

To terminate an application and generate a core dump:

# tools system app-management application mpls_mgr quit
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 3

To terminate an application immediately:

# tools system app-management application mpls_mgr kill
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 9

16.3.3. Reloading application configuration

Reloading an application causes the app_mgr to reread the application’s YAML configuration and restart the application using settings in its YAML file.

Example:

To reload the configuration of the app_mgr application:

--{ * running }--[  ]--
# tools system app-management application app_mgr reload
--{ * running }--[  ]--

16.3.4. Clearing application statistics

You can display statistics collected for an application with the info from state command. To reset the statistics counters for the application, use the statistics clear option in the tools system app-management command.

Example:

To reset the statistics counters for an application:

# tools system app-management application mpls_mgr statistics clear

16.3.5. Restricted operations for applications

An application may have one or more operations that are restricted by default. For example, the linux_mgr application has stop, quit, and kill as restricted operations, meaning that these options are not available when entering the tools system app-management command for the linux_mgr application.

Table 21 lists the restricted operations for each SR Linux application.

Table 21:  Restricted operations for SR Linux applications  

Application

Restricted operations

aaa_mgr

reload

acl_mgr

reload

app_mgr

start, stop, restart, quit, kill

arp_nd_mgr

reload

bfd_mgr

reload

bgp_mgr

reload

chassis_mgr

stop, quit, kill, reload

device_mgr

reload

dhcp_client_mgr

stop, reload

fib_mgr

reload

gnmi_server

reload

idb_server

start, stop, restart, quit, kill, reload

json_rpc_config

reload

linux_mgr

stop, quit, kill

lldp_mgr

reload

log_mgr

reload

mgmt_server

start, stop, quit, kill, reload

mpls_mgr

reload

net_inst_mgr

start, stop, quit, kill, reload

oam_mgr

reload

plcy_mgr

reload

qos_mgr

reload

sdk_mgr

reload

static_route_mgr

reload

supportd

reload

xdp_cpm

stop, quit, kill, reload

xdp_lc

reload

Restricted options are specified in the restricted-operations setting in the YAML file for the application.

16.4. Configuring an application

To configure an SR Linux application, edit settings in the application’s YAML file, then reload the application manager to activate the changes.

The example in this section shows how to configure an application to specify the action the SR Linux device takes when the application fails. If an SR Linux application fails a specified number of times over a specified time period, the SR Linux device can reboot the system or attempt to restart the application after waiting a specified number of seconds.

For example, if the aaa_mgr application crashes 5 times within a 500-second window, the SR Linux device can be configured to wait 100 seconds, then restart the aaa_mgr application.

The following actions can be taken if an SR Linux application fails:

  1. Reboot the system
  2. Wait a specified number of seconds, then attempt to restart the failed application
  3. Move the failed application to error state without rebooting the system or attempting to restart the application

If you stop or restart an application using the tools system app-management command in the SR Linux CLI, it is not considered an application failure; the failure action for the application, if one is configured, does not occur. However, if the failed application waits a specified period of time (or forever) to be restarted, or has been moved into error state, you can restart the application manually with the tools system app-management application restart CLI command.

To configure the failure action for an application:

  1. Check the status of the SR Linux applications:
    # show system application
    +-------------+-----+---------+-----------------------+--------------------------+
    |    Name     | PID |  State  |        Version        |       Last Change        |
    +=============+=====+=========+=======================+==========================+
    | aaa_mgr     | 242 | error   | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.529Z |
    | acl_mgr     | 261 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.530Z |
    | app_mgr     | 185 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.585Z |
    | arp_nd_mgr  | 270 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.530Z |
    | bfd_mgr     | 279 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.530Z |
    | bgp_mgr     | 850 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.823Z |
    | chassis_mgr | 288 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.530Z |
    | dev_mgr     | 194 | running |                       | 2021-03-05T20:34:48.803Z |
    | fib_mgr     | 311 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.531Z |
    | gnmi_server | 857 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.826Z |
    | idb_server  | 229 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.033Z |
    | json_rpc    | 864 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.828Z |
    | linux_mgr   | 320 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.531Z |
    | lldp_mgr    | 872 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.838Z |
    | log_mgr     | 330 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.532Z |
    | mgmt_server | 340 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.532Z |
    | mpls_mgr    | 357 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.532Z |
    | net_inst_mgr| 377 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.532Z |
    | oam_mgr     | 392 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.533Z |
    | plcy_mgr    | 401 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.533Z |
    | qos_mgr     | 842 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.750Z |
    | sdk_mgr     | 418 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.533Z |
    | sshd-mgmt   | 107 | running |                       | 2021-03-05T20:34:53.701Z |
    | supportd    | 480 | running |                       | 2021-03-05T20:34:49.534Z |
    | xdp_cpm     | 520 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.534Z |
    | xdp_lc_1    | 539 | running | v21.3.0-61-gd19567393 | 2021-03-05T20:34:49.535Z |
    +-------------+-----+---------+-----------------------+--------------------------+
  2. Use the info from state command to check the current failure action settings for the application to configure. These settings are highlighted in the following example:
    # info from state system app-management application aaa_mgr
        system {
            app-management {
                application aaa_mgr {
                    pid 242
                    state error
                    last-change 2021-03-05T20:34:49.529Z
                    author Nokia
                    failure-threshold 3
                    failure-window 300
                    failure-action reboot
                    path /opt/srlinux/bin
                    launch-command ./sr_aaa_mgr
                    search-command ./sr_aaa_mgr
                    version v21.3.0-61-gd19567393
                    restricted-operations [
                        reload
                    ]
                    statistics {
                        restart-count 0
                    }
                    yang {
                        modules [
                            srl_nokia-aaa
                            srl_nokia-aaa-types
                        ]
                        source-directories [
                            /opt/srlinux/models/ietf
                            /opt/srlinux/models/srl_nokia/models/common
                            /opt/srlinux/models/srl_nokia/models/network-instance
                            /opt/srlinux/models/srl_nokia/models/system
                        ]
                    }
                }
            }
        }
    The following failure action settings can be configured for an application:
    1. failure-threshold: Number of times that the application must fail during the failure-window period before the failure-action is taken; the default is three times.
    2. failure-window: Number of seconds over which the application must fail the failure-threshold number of times before the failure-action is taken; the default is 300 seconds.
    3. failure-action: Action to take if the application fails failure-threshold times over failure-window seconds. This can be one of the following:
      1. reboot: Reboot the system; this is the default failure-action.
      2. wait=seconds: Wait this number of seconds, then attempt to restart the application.
      3. wait=forever: Move the application to error state and do not reboot the system or attempt to restart the application.
  3. Edit the YAML configuration for the application.
    The YAML configuration files for SR Linux applications are located in the directory /opt/srlinux/appmgr on the SR Linux device. They are named sr_application_name_config.yml; for example, sr_aaa_mgr_config.yml.
  4. In the .yml file, add or change the settings for the failure-threshold, failure-window, and failure-action parameters.
    In the following example, the failure action settings in the sr_aaa_mgr_config.yml file are configured so that if the aaa_mgr application fails 5 times over a 500-second period, the SR Linux device waits 100 seconds, then attempts to restart the aaa_mgr application:
    aaa_mgr_setup:
       path: /opt/srlinux/bin
       launch-command: ./aaamgr_set_env.sh
       run-as-user: root
       never-show: Yes
       never-restart: Yes
       start-order: 1
    aaa_mgr:
       path: /opt/srlinux/bin
       launch-command:  ./sr_aaa_mgr
       search-command: ./sr_aaa_mgr
       run-as-user: root
       restricted-operations: ['reload']
       failure-threshold: 5
       failure-window : 500
       failure-action: "wait=100"
       yang-modules:
           names:
               - "srl_nokia-aaa"
               - "srl_nokia-aaa-types"
           source-directories:
               - "/opt/srlinux/models/ietf"
               - "/opt/srlinux/models/srl_nokia/models/common"
               - "/opt/srlinux/models/srl_nokia/models/system"
               - "/opt/srlinux/models/srl_nokia/models/network-instance"
  5. Save and close the .yml configuration file.
  6. In the SR Linux CLI, reload the application manager:
    # tools system app-management application app_mgr reload
    This command reloads any application whose .yml configuration file has changed. It does not affect any service.
  7. Use the info from state command to verify that the changes to the failure action settings are now in effect. For example:
    # info from state system app-management application aaa_mgr
        system {
            app-management {
                application aaa_mgr {
                    pid 242
                    state running
                    last-change 2021-03-05T20:44:31.403Z
                    author Nokia
                    failure-threshold 5
                    failure-window 500
                    failure-action wait=100
                    path /opt/srlinux/bin
                    launch-command ./sr_aaa_mgr
                    search-command ./sr_aaa_mgr
                    version v21.3.0-61-gd19567393
                    restricted-operations [
                        reload
                    ]
                    statistics {
                        restart-count 0
                    }
                    yang {
                        modules [
                            srl_nokia-aaa
                            srl_nokia-aaa-types
                        ]
                        source-directories [
                            /opt/srlinux/models/ietf
                            /opt/srlinux/models/srl_nokia/models/common
                            /opt/srlinux/models/srl_nokia/models/network-instance
                            /opt/srlinux/models/srl_nokia/models/system
                        ]
                    }
                }
            }
        }

16.5. Partioning and isolating application resources

The SR Linux protects system processes through the use of control groups (cgroups), which impose resource consumption limits on resource-intensive customer applications.

16.5.1. Cgroup profiles

Cgroup profiles define how usage limits are applied. On the SR Linux, cgroup profiles are supported for CPU and memory and are defined in cgroup_profile.json configuration files.

SR Linux provides a default cgroup profile; customers can configure additional cgroup profiles.

16.5.1.1. Default cgroup profile

The SR Linux-provided default cgroup profile is located in the /opt/srlinux/appmgr/cgroup_profile.json directory.

Note:

Editing the default cgroup profile is not recommended.

If the default cgroup profile fails to parse or be read by the app_mgr, the SR Linux will not start.

The default cgroup_profile.json file definition is shown below:

{
 "profiles": [
  {
   "name": "workload.primary",
   "path": "workload.slice/primary",
   "controller": {
    "memory": {
     "max": 0.8,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  },
  {
   "name": "workload.datapath",
   "path": "workload.slice/datapath",
   "controller": {
    "memory": {
     "max": 0.8,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "all",
     "mems": ""
    }
   }
  },
  {
   "name": "workload.secondary",
   "path": "workload.slice/secondary",
   "controller": {
    "memory": {
     "max": 0.5,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  },
  {
   "name": "user.default",
   "path": "user.slice/default",
   "controller": {
    "memory": {
     "max": 0.25,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "1000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  }
 ]
}

Table 22 describes the default cgroup profile parameters.

Table 22:   Default cgroup Profile Parameters 

Parameter

Description

name

The cgroup profile name.

Type: string

path

The cgroup directory path relative to a unified root path. A typical unified root path is /sys/fs/cgroup or /mnt/cgroup/unified

Type: string

controller

The memory controller configuration.

max: This number denotes the percentage of total memory. The actual memory value is calculated as (max. × total_memory) and is set in the memory.max interface file of the cgroup. If the value is 0, this configuration is ignored. The range is from 0 to 1, the default is 0.8.

low: This number denotes the percentage of total memory. The actual memory value is calculated as (max. × total_memory) and is set in the memory.low interface file of the cgroup. The range is from 0 to 1, the default is 0.8.

cpu

The CPU controller configuration.

weight: This value is set in the cpu.weight interface file of the cgroup. The range is from 1 to 10 000, the default is 100.

period: This value specifies a period of time, in microseconds, for how regularly a cgroup's access to CPU resources should be reallocated.This value is set in the cpu.max interface file of the cgroup. The range is from 1000 to 1 000 000, the default is 100 000.

quota: This value specifies the total length of time, in microseconds, for which all tasks in a cgroup can run during one period (as defined in the period parameter). If quota is set to 0, it translates to “max” in the cpu.max interface file. The range is from 1000 to 1 000 000, the default is max.

cpuset

CPU usage information for the cgroup.

cpus: This value indicates the CPUs used by the cgroup. This can be "", meaning use all CPUs except for the isolated CPUs; this is the default. The value all means include the isolated CPUs for cgroup usage. The value x, y-z, where x, y, and z are CPU numbers, means use a specific CPU or a range of CPUs.

mems: This value is used for scheduling multiple NUMA (non-uniform memory access) aware applications in the cgroup.

16.5.1.2. Customer-defined cgroup profile

Customers can configure cgroup profiles in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory. The app_mgr will create this directory at boot up if it does not exist.

If a customer-defined cgroup profile fails to load, the system will continue to function and applications that are loaded into the customer cgroup are loaded using Nokia defaults, listed below.

  1. Nokia-written applications run in the workload.slice/primary cgroup along with any processes that are started by linuxadmin, including sr_cli.
  2. Non-Nokia-written applications run in the workload.slice/secondary cgroup. If a customer builds an application and launches it using the app_mgr without specifying a cgroup, the application will run in this cgroup
  3. All interactive user applications run in the user.slice/default cgroup, including sr_cli when not started by linuxadmin.

The admin user is treated as any other user in the system. Its processes fall into the user.slice/default cgroup.

16.5.2. Configuring a cgroup

Customers can configure up to three cgroups in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory. Customer applications are assigned to these groups. Any more than three configured cgroups are ignored. The depth of cgroups is limited to two levels where, for example, workload is one level, and primary/secondary are two levels. Any levels beyond this are also ignored.

If a cgroup with the same name is used in multiple customer-defined profiles, the system ignores it and uses the cgroup defined in the default profile.

16.5.2.1. Cgroup configuration example

The following example shows the configuration of two customer-defined cgroups: one for a lightweight database that needs priority access to resources, and one for storing the users of the database.

The steps and the outputs are described below.

  1. Install the application, including the YAML binary file and optional YANG module.
    In this example, YAML defines two applications: dtw-database and dtw-frontend. These applications are placed into their own cgroups: distributetheweb.slice/database and distributetheweb.slice/frontend.
    The app-mgr creates the cgroups.
    The output below shows the installation of the database and a web front end.
    dtw-database:
       path: /opt/distributetheweb/bin/
       launch-command: ./run_db
       search-command: ./run_db
       failure-threshold: 100
       failure-action: "wait=60"
       cgroup: distributetheweb.slice/database
       oom-score-adj: -500
       yang-modules:
           names:
               - "database"
           source-directories:
               - "/opt/distributetheweb/yang/"
    dtw-frontend:
       path: /opt/distributetheweb/bin/
       launch-command: ./run_frontend
       search-command: ./run_frontend
       failure-threshold: 100
       failure-action: "wait=60"
       cgroup: distributetheweb.slice/frontend
       oom-score-adj: 200
       yang-modules:
           names:
               - "frontend"
           source-directories:
               - "/opt/distributetheweb/yang/"
  2. Configure the cgroup profiles in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory.
    The output below shows the configuration.
    {
     "profiles": [
      {
       "name": "distributetheweb.database",
       "path": "distributetheweb.slice/database",
       "controller": {
        "memory": {
         "max": 0.5,
         "swap_max": 0,
         "low": 0
        },
        "cpu": {
         "weight": "10000",
         "period": "100000",
         "quota": "0"
        }
       }
      },
      {
       "name": "distributetheweb.frontend",
       "path": "distributetheweb.slice/frontend",
       "controller": {
        "memory": {
         "max": 0.2,
         "swap_max": 0,
         "low": 0
        },
        "cpu": {
         "weight": "5000",
         "period": "100000",
         "quota": "0"
        }
       }
      }
     ]
    }

The configuration above created two cgroup profiles: one for the database slice and one for the frontend slice. The profile for the database slice is configured to limit the database to 50% of system memory. The profile for the frontend slice is configured to limit the web front end to 20% of system memory.

In addition, both cgroup profiles are configured to limit CPU resources for their respective cgroup. The database server CPU is weighted at 10000 (the maximum CPU weight) and the frontend server CPU is weighted at 5000 (half the CPU weight of the database). The weights are added together and each group is given its ratio of CPU as a proportion of the sum. The periods are kept the same, and no guaranteed CPU is granted.

16.5.3. Kernel low-memory killer

The kernel low-memory killer driver monitors the memory state of a running system. It reacts to high memory pressure by killing the least essential processes in order to keep the system performing at acceptable levels.

When the system is low in memory and cannot find free memory space, the out_of_memory function is called. The out_of_memory function makes memory available by killing one or more processes.

When an out-of-memory (OOM) failure occurs, the out_of_memory function is called and it obtains a score from the select_bad_process function. The process with the highest score is the one that is killed. Some of the criteria used to identify a bad process include the following:

  1. The kernel needs a minimum amount of memory for itself.
  2. Try to reclaim a large amount of memory.
  3. Do not kill a process that is using a small amount of memory.
  4. Try to kill the minimum number of processes.
  5. Algorithms that elevate the sacrifice priority on processes the user wants to kill.

In addition to this list, the OOM killer checks the out-of-memory (OOM) score. The OOM killer sets the OOM score for each process and then multiplies that value by memory usage. The processes with higher values have a high probability of being terminated by the OOM killer.

16.5.3.1. SR Linux kill strategy

The kernel calculates the oom_score using the formula 10 × percentage of memory used by the process. The maximum score is 10 × 100% = 1000. The oom_score of a process can be found in the /proc directory (/proc/$pid/oom_score). An oom_score of 1000 means the process is using all the memory, an oom_score of 500 means it is using half the memory, and an oom_score of 0 means it is using no memory.

The OOM killer checks the oom_score_adj file in the /proc/$pid/oom_score_adj directory to adjust its final calculated score. The default value is 0.

The oom_score_adj value can range from -1000 to 1000. A score of -1000 results in a process using 100% of the memory and in not being terminated by the OOM killer. However, a score of 1000 causes the Linux kernel to terminate the process even when it uses minimal memory. A score of -100 results in a process using 10% of the memory before it will be considered for termination, as its score will remain 0 until its unadjusted score reaches 100.

The oom_score_adj value for each process is defined in its corresponding YAML definition file. The system groups the processes based on their score, which the SR Linux OOM killer uses as the hierarchy for terminating a rogue process, as follows:

  1. group1 = -998, for processes that should never be killed, such as app_mgr, idb_server, and all processes on the IMM.
  2. group2 = -200, for processes that ideally should not be killed because doing so will have a big impact on the system, such as mgmt_server, chassis_mgr, and net_inst_mgr. A score of -200 means the process gets to use 20% of the memory before their memory use starts being counted.
  3. group3 = 0, for process that should be killed if they are using too much memory such as BGP, ISIS, and OSPF.
  4. group4 = 500, for processes that should be preferentially killed, such as cli and oam_mgr

Table 23 lists the OOM adjust score for each process.

Table 23:   OOM adjust score per process 

Process name

OOM adjust score

aaa_mgr

0

acl_mgr

0

app_mgr

-998

sarp_nd_mgr

-200

bfd_mgr

-200

bgp_mgr

0

chassis_mgr

-200

dev_mgr

-200

dhcp_client_mgr

0

dhcp_relay_mgr

0

eth_switch_mgr

-200

evpn_mgr

0

fib_mgr

0

gnmi_server

500

idb_server

-998

isis_mgr

0

json_rpc

500

l2_mac_learn_mgr

0

l2_mac_mgr

0

l2_static_mac_mgr

0

lag_mgr

0

linux_mgr

0

lldp_mgr

0

log_mgr

0

mcid_mgr

0

mgmt_server

-200

mpls_mgr

0

net_inst_mgr

-200

oam_mgr

500

ospf_mgr

0

plcy_mgr

0

qos_mgr

0

sdk_mgr

500

sflow_sample_mgr

500

sshd-mgmt

0

static_route_mgr

0

supportd

0

timesrv

0

vrrp_mgr

0

vxlan_mgr

0

xdp_cpm

-200

16.5.4. Application manager extensions

The cgroup feature provides two additional parameters in the app_mgr YAML file, the cgroup parameter and the oom-score-adj parameter.

The app_mgr uses the cgroup parameter to launch an application within a specific cgroup. A valid value for the cgroup parameter is the path of a cgroup as specified in the cgroup profile (equivalent to /profiles/name[]/path), from the cgroupv2 root. If this cgroup does not exist, the app_mgr launches the user application from the default cgroup profile path workload.slice/secondary.

The app_mgr uses the oom-score-adj parameter to set the out-of-memory adjust score for a process. This score is fed into the SR Linux OOM killer. Valid oom-score-adj scores are any value in the range of -1000 to 1000. A process with a score of -1000 is least likely to be killed while a process with a score of 1000 is most likely to be killed. At -1000, a process can use 100% of memory and still avoid being terminated by the OOM killer; however, SR Linux will kill the process more frequently.

The cgroup parameter and the oom-score-adj parameter are shown in the output below.

bgp_mgr:
   path: @SRLINUX_BINARY_INSTALL_PREFIX@
   launch-command: @YAML_LAUNCH_ENVIRONMENT@ ./sr_bgp_mgr
   search-command: ./sr_bgp_mgr
   oom-score-adj: 0
   wait-for-config: Yes
   author: 'Nokia'
   restricted-operations: ['reload']
   cgroup: workload.slice/primary
   yang-modules:
       names:
           - "srl_nokia-bgp"
           - "srl_nokia-bgp-vpn"
           - "srl_nokia-rib-bgp"
           - "srl_nokia-system-network-instance-bgp-vpn"
       tools:
           - "srl_nokia-tools-bgp"
       source-directories:
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/ietf"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/common"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/interfaces"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/network-
instance"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/routing-policy"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/system"

16.5.5. Debugging cgroups

Cgroup debugging capability is available through:

  1. SR Linux CLI commands
  2. Linux-provided CLI commands

16.5.5.1. SR Linux cgroup debugging commands

The SR Linux provides CLI commands to perform the following:

  1. check the usage of existing cgroups
  2. show information about the OOM adjust score of applications managed by the app_mgr
  3. show information about the cgroups that are associated with the applications that are managed by the app_mgr
  4. list all of the applications associated with a specified cgroup

16.5.5.1.1. Checking existing cgroup usage

The output below is an example of checking existing cgroup usage.

--{ running }--[  ]--  
A:rifa# info from state platform control A cgroup *
    platform {                    
        control A {                
            cgroup /mnt/cgroup/unified/user.slice/default {
                memory-statistics {
                    current 0 
                    current-swap 0    
                    anon 0        
                    kernel-stack 0  
                    slab 0      
                    sock 0
                    anon-thp 0      
                    file 0          
                    file-writeback 0  
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 0
                    system 0
                }
            }
            cgroup /mnt/cgroup/unified/workload.slice/primary {
                memory-statistics {
                    current 1621110784
                    current-swap 0
                    anon 1327427584
                    kernel-stack 2396160
                    slab 12939264
                    sock 151552
                    anon-thp 855638016
                    file 250916864
                    file-writeback 0
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 22082989551
                    system 11252275018
                }
            }
            cgroup /mnt/cgroup/unified/workload.slice/secondary {
                memory-statistics {
                    current 7827456
                    current-swap 0
                    anon 4501504
                    kernel-stack 73728
                    slab 2453504
                    sock 8192
                    anon-thp 0
                    file 0
                    file-writeback 0
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 34323091
                    system 12342430
                }
            }
        }
    }

16.5.5.1.2. Showing current OOM adjust scores

The output below is an example of showing information about the current OOM adjust scores for all applications that are managed by the app_mgr.

--{ running }--[  ]-- 
A:rifa# info from state system app-management application * oom-score-adj
    system {                     
        app-management {          
            application aaa_mgr {
                oom-score-adj 0          
            }                  
            application acl_mgr {
                oom-score-adj 0         
            }                  
            application app_mgr {
                oom-score-adj -998    
            }                  
            application arp_nd_mgr {
                oom-score-adj -200
            }                  
|
|
|
            application vxlan_mgr {
                oom-score-adj 0
            }
            application xdp_lc_1 {
                oom-score-adj -200
            }
        }
    }

16.5.5.1.3. Showing cgroup information

The output below is an example of showing information about cgroups that are associated with applications managed by the app_mgr.

--{ running }--[  ]--  
# info from state system app-management application * cgroup
    system {
        app-management {         
            application aaa_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary  
            }
            application acl_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application arp_nd_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application bgp_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }     
|
|
|
            application sshd-mgmt {
                cgroup /mnt/cgroup/unified/workload.slice/secondary  
            }
            application supportd {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application vxlan_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application xdp_lc_1 {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
        }
    }

16.5.5.1.4. Listing all the applications associated with a specified cgroup

Use the tools system cgroup command pgrep cgroup cgroupname command to list all the applications associated with a specified group; the output below shows an example.

--{ running }--[  ]--
# tools system cgroup command pgrep cgroup workload.slice/primary
+-------+----------------------+
|  Pid  |       Process        |
+=======+======================+
| 3373  | sr_app_mgr           |
| 3385  | sr_supportd          |
| 3402  | sr_device_mgr        |
| 3460  | sr_idb_server        |
| 3471  | sr_aaa_mgr           |
| 3482  | sr_acl_mgr           |
| 3518  | sr_arp_nd_mgr        |
| 3542  | sr_chassis_mgr       |
| 3560  | sr_dhcp_client_mgr   |
| 3574  | sr_evpn_mgr          |
| 3586  | sr_fib_mgr           |
| 3598  | sr_l2_mac_learn_mgr  |
| 3611  | sr_l2_mac_mgr        |
| 3621  | sr_lag_mgr           |
| 3631  | sr_linux_mgr         |
| 3641  | sr_log_mgr           |
| 3651  | sr_mcid_mgr          |
| 3681  | sr_mgmt_server       |
| 3702  | sr_net_inst_mgr      |
| 3724  | sr_oam_mgr           |
| 3743  | sr_sdk_mgr           |
| 3753  | sr_sflow_sample_mgr  |
| 3772  | sr_xdp_lc_1          |
| 3874  | sudo                 |
| 3895  | sudo                 |
| 3902  | sudo                 |
| 3921  | sudo                 |
| 3930  | sr_qos_mgr           |
| 3953  | sr_gnmi_server       |
| 3979  | sr_json_rpc          |
| 3990  | sr_vxlan_mgr         |
| 16740 | sr_json_wkr          |
| 16742 | python3              |
| 17013 | python3              |
| 18474 | sr_json_wkr          |
| 18478 | python3              |
| 18671 | python3              |
| 20671 | sr_lldp_mgr          |
| 21177 | sr_bgp_mgr           |
| 21190 | sr_plcy_mgr          |
| 12753 | sr_l2_static_mac_mgr |
| 27353 | rsyslogd             |
| 9777  | python               |
+-------+----------------------+

16.5.5.2. Linux-provided cgroup debugging commands

The following Linux-provided CLI commands are available for debugging cgroups:

  1. the systemd-cgls command
  2. the systemd-cgtop command

The systemd-cgls command dumps the cgroup hierarchy. The output below shows an example of the systemd-cgls command.

[linuxadmin@srlinux unified]$ systemd-cgls
+-1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
+-system.slice
  +-rngd.service
  ¦ +-2725 /sbin/rngd -f
  +-systemd-udevd.service
  ¦ +-868 /usr/lib/systemd/systemd-udevd
  +-system-serial\x2dgetty.slice
  ¦ +-serial-getty@ttyS0.service
  ¦   +-1120 login -- linuxadmin
  ¦   +-8320 -bash
  ¦   +-9262 sendacct "systemd-cgls"
  ¦   +-9263 systemd-cgls
  ¦   +-9264 less
  +-srlinux.service
  ¦ +- 3092 /usr/bin/sudo /opt/srlinux/bin/sr_linux
  ¦ +- 3120 runuser --user srlinux --group srlinux -- /opt/srlinux/bin/sr_linux
  ¦ +- 4019 /bin/bash /opt/srlinux/bin/sr_linux --user-env-switched
  ¦ +- 4039 ./sr_app_mgr
  ¦ +- 4051 ./sr_supportd --server-mode
  ¦ +- 4068 ./sr_device_mgr
  ¦ +- 4219 ./sr_idb_server
  ¦ +- 4229 ./sr_eth_switch
  ¦ +- 4288 ./sr_aaa_mgr
  ¦ +- 4299 ./sr_acl_mgr
  ¦ +- 4314 ./sr_arp_nd_mgr
  ¦ +- 4324 ./sr_chassis_mgr
  ¦ +- 4337 ./sr_dhcp_client_mgr
  ¦ +- 4365 ./sr_evpn_mgr
  ¦ +- 4381 ./sr_fib_mgr
  ¦ +- 4395 ./sr_l2_mac_learn_mgr
  ¦ +- 4411 ./sr_l2_mac_mgr
  ¦ +- 4423 ./sr_lag_mgr
  ¦ +- 4445 ./sr_linux_mgr
  ¦ +- 4494 ./sr_log_mgr
  ¦ +- 4510 ./ntpd -c /etc/ntps.conf -g
  ¦ +- 4526 ./sr_mcid_mgr
  ¦ +- 4574 ./sr_mgmt_server
  ¦ +- 4608 ./sr_net_inst_mgr
  ¦ +- 4626 ./sr_oam_mgr
  ¦ +- 4640 ./sr_sdk_mgr
  ¦ +- 4658 ./sr_sflow_sample_mgr
  ¦ +- 4681 ./sr_xdp_cpm
  ¦ +- 5197 /usr/bin/sudo -Enu root /usr/bin/sudo -Enu gnmirpc bash -c ./sr_gnmi
  ¦ +- 5224 /usr/bin/sudo -Enu root /usr/bin/sudo -Enu jsonrpc bash -c ./sr_json
  ¦ +- 5243 /usr/bin/sudo -Enu gnmirpc bash -c ./sr_gnmi_server
  ¦ +- 5248 ./sr_qos_mgr
  ¦ +- 5261 /usr/bin/sudo -Enu jsonrpc bash -c ./sr_json_rpc
  ¦ +- 5290 ./sr_gnmi_server
  ¦ +- 5302 ./sr_json_rpc
  ¦ +- 5693 /usr/sbin/sshd -f /etc/ssh/sshd_config_mgmt
  ¦ +- 6361 /usr/sbin/rsyslogd -i /var/run/srlinux/rsyslogd.pid
  ¦ +-21936 ./sr_bfd_mgr
  ¦ +-22018 ./sr_bgp_mgr
  ¦ +-22132 ./sr_isis_mgr
  ¦ +-22242 ./sr_lldp_mgr
  ¦ +-22321 ./sr_mpls_mgr
  ¦ +-22437 ./sr_ospf_mgr
  ¦ +-22588 ./sr_plcy_mgr
  ¦ +-22701 ./sr_static_route_mgr
  ¦ +-22796 sr_json_wkr
  ¦ +-22797 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  ¦ +-22953 ./ntpd -c /etc/mntp.conf -g
  ¦ +-23023 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  ¦ +-23736 sr_json_wkr
  ¦ +-23741 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  ¦ +-24009 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  ¦ +-27488 ./dnsmasq --conf-file=/etc/dnsmasq.conf
  +-polkit.service
  ¦ +-2701 /usr/lib/polkit-1/polkitd --no-debug
  +-ztpapi.service
  ¦ +-1114 /opt/srlinux/ztp/virtual-env/bin/python -m ztp.ztphttp
  +-systemd-journald.service
  ¦ +-840 /usr/lib/systemd/systemd-journald
  +-sshd.service
  ¦ +-985 /usr/sbin/sshd -D
  +-crond.service
  ¦ +-1133 /usr/sbin/crond -n
  +-sr_watchdog.service
  ¦ +-1155 sr_wd
  +-dbus.service
  ¦ +-991 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
  +-systemd-logind.service
    +-1062 /usr/lib/systemd/systemd-logind

The systemd-cgtop command dumps the current usage of each cgroup. The output below shows an example of the systemd-cgtop command.

Path                                     Tasks   %CPU   Memory  Input/s Output/s
 
/                                          186      -        -        -        -
/system.slice/crond.service                  1      -        -        -        -
/system.slice/dbus.service                   1      -        -        -        -
/system.slice/polkit.service                 1      -        -        -        -
/system.slice/rngd.service                   1      -        -        -        -
/system.slice/sr_watchdog.service            1      -        -        -        -
/system.slice/srlinux.service               53      -        -        -        -
/system.slice/sshd.service                   1      -        -        -        -
/system.slic...ial-getty@ttyS0.service       3      -        -        -        -
/system.slice/systemd-journald.service       1      -        -        -        -
/system.slice/systemd-logind.service         1      -        -        -        -
/system.slice/systemd-udevd.service          1      -        -        -        -
/system.slice/ztpapi.service                 1      -        -        -        -