9. gRPC

gRPC is a modern, open-source, high-performance RPC framework that runs in any environment. In SR OS, this framework is used to implement the gRPC server, which can then be used for configuration management or telemetry.

The gRPC transport service uses HTTP/2 bidirectional streaming between the gRPC client (the data collector) and the gRPC server (the SR OS device). A gRPC session is a single connection from the gRPC client to the gRPC server over the TCP/TLS port.

The gRPC service runs on port 57400 by default in SR OS. The service is not configurable.

A single gRPC server supports concurrent gRPC sessions and channels.

  1. There is a maximum of eight concurrent gRPC sessions for all of the gRPC clients.
  2. There is a maximum of 225 concurrent gRPC channels for all of the gRPC clients.

Figure 22 shows the gRPC protocol stack.

Figure 22:  Protocol Stack 

9.1. Security Aspects

9.1.1. TLS-Based Encryption

The gRPC server on SR OS can operate in two modes:

  1. without TLS encryption
  2. with TLS encryption

TLS encryption is used for added security. However, TLS encryption can be disabled in lab environments.

If TLS is not used, gRPC messages are not encrypted and user-names and passwords required in gRPC communication are visible to anyone capturing the packets. Therefore, Nokia recommends disabling TLS encryption only in a closed environment.

Before a gRPC connection will come up without TLS, the following conditions must both be met:

  1. no TLS server profile is assigned to the gRPC server
  2. the allow-unsecure-connection flag is set

The following summarizes the process of encryption:

  1. To use TLS encryption:
    1. The gRPC session must be in an encrypted state.
    2. If the gRPC client and gRPC server are unable to negotiate an encrypted gRPC session, the gRPC session fails and the gRPC server sends an error.
    3. Fallback from an encrypted to an unencrypted gRPC session is not allowed.
    For information about how to configure TLS with gRPC, see the TLS chapter.

9.1.2. Authentication

User authentication is based on following principles:

  1. Each RPC sent by the gRPC client carries a username and password.
  2. For the first RPC in the gRPC session, the gRPC server tries to authenticate the user using the specified authentication order, such as using the local user database, RADIUS, or TACACS+.
    For example, if TACACS+ is first in the authentication order, the gRPC server sends a request to the TACACs+ server to authenticate the gRPC user.
  3. For the subsequent RPCs on that same authenticated gRPC session, the username and password are re-authenticated only if changed.
  4. When no username and password are provided with the RPC, the gRPC server returns an error.
  5. If the RPC user is changed, any active subscriber RPCs on that same gRPC session are terminated by the gRPC server.
  6. If the RPC password is changed, the active gRPC session will continue to exist until a different username and password is sent in a subsequent RPC, or the gRPC session is terminated.
  7. Each message is carried over a gRPC session that was previously encrypted; the session is not re-encrypted.
  1. SR OS device authentication
    1. The gRPC clients do not share gRPC sessions. Each gRPC client starts a separate gRPC session.
    2. When a gRPC session is established, the gRPC server certificates are verified by the gRPC client to ensure that every gRPC server is authenticated by the gRPC client.
    3. If gRPC is shut down on the gRPC server and a gRPC client is trying to establish a gRPC session, the gRPC client will get an error for every RPC sent.
    4. If gRPC is shut down on the gRPC server and a gRPC session is established, all active RPCs are gracefully terminated and an error is returned for every active RPC.

9.2. gNMI Service

The gRPC Network Management Interface (gNMI) is a gRPC based protocol for network management functions, such as changing the configuration of network elements and retrieving state information. In addition, gNMI provides functionality necessary for supporting telemetry. The gNMI service is specified in the OpenConfig forum.

9.2.1. gNMI Service Definitions

The SR OS gRPC server supports gNMI version 0.7.0, and in particular, the following RPC operations:

  1. Capability RPC
  2. Set/Get RPCs
  3. Subscribe RPC

As in NETCONF RPCs, gNMI RPCs that are sent to the SR OS system are logged in security log and they are marked as authorized or unauthorized, and include information such as username, time, RPC type, and IP address of the client.

9.2.1.1. Capability Discovery

In gNMI service, the client discovers the capabilities of the gRPC server through a Capability-Discovery RPC, which consists of “CapabiltyRequest” and “CapabilityResponse” messages.

During this message exchange, the gRPC server informs the client about following attributes:

  1. supported gNMI version
  2. supported models
  3. supported encodings

The SR OS server announces the supported models based on the configuration under config>system>management-interface>yang-modules. The supported models includes the NOKIA-YANG or OpenConfig (OC) models.

The advertised module names and organizations are as follows:

  1. nokia-conf, org = "Nokia"
  2. nokia-state, org = "Nokia"
  3. openconfig, org = "OpenConfig working group" (as specified by the 'organization' in the YANG models)
  4. version - the version number is be defined as follows:
    1. for NOKIA YANG models, the version number corresponds to an SR OS release number, for example, "16.0.r1"
    2. for OC YANG models, the version number corresponds to a version number defined in "oc-ext:openconfig-version" that is included in the respective YANG models
    3. for OC-YANG models, including NOKIA deviations, the version number corresponds to an SR OS release number, for example, "16.0.r1"

The following is an example of a “Capabilities Response Message”:

Going to send message of type gnmi.CapabilityResponse:
 .gnmi_version: 0.4.0
 .supported_encodings (1):
  .encoding: 0 = JSON
 .supported_models (47):
  { .name: 'nokia-conf', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'nokia-state', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'openconfig-
bgp', .organization: 'OpenConfig working group', .version: '4.0.1' }
<snip>
  { .name: 'nokia-sr-openconfig-if-ethernet-deviations', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'nokia-sr-openconfig-if-ip-deviations', .organization: 'Nokia', .version: '16.0.r1'..."

9.2.1.2. Get/Set RPC

Information is retrieved from the NE using GET RPC messages, which consists of “GetRequest” and “GetResponse” messages. The client asks for a given information by specifying following:

  1. A set of paths — all rules to a path definition apply, as specified in the gNMI specification
  2. Type — configuration, state, or operational data
  3. Encoding — in accordance to server advertisement during capability discovery
  4. Use_models — this message is ignored

There is an upper limit on the size of the “GetResponse” message. This limit cannot exceed 100MB. If the limit is exceeded, the SR OS gRPC server responds with an error message.

In order to modify the information in an NE element, a SET gRPC message is used. This gRPC supports three types of transactions:

  1. delete
  2. replace
  3. update

With a gNMI SET RPC, SR OS authorizes all configuration changes, that is, it checks the YANG tree and authorizes every changed element.

The deletion of a container results in the deletion of any children containers that are authorized for deletion as well as their contents. Children containers that are not authorized for deletion, as well as their contents, are retained. For example, upon deletion of configure system, configure system security is not deleted because the deletion of that child container is not authorized.

Note:

Only changes to configuration values are checked for authorization. A configuration command that simple writes the same value to a leaf will succeed even if the user does not have access to that leaf (but there will be no resulting change to the configuration).

For example, when a user is not authorized to change access li, but attempts to change it for another user who already has access li, SR OS allows that action because there is no change in value.

9.2.1.3. Subscribe RPC

The subscribe RPC is part of the telemetry support in gNMI.

A subscription is initiated from the gRPC client by sending a Subscribe RPC that contains a "SubscribeRequest" message to the gRPC server. A prefix can be specified to be used with all paths specified in the "SubscribeRequest". If a prefix is present, it is appended to the start of every path to provide a full path.

A subscription contains:

  1. a list of one or more paths. The following conditions apply:
    1. A path represents the data tree as a series of repeated strings and elements. Each element represents a data tree node name and its associated attributes.
    2. A path must be syntactically valid within the set of schema modules that the gRPC server supports.
    3. The path list cannot be modified during the lifetime of the subscription.
    4. If the subscription path is to a container node, all child leafs of that container node are considered to be subscribed to.
    5. Any specified path must be unique within the list; paths cannot be repeated within the list. An error is returned if the same path is used more than one time in a single subscription.
    6. A specified path does not need to pre-exist within the current data tree on the gRPC server. If a path does not exist, the gRPC server continues to monitor for the existence of the path. Assuming that the path exists, the gRPC server transmits telemetry updates.
    7. The gRPC server does not send any data for a non-existent path; for example, if a path is non-existent at the time of subscription creation or if the path was deleted after the subscription was established.
    8. The maximum number of paths for all subscriptions on a single SR OS device is 14400. A path using a wildcard is still considered a single path.
  2. a subscription mode of one of the following types:
    1. ONCE mode — the server returns only one notification containing all information the client has subscribed to. In general, retrieving large amounts of information from the NE can be done using telemetry: “SubscribeRequest” message with ONCE subscription type.
    2. ON_CHANGE mode — the server returns notifications only when the value of the subscribed field changes. See ON_CHANGE Subscription Mode for more information.
    3. SAMPLE mode — the gRPC server sends notifications at the specified sampling interval
    4. TARGET_DEFINED mode — means ON_CHANGE for all states supporting ON_CHANGE notifications and SAMPLE mode for all other objects in the YANG tree
  3. a sample interval is supported for each path. If a sample interval of less than 1 s is specified, the gRPC server returns an error. If the sample interval is set to 0, the default value of 10 s is used. A sample interval is specified in nanoseconds (10 000 000 000 by default)

When a subscription is successfully initiated on the gRPC server, “SubscribeReponse” messages are sent from the gRPC server to the gRPC client. The “SubscribeResponse” message contains update notifications about the subscription's path list.

An update notification contains:

  1. a timestamp of the statistics collection time, represented in nanoseconds
  2. a prefix:
    1. If a prefix is present, it is logically appended to the start of every path to provide the full path.
    2. The presence of a prefix in the “SubscriptionResponse” message is not related to the presence of a prefix in the original “SubscriptionRequest” message. The prefix in the “SubscriptionResponse” message is optimized by the gRPC server.
  3. a list of updates (path and value pairs):
    1. A path represents the data tree path as a series of repeated strings or elements, where each element represents a data tree node name and its associated attributes. See Schema Paths for more information.
    2. The “TypedValue” message represents the data tree node’s value, where the encoding is “JSON” or “Bytes”, depending on the information in the “SubscribeRequest” message.

A sync response notification is sent one time, after the gRPC server sends all of the updates for the subscribed-to paths. The sync response must be set to “true” for the gRPC client to consider that the stream has synced one time. A sync response is used to signal the gRPC client that it has a full view of the subscribed-to data.

The gRPC server sends an error if required. The error contains a description of the problem.

AAA authorization is not performed when a subscription request is received. Authorization is only performed when telemetry data is being sent in notifications. For telemetry data, only configuration elements are subject to command authorization. State elements in telemetry data are not subject to command authorization.

9.2.1.3.1. Bytes Encoding

Bytes encoding is performed by the gRPC server, which encodes the values of the leafs as typed values. Table 119 lists the mapping of the individual YANG types to the typed values defined in the gNMI specification in the GitHub repository.

Table 119:  Mapping of YANG Types to gNMI-specified Typed Values  

YANG Type

Typed Value

binary

bytes_val

bits

string_val

boolean

bool_val

decimal64

decimal_val

empty

bool_val

enumeration

string_val

identityref

string_val

instance-identifier

string_val

int8

int_val

int16

int_val

int32

int_val

int64

int_val

leafref

type of referenced leaf

string

string_val

uint8

uint_val

uint16

uint_val

uint32

uint_val

uint64

uint_val

union

type of active union member

9.2.1.3.2. ON_CHANGE Subscription Mode

SR OS supports ON_CHANGE subscription mode. This subscription mode indicates that Notification messages are sent as follows:

  1. after the “SubscriptionRequest” message is received
  2. every time the corresponding leaf value is changed

The notification message, as a response to an ON_CHANGE subscription, always contains the new value of the corresponding leaf, as defined in gNMI specification.

The ON_CHANGE subscription is supported for all configuration events as well as for selected state leafs. The tools command can display all state leafs supporting the ON_CHANGE subscription.

ON_CHANGE subscription is accepted for all valid paths. The server sends ON_CHANGE notifications only for leafs within this path that support ON_CHANGE notifications.

9.2.1.4. Publish RPC

With dial-out telemetry, where the SR OS node is the gRPC client instead of the gRPC server, the SR OS node sends a Publish RPC with a “SubscribeResponse” message to the gRPC server. (See Dial-out Telemetry.)

Because the current gnmi.proto definition does not support dial-out mode, a protobuf definition is introduced, with a separate gRPC service, as follows:

NOKIA-DialOut.proto
      option (dialout_service) = 0.1.0
      service gMIDialOut {
              rpc Publish(stream SubscribeResponse) returns (stream PublishResponse)
      }
      message PublishResponse {
      }

The preceding proto file definition reuses the “SubscribeResponse” message defined for dial-in telemetry, in accordance with the gNMI specification.

9.2.1.5. Schema Paths

Telemetry subscriptions include a set of schema paths used to identify which data nodes are of interest to the collector.

The paths in Telemetry Subscribe RPC requests follow the basic conventions described in the OpenConfig gnmi-path-conventions.md published on github.com (version 0.2.0 from February 24th, 2017).

A path consists of a set of path segments often shown with a “/” character as a delimiter; for example, /configure/router[router-instance=Base]/interface[interface-name=my-interface1]/description.

These paths are encoded as a set of individual string segments in gnmi.proto (without any “/” characters); for example, ["configure", "router[router-instance=Base]", "interface[interface-name=my-interface1]", "description"].

A path selects an entire subtree of the data model and includes all descendants of the node indicated in the path. Table 120 describes the types of schema paths that are supported in SR OS telemetry.

Table 120:  Schema Paths  

Path example

Description

/configure/router[router-instance=Base]/interface[interface-name=abc]

Selects all config leafs of interface abc and all descendants.

/configure/router[router-instance=Base]/interface[interface-name=abc]/description

Selects only the description leaf of interface abc.

/state/router[router-instance=Base]/interface[interface-name=*]

Selects all state information for all base router interfaces using a wildcard in a single segment of a path.

/configure/router[router-instance=Base]/interface[interface-name=*]/description

Selects all state information for all base router interfaces using a wildcard in a single segment of a path.

/

The root path. This selects all config and state data from all models (in all namespaces) supported on the router. Encoded as “” in gRPC/gPB.

The following list describes types of telemetry paths that are not supported in SR OS.

  1. Wildcards for entire path segments are not supported.
    For example: /state/service/*/oper-status
  2. If a wildcard is used for any key of a list, a wildcard must be used for all the keys of that list. In a single path segment, all the keys must either have specific values or all the keys must have wildcards. A mix of wildcards and specific values for different parts of a list key is not supported.
    For example:
    Supported:
    /a/b[key1=*][key2=*]/c[key1=foo]
    /a/b[key1=foo][key2=bar]/c[key1=*]
    Not supported:
    /a/b[key1=foo][key2=*]
  3. Functions such as “current()”, “last()” and mathematical operators, such as stat<5 or octets>3 are not supported in paths. The “|” (OR operator, used to select multiple paths) is not supported.
  4. The “//” wildcard pattern is not supported.
    For example: /state//oper-status

The following list describes types of telemetry paths that are supported in SR OS.

  1. Wildcards in multiple segments of a path are supported.
    For example: /state/card[slot-number=*]/mda[mda-slot=*]

9.2.2. gNMI Service Use Cases

The gNMI Service can be used for the following:

  1. Telemetry
  2. NE Configuration Management

9.2.2.1. Telemetry

Telemetry is a network monitoring and fault management framework. Telemetry is driven by the need to use fresh data obtained from the network to make fast networking decisions such as traffic optimization and preventative troubleshooting.

9.2.2.1.1. Dial-in Telemetry

When the data collector initiates the gRPC connection, the SR OS node assumes the role of the gRPC server and the collector is the client. This is referred to as dial-in telemetry, where the SR OS node pushes data to the receiver (collector). Figure 23 shows the telemetry session initiated from the collector to the SR OS node via the Subscribe RPC.

Figure 23:  Dial-in Telemetry Session 

9.2.2.1.1.1. Dynamic Subscriptions

Dynamic subscriptions are created by the collector using the Subscribe RPC. These subscriptions are removed as soon as the gRPC session terminates. Dynamic subscriptions are currently supported only in dial-in mode.

9.2.2.1.1.2. Dial-in Telemetry Examples

This section contains examples of telemetry subscription requests and responses. The following examples are dumps of protobuf messages from a Python API. Formats may vary across different implementations.

Example 1 — Subscribe to a single path

2017-06-05 17:06:13,189 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=test]"
      element: "statistics"
      element: "ip"
      element: "in-packets"
    }
    mode: SAMPLE
    sample_interval: 10000000000
  }
}
2017-06-05 17:06:13,190 - RCVD::SubsribeResponse
2017-06-05 17:06:23,492 - RCVD::Subscribe
2017-06-05 17:06:23,492 - update {
  timestamp: 1496675183491595139
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=test]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:06:23,494 - RCVD::Subscribe
2017-06-05 17:06:23,494 - sync_response: true
2017-06-05 17:06:33,589 - RCVD::Subscribe
2017-06-05 17:06:33,589 - update {
  timestamp: 1496675213491595139
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=test]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““28””
  }
}
....
....

Example 2 — Subscribe to a single path with wildcard

2017-06-05 17:08:29,055 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=*]"
      element: "statistics"
      element: "ip"
      element: "in-packets"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-06-05 17:08:29,056 - RCVD::SubsribeResponse
2017-06-05 17:08:59,133 - RCVD::Subscribe
2017-06-05 17:08:59,133 - update {
  timestamp: 1496675339132056575
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=system]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:08:59,135 - RCVD::Subscribe
2017-06-05 17:08:59,135 - update {
  timestamp: 1496675339133006678
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:08:59,135 - RCVD::Subscribe
2017-06-05 17:08:59,135 - update {
  timestamp: 1496675339133006678
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_D]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:08:59,136 - RCVD::Subscribe
2017-06-05 17:08:59,136 - sync_response: true
2017-06-0517:09:29,139 - RCVD::Subscribe
2017-06-0517:09:29,139 - update {
  timestamp: 1496682569121314
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=system]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:09:29,142 - RCVD::Subscribe
2017-06-05 17:09:29,142 - update {
  timestamp: 1496682569124342
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:09:29,145 - RCVD::Subscribe
2017-06-05 17:09:29,145 - update {
  timestamp: 1496682569127344
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_D]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
....
....

Example 3: Subscribe to more than one path

2017-01-24 12:54:18,228 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "mpls"
      element: "statistics"
      element: "lsp-egress-stats[lsp-name=lsp_to_dest_f]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}

Example 4: Subscribe to a list with wildcard

2017-01-24 13:45:30,947 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=*]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}

Example 5: Subscribe to path where the object did not exist before subscription

2017-01-24 13:53:50,165 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-01-24 13:53:50,166 - RCVD::SubsribeResponse
2017-01-24 13:54:20,169 - RCVD::Subscribe
2017-01-24 13:54:20,169 - sync_response: true
2017-01-24 13:54:50,174 - RCVD::Subscribe
2017-01-24 13:54:50,174 - update {
  timestamp: 1485262490169309451
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
  }
  update {
...
...
  }
}

Example 6: Subscribe to a path where the object existed before subscription and then deleted after subscription

2017-01-24 14:00:41,292 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-01-24 14:00:41,294 - RCVD::SubsribeResponse
2017-01-24 14:01:11,295 - RCVD::Subscribe
2017-01-24 14:01:11,295 - update {
  timestamp: 1485262871290064704
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
  }
  update {
...
...
  }
}
2017-01-24 14:01:11,359 - RCVD::Subscribe
2017-01-24 14:01:11,359 - sync_response: true
2017-01-24 14:01:41,293 - RCVD::Subscribe
2017-01-24 14:02:11,296 - RCVD::Subscribe

9.2.2.1.2. Dial-out Telemetry

When the SR OS node initiates the gRPC connection, the SR OS node assumes the role of the gRPC client. This is referred to as dial-out telemetry. Figure 24 shows the telemetry session initiated from the SR OS node to the collector via a Publish RPC.

Figure 24:  Dial-out Telemetry Session 

9.2.2.1.2.1. Persistent Subscriptions

Persistent subscriptions are configured on the SR OS node and they are not cleared when the gRPC session terminates. Persistent subscriptions are supported only in dial-out mode.

A persistent subscription associates one or more paths with corresponding destinations via sensor groups.

Every subscription has an associated administrative state as well as an operational state. If a connection is lost, the operational state goes down. If the collector does not receive the data but the SR OS appears to have a connection and the subscription is up, the connection can be reset by setting the administrative state down and then back up.

Destinations are defined in the form of destination groups. A destination group supports up to two destinations, where the destinations are served in a round-robin fashion. SR OS attempts to connect the first destination, and if successful, the telemetry data is sent to that destination. If the connection to the first destination fails (initially or during operation), SR OS attempts to connect or reconnect to the second destination, if it is configured. All configured destinations and local addresses should be reachable in the specified routing instances.

When the SR OS node initiates the gRPC connection via the Publish RPC, it includes the subscription name and the configured system name in the metadata. The collector can use this information to associate individual notification messages with the node and subscription.

Modifying any parameter of the active subscription causes the SR OS node to close the gRPC connection before attempting a reconnection.

When a gRPC connection is lost, the SR OS node continually attempts to establish a new session with the collector.

9.2.2.1.2.2. QoS Marking

The QoS marking of the IP packets carrying notifications can be configured under persistent subscription. IP packets to a specified destination are marked according to the configuration of the first subscription opened to the destination. This DSCP marking is maintained, regardless of any configuration changes, as long as the dial-out connection to the specified destination is open. If the destination is disconnected for any reason, the DSCP marking must be redefined when the connection is reestablished.

9.2.2.1.2.3. Configuring Dial-out Telemetry

The dial-out telemetry configuration process includes the following elements:

  1. sensor group
    The sensor group specifies one or more schema paths from which data is streamed to the collector.
  2. destination group
    The destination group specifies the destination addresses (and ports) that the router uses to send the telemetry data.
  3. persistent subscription
    The persistent subscription associates a sensor group with a destination group and specifies streaming parameters for the telemetry data. For example, the subscription mode can be specified (ON_CHANGE, SAMPLE, or TARGET_DEFINED) for the subscription.

Dial-out telemetry can be configured via the MD-CLI. For more information about using the MD-CLI, refer to the 7450 ESS, 7750 SR, and 7950 XRS MD-CLI User Guide.

For more information about the MD-CLI configuration commands, refer to the 7450 ESS, 7750 SR, and 7950 XRS MD-CLI Command Reference Guide.

The following example shows a sample dial-out telemetry configuration in the MD-CLI.

[ex:configure system telemetry]
A:admin@node-2# info
    destination-group "quick_cfg_dg_1" {
        description "Destination Group 1"
        allow-unsecure-connection
        destination 192.168.65.5 port 40001 {
            router-instance "Base"
        }
        destination 192.168.65.5 port 40002 {
            router-instance "Base"
        }
    }
    persistent-subscriptions {
        subscription "quick_cfg_sub_1" {
            admin-state enable
            description "Subscription 1"
            sensor-group "quick_cfg_sg"
            mode sample
            sample-interval 1234
            destination-group "quick_cfg_dg_1"
            local-source-address 1.2.3.4
            originated-qos-marking cp19
            encoding bytes
        }
    }
    sensor-groups {
        sensor-group "quick_cfg_sg" {
            description "Sensor Group"
            path "/state/router[router-name=Base]/interface[interface-name=test]/statistics/ip" { }
        }
    }

The following sample outputs show telemetry information using the show command.

*A:node-2>show>system>telemetry>persistent# subscription "quick_cfg_sub_2"
===============================================================================
Telemetry persistent subscription
===============================================================================
Subscription Name     : quick_cfg_sub_2
Administrative State  : Enabled
Operational State     : Up
Subscription Id       : 198
Description           :
Sensor Group          : quick_cfg_sg_2
Destination Group     : quick_cfg_dg_2
Path Mode             : sample
Sample Interval       : 1000 ms
Encoding              : bytes
===============================================================================
*A:node-2>show>system>telemetry>persistent# subscription "quick_cfg_sub_2" paths
===============================================================================
Telemetry persistent subscriptionrsistent# subscription "quick_cfg_sub_2" paths
===============================================================================
Subscription Name     : quick_cfg_sub_1nt# subscription "quick_cfg_sub_2" paths
Administrative State  : Enabled
Operational State     : Up
Subscription Id       : 198
Description           :
Sensor Group          : quick_cfg_sg_2
Destination Group     : quick_cfg_dg_2
Path Mode             : sample
Sample Interval       : 1000 ms
Encoding              : bytes
-------------------------------------------------------------------------------
Paths
-------------------------------------------------------------------------------
Path                  : /state
Finished Samples      : 178
Deferred Samples      : 402
Total Collection Time : 405223 ms
Min Collection Time   : 2021 ms
Avg Collection Time   : 2276 ms
Max Collection Time   : 2956 ms
-------------------------------------------------------------------------------
No. of paths          : 1
===============================================================================
*A:node-2>show>system>telemetry>persistent# subscription "quick_cfg_sub_2" destinations
===============================================================================
Telemetry persistent subscription
===============================================================================
Subscription Name     : quick_cfg_sub_2
Administrative State  : Enabled
Operational State     : Up
Subscription Id       : 198
Description           :
Sensor Group          : quick_cfg_sg_2
Destination Group     : quick_cfg_dg_2
Path Mode             : sample
Sample Interval       : 1000 ms
Encoding              : bytes
-------------------------------------------------------------------------------
Destinations
-------------------------------------------------------------------------------
Destination           : 192.168.65.1
Port                  : 40001
Operational State     : Down
Last Oper Down Reason : MINOR: TELEMETRY #2353: RPC refused by peer
Last Oper Change      : 2020/04/06 21:19:29
Connection Attempts   : 22
Notification Count    : 0
Total Notification Co*: 2315653
-------------------------------------------------------------------------------
Destination           : 192.168.65.1
Port                  : 40002
Operational State     : Up
Last Oper Down Reason : MINOR: TELEMETRY #2356: Canceled by config change
Oper Router Instance  : management
Last Oper Change      : 2020/04/06 21:19:30
Connection Attempts   : 22
Notification Count    : 3783151
Total Notification Co*: 5034573
-------------------------------------------------------------------------------
No. of destinations   : 2
* indicates that the corresponding row element may have been truncated.
===============================================================================

9.2.2.2. NE Configuration Management

Figure 25 shows NE configuration and information retrieval using the gNMI service.

Figure 25:  NE Configuration and Information Retrieval using gNMI Service 

In the context of gNMI, every SET RPC appears as an single commit operation, regardless of the number of paths included in the message. Both, NOKIA and OC models are supported by gNMI SET/GET RPC.

An example of the SET RPC command (including the response message from the gRPC server) follows:

Example:
 gNMI_rpc - DEBUG - SENT::SetRequest
prefix {
}
update {
  path {
    elem {
      name: "configure"
    }
    elem {
      name: "system"
    }
  }
  val {
    json_val: {"location": "zurich"}
  }
}
gMI_rpc - DEBUG - RCVD::SetResponse
prefix {
}
response {
  path {
    elem {
      name: "configure"
    }
    elem {
      name: "system"
    }
  }
  op: UPDATE
}

An example of the GET RPC command (including the response message from the gRPC server) follows:

Example:
gNMI_rpc - INFO - SENT::GetRequest GET140550212650064
path {
  elem {
    name: "configure"
  }
  elem {
    name: "system"
  }
  elem {
    name: "location"
  }
}
type: CONFIG
2017-12-06 12:17:28,639 - gMI_rpc - INFO - RCVD::GetResponse GET140550212650064
notification {
  timestamp: 1512559048634751055
  update {
    path {
      elem {
        name: "configure"
      }
      elem {
        name: "system"
      }
      elem {
        name: "location"
      }
    }
    val {
      json_val: "zurich"
    }
  }
}

9.3. gNOI Services

The gRPC Network Operations Interface (gNOI) defines a set of gRPC-based micro-services for executing operational commands on network devices. This includes the gNOI CERT service, that provides certificate management. The individual RPCs and messages that perform the operations required for certificate management on the node are defined in the Git repository hosting service (GitHub).

9.3.1. Certificate Management for TLS Connections

This section describes the gNOI services certificates that SR OS supports for managing secure TLS connections.

The SR OS supports the following RPCs for managing certificates for secure TLS connections:

  1. RPC GetCertificates
  2. RPC CanGenerateCSR
  3. RPC Rotate
  4. RPC Install
Note:

The gNOI RPCs, are by default, disabled in the user profile.

9.3.1.1. RPC GetCertificates

RPC GetCertificates provide information to the controller about all active certificates on the server (SR OS node). Figure 26 shows the message sequence.

Figure 26:  RPC GetCertificates Message Flow 

The RPC GetCertificates messages include a GetCertificateRequest and a GetCertificateRespone message. The GetCertificatesResponse message shown in Figure 26 includes the following information:

  1. certificate_id — The SR OS uses a certificate file name as the certificate ID.
  2. CertificateType — This is always set to X509, because it is the only type that the SR OS supports.
  3. endpoint — Indicates the CERT profiles in the SR OS node that use this certificate; if multiple CERT profiles use the certificate, the names are concatenated with the separation character “/”.

9.3.1.2. RPC CanGenerateCSR

The RPC CanGenerateCSR message can be used to determine if the gRPC server (SR OS node) can generate a Certificate Signing Request (CSR). It is a simple request and response operation as shown in Figure 27.

Figure 27:  RPC CanGenerateCSR Message Flow 

The SR OS only supports RSA keys and X509 certificates, so it only responds positively if those values are filled in the respective fields. The key size must be between 512 and 8192. In all other cases, the SR OS responds negatively to the CanGenerateCSRRequest message.

9.3.1.3. RPC Rotate

RPC Rotate allows the controller to rotate an active certificate on the server. After the rotation is completed, a new certificate can be used without affecting existing TLS connections.

The following cases are supported for a certificate rotation:

  1. server capable of generating a CSR (see Figure 28)
  2. server not capable of generating a CSR (see Figure 29)

The SR OS supports both scenarios, although it is assumed that in most cases the CSR is generated on SR OS node.

The following steps apply to both scenarios:

  1. Generate the CSR
  2. Sign the CSR by the Certificate Authority (CA)
  3. Load the new certificate on the server
  4. Verify the new certificate by creating a new connection
  5. Finalize by confirming that the new certificate is being used

After the RPC Rotate is completed, all new connections use new keys.

Figure 28:  RPC Rotate Message Flow for CSRs Generated on the SR OS Node 
Figure 29:  RPC Rotate Message Flow When CSRs are Not Generated on the SR OS Node 

From the perspective of the interaction of the controller and the server (SR OS) two stages are the most important:

  1. message exchange to generate the CSR
  2. message exchange to load the new certificates on the server

Figure 30 shows a detailed content of the messages that are exchanged for CSR generation. The SR OS accepts requests only for the X509 certificate type, RSA key type, and a minimum key length of 512 bits.

Figure 30:  GenerateCSR Message Flow 

For RPC Rotate, the certificate_id points to an existing certificate on the node. All the other parameters in the GenerateCSRRequest message are not checked by the SR OS software explicitly. They are used by the internal API to generate the CSR and that result is transparently passed to the controller.

After the CA signs the certificates, the files are loaded to the server using LoadCSRRequest and LoadCSRResponse message exchange, as shown in Figure 31. If this message exchange is used in the context of RPC Rotate, the certificate_id should not be present in LoadCSRRequest message. When the SR OS receives the message, it performs all the necessary steps to load this certificate, including storing the certificate and key files on the disk.

Figure 31:  LoadCSRRequest/Response Message Flow 

The controller is responsible for verifying the connection with the new certificate (Step  4 in Figure 28 and Figure 29); SR OS treats this as an optional step.

After the whole RPC is successfully closed, the system can use the new certificate to start new TLS connections.

9.3.1.4. RPC Install

The controller can use RPC Install to install a new certificate on the server. After the certificate is installed, the server must be configured (assign a certificate and key files in the CERT profile) before the new certificate can be used.

The following two possible cases are supported for installing a certificate:

  1. server capable of generating a CSR (see Figure 32)
  2. server is not capable of generating a CSR (see Figure 33)

The SR OS supports both scenarios, although it is assumed that in most cases the CSR is generated on the SR OS node.

Both scenarios require the following steps:

  1. Generate the CSR
  2. Sign the CSR by the Certificate Authority (CA)
  3. Load the new certificate on the server
Figure 32:  RPC Install Message Flow for CSRs Generated on the SR OS Node 
Figure 33:  RPC Install Message Flow if CSRs are Not Generated on the SR OS Node) 

The message exchange during phases 1 and 3 is the same as shown in Figure 30 and Figure 31. The only difference, in the case of RPC Install, is that, a new certificate_id is used.

After new certificates are installed, the system must be configured before it can be used. Configuration is supported using the following methods:

  1. an existing gRPC session
  2. a CLI session, SNMP, and/or NETCONF

9.3.1.5. RPC RevokeCertificates

The purpose of the RPC RevokeCertificates is to render the existing certificate unusable by any client. In cases where the certificate being revoked by the client does not exist on the SR OS node, the corresponding RPC silently succeeds. The message flow is shown in Figure 34.

Figure 34:  RPC RevokeCertificates Message Flow 

9.4. gNOI System

In the gNOI System service, OpenConfig defines a generic interface to perform operational tasks on target network nodes. The specification can be found in following link: https://github.com/openconfig/gnoi/blob/master/system/system.proto.

These operations can be performed on individual targets, regardless of vendor. SR OS supports the following gNOI System RPCs:

  1. SetPackage RPC
  2. Reboot RPC
  3. CancelReboot RPC
  4. RebootStatus RPC
  5. SwitchControlProcessor RPC

9.4.1. SetPackage RPC

The SetPackage RPC allows the controller to place a software package on the target node. The file transfer is protected by the checksum. SR OS supports options where the controller can directly stream files to the target node. The remote download option is not supported.

9.4.2. Reboot, CancelReboot, and RebootStatus RPC

The Reboot RPC causes a target node to reboot. The controller can, in a RebootRequest message, specify the delay of the reboot and the actions that should be done with the system during reboot. SR OS supports only cold reboot, which means the whole node is shutdown and restarted.

The CancelReboot RPC allows the cancellation of any pending reboots.

The RebootStatus RPC allows the controller to query the status of a reboot for an individual component specified in the RebootStatus request. SR OS supports querying on a single component at the time.

9.4.3. SwitchControlProcessor RPC

The SwitchControlProcessor RPC allows the switching of the active Control-Processor to the Control-Processor that is provided in the request message. Since SR OS supports two Control Processors, one of the following paths should be given in the request message, depending which Control Processor is standby at that moment.

/state/cpm[cpm-slot=A]

/state/cpm[cpm-slot=B]

9.5. MD-CLI Service

The SR OS provides a proprietary management interface to use with the Network Interface Shell (NISH) tool which allows an MD-CLI style interface from a remote location in order to manage one or more SR OS nodes.

This feature is applicable only on SR OS platforms that support MD-CLI in Model-Driven mode.

This service operates using gRPC and therefore, the main gRPC service must also be enabled.

When enabled, the MD-CLI gRPC service provides MD-CLI schema information to the NISH client allowing users to remotely operate the SR OS device.

The MD-CLI gRPC service and the main gRPC service must be enabled on all nodes that will be managed using the nish client.

9.5.1. Remote Management Using a Remote Network Interface Shell Manager

When used together with the MD-CLI gRPC service, the remote management feature allows SR OS nodes to initiate communication with a remote NISH manager and announce to their availability to be managed using the NISH client.

This provides the NISH client with a dynamic view of the available nodes that it can manage. The remote management service does not perform or enable the actual management of the SR OS node using NISH. This communication is achieved directly from the NISH client to the SR OS node using the MD-CLI gRPC service.

This feature is particularly useful when deploying clusters of SR OS nodes that may dynamically join or leave a cluster, such as in scenarios using the Control and User Plane Separation (CUPS) BNG application with Virtualized Service Routers (VSRs).

A working NISH manager service is required on an external server to use the remote management feature. The absence of a working NISH manager will not stop remote management from being enabled within SR OS, nor will it stop the SR OS node from announcing its presence to the configured IP address or addresses of the NISH manager.

When a remote NISH manager is configured, the SR OS node initiates a gRPC session with the configured manager. The SR OS node sends a message to communicate its name, IP address (IPv4 and IPv6 are supported) and gRPC port to the NISH manager. The NISH manager responds with an acknowledgment message. The SR OS node periodically checks in with the NISH manager.

Figure 35 depicts the remote management initiation.

Figure 35:  Remote Management Service Initiation 

If the connection is interrupted, the SR OS node immediately attempts reconnection with the configured NISH managers.