8. gRPC

gRPC is a modern, open-source, high-performance RPC framework that runs in any environment. In SR OS, this framework is used to implement the gRPC server, which can be then be used for configuration management or telemetry.

The gRPC transport service uses HTTP/2 bidirectional streaming between the gRPC client (the data collector) and the gRPC server (the SR OS device). A gRPC session is a single connection from the gRPC client to the gRPC server over the TCP/TLS port.

The gRPC service runs on port 57400 by default in SR OS. The service is not configurable.

A single gRPC server supports concurrent gRPC sessions and channels.

  1. There is a maximum of eight concurrent gRPC sessions for all of the gRPC clients.
  2. There is a maximum of 225 concurrent gRPC channels for all of the gRPC clients.

Figure 21 shows the gRPC protocol stack.

Figure 21:  Protocol Stack 

8.1. Security Aspects

8.1.1. TLS-Based Encryption

The gRPC server on SR OS can operate in two modes:

  1. without TLS encryption
  2. with TLS encryption

TLS encryption is used for added security. However, TLS encryption can be disabled in lab environments.

If TLS is not used, gRPC messages are not encrypted and user-names and passwords required in gRPC communication are visible to anyone capturing the packets. Therefore, Nokia recommends disabling TLS encryption only in a closed environment.

Before a gRPC connection will come up without TLS, the following conditions must both be met:

  1. no TLS server profile is assigned to the gRPC server
  2. the allow-unsecure-connection flag is set

The following summarizes the process of encryption:

  1. To use TLS encryption:
    1. The gRPC session must be in an encrypted state.
    2. If the gRPC client and gRPC server are unable to negotiate an encrypted gRPC session, the gRPC session fails and the gRPC server sends an error.
    3. Fallback from an encrypted to an unencrypted gRPC session is not allowed.
    For information about how to configure TLS with gRPC, see the TLS chapter.

8.1.2. Authentication

User authentication is based on following principles:

  1. Each RPC sent by the gRPC client carries a username and password.
  2. For the first RPC in the gRPC session, the gRPC server tries to authenticate the user using the specified authentication order, such as using the local user database, RADIUS, or TACACS+.
    For example, if TACACS+ is first in the authentication order, the gRPC server sends a request to the TACACs+ server to authenticate the gRPC user.
  3. For the subsequent RPCs on that same authenticated gRPC session, the username and password are re-authenticated only if changed.
  4. When no username and password are provided with the RPC, the gRPC server returns an error.
  5. If the RPC user is changed, any active subscriber RPCs on that same gRPC session are terminated by the gRPC server.
  6. If the RPC password is changed, the active gRPC session will continue to exist until a different username and password is sent in a subsequent RPC, or the gRPC session is terminated.
  7. Each message is carried over a gRPC session that was previously encrypted; the session is not re-encrypted.
  1. SR OS device authentication
    1. The gRPC clients do not share gRPC sessions. Each gRPC client starts a separate gRPC session.
    2. When a gRPC session is established, the gRPC server certificates are verified by the gRPC client to ensure that every gRPC server is authenticated by the gRPC client.
    3. If gRPC is shut down on the gRPC server and a gRPC client is trying to establish a gRPC session, the gRPC client will get an error for every RPC sent.
    4. If gRPC is shut down on the gRPC server and a gRPC session is established, all active RPCs are gracefully terminated and an error is returned for every active RPC.

8.2. gNMI Service

gRPC Network Management Interface (gNMI) is a gRPC based protocol for network management functions, such as configuration and retrieval of information from network elements. In addition, it provides functionality necessary for supporting telemetry. gNMI service is specified in the OpenConfig forum.

8.2.1. gNMI Service Definitions

The SR OS gRPC server supports gNMI version 0.4.0, and in particular, the following RPC operations:

  1. Capability RPC
  2. Set/Get RPCs
  3. Subscribe RPC

8.2.1.1. Capability Discovery

In gNMI service, the client discovers the capabilities of the gRPC server through a Capability-Discovery RPC, which consists of “CapabiltyRequest” and “CapabilityResponse” messages.

During this message exchange, the gRPC server informs the client about following attributes:

  1. supported gNMI version
  2. supported models
  3. supported encodings

The SR OS server announces the supported models based on the configuration under configure>system>management-interface>yang-modules. The supported models includes the NOKIA-YANG or OpenConfig (OC) models.

The advertised module names and organizations are as follows:

  1. nokia-conf, org = "Nokia"
  2. nokia-state, org = "Nokia"
  3. openconfig, org = "OpenConfig working group" (as specified by the 'organization' in the YANG models)
  4. version - the version number will be defined as follows:
    1. for NOKIA YANG models, the version number will correspond to an SR OS release number, for example, "16.0r1"
    2. for OC YANG models, the version number will correspond to a version number defined in "oc-ext:openconfig-version" that is included in the respective YANG models
    3. for OC-YANG models, including NOKIA deviations, the version number will correspond to an SR OS release number, for example, "16.0r1"

The following is an example of a “Capabilities Response Message”:

Going to send message of type gnmi.CapabilityResponse:
 .gnmi_version: 0.4.0
 .supported_encodings (1):
  .encoding: 0 = JSON
 .supported_models (47):
  { .name: 'nokia-conf', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'nokia-state', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'openconfig-
bgp', .organization: 'OpenConfig working group', .version: '4.0.1' }
<snip>
  { .name: 'nokia-sr-openconfig-if-ethernet-deviations', .organization: 'Nokia', .version: '16.0.r1' }
  { .name: 'nokia-sr-openconfig-if-ip-deviations', .organization: 'Nokia', .version: '16.0.r1'..."

8.2.1.2. Get/Set RPC

Information is retrieved from the NE using GET RPC messages, which consists of “GetRequest” and “GetResponse” messages. The client asks for a given information by specifying following:

  1. A set of paths — all rules to a path definition apply, as specified in the gNMI specification
  2. Type — configuration, state, or operational data
  3. Encoding — in accordance to server advertisement during capability discovery
  4. Use_models — this message will be ignored

There is an upper limit on the size of the “GetResponse” message. This limit cannot exceed 100MB. If the limit is exceeded, the SR OS gRPC server responds with an error message.

In oder to modify the information in an NE element, a SET gRPC message is used. This gRPC supports three types of transactions:

  1. delete
  2. replace
  3. update

8.2.1.3. Subscribe RPC

A subscription is initiated from the gRPC client by sending a Subscribe RPC that contains a "SubscribeRequest" message to the gRPC server. A prefix can be specified to be used with all paths specified in the "SubscribeRequest". If a prefix is present, it is appended to the start of every path to provide a full path.

A subscription contains:

  1. a list of one or more paths. The following conditions apply:
    1. A path represents the data tree as a series of repeated strings and elements. Each element represents a data tree node name and its associated attributes.
    2. A path must be syntactically valid within the set of schema modules that the gRPC server supports.
    3. The path list cannot be modified during the lifetime of the subscription.
    4. If the subscription path is to a container node, all child leafs of that container node are considered to be subscribed to.
    5. Any specified path must be unique within the list; paths cannot be repeated within the list. An error is returned if the same path is used more than one time in a single subscription.
    6. A specified path does not need to pre-exist within the current data tree on the gRPC server. If a path does not exist, the gRPC server continues to monitor for the existence of the path. Assuming that the path exists, the gRPC server transmits telemetry updates.
    7. The gRPC server does not send any data for a non-existent path; for example, if a path is non-existent at the time of subscription creation or if the path was deleted after the subscription was established.
    8. The maximum number of paths for all subscriptions on a single SR OS device is 14400. A path using a wildcard is still considered a single path.
  2. a subscription mode of one of the following types:
    1. ONCE mode — the server returns only one notification containing all information the client has subscribed to. In general, retrieving large amounts of information from the NE can be done using telemetry: “SubscribeRequest” message with ONCE subscription type.
    2. ON-CHANGE mode — the server returns notifications only when the value of the subscribed field changes. See ON-CHANGE Subscription Mode for more information.
    3. SAMPLE mode — the gRPC server sends notifications at the specified sampling interval
    4. TARGET_DEFINED mode — means ON-CHANGE for all states supporting on-change notifications and SAMPLE mode for all other objects in the YANG tree
  3. a sample interval is supported for each path. If a sample interval of less than 10 s is specified, the gRPC server returns an error. If the sample interval is set to 0, the default value of 10 s is used. A sample interval is specified in nanoseconds (10 000 000 000 by default)

When a subscription is successfully initiated on the gRPC server, “SubscribeReponse” messages are sent from the gRPC server to the gRPC client. The “SubscribeResponse” message contains update notifications about the subscription's path list.

An update notification contains:

  1. a timestamp of the statistics collection time, represented in nanoseconds
  2. a prefix:
    1. If a prefix is present, it is logically appended to the start of every path to provide the full path.
    2. The presence of a prefix in the “SubscriptionResponse” message is not related to the presence of a prefix in the original “SubscriptionRequest” message. The prefix in the “SubscriptionResponse” message is optimized by the gRPC server.
  3. a list of updates (path and value pairs):
    1. A path represents the data tree path as a series of repeated strings or elements, where each element represents a data tree node name and its associated attributes. See Schema Paths for more information.
    2. The “TypedValue” message represents the data tree node’s value where encoding is always “JSON”.

A sync response notification is sent one time, after the gRPC server sends all of the updates for the subscribed-to paths. The sync response must be set to “true” for the gRPC client to consider that the stream has synced one time. A sync response is used to signal the gRPC client that it has a full view of the subscribed-to data.

The gRPC server sends an error if required. The error contains a description of the problem.

8.2.1.3.1. ON-CHANGE Subscription Mode

SR OS supports ON-CHANGE subscription mode. This subscription mode indicates that Notification messages are sent as follows:

  1. after the “SubscriptionRequest” message is received
  2. every time the corresponding leaf value is changed

The notification message, as a response to an ON-CHANGE subscription, always contains the new value of the corresponding leaf, as defined in gNMI specification.

The ON-CHANGE subscription is supported for all configuration events as well as for selected state leafs. The tools command can display all state leafs supporting the ON-CHANGE subscription.

ON-CHANGE subscription is accepted for all valid paths. The server sends ON-CHANGE notifications only for leafs within this path that support ON-CHANGE notifications.

8.2.1.4. Schema Paths

Telemetry subscriptions include a set of schema paths used to identify which data nodes are of interest to the collector.

The paths in Telemetry Subscribe RPC requests follow the basic conventions described in the OpenConfig gnmi-path-conventions.md published on github.com (version 0.2.0 from February 24th, 2017).

A path consists of a set of path segments often shown with a “/” character as a delimiter; for example, /configure/router[router-instance=Base]/interface[interface-name=my-interface1]/description.

These paths are encoded as a set of individual string segments in gnmi.proto (without any “/” characters); for example, ["configure", "router[router-instance=Base]", "interface[interface-name=my-interface1]", "description"].

A path selects an entire subtree of the data model and includes all descendants of the node indicated in the path. Table 102 describes the types of schema paths that are supported in SR OS telemetry.

Table 102:  Schema Paths  

Path example

Description

/configure/router[router-instance=Base]/interface[interface-name=abc]

Selects all config leafs of interface abc and all descendants.

/configure/router[router-instance=Base]/interface[interface-name=abc]/description

Selects only the description leaf of interface abc.

/state/router[router-instance=Base]/interface[interface-name=*]

Selects all state information for all base router interfaces using a wildcard in a single segment of a path.

/configure/router[router-instance=Base]/interface[interface-name=*]/description

Selects all state information for all base router interfaces using a wildcard in a single segment of a path.

/

The root path. This selects all config and state data from all models (in all namespaces) supported on the router. Encoded as “” in gRPC/gPB.

The following list describes types of telemetry paths that are not supported in SR OS.

  1. Wildcards for entire path segments are not supported.
    For example: /state/service/*/oper-status
  2. If a wildcard is used for any key of a list, a wildcard must be used for all the keys of that list. In a single path segment, all the keys must either have specific values or all the keys must have wildcards. A mix of wildcards and specific values for different parts of a list key is not supported.
    For example:
    Supported:
    /a/b[key1=*][key2=*]/c[key1=foo]
    /a/b[key1=foo][key2=bar]/c[key1=*]
    Not supported:
    /a/b[key1=foo][key2=*]
  3. Functions such as “current()”, “last()” and mathematical operators, such as stat<5 or octets>3 are not supported in paths. The “|” (OR operator, used to select multiple paths) is not supported.
  4. The “//” wildcard pattern is not supported.
    For example: /state//oper-status

The following list describes types of telemetry paths that are supported in SR OS.

  1. Wildcards in multiple segments of a path are supported.
    For example: /state/card[slot-number=*]/mda[mda-slot=*]

8.2.2. gNMI Service Use Cases

The gNMI Service can be used for the following:

  1. Telemetry
  2. NE Configuration Management

8.2.2.1. Telemetry

Telemetry is a network monitoring and fault management framework. Telemetry is driven by the need to use fresh data obtained from the network to make fast networking decisions such as traffic optimization and preventative troubleshooting.

Unlike legacy monitoring platforms such as SNMP, telemetry does not only rely on continuously pulling data from the network devices. Instead, network devices push and stream data (such as statistics) continuously to data collectors based on subscriptions. The data collectors can then filter, analyze, store, and make decisions using the collected data from the network devices. Figure 22 shows a telemetry application.

Figure 22:  Telemetry Application 

8.2.2.1.1. Telemetry Examples

This section contains examples of Telemetry subscription requests and responses. The following examples are dumps of protobuf messages from a Python API. Formats may vary across different implementations.

Example 1 — Subscribe to a single path

2017-06-05 17:06:13,189 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=test]"
      element: "statistics"
      element: "ip"
      element: "in-packets"
    }
    mode: SAMPLE
    sample_interval: 10000000000
  }
}
2017-06-05 17:06:13,190 - RCVD::SubsribeResponse
2017-06-05 17:06:23,492 - RCVD::Subscribe
2017-06-05 17:06:23,492 - update {
  timestamp: 1496675183491595139
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=test]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:06:23,494 - RCVD::Subscribe
2017-06-05 17:06:23,494 - sync_response: true
2017-06-05 17:06:33,589 - RCVD::Subscribe
2017-06-05 17:06:33,589 - update {
  timestamp: 1496675213491595139
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=test]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““28””
  }
}
....
....

Example 2 — Subscribe to a single path with wildcard

2017-06-05 17:08:29,055 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=*]"
      element: "statistics"
      element: "ip"
      element: "in-packets"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-06-05 17:08:29,056 - RCVD::SubsribeResponse
2017-06-05 17:08:59,133 - RCVD::Subscribe
2017-06-05 17:08:59,133 - update {
  timestamp: 1496675339132056575
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=system]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      jason_val: ““0””
    }
  }
}
2017-06-05 17:08:59,135 - RCVD::Subscribe
2017-06-05 17:08:59,135 - update {
  timestamp: 1496675339133006678
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:08:59,135 - RCVD::Subscribe
2017-06-05 17:08:59,135 - update {
  timestamp: 1496675339133006678
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_D]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:08:59,136 - RCVD::Subscribe
2017-06-05 17:08:59,136 - sync_response: true
2017-06-0517:09:29,139 - RCVD::Subscribe
2017-06-0517:09:29,139 - update {
  timestamp: 1496682569121314
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=system]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:09:29,142 - RCVD::Subscribe
2017-06-05 17:09:29,142 - update {
  timestamp: 1496682569124342
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
2017-06-05 17:09:29,145 - RCVD::Subscribe
2017-06-05 17:09:29,145 - update {
  timestamp: 1496682569127344
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_D]"
    element: "statistics"
    element: "ip"
  }
  update {
    path {
      element: "in-packets"
    }
    val {
      json_val: ““0””
    }
  }
}
....
....

Example 3: Subscribe to more than one path

2017-01-24 12:54:18,228 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "mpls"
      element: "statistics"
      element: "lsp-egress-stats[lsp-name=lsp_to_dest_f]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}

Example 4: Subscribe to a list with wildcard

2017-01-24 13:45:30,947 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=*]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}

Example 5: Subscribe to path where the object did not exist before subscription

2017-01-24 13:53:50,165 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-01-24 13:53:50,166 - RCVD::SubsribeResponse
2017-01-24 13:54:20,169 - RCVD::Subscribe
2017-01-24 13:54:20,169 - sync_response: true
2017-01-24 13:54:50,174 - RCVD::Subscribe
2017-01-24 13:54:50,174 - update {
  timestamp: 1485262490169309451
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
  }
  update {
...
...
  }
}

Example 6: Subscribe to a path where the object existed before subscription then was deleted after subscription

2017-01-24 14:00:41,292 - SENT::SubscribeRequest
subscribe {
  subscription {
    path {
      element: "state"
      element: "router[router-instance=Base]"
      element: "interface[interface-name=to_node_B]"
    }
    mode: SAMPLE
    sample_interval: 30000000000
  }
}
2017-01-24 14:00:41,294 - RCVD::SubsribeResponse
2017-01-24 14:01:11,295 - RCVD::Subscribe
2017-01-24 14:01:11,295 - update {
  timestamp: 1485262871290064704
  prefix {
    element: "state"
    element: "router[router-instance=Base]"
    element: "interface[interface-name=to_node_B]"
  }
  update {
...
...
  }
}
2017-01-24 14:01:11,359 - RCVD::Subscribe
2017-01-24 14:01:11,359 - sync_response: true
2017-01-24 14:01:41,293 - RCVD::Subscribe
2017-01-24 14:02:11,296 - RCVD::Subscribe

8.2.2.2. NE Configuration Management

Figure 23 shows NE configuration and information retrieval using the gNMI service.

Figure 23:  NE Configuration and Information Retrieval using gNMI Service 

In the context of gNMI, every SET RPC appears as an single commit operation, regardless of the number of paths included in the message. Both, NOKIA and OC models are supported by gNMI SET/GET RPC.

An example of the SET RPC command (including the response message from the gRPC server) follows:

Example:
 gNMI_rpc - DEBUG - SENT::SetRequest
prefix {
}
update {
  path {
    elem {
      name: "configure"
    }
    elem {
      name: "system"
    }
  }
  val {
    json_val: {"location": "zurich"}
  }
}
gMI_rpc - DEBUG - RCVD::SetResponse
prefix {
}
response {
  path {
    elem {
      name: "configure"
    }
    elem {
      name: "system"
    }
  }
  op: UPDATE
}

An example of the GET RPC command (including the response message from the gRPC server) follows:

Example:
gNMI_rpc - INFO - SENT::GetRequest GET140550212650064
path {
  elem {
    name: "configure"
  }
  elem {
    name: "system"
  }
  elem {
    name: "location"
  }
}
type: CONFIG
2017-12-06 12:17:28,639 - gMI_rpc - INFO - RCVD::GetResponse GET140550212650064
notification {
  timestamp: 1512559048634751055
  update {
    path {
      elem {
        name: "configure"
      }
      elem {
        name: "system"
      }
      elem {
        name: "location"
      }
    }
    val {
      json_val: "zurich"
    }
  }
}