NSO

NSO Adapter

The NSO Adapter connects to NSO using the web UI port and NETCONF port. Each time the NSO adapter establishes a connection to NSO, the adapter will read the service models advertised by NSO. To securely configure the NSO adapter, make sure the HTTP hostname is configured with the same hostname that is configured in the common name of the NSO server certificate.

• NSO needs to be installed on a server that IAP can access.
• NSO does not need to be installed on the same server that IAP is on.
• Refer to the Minimum System Requirements located in the IAP Installation guide for the required version.

Adapter Properties

Use the information in the table below to set NSO Adapter properties.

Sample Configuration

A sample NSO adapter configuration is provided for reference. Be sure to configure the following properties:

• http.host
• http.port
• ssl.enabled
• ssl.caFile
• ssl.ciphers
• netconf.host
• netconf.port
• credentials.user
• credentials.passwd

The default configuration is to use a machineLogin for all transactions. To enable individual user authorization to be handled by NSO, NSO must be configured with an external authentication script capable of returning external group memberships for a given user id. Ensure NSO is configured for external authentication and configure the authenticationStrategy.method to be tokenLogin instead of machineLogin.

{
  "id": "nso",
  "type": "NSO",
  "properties": {
    "http": {
      "host": "localhost",
      "port": 8888
    },
    "ssl": {
      "enabled": true,
      "acceptInvalidCerts": false,
      "caFile": "/etc/ssl/ca.cert",
      "ciphers": "DHE-RSA-AES256-SHA"
    },
    "netconf": {
      "host": "localhost",
      "port": 2022,
      "protocol": "ssh",
      "frame_size": 16736
    },
    "credentials": {
      "user": "admin",
      "passwd": "admin"
    },
    "authenticationStrategy": {
      "type": "dynamic",
      "location": "NSO",
      "method": "machineLogin"
    },
    "netconfSubscriptions": [ "ncs-events", "ncs-alarms", "device-notifications", "service-state-changes" ],
    "commitWait": 30000,
    "commitQueue": false,
    "max_reconnect_attempts": 0,
    "total_reconnect_window": 120,
    "min_reconnect_interval": 30,
    "max_reconnect_interval": 900,
    "entity_cache_TTL": 24,
    "max_cache_entries": 5000,
    "poolSize": 3,
    "yangdirs": []
  }
}

Itential Tools Package

The Itential Tools Package contains tools necessary for IAP to interact with NSO. Download this project from the Itential Software Repository.

1. Change to the NCS packages directory.

   cd /var/opt/ncs/packages
   

2. Extract the itential-tools package.

   tar xzf itential_tools_VERSION.tgz
   

3. Run the make command in the src directory of this project.

   cd itential-tools/src && make clean all -s && cd -
   

4. Perform a package reload using ncs_cli.

   ncs_cli -u admin -C --noaaa > package reload
   

NSO External Authentication Installation

To get started with external authentication, install the ph-auth script and then configure the ncs.conf file. The NSO external authentication script requires Python and the requests module to be installed.

Network connectivity is required between NSO and IAP (typically port 3000) for use with the External Authentication script.

1. Install these modules directly with yum.

   yum install -y python python-pip
   pip install requests
   

2. Configure the external authentication script found in.

   /var/opt/ncs/packages/itential-tools/external_auth
   

3. Configure the hostname and port of the IAP server.

   PH_ADDRESS = "localhost"
   PH_PORT = "3000"
   

4. Test the external authentication script.

   python /var/opt/ncs/packages/itential-tools/external_auth/ph-auth.py
   

5. Type the following command and press Enter.

   [test;test;]
   

6. You should receive the following result.

   reject
   

NSO External Authentication Configuration

Configure the ncs.conf file to use external authentication. Both external-authentication and either location-authentication or PAM should be enabled. IAP will initially make a connection using a global service account with admin privileges.

When performing transactions that can bind a particular request to a human user account, IAP will start making calls on behalf of a specific user, (e.g., bobjones instead of admin.

vi /etc/ncs/ncs.conf
  
...
<aaa>
    <ssh-server-key-dir>${NCS_CONFIG_DIR}/ssh</ssh-server-key-dir>

    <!-- Depending on OS - and also depending on user requirements -->
    <!-- the pam service value value must be tuned. -->

    <pam>
        <enabled>false</enabled>
        <service>system-auth</service>
    </pam>
    <external-authentication>
        <enabled>true</enabled>
        <executable>/var/opt/ncs/packages/itential-tools/external_auth/ph-auth.py</executable>
    </external-authentication>
    <local-authentication>
        <enabled>false</enabled>
    </local-authentication>

    <expiration-warning>prompt</expiration-warning>
</aaa>

Restart NSO.

service ncs restart

Configure Default Authgroup Mappings

If the tokenLogin feature of the IAP NSO adapter will be used, verify the users all properly map to the proper southbound credentials. IAP's reference configuration is accomplished by ensuring credentials are configured for the default umap inside NSO. Run this command for each authentication group; be sure to apply the appropriate device credentials as the remote-name and remote-password.

ncs_cli -u admin -C
config
devices authgroups group default default-map remote-name admin remote-password admin
commit

Sample NSO NACM Rules

To use IAP with an NSO NETCONF Access Control Model (NACM) implementation, you will need to reference the sample set of NACM rules provided with this package.

The NACM rules assume the following groups model.

These groups are shared across both IAP and NSO. Using the sample external authentication script, IAP will provide the set of groups defined inside IAP and discovered from the configured AAA system to NSO to be applied to the NACM rule-lists.

Note: It is important that the set of groups returned by the NSO AAA provider properly match NACM rule-list group assignments.

The following rule-lists are provided in the sample rules.

Be sure to adjust the groups associated with these rule-lists to ensure they are mapped to the user accounts appropriately.

Install NACM Rules in NSO

The attached file sample-nacm-rules.xml can be loaded into the NSO CDB using the load merge feature of the ncs_cli. Use the following commands to load the sample NACM rules.

ncs_cli -u admin -C
config
load merge sample-nacm-rules.xml
commit dry-run
commit

Configure Device Whitelists

The sample NACM rules contain a rule-list for whitelisting devices to the users group. Configure the sample rule-list or create your own rule-lists to associate devices with groups of users. Add devices to the users whitelist.

ncs_cli -u admin -C
config
nacm rule-list whitelist-devices rule permit-device-mydevicename path /devices/device[name='mydevicename'] action permit
commit

Note: In a NACM enabled system, any new device added to the system needs to have its groups defined at the time of device turn up. This applies to both manual configuration of the devices in NSO as well as any device turn up workflows that are turning up new devices on the network.

Configure Service Whitelists

The sample NACM rules contain a rule-list for whitelisting service instances to the users group. Configure the sample rule-list or create your own rule-lists to associate services with groups of users. Add all service instances for a model to the users whitelist.

ncs_cli -u admin -C
config
nacm rule-list whitelist-services rule permit-service-cisco-ios path /services/cisco-ios action permit
commit

Alternately, you can add a single instance to the users whitelist.

ncs_cli -u admin -C
config
nacm rule-list whitelist-services rule permit-service-cisco-ios-101 path /services/cisco-ios[vlan=101] action permit
commit

Note: In a NACM enabled system, any new service instance added to the system needs to have its groups defined. Service instances may be restricted to an individual group or made accessible to multiple groups with the appropriate NACM rules.

NSO NETCONF Events in Workflows

Configure workflows in IAP to wait for Cisco NSO northbound NETCONF event notifications before proceeding.

Cisco NSO emits notifications over NETCONF for various events that occur within its system. The NSO Adapter can be configured to listen to NSO NETCONF event streams and the IAP Event System can be configured to consume and process these events within a workflow.

Configure NSO NETCONF Event Streams

Event Streams are configured in NSO within the ncs.conf file. NSO allows the following event streams to be configured:

• ncs-alarms
• ncs-events
• device-notifications
• service-state-changes
• NETCONF

This example shows how to configure an ncs-alarms events stream inside the NSO ncs.conf file.

<notifications>
    <event-streams>

      <!-- This is the builtin stream used by NCS to generate northbound -->
      <!-- notifications whenever the alarm table is changed. -->
      <!-- See tailf-ncs-alarms.yang -->
      <!-- If you are not interested in NCS northbound netconf notifications -->
      <!-- remove this item since it does consume some CPU -->
      <stream>
        <name>ncs-alarms</name>
        <description>NCS alarms according to tailf-ncs-alarms.yang</description>
        <replay-support>false</replay-support>
        <builtin-replay-store>
          <enabled>false</enabled>
          <dir>./state</dir>
          <max-size>S10M</max-size>
          <max-files>50</max-files>
        </builtin-replay-store>
      </stream>
  </event-streams>
</notifications>

Note:: Every stream that is configured must be defined within ncs.conf regardless if it supports replay. NSO logs notifications and a NETCONF client can ask for logged notifications if replay support is enabled.

Configure NSO Adapter to Listen to NETCONF Streams

Adapter NSO can be configured to define which NSO NETCONF event streams to subscribe to. The list of event streams are defined in adapter properties, with the key netconfSubscriptions. The NSO adapter must be restarted after the adapter properties are updated in order to listen to the NSO event streams. The netconfSubscriptions values should correspond to the desired NSO NETCONF streams defined in NSO ncs.conf.

Example Configuration

This is an example of adapter-nso properties with the netconfSubscriptions key defined.

{
        "id": "NSO",
        "type": "NSO",
        "properties": {
            "netconfSubscriptions": [
                "ncs-events",
                "ncs-alarms",
                "device-notifications",
                "service-state-changes"
            ],
            "http": {
                "host": "localhost",
                "port": 8080
            },
            "netconf": {
                "host": "localhost",
                "port": 2022,
                "protocol": "ssh"
            },
            "ssh": {
                "port": 22
            },
            "credentials": {
                "user": "admin",
                "passwd": "admin"
            },
            "commitWait": 5000,
            "commitQueue": false
        },
        "groups": [],
        "brokers": [
            "device",
            "method",
            "service"
        ]
    }

Workflow Configuration

The IAP Workflow engine task called eventListener is used to listen for events from applications and adapters deployed inside the platform.

Figure 1: Event Listener Task

Note: Refer to the eventListenerJob task reference for more information.

When a job is created, it will commence to process all the tasks that it can, but upon reaching the eventListener task it will wait in a running state until it receives an event. The actual event that it is waiting for can be configured within the task itself.

Figure 2: Event Listener Input Dialog

Defined Event Listener Inputs

Event Schema Filter Examples

This example will process the eventListener task only when the following payload is received.

{
    "eventTime": "2019-09-19T18:56:14.227091+01:00",
    "plan-state-change": {
        "service": {
            "_": "/l3vpn:l3vpn-simple-svc[l3vpn:name='IAP']"
        },
        "component": "self",
        "state": {
            "_": "ncs:init"
        },
        "operation": "created",
        "status": "reached"
    }
}

The JSON schema for the Event Schema Filter would be defined as:

{
    "type": "object",
    "properties": {
        "eventTime": {
            "type": "string",
            "default": "2019-09-19T18:56:14.227091+01:00",
            "examples": [
                "2019-09-19T18:56:14.227091+01:00"
            ],
            "format": "date-time"
        },
        "plan-state-change": {
            "type": "object",
            "properties": {
                "service": {
                    "type": "object",
                    "properties": {
                        "_": {
                            "type": "string",
                            "default": "/l3vpn:l3vpn-simple-svc[l3vpn:name='IAP']",
                            "enum": [
                                "/l3vpn:l3vpn-simple-svc[l3vpn:name='IAP']"
                            ]
                        }
                    },
                    "required": [
                        "_"
                    ],
                    "additionalProperties": false
                },
                "component": {
                    "type": "string",
                    "default": "self",
                    "examples": [
                        "self"
                    ]
                },
                "state": {
                    "type": "object",
                    "properties": {
                        "_": {
                            "type": "string",
                            "default": "ncs:init",
                            "examples": [
                                "ncs:init"
                            ]
                        }
                    },
                    "required": [
                        "_"
                    ],
                    "additionalProperties": false
                },
                "operation": {
                    "type": "string",
                    "default": "created",
                    "examples": [
                        "created"
                    ]
                },
                "status": {
                    "type": "string",
                    "default": "reached",
                    "examples": [
                        "reached"
                    ]
                }
            },
            "required":
                "service",
                "component",
                "state",
                "operation",
                "status"
            ],
            "additionalProperties": false
        }
    },
    "required": [
        "eventTime",
        "plan-state-change"
    ],
    "additionalProperties": false
}

Note: If there is no need to filter on the payload and you simply want the task to execute when it receives a specific Event Topic, then an empty JSON object {} should be set within the Event Schema Filter field.

Itential Tools Rules Engine

The Itential Tools Rules Engine allows users to define scripts that can be executed prior to NSO command-sets being executed. These scripts will modify (add or change) values in the NSO command-sets prior to their execution.

The rules engine has two parts: a rule engine and a script engine. The script engine reads the script files that have been defined for the command that is passed in, and then reads in the defined rules before it finally passes the rules to the rule engine to be processed.

The following files are required:

Development Tools and Libraries

The following tools and libraries can be used for development and testing of the Itential Tools Rules Engine. The main framework for the Java-based application will be the Java Spring framework.

System Design

The Itential Tools Rules Engine has three functional areas:

• Script Engine
• Rule Engine
• Script Validation Engine

Figure 3: Itential Tools Rules Engine Design

Script Engine

The Script Engine is the main driver for script processing. It receives the command file that contains the request to modify the NSO command-set. The following processes are performed in the Script Engine:

• Read in the command file.
• Locate the command to get the script file to execute.
• Read in the script file that needs to be executed. This file will have the defined rules to execute against the NSO command-set.
• The rule engine will be called to process the rules.
• The NSO command-set will be modified appropriately.
• The modified command-set will be returned to the calling system.

Any errors encountered along the way will be handled appropriately and returned to the calling system.

Rule Engine

The Rule Engine is responsible for processing each rule. The input parameters will be the rule and the NSO command-set. The modified NSO command-set will be its output parameter. The following steps will be performed:

• Receive parameters.
• Determine which rule to process.
• Perform the rule process and modify the NSO command-set.
• Return the modified NSO command-set.
• Handle any errors and pass back to Script Engine for its error handler to process.

Script Validation Engine

The Script Validation Engine will validate if the script has the correct format and the commands that are contained in the script are valid commands with the correct number of parameters required for the command type.

If any of the validations fail, a custom java exception will be thrown with the appropriate error message and code, and then passed back to the calling system to be handled appropriately.

Validation Commands

$DLC_REPLACE

Four (4) parameters:

• 1st - Any String
• 2nd - Any String
• 3rd - Int value
• 4th - Int Value

$DLC_INSERT

Five (5) parameters:

• 1st - Any String
• 2nd - Any String
• 3rd - Before or After value
• 4th - Int Value
• 5th - Int value

Rule Command Format and Usage

Arguments

Supported Options within the "new string" Argument

• $STRING

  Syntax: $STRING['my string'|'2nd string'|'3rd string']

  Parameters:

  • $STRING - A single value or and array or string separated by a a pipe ("|") symbol that will be printed out. An array will cause a loop will occur.

• $LOOP

  Syntax: $LOOP[{start}-{end}|$STRING['{string to print out}']$LOOP_VAR]

  Parameters:

  • {start}:[int] - The starting number of the loop.
  • {end}:[int] - The ending number of the loop.
  • {string to print out}:[string] - Any string you want to print out.
  • $LOOP_VAR:[int] - The integer value of the current loop iteration that will be printed out where it appears either before or after the $STRING[] option. Note: Multiple references to $LOOP_VAR are allowed and supported.

Example

$DLC_INSERT("access-list","new text here",AFTER,1);
$DLC_INSERT("access-list","new text here",BEFORE,1);
$DLC_INSERT("","\n$LOOP[1-100|$STRING['vlan ']$LOOP_VAR]\n",AFTER,1,1);
$DLC_INSERT("","\n$STRING['my string'|'2nd string'|'3rd string']\n",BEFORE,1,1)

Implementation Design

Implementation of the designed Rule Engine will include the following features:

• Integration into the current itential-tools java application.
• A mechanism for the system to refresh the list of commands available to it without a required restart of itential-tools.
• An option to bypass the refresh feature.
• The ability of users to update the commands and scripts.

Rule Engine Integration

To integrate the new Rule Engine into the itential-tools application, the java jar file generated by the Rule Engine application would need to be included in the private-jar or the shared-jar directory to build the itential-tools application.

The com.itential.datalore_utils.ScriptEngine package would need to be imported into the class that is calling it. Below is a code sample to execute the command.

import com.itential.datalore_utils.ScriptEngine;

ScriptEngine scriptEngine = new ScriptEngine(sCommand, sNSOCommandSet);

try {
    sNSOCommandSet = scriptEngine.processCommand();
} catch (Exception e) {
    e.printStackTrace();
}

Integration Features

Configure NSO for SSL

The following steps provide instructions to configure SSL between NSO and IAP.

NSO SSL Configuration

1. Upload NSO host and CA X.509 certificates to the NSO host $CERT_DIR.

2. Copy the NSO host private key and signed certificate to the NSO configuration directory.

   NCS_CONFIG_DIR=/etc/ncs
   cp $NSO_SERVER_PRIV_KEY $NCS_CONFIG_DIR/ssl/cert/host.key
   cp $CERT_DIR/$NSO_SERVER_CERTIFICATE $NCS_CONFIG_DIR/ssl/cert/host.cert
   

3. Trust the CA certificate and adjust for your platform. These instructions apply to a CentOS 7 host.

   cp $CA_SELF_SIGNED_CERT /etc/pki/ca-trust/source
   update-ca-trust
   

4. Edit the configuration file.

   /etc/ncs/ncs.conf
   

5. Enable SSL for NSO REST and JSON RPC APIs.

   <ncs-config xmlns="http://tail-f.com/yang/tailf-ncs-config">
     <webui>
       <transport>
         <ssl>
           <enabled>true</enabled>
           <ip>0.0.0.0</ip>
           <port>8888</port>
           <key-file>${NCS_CONFIG_DIR}/ssl/cert/host.key</key-file>
           <cert-file>${NCS_CONFIG_DIR}/ssl/cert/host.cert</cert-file>
         </ssl>
       </transport>
     </webui>
   </ncs-config>
   

6. Enable NETCONF.

   <ncs-config xmlns="http://tail-f.com/yang/tailf-ncs-config">
     <netconf-north-bound>
       <enabled>true</enabled>
       <transport>
         <ssh>
           <enabled>true</enabled>
           <ip>0.0.0.0</ip>
           <port>2022</port>
         </ssh>
         <tcp>
           <enabled>false</enabled>
           <ip>127.0.0.1</ip>
           <port>2023</port>
         </tcp>
       </transport>
     </netconf-north-bound>
   </ncs-config>
   

7. Restart NSO.

Note: Refer to Cisco NSO Admin and Installation guides for detailed instructions on NSO SSL configuration.

Pronghorn NSO SSL Configuration

1. Upload the CA X.509 certificate to Pronghorn host $CERT_DIR.

   CERT_DIR=/etc/ssl/certs
   CA_SELF_SIGNED_CERT=${CERT_DIR}/certificate_authority.crt
   NSO_HOST=nso-hostname
   

2. Update your desired NSO adapter under Settings -> Services -> Adapters with the below configuration.

   {
       "loggerProps": {
           "description": "Logging",
           "log_max_files": 10,
           "log_max_file_size": 10485760,
           "log_level": "debug",
           "log_directory": "/var/log/pronghorn",
           "log_filename": "my_nso.log",
           "log_timezone_offset": 0,
           "console_level": "info",
           "syslog": {
               "level": "info",
               "host": "10.1.0.211",
               "port": 1514,
               "protocol": "udp4",
               "facility": "local0",
               "type": "BSD"
           }
       },
       "isEncrypted": true,
       "model": "@itential/adapter-nso",
       "name": "my_nso",
       "type": "Adapter",
       "properties": {
           "id": "my_nso",
           "type": "NSO",
           "properties": {
               "http": {
                   "host": "nso-hostname",
                   "port": 8888
               },
               "ssl": {
                   "enabled": true,
                   "acceptInvalidCerts": false,
                   "caFile": "/etc/ipa/ca.crt",
                   "ciphers": "ECDHE-RSA-AES128-GCM-SHA256"
               },
               "netconf": {
                   "host": "nso-hostname",
                   "port": 2022,
                   "protocol": "ssh"
               },
               "ssh": {
                   "port": 22
               },
               "credentials": {
                   "user": "nso_user",
                   "passwd": "nso_pass"
               },
               "authenticationStrategy": {
                   "type": "dynamic",
                   "location": "NSO",
                   "method": "tokenLogin"
               },
               "commitWait": 5000,
               "commitQueue": false,
               "max_reconnect_attempts": 0,
               "total_reconnect_window": 0,
               "min_reconnect_interval": 30,
               "max_reconnect_interval": 900
           },
           "groups": [],
           "brokers": [
               "device",
               "method",
               "service"
           ]
       },
       "rabbitmq": {
           "protocol": "amqp",
           "hostname": "localhost",
           "port": 5672,
           "username": "guest",
           "password": "guest",
           "locale": "en_US",
           "frameMax": 0,
           "heartbeat": 0,
           "vhost": "/",
           "certPath": "",
           "keyPath": "",
           "passphrase": "guest",
           "caPath": ""
       }
   }
   

Note: While updating the cipher in SSL properties, you can use the preferred cipher in your SSL handshake with the following command from IAP server:

$ openssl s_client -connect NSOServer.com(localhost):portNumber(8080)

In the example below, the preferred cipher is ECDHE-RSA-AES128-GCM-SHA256.

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 1A1D51D03BFB624A617626DCEEDACAF6AD0317DF9C34DF7F55802BAA776EC112
    Master-Key: 4B7827941C
    Start Time: 1595966521
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)

NSO LSA (Layered Service Architecture) Setup

The following principles drive adapter NSO and apply to NSO LSA architecture:

• Itential does not support multiple NSO instances having devices of the same name. We assume only one NSO instance manages a given device, and no other devices use that exact same name.

• Itential does not support having multiple NSO instances having service models of the same name/path. We assume only one NSO instance manages a given service model, and no other service models use that exact same name/path.

Assuming the following LSA architecture:

1. An “upper” NSO service that has service models defined.
2. One or more “lower” NSO services that connect to devices.

You can then do the following setup in IAP:

1. Create an adapter-nso instance and point it to the upper node.
2. Create an additional adapter-nso instance for every lower node and point it to a lower node.

With this setup, you should then be able to use service models (e.g., with tasks such as saveInstances) and perform device management (e.g., with Configuration Manager).

For more context, NSO can also utilize an IAP “broker” to further support this setup. When an app task like saveInstances is used, you provide the device name to the task. At that point the IAP broker will query all adapter-nso instances to see which one has that service model, and then use that adapter instance. The same principle applies to any tasks which directly interact with devices. For example, the golden config feature in Configuration Manager utilizes a broker to know which adapter-nso instances manage which device.