Stage 1: Prestaging

These are the necessary activities to prepare for the upgrade and perform a backup of dependencies and other essential platform tasks.

Review Product Notices

1. Review all feature release information related to your IAP target release.

   Feature Release
   2022.1
   2021.2
   2021.1
   2020.2

2. Review all breaking changes in the Itential software that can impact the upgrade timeline for your target release.

   Breaking Changes
   Release 2022.1.x
   Release 2021.2.x
   Release 2021.1.x
   Release 2020.2.x

3. Audit (review) all deprecation tasks prior to initiating the upgrade.

   Deprecations
   Release 2022.1.x
   Release 2021.2.x
   Release 2021.1.x
   Release 2020.2.x

4. Review all release alerts related to the Itential software.

   Release Alerts
   Release 2022.1.x
   Release 2021.2.x
   Release 2021.1.x
   Release 2020.2.x

Confirm Dependencies

Audit (confirm) all dependencies meet the required minimum version.

1. Backup and copy all existing MongoDB, Redis and RabbitMQ configuration files prior to upgrading the following dependencies.

   Dependency
   OS
   Node.js
   MongoDB
   Redis
   RabbitMQ
   RabbitMQ Erlang
   Python

2. Upgrade the dependencies to align with the required pre-requisites for the target IAP version.
   
   

Follow Upgrade Sequence for Environment

Itential assumes you are using different type of environments. Follow this sequence when upgrading to a newer IAP version: DEV-UAT-PROD.

1. Upgrade DEV to continue with new development. Upgrade UAT for sanity checks and regressions of Itential apps and custom apps. The DEV and UAT upgrade will serve as a source of truth database.
2. As a pre-requisite for PROD upgrade, allow the IAP platform to soak for 24-hours after upgrading dependencies. Be sure to monitor MongoDB, Redis, and RabbitMQ behavior during that time.
   
   

Upgrade MongoDB

Upgrade MongoDB and enable the compatibility settings.

1. Follow the MongoDB upgrade path (4.2 → 4.4 → 5.0.0)

   db.adminCommand( { setFeatureCompatibilityVersion: <version> } )
   

   Example

   db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
   

2. Check MongoDB version compatibility.

   db.version()
   db.getCollection.(“system.version”).({})
   

Confirm Workflow Collections

1. Audit the workflow collections.
2. Audit your existing workflows for Schema Validation Errors (Warnings and Drafts).
3. Audit your existing workflows for deprecated tasks. These can be found in the Deprecations section of the Itential product documentation site.
   
   

Validate Indexes

Validate the indexes for jobs, tasks, and workflows from the MongoDB CLI.

1. Connect to Mongo.

   mongo > use pronghorn
   pronghorn.db>
   

2. Run the following commands.

   db.jobs.getIndexes()
   db.tasks.getIndexes()
   db.workflows.getIndexes()
   

3. Expect, at a minimum, 14 Jobs, 8 Tasks, and 4 Workflows.
   
   

Stop Jobs

1. Stop all scheduled jobs and capture the input and groups data from all Automation Catalog jobs.

2. Before any major version upgrade, all jobs should be paused, canceled or completed before starting the upgrade process. For any jobs started on IAP 2019.3, or any version prior to 2019.3, it is important to ensure no tasks are left in a primed state. Tasks in the primed state are not supported with the migration script, so it is imperative that all jobs are paused and no primed tasks exist prior to the upgrade. Jobs with primed tasks will hang and the primed tasks will never run. Any tasks that are left on the system in a primed state must be removed manually.
   
   

Confirm Task Worker Settings

1. Confirm the TaskWorker settings in workFlowEngine and the properties.json file.

   workFlowEngine  Service Config properties - “activate”: false
   

   properties.json file [ “processTasksOnStart” : false ]
   

2. Stop TaskWorker and wait 30-minutes to complete the current running jobs in progress.

3. Stop IAP.
   
   

Backup MongoDB

Perform a mongoDump and backup the MongoDB Collections.

1. Run the following command.

   mongodump --host <MONGO HOSTNAME> --port=<MONGO PORT> --db=<PRONGHORN DB> --archive=<BACKUP LOC>/mongoDump<date>.bson
   

2. Alternatively, you can run this command.

   mongodump --authenticationDatabase admin -u admin -p admin --db pronghorn -o /tmp/mongo-backup --excludeCollection jobs --excludeCollection tasks --gzip
   

3. Backup and copy all MongoDB Collections (Worfklows, Transformations, ac_agenda_job, Jobs, Tasks, Forms and Templates).

4. Confirm there is sufficient disk space in the target directory (/tmp).
   
   

Backup Pronghorn Files and Install Directory

1. Backup the IAP pronghorn file system or installation directory (server.js, properties.json, all applications, all adapters).

2. Take a symlink (symbolic link) snapshot of the @itentialopensource and @<customer custom folder> namespaces.

   ls -lart
   

Figure 1: Custom Directory

Figure 2: Symlink Mapping

Backup NSO

Run ncs-backup to backup NSO.

Download IAP Target Version

Download the IAP target version bin file from the Itential Software Repository.

Audit Dependency Vitals

Complete the following to bounce (soft restart) Redis, Rabbit MQ, and MongoDB, and then audit the dependency vitals. This is done to clean-up logs history and the logging buffer.

1. Run this vital check curl command to capture the dependencies current status.

   curl -s http://<server>:port/status
   

2. Run the following to audit the MongoDB Cluster and Replica status.

   sudo systemctl mongod stop
   

3. Wait 5 minutes and then start MongoDB.

   sudo systemctl mongod start
   

4. Run the monitoring checkpoints to audit the MongoDB Cluster.

   rs.status()
   rs.printSlaveReplicationInfo()
   

5. Bounce Redis, and then audit Sentinel and the Cluster status.

   sudo systemctl stop redis-sentinal.service
   

   sudo systemctl stop redis.service
   

6. Wait 5 minutes and then start the services.

   sudo systemctl start redis-sentinal.service
   

   sudo systemctl start redis.service
   

7. Audit the Sentinel status.

   redis-cli -h redis-XX -p 26379 info|grep status
   

8. Run this command on all Redis servers.

   redis-cli -h redis-XX -p 6379 info|grep role
   

9. Bounce RabbitMQ, and then audit the Cluster status.

   sudo service rabbitmq-server stop
   

10. Wait 5 minutes and then start the RabbitMQ server.

    sudo service rabbitmq-server start
    

11. Audit the RabbitMQ Cluster.

    sudo rabbitmqctl cluster_status
    

12. Run this vital check curl command after bounce to capture the current status of dependencies.

    curl -s http://<server>:port/status
    

Disable Deprecated Apps and Adapters

Disable all deprecated applications and adapters (as listed below) to prepare for a clean IAP start.

1. Go to Admin Essentials in IAP and select the active profile (i.e., profile1).
2. Select the Configure tab and disable (uncheck) all deprecated apps/adapter under Edit Profile Services.
3. Restart IAP.

Figure 3: Edit Profile to Disable Apps/Adapters

Complete Remaining Activites for Platform

1. Audit the platform vitals for rogue processes, diskspace, CPU utilities, and memory.
2. Stop the cron jobs.
3. Archive the log files.
4. Confirm DB backup is complete.