Stage 1: Prestaging
  • 13 Dec 2023
  • Dark
    Light
  • PDF

Stage 1: Prestaging

  • Dark
    Light
  • PDF

Article Summary

These are the necessary activities to prepare for the upgrade and perform a backup of dependencies and other essential platform tasks.

Review Product Notices

  1. Review all feature release information related to your IAP target release.

    Feature Release
    2022.1
    2021.2
    2021.1
    2020.2
  2. Review all breaking changes in the Itential software that can impact the upgrade timeline for your target release.

    Breaking Changes
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x
  3. Audit (review) all deprecation tasks prior to initiating the upgrade.

    Deprecations
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x
  4. Review all release alerts related to the Itential software.

    Release Alerts
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x

Confirm Dependencies

Audit (confirm) all dependencies meet the required minimum version.

Note: The latest dependency versions can be found on the Itential Dependencies page of the Itential Documentation site.

  1. Backup and copy all existing MongoDB, Redis and RabbitMQ configuration files prior to upgrading the following dependencies.

    Dependency
    OS
    Node.js
    MongoDB
    Redis
    RabbitMQ
    RabbitMQ Erlang
    Python
  2. Upgrade the dependencies to align with the required pre-requisites for the target IAP version.

Follow Upgrade Sequence for Environment

Itential assumes you are using different type of environments. Follow this sequence when upgrading to a newer IAP version: DEV-UAT-PROD.

  1. Upgrade DEV to continue with new development. Upgrade UAT for sanity checks and regressions of Itential apps and custom apps. The DEV and UAT upgrade will serve as a source of truth database.
  2. As a pre-requisite for PROD upgrade, allow the IAP platform to soak for 24-hours after upgrading dependencies. Be sure to monitor MongoDB, Redis, and RabbitMQ behavior during that time.

Upgrade MongoDB

Upgrade MongoDB and enable the compatibility settings.

  1. Follow the MongoDB upgrade path (4.2 → 4.4 → 5.0.0)

    db.adminCommand( { setFeatureCompatibilityVersion: <version> } )
    

    Example

    db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
    

  1. Check MongoDB version compatibility.

    db.version()
    db.getCollection.(“system.version”).({})
    

Confirm Workflow Collections

  1. Audit the workflow collections.
  2. Audit your existing workflows for Schema Validation Errors (Warnings and Drafts).
  3. Audit your existing workflows for deprecated tasks. These can be found in the Deprecations section of the Itential product documentation site.

Validate Indexes

Validate the indexes for jobs, tasks, and workflows from the MongoDB CLI.

See Database Indexes for additional guidance.

  1. Connect to Mongo.

    mongo > use pronghorn
    pronghorn.db>
    
  2. Run the following commands.

    db.jobs.getIndexes()
    db.tasks.getIndexes()
    db.workflows.getIndexes()
    
  3. Expect, at a minimum, 14 Jobs, 8 Tasks, and 4 Workflows.

Stop Jobs

  1. Stop all scheduled jobs and capture the input and groups data from all Automation Catalog jobs.

  2. Before any major version upgrade, all jobs should be canceled or completed before starting the upgrade process. For any jobs started on IAP 2019.3, or any version prior to 2019.3, it is important to ensure no tasks are left in a primed state. Tasks in the primed state are not supported with the migration script, so it is imperative that all jobs are paused and no primed tasks exist prior to the upgrade. Jobs with primed tasks will hang and the primed tasks will never run. Any tasks that are left on the system in a primed state must be removed manually.

Confirm Task Worker Settings

  1. Confirm the TaskWorker settings in workFlowEngine and the properties.json file.

    workFlowEngine  Service Config properties - “activate”: false
    
    properties.json file [ “processTasksOnStart” : false ]
    
  2. Stop TaskWorker and wait 30-minutes to complete the current running jobs in progress.

  3. Stop IAP.

Backup MongoDB

Perform a mongoDump and backup the MongoDB Collections.

  1. Run the following command.

    mongodump --host <MONGO HOSTNAME> --port=<MONGO PORT> --db=<PRONGHORN DB> --archive=<BACKUP LOC>/mongoDump<date>.bson
    
  2. Alternatively, you can run this command.

    mongodump --authenticationDatabase admin -u admin -p admin --db pronghorn -o /tmp/mongo-backup --excludeCollection jobs --excludeCollection tasks --gzip
    
  3. Backup and copy all MongoDB Collections (Worfklows, Transformations, ac_agenda_job, Jobs, Tasks, Forms and Templates).

  4. Confirm there is sufficient disk space in the target directory (/tmp).

Backup Pronghorn Files and Install Directory

  1. Backup the IAP pronghorn file system or installation directory (server.js, properties.json, all applications, all adapters).

  2. Take a symlink (symbolic link) snapshot of the @itentialopensource and @<customer custom folder> namespaces.

    ls -lart
    
Best Practice:

To avoid missing adapters in @itentialopensource, manually create all symlinks first before starting the upgrade.

➤ Install all opensoure applications and adapters in a custom placeholder directory under /opt/pronghorn/current/custom.

➤ Go to @itentialopensource <dir> and perform a symlink mapping of the installed apps/adapters to /opt/pronghorn/current/custom.

➤ During the upgrade process, when the bin installer is run with -p, you will be prompted to select Y (yes) to copy the entire custom folder and automatically recreate the symlinks.

Figure 1: Custom Directory
01_prestaging_Upgrade_MOP


Figure 2: Symlink Mapping
02_prestaging_Upgrade_MOP


Backup NSO

Run ncs-backup to backup NSO.

Download IAP Target Version

Download the IAP target version bin file from the Itential Software Repository.

Audit Dependency Vitals

Complete the following to bounce (soft restart) Redis, Rabbit MQ, and MongoDB, and then audit the dependency vitals. This is done to clean-up logs history and the logging buffer.

  1. Run this vital check curl command to capture the dependencies current status.

    curl -s http://<server>:port/status
    
  2. Run the following to audit the MongoDB Cluster and Replica status.

    sudo systemctl mongod stop
    
  3. Wait 5 minutes and then start MongoDB.

    sudo systemctl mongod start
    
  4. Run the monitoring checkpoints to audit the MongoDB Cluster.

    rs.status()
    rs.printSlaveReplicationInfo()
    
  5. Bounce Redis, and then audit Sentinel and the Cluster status.

    sudo systemctl stop redis-sentinal.service
    
    sudo systemctl stop redis.service
    
  6. Wait 5 minutes and then start the services.

    sudo systemctl start redis-sentinal.service
    
    sudo systemctl start redis.service
    
  7. Audit the Sentinel status.

    redis-cli -h redis-XX -p 26379 info|grep status
    
  8. Run this command on all Redis servers.

    redis-cli -h redis-XX -p 6379 info|grep role
    
  9. Bounce RabbitMQ, and then audit the Cluster status.

    sudo service rabbitmq-server stop
    
  10. Wait 5 minutes and then start the RabbitMQ server.

    sudo service rabbitmq-server start
    
  11. Audit the RabbitMQ Cluster.

    sudo rabbitmqctl cluster_status
    
  12. Run this vital check curl command after bounce to capture the current status of dependencies.

    curl -s http://<server>:port/status
    

Disable Deprecated Apps and Adapters

Disable all deprecated applications and adapters (as listed below) to prepare for a clean IAP start.

Deprecated Applications
@itential/app-smart_template
@itential/app-object/
@itential/app-json/pronghorn.json
@itential/app-tools/
@itential/app-string
@itential/app-number
@itential/app-golden_services
@itential/app-golden_config
@itential/app-array/
@itential/app-TaskWorker
Deprecated Adapters
adapter @itential/adapter-mongo

See Adapter Mongo for more detail.


  1. Go to Admin Essentials in IAP and select the active profile (i.e., profile1).
  2. Select the Configure tab and disable (uncheck) all deprecated apps/adapter under Edit Profile Services.
  3. Restart IAP.

Figure 3: Edit Profile to Disable Apps/Adapters
03_prestaging_Upgrade_MOP


Complete Remaining Activites for Platform

  1. Audit the platform vitals for rogue processes, diskspace, CPU utilities, and memory.
  2. Stop the cron jobs.
  3. Archive the log files.
  4. Confirm DB backup is complete.

Was this article helpful?

What's Next
Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.