Stage 1: Prestaging
  • Dark
    Light
  • PDF

Stage 1: Prestaging

  • Dark
    Light
  • PDF

These are the necessary activities to prepare for the upgrade and perform a backup of dependencies and other essential platform tasks.

  1. Review all feature release information related to your IAP target release.

    Feature Release
    2022.1
    2021.2
    2021.1
  2. Review all breaking changes in the Itential software that can impact the upgrade timeline for your target release.

    Breaking Changes
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x
  3. Audit (review) all deprecation tasks prior to initiating the upgrade.

    Deprecations
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x
  4. Review all release alerts related to the Itential software.

    Release Alerts
    Release 2022.1.x
    Release 2021.2.x
    Release 2021.1.x
    Release 2020.2.x
  5. Audit (confirm) all dependencies meet the required minimum version. A consolidated list of dependencies is available on the Itential documentation site.

    • Backup and copy all existing MongoDB, Redis and RabbitMQ configuration files prior to upgrading dependencies.

      Dependency Minimum Version
      OS Centos 7
      Node.js >=14.18.2 <15.0.0
      MongoDB >=5.0.0 <5.1.0
      Redis >=5.0.12 <6.0.0
      RabbitMQ >=3.8.14 <3.9.0
      RabbitMQ Erlang >=22.3.0 <25.0.0
      Python >=3.9.5
    • Upgrade the dependencies to align with the required pre-requisites for the target IAP version.

  6. Itential assumes you are using different type of environments. Follow this sequence when upgrading to a newer IAP version: DEV-UAT-PROD.

    • Upgrade DEV to continue with new development. Upgrade UAT for sanity checks and regressions of Itential apps and custom apps. The DEV and UAT upgrade will serve as a source of truth database.
    • As a pre-requisite for PROD upgrade, allow the IAP platform to soak for 24-hours after upgrading dependencies. Be sure to monitor MongoDB, Redis, and RabbitMQ behavior during that time.
  7. Upgrade MongoDB and enable the compatibility settings.

    • MongoDB upgrade path (4.2 → 4.4 → 5.0.0)

      db.adminCommand( { setFeatureCompatibilityVersion: <version> } )
      

      Example

      db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
      
  8. Check MongoDB version compatibility.

    db.version()
    db.getCollection.(“system.version”).({})
    
  9. Audit the workflow collections.

    • Audit your existing workflows for Schema Validation Errors (Warnings and Drafts).
    • Audit your existing workflows for deprecated tasks. These can be found in product Deprecations on the Itential documentation site.
  10. Validate the indexes for jobs, tasks, and workflows from the MongoDB CLI.

    • Connect to Mongo.

      mongo > use pronghorn
      pronghorn.db>
      
    • Run the following commands.

      db.jobs.getIndexes()
      db.tasks.getIndexes()
      db.workflows.getIndexes()
      
    • Expect, at a minimum, 14 Jobs, 8 Tasks, and 4 Workflows.

  11. ⚠ Stop all scheduled jobs and capture the input and groups data from all Automation Catalog jobs.

  12. ⚠ Before any major version upgrade, all jobs should be paused, canceled or completed before starting the upgrade process. For any jobs started on IAP 2019.3, or any version prior to 2019.3, it is important to ensure no tasks are left in a primed state. Tasks in the primed state are not supported with the migration script, so it is imperative that all jobs are paused and no primed tasks exist prior to the upgrade. Jobs with primed tasks will hang and the primed tasks will never run. Any tasks that are left on the system in a primed state must be removed manually.

  13. Confirm the TaskWorker settings in workFlowEngine and the properties.json file.

    workFlowEngine  Service Config properties - “activate”: false
    
    properties.json file [ “processTasksOnStart” : false ]
    
    • Stop TaskWorker and wait 30-minutes to complete the current running jobs in progress.
    • Stop IAP.
  14. Perform a MongoDump and backup the MongoDB Collections.

    • Run the following command.

      mongodump --host <MONGO HOSTNAME> --port=<MONGO PORT> --db=<PRONGHORN DB> --archive=<BACKUP LOC>/mongoDump<date>.bson
      

      or

      mongodump --authenticationDatabase admin -u admin -p admin --db pronghorn -o /tmp/mongo-backup --excludeCollection jobs --excludeCollection tasks --gzip
      
    • Backup and copy all MongoDB Collections (Worfklows, Transformations, ac_agenda_job, Jobs, Tasks, Forms and Templates).

    • Confirm there is sufficient disk space in the target directory (/tmp).

  15. Backup the pronghorn database.

    tar -czvf <BACKUP LOC>/iapBackup.tar.gz /opt/pronghorn/itential-<P/E>-release
    
  16. Perform a NSO backup.

    ncs-backup
    
  17. Download the IAP target version bin file from the Nexus repository.

  18. Complete the following to bounce (soft restart) Redis, Rabbit MQ, and MongoDB, and then audit the dependency vitals. This is done to clean-up logs history and the logging buffer.

    • Run this vital check curl command to capture the dependencies current status.

      curl -s http://<server>:port/status
      
    • Run the following to audit the MongoDB Cluster and Replica status.

      sudo systemctl mongod stop
      
    • Wait 5 minutes and then start MongoDB.

      sudo systemctl mongod start
      
    • Run the monitoring checkpoints to audit the MongoDB Cluster.

      rs.status()
      rs.printSlaveReplicationInfo()
      
    • Bounce Redis, and then audit Sentinel and the Cluster status.

      sudo systemctl stop redis-sentinal.service
      
      sudo systemctl stop redis.service
      
    • Wait 5 minutes and then start the services.

      sudo systemctl start redis-sentinal.service
      
      sudo systemctl start redis.service
      
    • Audit the Sentinel status.

      redis-cli -h redis-XX -p 26379 info|grep status
      
    • Run this command on all Redis servers.

      redis-cli -h redis-XX -p 6379 info|grep role
      
    • Bounce RabbitMQ, and then audit the Cluster status.

      sudo service rabbitmq-server stop
      
    • Wait 5 minutes and then start the RabbitMQ server.

      sudo service rabbitmq-server start
      
    • Audit the RabbitMQ Cluster.

      sudo rabbitmqctl cluster_status
      
    • Run this vital check curl command after bounce to capture the current status of dependencies.

      curl -s http://<server>:port/status
      
  19. Clean-up the deprecated (old) applications and adapters (as listed below) to prepare for a clean IAP start.

    Deprecated Applications
    @itential/app-smart_template
    @itential/app-object/
    @itential/app-json/pronghorn.json
    @itential/app-tools/
    @itential/app-string
    @itential/app-number
    @itential/app-golden_services
    @itential/app-golden_config
    @itential/app-array/
    Deprecated Adapters
    adapter @itential/adapter-mongo


    • From the MongoDB CLI, remove the deprecated apps/adapter from service_configs.

      db.getCollection('service_configs').remove({"name" : "<app-name>"})
      
    • From the MongoDB CLI, remove the deprecated apps/adapter from iap_profiles - profile1.

      db.getCollection('iap_profiles -profile1').remove({"name" : "<app-name>"})
      
    • From IAP, remove the deprecated apps/adapter from the @itential scope in pronghorn.

      /opt/pronghorn/current/node_modules/@itential
      rm -rf <app-name>
      
  20. Audit the platform vitals for rogue processes, diskspace, CPU utilities, and memory.

  21. Stop the cron jobs.

  22. Archive the log files.

  23. Confirm DB backup is complete.


What's Next