Common Upgrade Errors

This guide discusses some of the errors you may encounter in the software upgrade, along with various troubleshooting solutions.

Deprecated App Not Loading

When loading IAP, you might see this error message in the logs. In this case, the depreciated method app-number is loaded, but does not exist.

Error: ENOENT: no such file or directory, open '/opt/pronghorn/itential-bundle-5-20193_2019.3.6/node_modules/@itential/app-number/pronghorn.json'
    at Object.openSync (fs.js:443:3)
    at Object.readFileSync (fs.js:343:35)
    at initialize (/opt/pronghorn/itential-bundle-5-20193_2019.3.6/node_modules/@itential/service/lib/Service.js:787:24)
    at new Service (/opt/pronghorn/itential-bundle-5-20193_2019.3.6/node_modules/@itential/service/lib/Service.js:886:3)
    at Object.<anonymous> (/opt/pronghorn/itential-bundle-5-20193_2019.3.6/node_modules/@itential/service/lib/ServiceBootstrap.js:35:19)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/opt/pronghorn/itential-bundle-5-20193_2019.3.6/node_modules/@itential/service/index.js:1:26)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)

Solution

You need to remove the entry from the service_configs collection.

1. Select the collection service_configs in your Pronghorn (IAP) database.

2. Search for the model number shown in the error. In this example, we would search for: db.getCollection('service_configs').find({"model" : "@itential/app-number"})

   

3. Once you find the service config document, right-click and delete the document.

   

Admin Group Not in Proper Format

When assigning roles to user groups the GUI is not correctly reflecting the selected roles as "checked". An error box appears at the bottom of the screen.

Upon inspecting the console, an error message displays.

After running the journalctl -fu pronghorn command, the following error returns.

message=[ 'Error in web handler: TypeError: kg._id.equals is not a function' ]

Solution

This issue is caused by the "admin" group having an _id that is a string, and not an ObjectID. To address the issue, you will need delete this group.

1. Before proceeding, backup the DB using mongodump. Use a GUI mongo client to run the commands below. Alternatively, you can access the mongo shell with these commands:

   mongo --host localhost --authenticationDatabase <authdb> -u admin -p <password> use pronghorn
   

   db.groups.find({name: "admin"}).pretty()
   

   A single document should return matching the admin user.

   db.groups.remove({name: "admin"})
   

   This should reflect 1 record removed.

2. Return to the Pronghorn (IAP) UI and click the REFRESH button to fully refresh the page and reload the page data.

3. At this point the "admin" group should no longer be present and you should now be able to use your groups normally.

Jobs (Operations Manager) Not Showing Updates

The Jobs UI is not showing any updates after running the upgrade.

Solution

The migration scripts have not ran. Change directory to the migration script database, then run the scripts again.

cd <PRONGHORN_HOME>
cd ./current/node_modules/@itential/pronghorn-core/migration_scripts
node migratePropertiesToDatabase.js

Devices Not Visible in Configuration Manager

The following error message is seen in the ConfigurationManager.log.

No group access for [object Object] for action XXXX on entity of type device

When you look in the Configuration Manager app, devices are unavailable.

Solution

The migration scripts have not ran. Change directory to the migration script database, then run the scripts again.

cd <PRONGHORN_HOME>
cd ./current/node_modules/@itential/pronghorn-core/migration_scripts
node migratePropertiesToDatabase.js

Failed to Load Schema Warning Message

Beginning with release version 2019.3, IAP included a built-in schema validation system for inputs/outputs to the pronghorn.json. This system will identify invalid schema formats and automatically run a script to convert the schema to the appropriate format.

In the majority of cases the script works and makes updating the pronghorn.json easier. However there may be some schema errors the script cannot handle, in which case an error warning will display alerting you to the invalid data and the need for it to be fixed. Once the pronghorn.json is updated, the schema should load.

The following are valid types for input and outputs to the pronghorn.json:

['*', 'enum', 'object', 'string', 'array', 'boolean', 'integer', 'number']

Notes:

• Type 'enum' is not a valid json schema so the type will be set to array
• Type '*' is a valid type, however it should ONLY be used when you really intend for an input/output to be any type. Using this type will limit json validation and could result in errors if bad type is allowed in.
• Common invalid types include: json, json[], str, string[] and many other variations.

Example: Valid pronghorn.json

"methods": [
  {
    "name": "getDevices",
    "summary": "Get the list of supported devices from Inventory for the siteId",
    "description": "Get all microwave devices for the given siteId defined in inventory, then filter and
only return those supported.",
    "roles": [
      "admin"
    ],
    "route": {
      "path": "/:siteId/devices",
      "verb": "GET"
    },
    "input": [
      {
        "name": "siteId",
        "type": "string",
        "description": "Site identifier to get equipment for",
        "required": "true",
        "in": "path",
        "schema": {
          "title": "devices",
          "type": "object"
        },
      },
    ],
   "output": {
         "name": "devices",
         "description": "success/failure response object",
         "type": "object",
         "schema": {
          "title": "devices",
          "type": "object"
       }
   },
   "task": true
},

Solution

Run the fix-pronghorn_json.js.gz script to fix issues in pronghorn.json.

1. Download the script to your folder.

   app-folder_name
   

2. Execute the node command.

   node fix-pronghorn_json.js pronghorn.json
   

3. Possible responses include:

   Response	Message
   Success	pronghorn.json has been updated
   Success (no updates required)	No methods detected in: pronghorn.json, file skipped
   Failure	pronghorn.json update cancelled due to previous errors

4. Should a failure occur, an error should display that provides a hint on how to fix it. Typically invalid JSON or invalid types are the cause; once you have fixed the issue repeat Step 2 (execute node command).

5. If the pronghorn.json update is successful, be sure to push the pronghorn.json back to your repository.

6. Change directory to the migration script database, then run the migration scripts.

   cd <PRONGHORN_HOME>
   cd ./current/node_modules/@itential/pronghorn-core/migration_scripts
   node migratePropertiesToDatabase.js
   

Task Is Causing Workflow Engine to Stall or Crash

When an upgraded workflow is ran, Workflow Engine (WFE) will either stall or crash when hitting a specific task.

Solution

Delete the task and then recreate it.

Error Retrieving Backups, Device Groups, and Golden Configurations

In Configuration Manager errors occur when trying to retrieve backups, device groups, or Golden Configurations.

Solution

A migration script is needed.

1. Confirm MongoDB version is 5.0.

   mongo --version
   

2. Run the migration script in the Configuration Manager folder.

   /node_modules/@itential/app-configuration_manager/migration_scripts
   

3. For DEV / STAGING environment, execute the node command:

   node migrate_to_jinja.js dev
   

4. For PROD environment, execute the node command:

   node migrate_to_jinja.js
   

5. Restart IAP.

   sudo systemctl restart pronghorn
   

6. Try to retrieve backups, device groups, and Golden Configurations.

RabbitMQ Not Starting

RabbitMQ will not start after being upgraded. When investigating the RabbitMQ log file, error messages similar to the following are found:

[error] <0.237.0> Feature flags: `classic_mirrored_queue_version`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
[error] <0.237.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
[error] <0.237.0>                                               classic_mirrored_queue_version}
[error] <0.237.0>
[error] <0.237.0> BOOT FAILED

Solution

Enable the feature flag indicated in the error messages by using the rabbitmqctl enable_feature_flag command. For example, to enable the classic_mirrored_queue_version feature flag, issue the following:

rabbitmqctl enable_feature_flag classic_mirrored_queue_version

For more information about feature flags, refer to the official RabbitMQ documentation.