Manage database indexes

2023.2 2023.1 2022.1

Index collections

Itential Platform requires indexes to be created in MongoDB collections for optimal performance. The following collections require predefined indexes. The owning application is shown in parentheses.

accounts (pronghorn-core)
catalog_store (app-service_catalog)
forms (app-form_builder)
groups (pronghorn-core)
iap_profiles (pronghorn-core)
integration_models (pronghorn-core)
jobs (app-workflow_engine)
job_history (app-workflow_engine)
mop_templates (app-mop)
mop_analytic_templates (app-mop)
roles (pronghorn-core)
service_configs (pronghorn-core)
tags (tags)
tasks (app-workflow_engine)
transformations (app-jst)
ucm_configs (app-configuration_manager)
wfe_job_metrics (app-workflow_engine)
wfe_task_metrics (app-workflow_engine)
workflows (app-workflow_engine)

Indexing APIs and seeding

Indexing in Itential Platform is managed via indexing APIs that can be called while the platform is running — not via indexing scripts. If an application is not installed, indexes for its collections are not required.

Collections that do not already exist are automatically created and properly indexed when Itential Platform starts. For new installations, manual indexing is not required. Use the indexing APIs when additional indexes are needed or an index has been accidentally dropped.

Validate indexes

To verify that a collection is properly indexed, use the GET /indexes/:collectionName/status API. This returns an object containing all missing or incorrectly named indexes. For example, to check the jobs collection:

GET /indexes/jobs/status

Any collection that is incorrectly indexed should go through the index creation API.

Create indexes

Create indexes only during a maintenance window, especially for large collections like jobs and tasks. Indexing these collections can take time and may impact MongoDB performance until the process completes.

If a collection is not properly indexed, the POST /indexes/:collectionName API creates any missing indexes and drops then recreates any indexes with an incorrect name. For the jobs collection:

POST /indexes/jobs

Submit an empty request body. All collections that are already properly indexed, or that are not included in the predefined index set, are ignored. If index creation has already been started, this call will not trigger a second creation in MongoDB. The API initiates the process but does not wait for it to complete before responding, since index creation can take significant time.

Validate indexes after creation

After running the index creation API, validate that indexes were created successfully. For large collections such as jobs and tasks, use GET /indexes/:collectionName/status to monitor progress and confirm when the process is complete.

Legacy npm script validation (deprecated)

The npm script method is deprecated. Use the indexing APIs described above instead.

The following applications support the legacy npm run index script:

Command Templates/MOP
Service Catalog
Workflow Engine

Automation Engine indexes

Run the index script for Workflow Engine:

$ cd /opt/pronghorn/current/node_modules/@itential/app-workflow_engine
$ npm run index

After the script completes, verify the indexes in MongoDB:

1 db.jobs.getIndexes()
2 db.tasks.getIndexes()
3 db.workflows.getIndexes()

Example output:

$ $ mongo mongo01.zone1.itential.io:27017/pronghorn --ssl -u pronghorn
$ MongoDB shell version v3.4.18
$ Enter password:
$ connecting to: mongodb://mongo01.zone1.itential.io:27017/pronghorn
$ MongoDB server version: 3.4.18
$ rs0:PRIMARY> db.jobs.getIndexes()
$ 
$ [{
>     "v": 2,
>     "key": { "_id": 1 },
>     "name": "_id_",
>     "ns": "pronghorn.jobs"
>   },
>   {
>     "v": 2,
>     "key": {
>       "name": 1, "type": 1, "groups": 1, "status": 1,
>       "metrics.start_time": -1, "metrics.progress": 1, "metrics.user": 1
>     },
>     "name": "name_1_type_1_groups_1_status_1_metrics.start_time_-1_metrics.progress_1_metrics.user_1",
>     "ns": "pronghorn.jobs",
>     "background": true
>   },
>   {
>     "v": 2,
>     "key": {
>       "name": 1, "type": 1, "watchers": 1, "status": 1,
>       "metrics.start_time": -1, "metrics.progress": 1, "metrics.user": 1
>     },
>     "name": "name_1_type_1_watchers_1_status_1_metrics.start_time_-1_metrics.progress_1_metrics.user_1",
>     "ns": "pronghorn.jobs",
>     "background": true
>   }
> ]
$ 
$ rs0:PRIMARY> db.tasks.getIndexes()
$ [{
>     "v": 2,
>     "key": { "_id": 1 },
>     "name": "_id_",
>     "ns": "pronghorn.tasks"
>   },
>   {
>     "v": 2,
>     "key": {
>       "name": 1, "status": 1, "groups": 1, "type": 1,
>       "job.name": 1, "job._id": 1, "job.task": 1,
>       "metrics.owner": 1, "metrics.start_time": -1
>     },
>     "name": "name_1_status_1_groups_1_type_1_job.name_1_job._id_1_job.task_1_metrics.owner_1_metrics.start_time_-1",
>     "ns": "pronghorn.tasks",
>     "background": true
>   },
>   {
>     "key": { "status": 1, "locked": 1 },
>     "background": true
>   },
>   {
>     "key": { "job._id": 1, "job.task": 1 },
>     "background": true
>   }
> ]
$ 
$ rs0:PRIMARY> db.workflows.getIndexes()
$ [{
>     "v": 2,
>     "key": { "_id": 1 },
>     "name": "_id_",
>     "ns": "pronghorn.workflows"
>   },
>   {
>     "v": 2,
>     "unique": true,
>     "key": { "name": 1, "type": 1 },
>     "name": "name_1_type_1",
>     "ns": "pronghorn.workflows",
>     "background": true
>   },
>   {
>     "v": 2,
>     "key": {
>       "groups": 1, "created": -1, "created_by": 1,
>       "last_updated": -1, "last_updated_by": 1
>     },
>     "name": "groups_1_created_-1_created_by_1_last_updated_-1_last_updated_by_1",
>     "ns": "pronghorn.workflows",
>     "background": true
>   }
> ]

Command Template/MOP indexes

$ cd /opt/pronghorn/current/node_modules/@itential/app-mop
$ npm run index

Service Catalog indexes

$ cd /opt/pronghorn/current/node_modules/@itential/app-service_catalog
$ npm run index