MongoDB backup and recovery

This article describes how to back up and restore the MongoDB database used by the Itential Platform. The following high-level strategy minimizes downtime, maximizes efficiency, and helps prevent service level agreement (SLA) breaches.

Backup strategy

Itential recommends these backup practices:

Perform a daily mongodump to capture all database data once per day
Take hourly snapshots of the jobs and tasks collections to capture automation results throughout the day
Retain backups for no longer than seven days, as backup files can become large

Note

You can capture hourly snapshots in binary format or JSON format. However, Itential prefers binary format and recommends using binary whenever possible.

Identify critical directories and files

You need current configuration and data files for restoration. Ensure you identify and backup these files. The following locations are defaults and might vary in your installation:

File type	Location	Description
Configuration files	`/etc/mongod.conf`	Main configuration
Security and authentication files	`/etc/ssl/certs/mongo`	TLS certificates and keys
Security and authentication files	`/etc/ssl/`	Additional security files
Data directory	`/var/lib/mongo`	Database files

Back up MongoDB data

You can only export and import from the primary MongoDB server in the replica set. Connect to the primary server, then use mongoexport or mongodump to export content to the appropriate directory. Move the data to the target server. You'll use mongoimport or mongorestore to recover from these backups.

For complete MongoDB documentation, see the MongoDB backup and restore tools guide.

Back up the entire database

Export the database to a ./dump/ directory using the hostname of the current primary server in the replica set:

mongodump --db=itential --oplog <other tls options> --uri='mongodb://admin:<password>@<mongo-host1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin&readPreference=secondary'

Package the backup for transfer:

tar -czvf mongodump-20230906.tgz dump
scp mongodump-20230906.tgz <user>@<target-server>:<target-directory>

Back up a specific collection in the Itential database

Use mongodump for binary backups or mongoexport for human-readable JSON format.

Binary format

Export to a binary formatted file. The mongodump command dumps the workflows collection to a dump directory. You can run this directly against a secondary server or connect to the replica set and prefer the secondary:

mongodump --ssl --collection=workflows --uri='mongodb://admin:<password>@<mongo-host-1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin&readPreference=secondary'

Package the backup for transfer:

tar -czvf mongodump-20230906.tgz dump
scp mongodump-20230906.tgz <user>@<target-server>:<target-directory>

JSON format

Export to a JSON formatted file. You can use --ssl or --tls options:

mongoexport --ssl --db=itential --collection=workflows --out=workflows/workflows.json --uri='mongodb://admin:<password>@<mongo-host-1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin&readPreference=secondary'

Package the backup for transfer:

tar -czvf workflows.json.tgz workflows/*
scp workflows.json.tgz <user>@<target-server>:<target-directory>

Restore the data

When you restore a member of a replica set, MongoDB uses ensures that the new member doesn't become the primary member until it syncs with the existing replica set. MongoDB does this automatically by controlling the member's state. However, you can manually control whether a member can become primary by setting the priority.

Member states

If you add a new member to a replica set, or if an existing member has been out of service for an extended period, MongoDB initiates a series of steps to bring that member in line with the existing members. MongoDB achieves this by controlling the member's state to reflect its status within the set.

STARTUP

Each replica set member starts in this state
mongod loads the member's replica set configuration and transitions the member's state to STARTUP2
Members can't vote and aren't recognized as replica set members yet

STARTUP2

Members enter this state after mongod finishes loading the configuration
The member becomes an active replica set member
If the member begins initial sync, it remains in STARTUP2 until all data is copied and all indexes are built
The member then transitions to RECOVERING

RECOVERING

Members aren't ready to accept reads
This state can occur during normal operation and doesn't indicate an error
Members can vote in elections but can't become primary
Members transition to SECONDARY after replicating enough data for consistent client reads
The only difference between RECOVERING and SECONDARY states is that RECOVERING prohibits client reads and SECONDARY permits them

Control member priority

To prevent a secondary member from becoming primary during failover, assign it a priority lower than the primary member's priority.

Example scenario: Your current primary is m1.example.net and you want to make m3.example.net the primary. You have a three-member replica set with the following configuration:

{
  "_id" : "rs0",
  "version" : 7,
  "members" : [
    {
      "_id" : 0,
      "host" : "m1.example.net:27017"
    },
    {
      "_id" : 1,
      "host" : "m2.example.net:27017"
    },
    {
      "_id" : 2,
      "host" : "m3.example.net:27017"
    }
  ]
}

To make m3.example.net the primary:

In a mongo shell connected to the primary, run the following sequence of operations:

cfg = rs.conf();
cfg.members[0].priority = 5;
cfg.members[1].priority = 5;
cfg.members[2].priority = 10;
rs.reconfig(cfg);

The following sequence occurs:
1. m3.example.net and m2.example.net sync with m1.example.net (typically within 10 seconds)
2. m1.example.net sees it no longer has the highest priority and steps down
  1. If m3.example.net's sync is far behind, m1.example.net waits until m3.example.net is within 10 seconds of its optime before stepping down
3. The step down forces an election where m3.example.net becomes primary based on its priority setting

Complete the restoration

Use the appropriate restore command based on your backup method:

Restore from binary format

If you used mongodump to create the backup, use mongorestore to restore it:

mongorestore --ssl --drop --uri='mongodb://admin:<password>@<mongo-host-1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin' dump/itential

Restore from JSON format

If you used mongoexport to create the backup, use mongoimport to restore it:

mongoimport --ssl --db=itential --collection=workflows --file=workflows/workflows.json --uri='mongodb://admin:<password>@<mongo-host-1>:27017,<mongo-host-2>:27017,<mongo-host-3>:27017/itential?replicaSet=rs0&authSource=admin'

For more information, see the MongoDB replica set restoration documentation.

Next steps

After completing the backup and recovery process, consider implementing these additional measures:

Set up automated backup monitoring to ensure backups complete successfully
Test your recovery procedures regularly to verify they work as expected
Document your specific configuration details for future reference
Consider implementing additional security measures for backup storage