Administer MongoDB

MongoDB serves as the primary data store for Itential Platform. Platform stores jobs, tasks, workflow definitions, adapter configurations, and operational data in a MongoDB replica set. This guide covers the routine maintenance tasks required to keep the replica set healthy, performant, and secure.

For backup and retention policy guidance, see Archive and purge data. For detailed metrics and alert thresholds, see MongoDB metrics reference.

Before you begin

The commands in this guide assume the following environment variables are set:

$
MONGO_PRIMARY=<primary-hostname>
$
MONGO_NODES="<node1> <node2> <node3>"
$
MONGO_PORT=27017
$
MONGO_RS=<replica-set-name>
$
SSH_KEY_PATH=~/.ssh/<your-key>.pem
$
MONGO_SSH_USER=<ssh-user>

Retrieve MONGO_ADMIN_PASS from your secrets manager at the start of each maintenance session and hold it only in a shell variable for the session’s duration.

The example below uses HashiCorp Vault syntax. Adjust the retrieval command to match your secrets manager’s API:

$
source vault-env.sh
$
MONGO_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>
  $VAULT_ADDR/<platform-credentials-path> \
>
  | jq -r '.data.data.mongoDbAdmin')
$
$
MONGODB_URI="mongodb://admin:$MONGO_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS"

If TLS is enabled, append TLS flags to every mongosh command:

$
TLS_FLAGS="--tls --tlsCAFile /path/to/ca.pem"

For information about setting environment variables and Platform properties, see Platform properties and environment variables reference.

Check health and replica set status

MongoDB continuously synchronizes data across replica set members. If a node falls behind, loses connectivity, or enters a degraded state, Platform can experience job failures, write errors, or data inconsistency.

Check replica set member health

rs.status() is the single most important MongoDB health command. Run it in the mongosh shell connected to any replica set member:

1
rs.status()

Flag any member with health: 0 or stateStr: RECOVERING or UNKNOWN. For a concise view:

1
rs.status().members.forEach(m => print(m.name, m.stateStr, m.health, m.uptime));

Identify the current primary

All writes go to the primary. Know which node it is before running any write-intensive maintenance, credential rotation, or compaction; routing writes to a secondary fails.

1
rs.isMaster().primary

From a Bash prompt, useful in scripts:

$
mongosh "mongodb://admin:$MONGO_ADMIN_PASS@<any-node>:$MONGO_PORT/?authSource=admin" \
>
  $TLS_FLAGS --quiet --eval "rs.isMaster().primary" 2>/dev/null | tr -d '"' | cut -d: -f1

Always resolve the primary at runtime; never hardcode a hostname.

Check replication lag

1
rs.printSecondaryReplicationInfo()

A lag under 10 seconds is generally acceptable. Persistent lag above 60 seconds warrants investigation. Initiating an election while a secondary is far behind can leave it unable to vote or let it become primary with stale data, so check this before any planned maintenance.

Check WiredTiger cache utilization

WiredTiger’s in-memory cache serves reads without hitting disk. operations_timed_out above zero indicates the cache is starving and queries are waiting for eviction, a leading indicator of performance degradation before it’s visible to Platform users.

1
const cache = db.serverStatus().wiredTiger.cache;
2
print(JSON.stringify({
3
  bytes_in_cache: cache['bytes currently in the cache'],
4
  max_bytes_configured: cache['maximum bytes configured'],
5
  cache_full_pct: (
6
    cache['bytes currently in the cache'] / cache['maximum bytes configured'] * 100
7
  ).toFixed(1) + '%',
8
  operations_timed_out: cache['operations timed out waiting for space in cache'],
9
  eviction_unable_to_reach_goal: cache['eviction server unable to reach eviction goal']
10
}, null, 2));

Flag operations_timed_out greater than 0 and cache_full_pct above 95%.

Check system resource health

CPU saturation delays queries, disk I/O pressure slows writes and cache eviction, and swap usage causes severe latency spikes because MongoDB must wait for pages to load from disk.

$
for node in $MONGO_NODES; do
$
  echo "=== $node: CPU and memory ==="
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'vmstat 1 5; echo "---"; free -h' 2>/dev/null
$
  echo "=== $node: disk I/O ==="
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'iostat -x 1 5 2>/dev/null || echo "iostat not available"'
$
done

MongoDB must never use swap. If swap is non-zero, investigate memory pressure before the next maintenance window.

Review replica set configuration

A misconfigured member (wrong priority, an unexpected hidden flag, missing votes) can silently prevent failover from working correctly. Review the configuration periodically and always before a planned maintenance window.

View the configuration

1
rs.conf()

For election-relevant fields:

1
const cfg = rs.conf();
2
cfg.members.forEach(m =>
3
  print(JSON.stringify({
4
    host: m.host, priority: m.priority, votes: m.votes,
5
    hidden: m.hidden, slaveDelay: m.slaveDelay || m.secondaryDelaySecs
6
  }))
7
);

Review the index inventory

The jobs, job_data, and tasks collections in the itential database are the highest-traffic collections in a typical deployment. Without the right indexes, Platform queries fall back to full collection scans that slow down as data grows.

1
const itential = db.getSiblingDB('itential');
2
itential.jobs.getIndexes().forEach(i => print(JSON.stringify(i)));
3
itential.job_data.getIndexes().forEach(i => print(JSON.stringify(i)));
4
itential.tasks.getIndexes().forEach(i => print(JSON.stringify(i)));

Index existence doesn’t confirm usefulness; if a query runs a COLLSCAN despite an index being present, use explain("executionStats") (see Diagnose performance issues).

Check database sizes and connection counts

1
const dbs = db.adminCommand({ listDatabases: 1 });
2
const conn = db.serverStatus().connections;
3
print(JSON.stringify({
4
  connections: { current: conn.current, available: conn.available, totalCreated: conn.totalCreated },
5
  databases: dbs.databases
6
    .sort((a, b) => b.sizeOnDisk - a.sizeOnDisk)
7
    .map(d => ({ name: d.name, sizeOnDisk_MB: (d.sizeOnDisk / 1024 / 1024).toFixed(1) }))
8
}, null, 2));

Perform a manual failover

Without a controlled failover, stopping the primary causes the remaining members to detect the outage and hold an unplanned election, which can take 10 to 30 seconds and interrupt active Platform jobs. A manual failover moves the primary role to a specific node on your schedule, before any disruption occurs. This procedure is safe and non-destructive; no data is lost, and clients using a replica set connection string rediscover the new primary automatically.

Pre-failover checklist

Before stepping down the primary, confirm:

1. All members are healthy (health: 1, stateStr: SECONDARY or PRIMARY).
2. Replication lag is under 10 seconds on all secondaries.
3. At least two members are eligible to become primary (priority > 0, votes > 0).

1
rs.status()
2
rs.printSecondaryReplicationInfo()

Stepping down while a secondary is far behind can leave the replica set with no primary.

Bias the election toward a specific secondary (optional)

With no priority differences, the member that wins an election is the one with the most current oplog, which may not be the node you intend. Raise a specific member’s priority to bias the outcome:

1
cfg = rs.conf();
2
// Set priority to 10 on the member you want to win. Adjust the index to match.
3
cfg.members[1].priority = 10;
4
rs.reconfig(cfg);

The default priority is 1. Revert this after the failover completes.

Step down the primary

rs.stepDown() notifies the replica set gracefully, closes client connections cleanly, and gives secondaries time to catch up before the election begins:

1
rs.stepDown(60)

The argument is the number of seconds the stepped-down node refuses to become primary again, giving the election time to complete. Expect MongoServerError: not primary; this is normal, and mongosh reconnects automatically.

Confirm the new primary

Wait 10 to 15 seconds, then confirm a different member shows stateStr: "PRIMARY", the former primary shows "SECONDARY", and no member is stuck in RECOVERING or UNKNOWN:

1
rs.status()
2
rs.printSecondaryReplicationInfo()

Restore priority (if you raised it)

1
cfg = rs.conf();
2
cfg.members[1].priority = 1;
3
rs.reconfig(cfg);

Leaving priority elevated means that member always wins future elections, which can surprise the next person running a failover.

Manage the MongoDB service

The mongod process is managed by the systemd service of the same name. Always act on secondaries before the primary; restarting the primary triggers an election.

Check status on all nodes

$
for node in $MONGO_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'sudo systemctl status mongod' 2>/dev/null
$
done

A member that isn’t running is invisible to rs.status() heartbeats until it reconnects, so an OS-level check catches cases the replica set hasn’t yet registered.

Start, stop, or restart on a single node

$
# Replace <action> with: start, stop, or restart
$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> 'sudo systemctl <action> mongod'

After a start or restart, verify the node rejoins:

1
rs.status().members.find(m => m.name.startsWith('<target-node>'))

Expect stateStr: "SECONDARY" (or "PRIMARY" if the node won an election). Don’t touch the next node until this one is healthy.

Review the configuration file

The configuration file at /etc/mongod.conf records where data is stored, cache allocation, TLS settings, and the replica set name.

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> 'sudo cat /etc/mongod.conf'

Enable mongod to start on boot

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> 'sudo systemctl enable mongod'

Diagnose performance issues

As job and task collections grow, queries that were fast at low volume become slow, and slow queries compound under load. None of the diagnostics below modify data; they’re safe to run on a live system.

Check active operations

A stuck aggregation on jobs or tasks holds locks that block new job writes, causing Platform to queue requests and eventually time out.

1
const slow = db.currentOp({ secs_running: { $gt: 1 } });
2
print(JSON.stringify({
3
  total_active: db.currentOp().inprog.length,
4
  slow_ops_over_1s: slow.inprog.map(op => ({
5
    opid: op.opid, op: op.op, ns: op.ns, secs_running: op.secs_running,
6
    planSummary: op.planSummary, client: op.client
7
  }))
8
}, null, 2));

To filter specifically for aggregations on the jobs or tasks collections:

1
db.currentOp({ 'command.aggregate': { $in: ['jobs', 'tasks'] } })

Not every COLLSCAN is a problem: tailable/awaitData/$changeStream/oplog readers scan the oplog by design. A COLLSCAN on an aggregate: "jobs" or "tasks" command with secs_running > 1 seen repeatedly is the pattern to investigate.

Review slow query history (profiler)

The profiler records queries that took longest to complete, useful for intermittent slowdowns no longer visible in currentOp.

1
const db_name = 'itential';
2
const level = db.getSiblingDB(db_name).getProfilingStatus();
3
if (level.was === 0) {
4
  print('Profiler is OFF. To enable: db.getSiblingDB("itential").setProfilingLevel(1, {slowms: 100})');
5
} else {
6
  db.getSiblingDB(db_name).system.profile
7
    .find({}, { ns: 1, millis: 1, op: 1, 'command.aggregate': 1, 'command.find': 1 })
8
    .sort({ millis: -1 }).limit(20)
9
    .forEach(doc => print(JSON.stringify(doc)));
10
}

The profiler data is stored in the system.profile capped collection within each database; it isn’t persisted to disk separately from the MongoDB data files.

Explain a query

explain("executionStats") shows how MongoDB actually executed a query and how much work it did:

1
db.getSiblingDB('itential').<collection>.explain('executionStats').find({ <query> });
2
db.getSiblingDB('itential').<collection>.explain('executionStats').aggregate([ <pipeline> ]);

Don’t add an index if a similar leading key already exists in the index inventory.

Check write contention and locks

1
print(JSON.stringify({
2
  writeConflicts: db.serverStatus().metrics.operation.writeConflicts,
3
  locks: db.serverStatus().locks
4
}, null, 2));

Rising write conflicts mean multiple workers are updating the same document concurrently, often job status or task queue updates competing on a small set of shared documents. This is an application-level pattern that hardware can’t fix; correlate with currentOp to identify the namespace.

Configure Transparent Huge Pages (THP)

MongoDB 8.0 introduced a new version of TCMalloc that reversed long-standing THP guidance: THP must be disabled on MongoDB 7 and below, and enabled on MongoDB 8 and above. A misconfigured setting is hard to diagnose because its effects (latency spikes, fragmentation, increased I/O) resemble many other issues.

$
# Check the running version
$
mongosh "$MONGODB_URI" $TLS_FLAGS --quiet --eval "db.version()"
$
$
# Check current THP settings on all nodes
$
for node in $MONGO_NODES; do
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \
>
    'cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null;
>
     cat /sys/kernel/mm/transparent_hugepage/defrag 2>/dev/null'
$
done

$
# MongoDB 7 and below: disable THP
$
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
$
$
# MongoDB 8 and above: enable THP
$
echo always        | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$
echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

MongoDB 8 also recommends setting the following on each node:

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> \
>
  'echo 0 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none;
>
   echo 1 | sudo tee /proc/sys/vm/overcommit_memory'

These changes aren’t persistent across reboots; configure them in a systemd unit or /etc/rc.local. If you’re upgrading from MongoDB 7 to 8, change THP configuration to the MongoDB 8 targets as part of the upgrade; running MongoDB 8 with THP disabled negates the TCMalloc improvements introduced in that version.

Reclaim storage space

MongoDB doesn’t automatically reclaim disk space after documents are deleted. When large numbers of jobs or tasks are archived or purged, the freed space remains allocated as fragmented free space inside the data files, which can grow storage usage even as document counts fall.

Compact a collection

1
db.getSiblingDB('itential').runCommand({ compact: 'jobs' })

Verify the reduction afterward:

1
db.getSiblingDB('itential').jobs.stats().storageSize

Compact after a large delete (for example, archiving completed jobs), or when storageSize is significantly larger than size in collection.stats(). Don’t compact during active job processing or business hours in production.

Manage users and credentials

This deployment uses two accounts:

Your secrets manager is the authoritative source for both passwords. Update your secrets manager first, then update MongoDB to match. Updating MongoDB first creates a window where the two are out of sync.

Rotate the admin password

$
source vault-env.sh
$
NEW_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>
  $VAULT_ADDR/<platform-credentials-path> \
>
  | jq -r '.data.data.mongoDbAdmin')

$
mongosh "$MONGODB_URI" $TLS_FLAGS --quiet \
>
  --eval "db.adminCommand({ updateUser: 'admin', pwd: '$NEW_ADMIN_PASS' })"

$
mongosh "mongodb://admin:$NEW_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS" \
>
  $TLS_FLAGS --quiet --eval "db.adminCommand({ ping: 1 })"

Rotate the itential service account password

The itential account is the credential Platform uses for every database read and write. Because Platform reads it from your secrets manager at startup, rotating it requires a Platform restart to pick up the change.

$
source vault-env.sh
$
NEW_PLATFORM_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>
  $VAULT_ADDR/<platform-credentials-path> \
>
  | jq -r '.data.data.mongoDb')

$
mongosh "$MONGODB_URI" $TLS_FLAGS --quiet \
>
  --eval "db.getSiblingDB('itential').updateUser('itential', { pwd: '$NEW_PLATFORM_PASS' })"

$
mongosh "mongodb://itential:$NEW_PLATFORM_PASS@$MONGO_PRIMARY:$MONGO_PORT/itential?authSource=itential&replicaSet=$MONGO_RS" \
>
  $TLS_FLAGS --quiet --eval "db.adminCommand({ ping: 1 })"

Inspect logs

MongoDB logs are written in structured JSON, with a severity field ("s") where "I" is informational, "W" is warning, and "E" is error. The log file is at /var/log/mongodb/mongod.log by default; confirm against systemLog.path in /etc/mongod.conf.

Search for errors and warnings

$
for node in $MONGO_NODES; do
$
  echo "=== $node: recent errors and warnings ==="
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \
>
    'sudo grep -E "\"s\":\"E\"|\"s\":\"W\"|Slow query|NETWORK|REPL|assertion|INITSYNC" \
>
     /var/log/mongodb/mongod.log | tail -30' 2>/dev/null
$
done

Search for a specific pattern

$
for node in $MONGO_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \
>
    "sudo grep -E '<your-pattern>' /var/log/mongodb/mongod.log | tail -50" 2>/dev/null
$
done

View recent log entries on a single node

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> \
>
  'sudo tail -100 /var/log/mongodb/mongod.log'

For a human-readable format, pipe through jq to extract key fields:

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> \
>
  'sudo tail -100 /var/log/mongodb/mongod.log' \
>
  | jq -r '[.t["$date"], .s, .c, .msg] | @tsv' 2>/dev/null

Rotate TLS certificates

An expired certificate breaks replication, interrupts Platform’s database access, and can cause a primary election to fail if secondaries can’t reach the primary. TLS certificate files are referenced in /etc/mongod.conf under net.tls:

$
for node in $MONGO_NODES; do
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \
>
    'sudo openssl x509 -in /etc/ssl/mongodb/mongod.pem -noout -dates 2>/dev/null'
$
done

$
openssl x509 -in new-mongod.pem -noout -text | grep -E "Subject:|Not Before:|Not After :"
$
openssl verify -CAfile new-ca.pem new-mongod.pem
$
openssl x509 -noout -modulus -in new-mongod.pem | openssl md5
$
openssl rsa  -noout -modulus -in new-mongod.pem | openssl md5

$
for node in $MONGO_NODES; do
$
  scp -i $SSH_KEY_PATH new-mongod.pem new-ca.pem $MONGO_SSH_USER@$node:/tmp/
$
$
  ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node '
>
    sudo cp /tmp/new-mongod.pem /etc/ssl/mongodb/mongod.pem
>
    sudo cp /tmp/new-ca.pem    /etc/ssl/mongodb/ca.pem
>
    sudo chown mongod:mongod   /etc/ssl/mongodb/mongod.pem /etc/ssl/mongodb/ca.pem
>
    sudo chmod 400             /etc/ssl/mongodb/mongod.pem
>
    sudo chmod 444             /etc/ssl/mongodb/ca.pem
>
  '
$
done

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<secondary-node> 'sudo systemctl restart mongod'

1
rs.status().members.find(m => m.name.startsWith('<secondary-node>'))

1
rs.stepDown(60)

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<former-primary> 'sudo systemctl restart mongod'

$
mongosh "mongodb://admin:$MONGO_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS" \
>
  --tls --tlsCAFile /etc/ssl/mongodb/ca.pem --quiet \
>
  --eval "rs.status().members.forEach(m => print(m.name, m.stateStr, m.health))"

Rollback

$
ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@<target-node> '
>
  sudo cp /etc/ssl/mongodb/mongod.pem.bak /etc/ssl/mongodb/mongod.pem
>
  sudo cp /etc/ssl/mongodb/ca.pem.bak     /etc/ssl/mongodb/ca.pem
>
  sudo systemctl restart mongod
>
'