> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.itential.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.itential.com/_mcp/server. # Administer MongoDB > Routine administration and maintenance procedures for the MongoDB replica set backing Itential Platform on-prem, including health checks, manual failover, performance diagnostics, and TLS certificate rotation. MongoDB serves as the primary data store for Itential Platform. Platform stores jobs, tasks, workflow definitions, adapter configurations, and operational data in a MongoDB replica set. This guide covers the routine maintenance tasks required to keep the replica set healthy, performant, and secure. For backup and retention policy guidance, see [Archive and purge data](/itential-platform/maintain/archive-purge-data). For detailed metrics and alert thresholds, see [MongoDB metrics reference](/itential-platform/monitor/metrics-reference/mongodb). ## Before you begin The commands in this guide assume the following environment variables are set: ```bash MONGO_PRIMARY= MONGO_NODES=" " MONGO_PORT=27017 MONGO_RS= SSH_KEY_PATH=~/.ssh/.pem MONGO_SSH_USER= ``` Retrieve `MONGO_ADMIN_PASS` from your secrets manager at the start of each maintenance session and hold it only in a shell variable for the session's duration. The example below uses HashiCorp Vault syntax. Adjust the retrieval command to match your secrets manager's API: ```bash source vault-env.sh MONGO_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.mongoDbAdmin') MONGODB_URI="mongodb://admin:$MONGO_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS" ``` If TLS is enabled, append TLS flags to every `mongosh` command: ```bash TLS_FLAGS="--tls --tlsCAFile /path/to/ca.pem" ``` Passwords must never be stored in scripts, configuration files, or shell history. Retrieve credentials from your secrets manager at runtime and let them leave scope when the session ends. For information about setting environment variables and Platform properties, see [Platform properties and environment variables reference](https://docs.itential.com/itential-platform/6/configure/environment-variables-properties-reference). ## Check health and replica set status MongoDB continuously synchronizes data across replica set members. If a node falls behind, loses connectivity, or enters a degraded state, Platform can experience job failures, write errors, or data inconsistency. ### Check replica set member health `rs.status()` is the single most important MongoDB health command. Run it in the `mongosh` shell connected to any replica set member: ```js rs.status() ``` | Field | Expected value | | -------------------------------- | ---------------------------------------- | | `members[].health` | `1` for all members | | `members[].stateStr` | One `PRIMARY`, remainder `SECONDARY` | | `members[].lastHeartbeatMessage` | Empty string | | `members[].syncSourceHost` | Each secondary names a valid sync source | Flag any member with `health: 0` or `stateStr: RECOVERING` or `UNKNOWN`. For a concise view: ```js rs.status().members.forEach(m => print(m.name, m.stateStr, m.health, m.uptime)); ``` ### Identify the current primary All writes go to the primary. Know which node it is before running any write-intensive maintenance, credential rotation, or compaction; routing writes to a secondary fails. ```js rs.isMaster().primary ``` From a Bash prompt, useful in scripts: ```bash mongosh "mongodb://admin:$MONGO_ADMIN_PASS@:$MONGO_PORT/?authSource=admin" \ $TLS_FLAGS --quiet --eval "rs.isMaster().primary" 2>/dev/null | tr -d '"' | cut -d: -f1 ``` Always resolve the primary at runtime; never hardcode a hostname. ### Check replication lag ```js rs.printSecondaryReplicationInfo() ``` A lag under 10 seconds is generally acceptable. Persistent lag above 60 seconds warrants investigation. Initiating an election while a secondary is far behind can leave it unable to vote or let it become primary with stale data, so check this before any planned maintenance. ### Check WiredTiger cache utilization WiredTiger's in-memory cache serves reads without hitting disk. `operations_timed_out` above zero indicates the cache is starving and queries are waiting for eviction, a leading indicator of performance degradation before it's visible to Platform users. ```js const cache = db.serverStatus().wiredTiger.cache; print(JSON.stringify({ bytes_in_cache: cache['bytes currently in the cache'], max_bytes_configured: cache['maximum bytes configured'], cache_full_pct: ( cache['bytes currently in the cache'] / cache['maximum bytes configured'] * 100 ).toFixed(1) + '%', operations_timed_out: cache['operations timed out waiting for space in cache'], eviction_unable_to_reach_goal: cache['eviction server unable to reach eviction goal'] }, null, 2)); ``` Flag `operations_timed_out` greater than 0 and `cache_full_pct` above 95%. ### Check system resource health CPU saturation delays queries, disk I/O pressure slows writes and cache eviction, and swap usage causes severe latency spikes because MongoDB must wait for pages to load from disk. ```bash for node in $MONGO_NODES; do echo "=== $node: CPU and memory ===" ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'vmstat 1 5; echo "---"; free -h' 2>/dev/null echo "=== $node: disk I/O ===" ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'iostat -x 1 5 2>/dev/null || echo "iostat not available"' done ``` | Metric | Concern | | ---------------- | --------------------------------------------------- | | `wa` in `vmstat` | Consistently above 5% indicates disk wait | | `r` (run queue) | High relative to CPU count indicates CPU saturation | | Swap used | Any swap usage on a MongoDB host is a warning sign | | Disk `%util` | Near 100% indicates an I/O bottleneck | MongoDB must never use swap. If swap is non-zero, investigate memory pressure before the next maintenance window. ## Review replica set configuration A misconfigured member (wrong priority, an unexpected hidden flag, missing votes) can silently prevent failover from working correctly. Review the configuration periodically and always before a planned maintenance window. ### View the configuration ```js rs.conf() ``` For election-relevant fields: ```js const cfg = rs.conf(); cfg.members.forEach(m => print(JSON.stringify({ host: m.host, priority: m.priority, votes: m.votes, hidden: m.hidden, slaveDelay: m.slaveDelay || m.secondaryDelaySecs })) ); ``` | Field | Notes | | ----------------------------------- | ----------------------------------------------------------------- | | `priority` | Members with `priority: 0` can't become primary | | `votes` | Members with `votes: 0` don't participate in elections | | `hidden` | Hidden members are invisible to clients but can vote | | `slaveDelay` / `secondaryDelaySecs` | Delayed members lag intentionally; confirm this matches intention | ### Review the index inventory The `jobs`, `job_data`, and `tasks` collections in the `itential` database are the highest-traffic collections in a typical deployment. Without the right indexes, Platform queries fall back to full collection scans that slow down as data grows. ```js const itential = db.getSiblingDB('itential'); itential.jobs.getIndexes().forEach(i => print(JSON.stringify(i))); itential.job_data.getIndexes().forEach(i => print(JSON.stringify(i))); itential.tasks.getIndexes().forEach(i => print(JSON.stringify(i))); ``` Index existence doesn't confirm usefulness; if a query runs a `COLLSCAN` despite an index being present, use `explain("executionStats")` (see [Diagnose performance issues](#diagnose-performance-issues)). ### Check database sizes and connection counts ```js const dbs = db.adminCommand({ listDatabases: 1 }); const conn = db.serverStatus().connections; print(JSON.stringify({ connections: { current: conn.current, available: conn.available, totalCreated: conn.totalCreated }, databases: dbs.databases .sort((a, b) => b.sizeOnDisk - a.sizeOnDisk) .map(d => ({ name: d.name, sizeOnDisk_MB: (d.sizeOnDisk / 1024 / 1024).toFixed(1) })) }, null, 2)); ``` ## Perform a manual failover Without a controlled failover, stopping the primary causes the remaining members to detect the outage and hold an unplanned election, which can take 10 to 30 seconds and interrupt active Platform jobs. A manual failover moves the primary role to a specific node on your schedule, before any disruption occurs. This procedure is safe and non-destructive; no data is lost, and clients using a replica set connection string rediscover the new primary automatically. ### Pre-failover checklist Before stepping down the primary, confirm: 1. All members are healthy (`health: 1`, `stateStr: SECONDARY` or `PRIMARY`). 2. Replication lag is under 10 seconds on all secondaries. 3. At least two members are eligible to become primary (`priority > 0`, `votes > 0`). ```js rs.status() rs.printSecondaryReplicationInfo() ``` Stepping down while a secondary is far behind can leave the replica set with no primary. ### Bias the election toward a specific secondary (optional) With no priority differences, the member that wins an election is the one with the most current oplog, which may not be the node you intend. Raise a specific member's priority to bias the outcome: ```js cfg = rs.conf(); // Set priority to 10 on the member you want to win. Adjust the index to match. cfg.members[1].priority = 10; rs.reconfig(cfg); ``` The default priority is `1`. Revert this after the failover completes. ### Step down the primary `rs.stepDown()` notifies the replica set gracefully, closes client connections cleanly, and gives secondaries time to catch up before the election begins: ```js rs.stepDown(60) ``` The argument is the number of seconds the stepped-down node refuses to become primary again, giving the election time to complete. Expect `MongoServerError: not primary`; this is normal, and `mongosh` reconnects automatically. ### Confirm the new primary Wait 10 to 15 seconds, then confirm a different member shows `stateStr: "PRIMARY"`, the former primary shows `"SECONDARY"`, and no member is stuck in `RECOVERING` or `UNKNOWN`: ```js rs.status() rs.printSecondaryReplicationInfo() ``` ### Restore priority (if you raised it) ```js cfg = rs.conf(); cfg.members[1].priority = 1; rs.reconfig(cfg); ``` Leaving priority elevated means that member always wins future elections, which can surprise the next person running a failover. ## Manage the MongoDB service The `mongod` process is managed by the systemd service of the same name. Always act on secondaries before the primary; restarting the primary triggers an election. ### Check status on all nodes ```bash for node in $MONGO_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node 'sudo systemctl status mongod' 2>/dev/null done ``` A member that isn't running is invisible to `rs.status()` heartbeats until it reconnects, so an OS-level check catches cases the replica set hasn't yet registered. ### Start, stop, or restart on a single node Restarting a secondary is low-risk. Restarting the primary triggers an election and temporarily interrupts write availability. Restarting multiple nodes simultaneously can leave the replica set without enough voting members to elect a primary. ```bash # Replace with: start, stop, or restart ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ 'sudo systemctl mongod' ``` After a start or restart, verify the node rejoins: ```js rs.status().members.find(m => m.name.startsWith('')) ``` Expect `stateStr: "SECONDARY"` (or `"PRIMARY"` if the node won an election). Don't touch the next node until this one is healthy. ### Review the configuration file The configuration file at `/etc/mongod.conf` records where data is stored, cache allocation, TLS settings, and the replica set name. ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ 'sudo cat /etc/mongod.conf' ``` | Section | What to check | | --------------------------------------------- | ---------------------------------------- | | `storage.dbPath` | Data directory location | | `storage.wiredTiger.engineConfig.cacheSizeGB` | WiredTiger cache size, if explicitly set | | `replication.replSetName` | Must match on all nodes | | `net.tls` | TLS configuration | | `systemLog.path` | Log file location | ### Enable mongod to start on boot ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ 'sudo systemctl enable mongod' ``` ## Diagnose performance issues As job and task collections grow, queries that were fast at low volume become slow, and slow queries compound under load. None of the diagnostics below modify data; they're safe to run on a live system. ### Check active operations A stuck aggregation on `jobs` or `tasks` holds locks that block new job writes, causing Platform to queue requests and eventually time out. ```js const slow = db.currentOp({ secs_running: { $gt: 1 } }); print(JSON.stringify({ total_active: db.currentOp().inprog.length, slow_ops_over_1s: slow.inprog.map(op => ({ opid: op.opid, op: op.op, ns: op.ns, secs_running: op.secs_running, planSummary: op.planSummary, client: op.client })) }, null, 2)); ``` To filter specifically for aggregations on the `jobs` or `tasks` collections: ```js db.currentOp({ 'command.aggregate': { $in: ['jobs', 'tasks'] } }) ``` Not every `COLLSCAN` is a problem: tailable/`awaitData`/`$changeStream`/oplog readers scan the oplog by design. A `COLLSCAN` on an `aggregate: "jobs"` or `"tasks"` command with `secs_running > 1` seen repeatedly is the pattern to investigate. ### Review slow query history (profiler) The profiler records queries that took longest to complete, useful for intermittent slowdowns no longer visible in `currentOp`. ```js const db_name = 'itential'; const level = db.getSiblingDB(db_name).getProfilingStatus(); if (level.was === 0) { print('Profiler is OFF. To enable: db.getSiblingDB("itential").setProfilingLevel(1, {slowms: 100})'); } else { db.getSiblingDB(db_name).system.profile .find({}, { ns: 1, millis: 1, op: 1, 'command.aggregate': 1, 'command.find': 1 }) .sort({ millis: -1 }).limit(20) .forEach(doc => print(JSON.stringify(doc))); } ``` The profiler data is stored in the `system.profile` capped collection within each database; it isn't persisted to disk separately from the MongoDB data files. ### Explain a query `explain("executionStats")` shows how MongoDB actually executed a query and how much work it did: ```js db.getSiblingDB('itential')..explain('executionStats').find({ }); db.getSiblingDB('itential')..explain('executionStats').aggregate([ ]); ``` | Scenario | Meaning | | ----------------------------------------------------- | ------------------------------------------------------------- | | `COLLSCAN` + `totalDocsExamined` near collection size | Full table scan; fix the query shape or add a selective index | | `IXSCAN` + `totalDocsExamined` near collection size | Index exists but isn't selective; fix the query | | `IXSCAN` + low `totalDocsExamined` | Index is working correctly | Don't add an index if a similar leading key already exists in the index inventory. ### Check write contention and locks ```js print(JSON.stringify({ writeConflicts: db.serverStatus().metrics.operation.writeConflicts, locks: db.serverStatus().locks }, null, 2)); ``` Rising write conflicts mean multiple workers are updating the same document concurrently, often job status or task queue updates competing on a small set of shared documents. This is an application-level pattern that hardware can't fix; correlate with `currentOp` to identify the namespace. ### Configure Transparent Huge Pages (THP) MongoDB 8.0 introduced a new version of TCMalloc that reversed long-standing THP guidance: THP must be disabled on MongoDB 7 and below, and enabled on MongoDB 8 and above. A misconfigured setting is hard to diagnose because its effects (latency spikes, fragmentation, increased I/O) resemble many other issues. ```bash # Check the running version mongosh "$MONGODB_URI" $TLS_FLAGS --quiet --eval "db.version()" # Check current THP settings on all nodes for node in $MONGO_NODES; do ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \ 'cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null; cat /sys/kernel/mm/transparent_hugepage/defrag 2>/dev/null' done ``` | Setting | MongoDB 7 and below | MongoDB 8 and above | | ------------------------------ | ------------------- | ------------------- | | `transparent_hugepage/enabled` | `never` | `always` | | `transparent_hugepage/defrag` | `never` | `defer+madvise` | Confirm with your team before changing THP settings on a production node. ```bash # MongoDB 7 and below: disable THP echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag # MongoDB 8 and above: enable THP echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag ``` MongoDB 8 also recommends setting the following on each node: ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ \ 'echo 0 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none; echo 1 | sudo tee /proc/sys/vm/overcommit_memory' ``` These changes aren't persistent across reboots; configure them in a systemd unit or `/etc/rc.local`. If you're upgrading from MongoDB 7 to 8, change THP configuration to the MongoDB 8 targets as part of the upgrade; running MongoDB 8 with THP disabled negates the TCMalloc improvements introduced in that version. ## Reclaim storage space MongoDB doesn't automatically reclaim disk space after documents are deleted. When large numbers of jobs or tasks are archived or purged, the freed space remains allocated as fragmented free space inside the data files, which can grow storage usage even as document counts fall. ### Compact a collection `compact` blocks all reads and writes on the collection for its duration. Run it on a secondary during a maintenance window, or on the primary only when there are no active workflows. ```js db.getSiblingDB('itential').runCommand({ compact: 'jobs' }) ``` Verify the reduction afterward: ```js db.getSiblingDB('itential').jobs.stats().storageSize ``` Compact after a large delete (for example, archiving completed jobs), or when `storageSize` is significantly larger than `size` in `collection.stats()`. Don't compact during active job processing or business hours in production. ## Manage users and credentials This deployment uses two accounts: | Username | Auth database | Purpose | | ---------- | ------------- | ------------------------------------------------------ | | `admin` | `admin` | Root admin account used for all maintenance operations | | `itential` | `itential` | Application service account used by Platform | Your secrets manager is the authoritative source for both passwords. Update your secrets manager first, then update MongoDB to match. Updating MongoDB first creates a window where the two are out of sync. ### Rotate the admin password High-risk. The `admin` account has full control over the instance, including creating users, dropping databases, and modifying the replica set configuration. #### Update the secret in your secrets manager Generate a new password and store it under the key `mongoDbAdmin` before making any change to MongoDB. #### Retrieve the new password Don't type the password directly into the shell; read it into a variable. The example below uses HashiCorp Vault syntax; adjust the retrieval command to match your secrets manager's API: ```bash source vault-env.sh NEW_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.mongoDbAdmin') ``` #### Apply the new password ```bash mongosh "$MONGODB_URI" $TLS_FLAGS --quiet \ --eval "db.adminCommand({ updateUser: 'admin', pwd: '$NEW_ADMIN_PASS' })" ``` #### Verify connectivity ```bash mongosh "mongodb://admin:$NEW_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS" \ $TLS_FLAGS --quiet --eval "db.adminCommand({ ping: 1 })" ``` ### Rotate the itential service account password The `itential` account is the credential Platform uses for every database read and write. Because Platform reads it from your secrets manager at startup, rotating it requires a Platform restart to pick up the change. #### Update the secret in your secrets manager Store the new password under the key `mongoDb`. #### Retrieve the new password The example below uses HashiCorp Vault syntax; adjust the retrieval command to match your secrets manager's API: ```bash source vault-env.sh NEW_PLATFORM_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.mongoDb') ``` #### Apply the new password ```bash mongosh "$MONGODB_URI" $TLS_FLAGS --quiet \ --eval "db.getSiblingDB('itential').updateUser('itential', { pwd: '$NEW_PLATFORM_PASS' })" ``` #### Restart Platform Restart Platform on each node so it reconnects with the new credential from your secrets manager. See [Manage the Platform service](/itential-platform/administer/platform-on-prem#manage-the-platform-service). #### Verify connectivity ```bash mongosh "mongodb://itential:$NEW_PLATFORM_PASS@$MONGO_PRIMARY:$MONGO_PORT/itential?authSource=itential&replicaSet=$MONGO_RS" \ $TLS_FLAGS --quiet --eval "db.adminCommand({ ping: 1 })" ``` ## Inspect logs MongoDB logs are written in structured JSON, with a severity field (`"s"`) where `"I"` is informational, `"W"` is warning, and `"E"` is error. The log file is at `/var/log/mongodb/mongod.log` by default; confirm against `systemLog.path` in `/etc/mongod.conf`. ### Search for errors and warnings ```bash for node in $MONGO_NODES; do echo "=== $node: recent errors and warnings ===" ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \ 'sudo grep -E "\"s\":\"E\"|\"s\":\"W\"|Slow query|NETWORK|REPL|assertion|INITSYNC" \ /var/log/mongodb/mongod.log | tail -30' 2>/dev/null done ``` | Pattern | Meaning | | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `"s":"E"` | Error; review immediately | | `"s":"W"` | Warning; review during the next maintenance window | | `REPL` | Replication event, including elections and state changes | | `assertion` | Internal assertion failure; may precede a crash | | `INITSYNC` | A member is performing initial sync to catch up; expected after a restart, concerning if unexpected on a long-running member | | `Slow query` | Query exceeded the slow query threshold | ### Search for a specific pattern ```bash for node in $MONGO_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \ "sudo grep -E '' /var/log/mongodb/mongod.log | tail -50" 2>/dev/null done ``` ### View recent log entries on a single node ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ \ 'sudo tail -100 /var/log/mongodb/mongod.log' ``` For a human-readable format, pipe through `jq` to extract key fields: ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ \ 'sudo tail -100 /var/log/mongodb/mongod.log' \ | jq -r '[.t["$date"], .s, .c, .msg] | @tsv' 2>/dev/null ``` ## Rotate TLS certificates An expired certificate breaks replication, interrupts Platform's database access, and can cause a primary election to fail if secondaries can't reach the primary. TLS certificate files are referenced in `/etc/mongod.conf` under `net.tls`: | Config key | Typical path | Contents | | ---------------------------- | ----------------------------- | ------------------------------------------------- | | `net.tls.certificateKeyFile` | `/etc/ssl/mongodb/mongod.pem` | Server certificate and private key (combined PEM) | | `net.tls.CAFile` | `/etc/ssl/mongodb/ca.pem` | CA bundle used to verify clients and peers | #### Check current certificate expiry Plan the rotation at least two weeks before the earliest expiry. ```bash for node in $MONGO_NODES; do ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node \ 'sudo openssl x509 -in /etc/ssl/mongodb/mongod.pem -noout -dates 2>/dev/null' done ``` #### Verify the new certificate before deploying ```bash openssl x509 -in new-mongod.pem -noout -text | grep -E "Subject:|Not Before:|Not After :" openssl verify -CAfile new-ca.pem new-mongod.pem openssl x509 -noout -modulus -in new-mongod.pem | openssl md5 openssl rsa -noout -modulus -in new-mongod.pem | openssl md5 ``` Don't proceed if `openssl verify` errors or the modulus hashes don't match. #### Copy new certificate files to each node ```bash for node in $MONGO_NODES; do scp -i $SSH_KEY_PATH new-mongod.pem new-ca.pem $MONGO_SSH_USER@$node:/tmp/ ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@$node ' sudo cp /tmp/new-mongod.pem /etc/ssl/mongodb/mongod.pem sudo cp /tmp/new-ca.pem /etc/ssl/mongodb/ca.pem sudo chown mongod:mongod /etc/ssl/mongodb/mongod.pem /etc/ssl/mongodb/ca.pem sudo chmod 400 /etc/ssl/mongodb/mongod.pem sudo chmod 444 /etc/ssl/mongodb/ca.pem ' done ``` Back up the previous files first (`mongod.pem.bak`, `ca.pem.bak`) so rollback is a one-file restore. #### Roll out the restart: secondaries first, primary last ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ 'sudo systemctl restart mongod' ``` Confirm it rejoined as `SECONDARY` before touching the next node: ```js rs.status().members.find(m => m.name.startsWith('')) ``` For the primary, step down first (see [Perform a manual failover](#perform-a-manual-failover)), then restart what is now a secondary: ```js rs.stepDown(60) ``` ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ 'sudo systemctl restart mongod' ``` #### Verify TLS connectivity ```bash mongosh "mongodb://admin:$MONGO_ADMIN_PASS@$MONGO_PRIMARY:$MONGO_PORT/?authSource=admin&replicaSet=$MONGO_RS" \ --tls --tlsCAFile /etc/ssl/mongodb/ca.pem --quiet \ --eval "rs.status().members.forEach(m => print(m.name, m.stateStr, m.health))" ``` All members should show `health: 1` and the expected `stateStr`. Update `TLS_FLAGS` in any local scripts to the new CA path. ### Rollback ```bash ssh -i $SSH_KEY_PATH $MONGO_SSH_USER@ ' sudo cp /etc/ssl/mongodb/mongod.pem.bak /etc/ssl/mongodb/mongod.pem sudo cp /etc/ssl/mongodb/ca.pem.bak /etc/ssl/mongodb/ca.pem sudo systemctl restart mongod ' ```