Administer etcd

An etcd cluster serves as the shared store backend for Itential Gateway when running distributed execution or active-standby high availability. Gateway requires etcd v3.5, the version these procedures target. Examples assume a three-node cluster, the smallest topology that tolerates a single-node failure while preserving write quorum.

Procedures that modify state, restart processes, or alter on-disk data are marked with a warning.

Before you begin

The commands here assume the following environment variables are set:

$
ETCD_NODES="<node1> <node2> <node3>"
$
ETCD_CLIENT_PORT=2379
$
ETCD_PEER_PORT=2380
$
ETCD_ENDPOINTS="https://<node1>:2379,https://<node2>:2379,https://<node3>:2379"
$
SSH_KEY_PATH=~/.ssh/<your-key>.pem
$
ETCD_SSH_USER=<ssh-user>
$
$
export ETCDCTL_API=3
$
ETCDCTL_TLS="--cacert=/etc/etcd/pki/ca.crt"

If TLS isn’t enabled in your deployment, set ETCDCTL_TLS="" and use http:// URLs in ETCD_ENDPOINTS. A non-TLS etcd deployment isn’t recommended for any environment that isn’t air-gapped.

ETCDCTL_TLS above assumes transport TLS only, on both the client and peer planes: etcd nodes present certificates but don’t require clients or peers to present client certificates back. In etcd terms, --cert-file / --key-file and --peer-cert-file / --peer-key-file are set, but neither --client-cert-auth nor --peer-client-cert-auth is enabled.

If your deployment uses full mTLS (--client-cert-auth=true), add --cert and --key flags to ETCDCTL_TLS, pointing at a client certificate and key trusted by the cluster’s CA. Reference: etcd v3.5 transport security. TLS private keys must never be stored in version control or distributed outside your secrets management process.

Deployment types

RPM/DEB and container deployments differ mainly in how the process is controlled. The underlying etcdctl and etcdutl commands are identical either way.

Confirm the actual paths against your deployment before running any command. etcd’s documented data directory default is ${name}.etcd relative to the working directory. /var/lib/etcd is a package convention, not an etcd default.

Check cluster health and status

etcd is a Raft-based consensus system. As long as a majority of members are healthy and reachable, the cluster accepts writes. Regular health checks confirm every member is reachable, the leader is stable, and no follower is lagging far enough behind to compromise availability if the leader fails.

Check cluster-wide health

etcdctl endpoint health contacts every endpoint, measures round-trip time, and reports whether each member would accept a write:

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint health --write-out=table

Every endpoint should report true under HEALTH with low TOOK latency. In v3.5, etcd also exposes HTTP /health, /livez, and /readyz endpoints at the address set by --listen-metrics-urls, useful for external monitoring. This guide uses etcdctl endpoint health for operator-facing checks. Reference: etcd v3.5 monitoring.

Check per-endpoint status

etcdctl endpoint status reveals which member holds the leader role, each member’s Raft term and index, and database size on disk:

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=table

The default --quota-backend-bytes is 0, which etcd interprets as a low space quota. Confirm the configured value before relying on absolute numbers; a DB SIZE approaching the quota is the most common cause of cluster read-only events.

List cluster members and the current leader

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  member list --write-out=table
$
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=json \
>
  | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint'

Each member-list row shows the member ID, name, peer URL, client URL, and whether it’s a learner (a non-voting replica being added). No member should be a learner in a steady-state cluster. Resolve the leader at runtime before any operation that will restart a node, so you can plan to handle it last.

Verify read and write availability

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS put /healthcheck/$(date +%s) ok
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS get /healthcheck/ --prefix --keys-only | tail -5

This exercises the full Raft commit path. Health-check keys accumulate, so clean them up periodically.

Manage the etcd service

In a 3-node cluster, restarting a follower is low-risk; quorum holds and the cluster keeps serving. Restarting the leader briefly pauses writes during the election that follows. Restarting two members at once breaks quorum.

Check status on all nodes

$
# RPM/DEB
$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl status etcd' 2>/dev/null
$
done
$
$
# Container
$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \
>
    'docker ps --filter name=etcd --format "table {{.Names}}\t{{.Status}}"' 2>/dev/null
$
done

An etcd process that’s crashed but hasn’t been restarted reduces the cluster to two members without raising an alarm at the application layer until the next Gateway read fails.

Restart on a single node

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl restart etcd'
$
$
# Container (adjust service name to match your compose file)
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'docker compose -f /path/to/docker-compose.yml restart etcd'

Verify the node rejoined and cluster-wide health, then confirm the rejoined member caught up to the leader’s Raft index before touching another member:

$
etcdctl --endpoints=https://<target-node>:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health
$
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint status --write-out=table

The RAFT INDEX of the rejoined member should be within a few hundred of the leader’s; a persistent gap warrants investigation in the etcd logs.

Start and stop on a single node

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl stop etcd'    # or: start
$
$
# Container
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'docker compose -f /path/to/docker-compose.yml stop etcd'   # or: start

Enable etcd to start on boot (RPM/DEB)

$
for node in $ETCD_NODES; do
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl is-enabled etcd' 2>/dev/null
$
done

Enable any node that returns disabled:

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl enable etcd'

A single offline member in a 3-node cluster leaves zero redundancy; the loss of any other member breaks quorum. For container deployments, configure the runtime’s restart policy (restart: always in Compose, or --restart=always for docker run) to achieve the same effect.

Review the configuration

etcd has no canonical default config-file path.

Inspect what the running process actually sees:

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'sudo cat /etc/etcd/etcd.conf.yml 2>/dev/null; sudo systemctl cat etcd'
$
$
# Container
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'docker inspect etcd | jq ".[0].Args, .[0].Config.Env"'

References: etcd v3.5 configuration, clustering.

Shut down etcd safely

$
# Confirm whether the target node is the leader
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=json \
>
  | jq -r '.[] | "\(.Endpoint) leader=\(.Status.leader == .Status.header.member_id)"'

$
NEW_LEADER_ID=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  member list --write-out=json \
>
  | jq -r '.members[] | select(.name != "<target-node-name>") | .ID' | head -1)
$
$
etcdctl --endpoints=https://<target-node>:$ETCD_CLIENT_PORT $ETCDCTL_TLS \
>
  move-leader $(printf '%x' $NEW_LEADER_ID)

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl stop etcd'
$
$
# Container
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'docker compose -f /path/to/docker-compose.yml stop etcd'

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl status etcd'
$
$
# Container
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'docker ps -a --filter name=etcd'

$
# RPM/DEB
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl start etcd'
$
$
# Container
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'docker compose -f /path/to/docker-compose.yml start etcd'

$
etcdctl --endpoints=https://<target-node>:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health
$
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint status --write-out=table

Back up and restore etcd

etcd is the source of truth for Gateway cluster coordination state. A corrupted data directory, a lost majority of members, or an inadvertent destructive operation against the keyspace can require restoring from a snapshot. With a current snapshot, recovery is typically a single multi-node procedure; without one, the recovery path is rebuilding Gateway cluster state from scratch.

References: etcd v3.5 disaster recovery, maintenance.

What to back up

A snapshot alone isn’t sufficient if a host is also lost; without the TLS material, a restored cluster can’t be reached by clients still configured against the old CA. Back up /etc/etcd/ (or the equivalent host-mounted volume) alongside each snapshot.

Take and verify a snapshot

etcdctl snapshot save produces a consistent snapshot without blocking writes. Target a single endpoint, not a comma-separated list; any healthy member works, though targeting the leader produces the most up-to-date snapshot.

$
SNAPSHOT_DIR=/var/backups/etcd
$
SNAPSHOT_FILE=$SNAPSHOT_DIR/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db
$
$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> "
>
  sudo mkdir -p $SNAPSHOT_DIR
>
  sudo ETCDCTL_API=3 etcdctl \
>
    --endpoints=https://<target-node>:$ETCD_CLIENT_PORT \
>
    --cacert=/etc/etcd/pki/ca.crt \
>
    snapshot save $SNAPSHOT_FILE
>
"

For container deployments, prefix the snapshot command with docker exec <etcd-container> and write the file inside the container’s volume so it lands on the host filesystem.

A snapshot that’s corrupt or truncated fails at restore time, so verify immediately after creation, while the healthy cluster is still available to compare against. Snapshot inspection lives in etcdutl, the offline data-management binary shipped alongside etcdctl:

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  "sudo etcdutl --write-out=table snapshot status $SNAPSHOT_FILE"

A non-zero hash and key count confirm the snapshot is intact. Compare its revision against the live cluster’s revision (from endpoint status) to confirm it captured recent state.

Copy the snapshot off the host and schedule regular runs

A snapshot stored only on the host that produced it doesn’t protect against host loss:

$
scp -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node>:$SNAPSHOT_FILE /<offsite-backup-location>/

Schedule a daily snapshot, verify it, copy it off-host, and prune local copies older than your retention window:

$
# /etc/cron.d/etcd-snapshot
$
0 2 * * * root /usr/local/bin/etcd-snapshot.sh >> /var/log/etcd-snapshot.log 2>&1

The wrapper script should run snapshot save, run snapshot status and exit non-zero on a zero hash or key count, copy the snapshot offsite and verify the copy, delete local snapshots past the retention window, and log success or failure. Pair the cron job with monitoring that alerts when the most recent successful snapshot exceeds your recovery point objective; a silently broken cron job is not a data protection strategy.

Restore from a snapshot

A restore is a four-phase process: stop etcd everywhere, restore the data directory on each member, start etcd, and verify the new cluster.

$
for node in $ETCD_NODES; do
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl stop etcd'
$
done

$
for node in $ETCD_NODES; do
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \
>
    "sudo mv /var/lib/etcd /var/lib/etcd.preRestore-$(date +%Y%m%d-%H%M%S)"
$
done

$
SNAPSHOT_FILE=/tmp/etcd-snapshot.db
$
INITIAL_CLUSTER="<node1>=https://<node1>:2380,<node2>=https://<node2>:2380,<node3>=https://<node3>:2380"
$
RESTORE_TOKEN="etcd-restored-$(date +%Y%m%d)"
$
$
for node in $ETCD_NODES; do
$
  scp -i $SSH_KEY_PATH /<offsite-backup-location>/etcd-snapshot.db \
>
    $ETCD_SSH_USER@$node:$SNAPSHOT_FILE
$
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node "
>
    sudo etcdutl snapshot restore $SNAPSHOT_FILE \
>
      --name=$node \
>
      --initial-cluster='$INITIAL_CLUSTER' \
>
      --initial-cluster-token=$RESTORE_TOKEN \
>
      --initial-advertise-peer-urls=https://$node:2380 \
>
      --data-dir=/var/lib/etcd
>
    sudo chown -R etcd:etcd /var/lib/etcd
>
  "
$
done

$
for node in $ETCD_NODES; do
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl start etcd'
$
done

$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS member list --write-out=table
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS get /<known-key-path> --prefix --keys-only | head

Compact and defragment the keyspace

etcd retains every revision of every key by default. For a Gateway store backend, that history is unused (Gateway reads current values, never historical ones), so accumulating revisions only grows disk usage. Without compaction, the database eventually hits --quota-backend-bytes and refuses writes. Compaction marks old revisions as free; defragmentation reclaims the freed space on disk. Both are needed.

Reference: etcd v3.5 maintenance.

Configure auto-compaction

Configure etcd to compact continuously, retaining one hour of revision history, a comfortable safety window for incident triage without letting the database bloat:

$
--auto-compaction-mode=periodic
$
--auto-compaction-retention=1h

Or in YAML:

1
auto-compaction-mode: periodic
2
auto-compaction-retention: "1h"

Restart each member one at a time (see Manage the etcd service) to pick up the change, then confirm the configured value loaded:

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'sudo journalctl -u etcd --no-pager | grep -i "auto-compaction" | tail -5'

Run manual compaction

If auto-compaction isn’t configured yet, or the database grew faster than the auto-compaction interval can keep up with:

$
CURRENT_REV=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=json | jq -r '.[0].Status.header.revision')
$
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS compact $CURRENT_REV

Schedule periodic defragmentation

Compaction frees revisions inside the bbolt backend file but doesn’t shrink it; defragmentation rewrites the backend in place to reclaim physical disk space. It pauses requests on the targeted member while it runs, so target one member at a time, preferring followers before the leader.

Schedule a weekly defrag during a low-activity window:

$
# /etc/cron.d/etcd-defrag
$
0 3 * * 0 root /usr/local/bin/etcd-defrag.sh >> /var/log/etcd-defrag.log 2>&1

$
#!/bin/bash
$
set -euo pipefail
$
$
ETCDCTL_API=3
$
ETCD_ENDPOINTS="https://<node1>:2379,https://<node2>:2379,https://<node3>:2379"
$
ETCDCTL_TLS="--cacert=/etc/etcd/pki/ca.crt"
$
$
LEADER=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=json \
>
  | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint')
$
$
for endpoint in $(echo $ETCD_ENDPOINTS | tr ',' ' '); do
$
  [ "$endpoint" = "$LEADER" ] && continue
$
  echo "=== defragmenting $endpoint ==="
$
  etcdctl --endpoints=$endpoint $ETCDCTL_TLS defrag
$
  sleep 30
$
done
$
$
echo "=== defragmenting leader $LEADER ==="
$
etcdctl --endpoints=$LEADER $ETCDCTL_TLS defrag

Confirm DB SIZE decreased on each member with endpoint status --write-out=table afterward.

Recover from a quota alarm

If etcd has already filled and refused writes, recovery is required even with auto-compaction configured: compact, defrag every member, then disarm the alarm.

$
CURRENT_REV=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \
>
  endpoint status --write-out=json | jq -r '.[0].Status.header.revision')
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS compact $CURRENT_REV
$
$
for node in $ETCD_NODES; do
$
  echo "=== $node: defragmenting ==="
$
  etcdctl --endpoints=https://$node:$ETCD_CLIENT_PORT $ETCDCTL_TLS defrag
$
done
$
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS alarm list
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS alarm disarm

Manage logs

The etcd log records leader elections, member additions and removals, slow disk warnings, and any error that prevents a write from committing, the first place to look when the cluster behaves unexpectedly.

Reference: etcd v3.5 configuration logging flags.

Log destinations

etcd v3.5 supports two log destinations, controlled by --log-outputs:

stdout and stderr are also valid --log-outputs values, typical in container deployments where the runtime captures the stream. Confirm the configured destination before relying on a specific path:

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> \
>
  'sudo systemctl cat etcd | grep -E "log-outputs|StandardOutput"'

View and search logs

$
# journald
$
journalctl -u etcd -n 100 --no-pager
$
journalctl -u etcd -p err -n 200 --no-pager
$
$
# File
$
sudo tail -n 100 /var/log/etcd/etcd.log
$
sudo grep -iE "ERROR|FATAL|PANIC" /var/log/etcd/etcd.log | tail -50

Search across all nodes for the events most worth flagging:

$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \
>
    'sudo journalctl -u etcd --no-pager \
>
       | grep -iE "elected leader|leader changed|lost leader|slow|took too long|database space exceeded|corrupt|panic|fatal" \
>
       | tail -30' 2>/dev/null
$
done

Configure log rotation

journald (default): rotation is controlled in /etc/systemd/journald.conf:

1
[Journal]
2
SystemMaxUse=2G
3
SystemKeepFree=1G
4
MaxRetentionSec=30day

Apply with sudo systemctl restart systemd-journald, and confirm with journalctl --disk-usage.

File output: etcd v3.5 has built-in rotation when enabled:

1
log-outputs: /var/log/etcd/etcd.log
2
enable-log-rotation: true
3
log-rotation-config-json: '{"maxsize": 100, "maxage": 7, "maxbackups": 14, "localtime": false, "compress": true}'

Restart each member one at a time to pick up the change. If your deployment can’t enable --enable-log-rotation (for example, a locked-down vendor wrapper), fall back to external logrotate with copytruncate, since etcd holds the file open:

/var/log/etcd/etcd.log {
    daily
    rotate 14
    size 100M
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    su etcd etcd
}

Save as /etc/logrotate.d/etcd on each node. This is the third-best option; prefer journald or --enable-log-rotation when possible.

Rotate TLS certificates

TLS certificates come in two planes:

This guide assumes transport TLS only on both planes. If your deployment uses full mTLS, the same rotation steps apply to the client and peer cert files plus their CA trust bundles.

Since v3.2, etcd reloads client-plane certificates on every new client connection: replacing cert-file and key-file on a member takes effect for the next client connection without a restart. The peer plane, where connections are long-lived between members, isn’t documented as hot-reloadable; treat peer-cert changes as requiring a rolling restart. When the CA changes, deploy a bundled CA file (old and new concatenated) so every member trusts either cert during the rollout; this is an operator pattern, not an etcd-documented one, but it’s the standard way to roll a CA without a window of mutual rejection.

Reference: etcd v3.5 transport security.

$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \
>
    'sudo systemctl cat etcd | grep -iE "cert-file|key-file|trusted-ca-file"' 2>/dev/null
$
done

$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node '
>
    for cert in /etc/etcd/pki/server.crt /etc/etcd/pki/peer.crt /etc/etcd/pki/ca.crt; do
>
      [ -f "$cert" ] || continue
>
      echo "--- $cert ---"
>
      sudo openssl x509 -in "$cert" -noout -dates -subject 2>/dev/null
>
    done
>
  '
$
done

$
for node in $ETCD_NODES; do
$
  echo "=== $node: served client certificate ==="
$
  echo | openssl s_client -servername $node -connect $node:$ETCD_CLIENT_PORT 2>/dev/null \
>
    | openssl x509 -noout -dates -subject
$
done

$
openssl x509 -in new-server.crt -noout -text \
>
  | grep -E "Subject:|Not Before:|Not After :|DNS:|IP Address:"
$
openssl x509 -in new-peer.crt -noout -text \
>
  | grep -E "Subject:|Not Before:|Not After :|DNS:|IP Address:"
$
$
openssl verify -CAfile new-ca.crt new-server.crt
$
openssl verify -CAfile new-ca.crt new-peer.crt
$
$
openssl x509 -noout -modulus -in new-server.crt | openssl md5
$
openssl rsa  -noout -modulus -in new-server.key | openssl md5
$
openssl x509 -noout -modulus -in new-peer.crt | openssl md5
$
openssl rsa  -noout -modulus -in new-peer.key | openssl md5

$
for node in $ETCD_NODES; do
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node '
>
    sudo cp /etc/etcd/pki/server.crt /etc/etcd/pki/server.crt.bak
>
    sudo cp /etc/etcd/pki/server.key /etc/etcd/pki/server.key.bak
>
    sudo cp /etc/etcd/pki/peer.crt   /etc/etcd/pki/peer.crt.bak
>
    sudo cp /etc/etcd/pki/peer.key   /etc/etcd/pki/peer.key.bak
>
    sudo cp /etc/etcd/pki/ca.crt     /etc/etcd/pki/ca.crt.bak
>
  '
$
done

$
for node in $ETCD_NODES; do
$
  scp -i $SSH_KEY_PATH new-server.crt new-server.key new-peer.crt new-peer.key new-ca.crt \
>
    $ETCD_SSH_USER@$node:/tmp/
$
$
  ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node '
>
    sudo cp /tmp/new-server.crt /etc/etcd/pki/server.crt
>
    sudo cp /tmp/new-server.key /etc/etcd/pki/server.key
>
    sudo cp /tmp/new-peer.crt   /etc/etcd/pki/peer.crt
>
    sudo cp /tmp/new-peer.key   /etc/etcd/pki/peer.key
>
    sudo cp /tmp/new-ca.crt     /etc/etcd/pki/ca.crt
>
    sudo chown etcd:etcd /etc/etcd/pki/*.crt /etc/etcd/pki/*.key
>
    sudo chmod 400       /etc/etcd/pki/*.key
>
    sudo chmod 444       /etc/etcd/pki/*.crt
>
  '
$
done

$
for node in $ETCD_NODES; do
$
  echo "=== $node ==="
$
  echo | openssl s_client -servername $node -connect $node:$ETCD_CLIENT_PORT 2>/dev/null \
>
    | openssl x509 -noout -dates -subject
$
done

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> 'sudo systemctl restart etcd'
$
$
echo | openssl s_client -servername <target-node> -connect <target-node>:$ETCD_CLIENT_PORT 2>/dev/null \
>
  | openssl x509 -noout -dates -subject
$
$
etcdctl --endpoints=https://<target-node>:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health
$
etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table

Rollback

$
ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@<target-node> '
>
  sudo cp /etc/etcd/pki/server.crt.bak /etc/etcd/pki/server.crt
>
  sudo cp /etc/etcd/pki/server.key.bak /etc/etcd/pki/server.key
>
  sudo cp /etc/etcd/pki/peer.crt.bak   /etc/etcd/pki/peer.crt
>
  sudo cp /etc/etcd/pki/peer.key.bak   /etc/etcd/pki/peer.key
>
  sudo cp /etc/etcd/pki/ca.crt.bak     /etc/etcd/pki/ca.crt
>
  sudo systemctl restart etcd
>
'

If the cluster fully fragmented because rotation was attempted on multiple nodes simultaneously, restore from the snapshot taken before the rotation using Restore from a snapshot.

Related resources

• etcd v3.5 op-guide
• Configure etcd shared database