Administer Itential Gateway

Itential Gateway executes Ansible playbooks, Python scripts, and OpenTofu plans on behalf of Itential Platform, retrieving content from Git and building isolated runtime environments at execution time. These procedures assume a Linux deployment installed using the Itential Deployer RPM/DEB package, which provides the iagctl binary and the iagctl systemd service.

Procedures that modify state, restart services, or alter on-disk data are marked with a warning.

Before you begin

The commands here assume the following environment variables are set:

$ GATEWAY_SERVER_NODES="<server1> <server2> <server3>"   # Empty in single-node deployments
$ GATEWAY_RUNNER_NODES="<runner1> <runner2>"             # Empty if no separate runners
$ GATEWAY_NODES="$GATEWAY_SERVER_NODES $GATEWAY_RUNNER_NODES"
$ GATEWAY_ACTIVE_SERVER=<active-server-hostname>
$ SSH_KEY_PATH=~/.ssh/<your-key>.pem
$ GATEWAY_SSH_USER=<ssh-user>

Most procedures here are host-side such as SSH plus systemctl or filesystem operations, rather than calls to a remote API. Gateway’s control plane is gRPC on port 50051, accessed through the iagctl CLI; there’s no REST health endpoint. Where a procedure runs iagctl commands, run them from a node that’s logged in (see iagctl login).

Default file system layout for an RPM/DEB install:

Path	Purpose
`/etc/gateway/gateway.conf`	Configuration file (INI-style sections)
`/etc/gateway/certificates/`	Default certificate location
`/var/log/gateway/gateway.log`	Default log file (server/runner mode)
`/var/lib/gateway/`	`runtime_data_dir`, cached virtual environments built per execution

The Gateway service runs as the itential user. Add yourself to the itential group during installation if you need read access to certificate and configuration files.

Passwords, API keys, and TLS private keys must never be stored in scripts, configuration files, or shell history. Retrieve them from your secrets manager at runtime.

Topologies covered

Gateway supports several deployment models; the procedures here apply to all of them, and topology-specific differences are called out inline.

Topology	Composition
All-in-one single-node	One gateway server (controller node) handling both management and execution.
All-in-one active-standby	Multiple gateway servers sharing an etcd or DynamoDB store backend. One server is active and holds the connection to Gateway Manager; the rest are standby. All nodes perform both management and execution.
Distributed service execution	One or more gateway servers plus dedicated runner nodes, sharing an etcd or DynamoDB store backend. Servers manage; runners execute.
HA with distributed execution	Active-standby gateway servers plus dedicated runner nodes.

See Choose a deployment architecture for the full decision framework, including multi-cluster deployments.

Where a step depends on topology, look for a Topology notes block. In an all-in-one single-node deployment, treat the single node as both server and runner wherever the two are distinguished.

Check health and status

A degraded gateway node can leave Ansible playbooks half-applied, OpenTofu state locked, or scripts hung against managed devices. Check health before and after any maintenance activity.

Gateway 5.4+ iagctl inspect cluster is the fastest way to check both health and active execution together; see Monitor gateway cluster health and activity for full command reference and output fields.

$ iagctl inspect cluster

On earlier versions, or for host-level detail, use the signals below.

Check service status (all nodes)

$ for node in $GATEWAY_NODES; do
$   echo "=== $node ==="
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node 'sudo systemctl status iagctl' 2>/dev/null
$ done

The Active: line on every node should read active (running).

Check Gateway Manager connectivity

A server node that’s lost its connection to Gateway Manager won’t receive task requests from Platform. Look for connected to gateway manager at <host:port> in the journal:

$ for node in $GATEWAY_SERVER_NODES; do
$   echo "=== $node: Gateway Manager connection ==="
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node \
>     'sudo journalctl -u iagctl --no-pager \
>        | grep -E "connected to gateway manager|disconnected from gateway manager" \
>        | tail -5' 2>/dev/null
$ done

In an active/standby deployment, only the active server holds the connection at any time. Standby servers log not the active node until they win an election; a standby with no recent connection messages is normal.

Check the version and registered runners

$ # Version (and effective configuration)
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'iagctl version --show-config'

In a distributed-execution topology, run iagctl get runners from a logged-in client to confirm every expected runner is registered with the store backend. A runner that fails to register won’t receive work even though its iagctl service appears healthy.

$ iagctl get runners

Topology notes: returns no runners in an all-in-one deployment (skip it). In distributed execution (with or without active/standby), run this after every service restart that affects a runner.

Manage the Gateway service

The order in which nodes are restarted matters and depends on topology. Restarting a runner only interrupts whatever execution is currently on it. Restarting a server interrupts every execution on that server in an all-in-one topology. Restarting the active server in an active/standby cluster forces a leadership election, which briefly pauses the Gateway Manager connection.

Restart on a single node

Restarting iagctl terminates any in-flight execution on that node. Use Check health and status to confirm the node is idle first, and plan restarts during maintenance windows when possible.

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl restart iagctl'

Verify the service came back up, and for server nodes, that the Gateway Manager connection re-established (active node only):

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl status iagctl'
$ 
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> \
>   'sudo journalctl -u iagctl --since "2 minutes ago" \
>      | grep -E "connected to gateway manager|elected as the leader"'

Topology-specific restart ordering

When restarting more than one node, for example after a cluster-wide configuration change, follow the order below. The general principle is restart standbys first, active last, so the active server keeps Gateway Manager connectivity for as long as possible.

Topology	Restart order
All-in-one single-node	The one node
All-in-one active/standby	Standby servers first, one at a time; active server last
Distributed execution (no active/standby)	Runners first, one at a time; server last
Distributed execution + active/standby	Runners first, then standby servers, then the active server

Verify health after each node before moving to the next. Don’t parallelize restarts.

Start and stop on a single node

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl stop iagctl'   # or: start

Enable iagctl to start on boot

$ for node in $GATEWAY_NODES; do
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node 'sudo systemctl is-enabled iagctl' 2>/dev/null
$ done

Enable any node that returns disabled:

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl enable iagctl'

A node that reboots without this enabled stays offline until someone manually starts it, reducing redundancy in an active/standby cluster or execution capacity in a distributed one.

Review the configuration file

/etc/gateway/gateway.conf is the canonical configuration source. Environment variables of the form GATEWAY_* override values from the file. iagctl version --show-config shows the effective configuration after those overrides are applied.

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo cat /etc/gateway/gateway.conf'

Section	Purpose
`[application]`	Mode, cluster ID, working directory, CA certificate file
`[server]`	Listen address/port, TLS, runtime data dir, distributed execution flag
`[runner]`	Listen address/port, TLS, runtime data dir, announcement address
`[connect]`	Gateway Manager connection (server nodes); active/standby flags
`[store]`	Store backend (`local`, `memory`, `etcd`, `dynamodb`)
`[log]`	Log level, file enabled/disabled, server log directory
`[secrets]`	Encryption key file location

Shut down Gateway safely

As of Gateway 5, there’s no drain mechanism to stop accepting new execution requests on a node while letting in-flight work finish. Stopping iagctl terminates whatever automation is currently running on that node. In production, stop nodes only during a maintenance window, or use iagctl inspect cluster activity (below) to confirm the node is idle first.

A safe shutdown is a sequencing exercise: check activity, confirm the rest of the cluster can absorb the loss of the target node, then stop it at the right point in the topology order.

Check for active work on the target node

Gateway 5.4+

$ iagctl inspect cluster activity

Confirm no running executions are assigned to the node you intend to stop. On earlier versions, check for orphan processes instead (Step 4 below covers this after shutdown; there’s no pre-shutdown equivalent without this command).

Confirm cluster health

Every node other than the target should be active (running) under Check health and status. If any other node is unhealthy, repair it before continuing; stopping a server while another is already down may leave the cluster without quorum on its store backend or without an active node to hold the Gateway Manager connection.

Identify the active server (active/standby only)

If you intend to stop the active server, plan to do it last and expect a brief Gateway Manager disconnection while a standby takes over.

$ for node in $GATEWAY_SERVER_NODES; do
$   echo "=== $node ==="
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node \
>     'sudo journalctl -u iagctl --no-pager \
>        | grep -E "connected to gateway manager|not the active node|elected as the leader" \
>        | tail -3' 2>/dev/null
$ done

Notify users and stop the service

Anything currently executing on the target node is killed mid-step, which can leave managed systems partially configured for Ansible playbooks and OpenTofu plans. Notify users before proceeding.

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl stop iagctl'

iagctl handles SIGTERM and logs got signal for shutdown.... terminated.

Verify shutdown and check for orphans

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl status iagctl'
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'ps -u itential -f'

The Active: line should show inactive (dead). Any ansible-playbook, python, or tofu processes still listed are leftovers from killed executions; review and terminate them before continuing.

Verify the rest of the cluster is still healthy

Re-run Check health and status: every other node still running, every remaining runner still registered (distributed execution), and, if you stopped the active server, a standby has taken over (elected as the leader in its journal). If any other node became unhealthy as a side effect, restart the target node immediately and investigate.

Manage logs

References: Logging, Monitoring.

Gateway writes a single log file. When running as a server or runner, that’s <GATEWAY_LOG_SERVER_DIR>/gateway.log, defaulting to /var/log/gateway/gateway.log. systemd also captures stdout/stderr in the journal; use it as a secondary source when file logs don’t contain enough detail, such as during a startup failure before file logging initializes.

$ sudo tail -n 100 /var/log/gateway/gateway.log
$ sudo grep -iE "ERR|FATAL" /var/log/gateway/gateway.log | tail -50
$ 
$ journalctl -u iagctl -n 100 --no-pager
$ journalctl -u iagctl -p err -n 200 --no-pager

See Logging for log levels and JSON format.

Adjust the log level

Gateway supports the following log levels from least to most verbose: DISABLED, FATAL, ERROR, WARN, INFO, DEBUG, TRACE.

The production default is INFO. The DEBUG and TRACE levels produce significant log volume; don’t leave them enabled after completing a troubleshooting session.

Set the level in /etc/gateway/gateway.conf under [log]:

1 [log]
2 level = INFO

Or via the GATEWAY_LOG_LEVEL environment variable. Restart iagctl after changing either so the new value takes effect.

Configure log rotation

Gateway doesn’t rotate gateway.log internally; that’s the operator’s responsibility, typically with the host’s logrotate package. Without a policy, the file grows without bound and can eventually consume all available disk space.

Create a logrotate policy

$ sudo tee /etc/logrotate.d/iagctl <<'EOF'
$ /var/log/gateway/gateway.log {
>     daily
>     rotate 14
>     size 100M
>     compress
>     delaycompress
>     missingok
>     notifempty
>     copytruncate
>     su itential itential
> }
$ EOF

Directive	Purpose
`daily`	Rotate at least once per day
`rotate 14`	Keep 14 rotated files before deleting the oldest
`size 100M`	Force rotation if the file exceeds 100 MB regardless of date
`compress` / `delaycompress`	gzip rotated files; defer compression a cycle so a tail-following session doesn’t lose track
`copytruncate`	Truncate the active log in place rather than renaming it

copytruncate matters here because iagctl holds an open file descriptor on gateway.log; a rename-based rotation without it would leave Gateway writing to the renamed file until the next restart.

Test the policy

$ sudo logrotate -d /etc/logrotate.d/iagctl

Review the output for errors and confirm the listed path matches gateway.log; nothing is rotated in debug mode.

Force a rotation if disk space is already tight

Only do this if the log has already grown large enough to threaten disk capacity.

$ sudo logrotate -f /etc/logrotate.d/iagctl
$ sudo ls -lh /var/log/gateway/

A newly created gateway.log.1 (or .gz) should appear alongside a freshly truncated gateway.log.

Verify disk space

$ df -h /var/log

If disk usage remains high after rotation, inspect the log directory for old files outside the rotation policy, or for journal files; the systemd journal is rotated separately, see journalctl --disk-usage and /etc/systemd/journald.conf.

Deploy to every node

logrotate policies are local to each host.

$ for node in $GATEWAY_NODES; do
$   scp -i $SSH_KEY_PATH /etc/logrotate.d/iagctl $GATEWAY_SSH_USER@$node:/tmp/iagctl-logrotate
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node '
>     sudo mv /tmp/iagctl-logrotate /etc/logrotate.d/iagctl
>     sudo chown root:root /etc/logrotate.d/iagctl
>     sudo chmod 644       /etc/logrotate.d/iagctl
>   '
$ done

Clean up Python virtual environments

When Gateway executes a Python or Ansible service, it builds an isolated runtime environment from the dependencies declared in the service’s requirements file (requirements.txt, pyproject.toml, or requirements.yml). The environment is cached under runtime_data_dir (/var/lib/gateway by default) and rebuilt only when requirements change. The cache accumulates over time as services are added, removed, or have dependencies updated.

References: GATEWAY_SERVER_RUNTIME_DATA_DIR, GATEWAY_RUNNER_RUNTIME_DATA_DIR.

Environments live on whichever node performs execution: runners in a distributed-execution topology, servers otherwise.

$ EXEC_NODES="$GATEWAY_RUNNER_NODES"
$ [ -z "$EXEC_NODES" ] && EXEC_NODES="$GATEWAY_SERVER_NODES"
$ 
$ for node in $EXEC_NODES; do
$   echo "=== $node ==="
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node \
>     'sudo du -sh /var/lib/gateway 2>/dev/null; sudo ls -la /var/lib/gateway 2>/dev/null' 2>/dev/null
$ done

Clean the environment cache

Cleaning the cache forces Gateway to rebuild every environment from scratch on the next execution that needs it, so the first execution after cleanup is slower. The procedure must run with the service stopped; deleting cache contents while an execution is in progress can corrupt it.

Topology notes: in an all-in-one deployment, the whole node (and, in single-node, the whole cluster) is unavailable during the procedure. In distributed execution, run one runner at a time; servers are unaffected and the cluster keeps serving with reduced execution capacity.

Stop iagctl

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl stop iagctl'

Confirm no leftover execution processes

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'ps -u itential -f'

Terminate any remaining ansible-playbook, python, or tofu processes before proceeding so they don’t hold open files in the cache directory.

Confirm the runtime data directory

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> \
>   'sudo grep -E "runtime_data_dir|working_dir" /etc/gateway/gateway.conf'

Substitute the configured path for /var/lib/gateway below if it’s non-default.

Delete the contents of the runtime data directory

Delete the contents, not the directory itself, so ownership and mode survive.

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> '
>   if [ -d /var/lib/gateway ]; then
>     sudo find /var/lib/gateway -mindepth 1 -delete
>   fi
> '

Restart iagctl and verify

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl start iagctl'
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl status iagctl'

For a runner, confirm it re-registers:

$ iagctl get runners

The first execution dispatched to it is slower than usual while its environment rebuilds.

Move to the next node in topology-required order; don’t parallelize.

Rotate TLS certificates

Gateway uses mutual TLS (mTLS) across three communication planes:

Plane	Endpoints	Default cert variables
Gateway Manager	Server node ↔ Gateway Manager / Platform	`GATEWAY_CONNECT_CERTIFICATE_FILE`, `GATEWAY_CONNECT_PRIVATE_KEY_FILE`
Client-Server	`iagctl` clients ↔ server node	`GATEWAY_SERVER_CERTIFICATE_FILE` / `GATEWAY_CLIENT_CERTIFICATE_FILE` (and matching key files)
Server-Runner	Server node ↔ runner node	`GATEWAY_RUNNER_CERTIFICATE_FILE`, `GATEWAY_RUNNER_PRIVATE_KEY_FILE`

A single CA certificate is shared across all three planes via GATEWAY_APPLICATION_CA_CERTIFICATE_FILE, under /etc/gateway/certificates/ by default. References: Create certificates, Manage certificates (Gateway Manager UI), iagctl cert-gen.

iagctl reads certificate files at startup; there’s no reload command, so a restart is required for new files to take effect. Restart standby nodes first and the active node last.

Inspect current certificate configuration

$ for node in $GATEWAY_NODES; do
$   echo "=== $node ==="
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node \
>     'sudo grep -E "certificate_file|private_key_file|ca_certificate_file" \
>      /etc/gateway/gateway.conf' 2>/dev/null
$ done

Check current expiry

Plan the rotation at least two weeks before the earliest expiry.

$ for node in $GATEWAY_NODES; do
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node '
>     for cert in /etc/gateway/certificates/*.pem; do
>       [ -f "$cert" ] || continue
>       echo "--- $cert ---"
>       sudo openssl x509 -in "$cert" -noout -dates -subject 2>/dev/null
>     done
>   '
$ done

For the Gateway Manager certificate, cross-check the expiry shown in the Gateway Manager app’s Certificates tab in the Platform UI against gw-manager.pem on the active server.

Verify new certificates locally before deploying

$ openssl x509 -in new-server.pem -noout -text \
>   | grep -E "Subject:|Not Before:|Not After :|DNS:|IP Address:"
$ openssl verify -CAfile new-ca.pem new-server.pem
$ openssl x509 -noout -modulus -in new-server.pem | openssl md5
$ openssl rsa  -noout -modulus -in new-server-key.pem | openssl md5

Repeat for the runner and Gateway Manager certificates. Confirm the SANs on server and runner certificates include every hostname or IP that other nodes use to reach this node; a missing SAN causes those clients to reject the connection. Don’t proceed if openssl verify errors or any modulus pair doesn’t match.

Back up existing certificate files

$ for node in $GATEWAY_NODES; do
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node '
>     cd /etc/gateway/certificates && \
>     sudo bash -c "for f in *.pem; do cp \"\$f\" \"\$f.bak\"; done"
>   '
$ done

Distribute new certificate files

Copy new files to all affected nodes before restarting any service. If the CA changed, deploy a bundled CA file (old and new CA certificates concatenated) so every node trusts either during the rolling restart; replace it with a new-CA-only file in a follow-up rotation once every node is rotated.

$ for node in $GATEWAY_NODES; do
$   scp -i $SSH_KEY_PATH new-ca.pem new-server.pem new-server-key.pem $GATEWAY_SSH_USER@$node:/tmp/
$ 
$   ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@$node '
>     sudo cp /tmp/new-ca.pem         /etc/gateway/certificates/ca.pem
>     sudo cp /tmp/new-server.pem     /etc/gateway/certificates/server.pem
>     sudo cp /tmp/new-server-key.pem /etc/gateway/certificates/server-key.pem
>     sudo chown itential:itential /etc/gateway/certificates/*.pem
>     sudo chmod 400 /etc/gateway/certificates/*-key.pem
>     sudo chmod 444 /etc/gateway/certificates/ca.pem /etc/gateway/certificates/server.pem
>     sudo rm /tmp/new-ca.pem /tmp/new-server.pem /tmp/new-server-key.pem
>   '
$ done

Repeat with new-runner.pem / new-runner-key.pem against $GATEWAY_RUNNER_NODES, and new-gw-manager.pem / new-gw-manager-key.pem against $GATEWAY_SERVER_NODES.

Upload the rotated Gateway Manager certificate

If the certificate a server presents to Gateway Manager changed, upload the new certificate in the Platform UI, or the active server can’t register.

Open the Gateway Manager app in Itential Platform and go to the Certificates tab.
Click Upload Certificate and select the new gw-manager.pem (the public certificate, not the private key).
Click Add Certificate.

Roll out the restart

Follow Topology-specific restart ordering: runners first, then standby servers, then the active server. Verify health between every restart.

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl restart iagctl'
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl status iagctl'
$ 
$ echo | openssl s_client -connect <target-node>:50051 2>/dev/null \
>   | openssl x509 -noout -dates -subject

Confirm notAfter matches the new certificate. For server nodes, also confirm Gateway Manager reconnected (active only); for runners, confirm re-registration with iagctl get runners. Don’t restart the next node until this one reports healthy.

Update references to the old CA path

Update any local scripts, environment files, or monitoring configuration that pin the old CA path or trust bundle.

Rollback

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> '
>   cd /etc/gateway/certificates && \
>   sudo bash -c "for f in *.pem.bak; do cp \"\$f\" \"\${f%.bak}\"; done"
>   sudo systemctl restart iagctl
> '

Verify the node comes back up before treating the incident as resolved:

$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> 'sudo systemctl status iagctl'
$ 
$ ssh -i $SSH_KEY_PATH $GATEWAY_SSH_USER@<target-node> \
>   'sudo journalctl -u iagctl --since "2 minutes ago" | tail -30'

If the Gateway Manager certificate was already replaced in the Platform UI but is being rolled back on the server, re-upload the old gw-manager.pem to Gateway Manager so both ends agree again.

For architecture and deployment models, see Choose a deployment architecture.
For day-to-day cluster health and activity monitoring, see Monitor gateway cluster health and activity.
For log levels and format, see Logging and Monitoring.