Gateway cluster TLS

This topic demonstrates how to establish secure communication between the different nodes in your gateway cluster.

For more information on node types and the deployment models that use them, see Architecture & Deployment Models.

Communication scenarios

Itential Automation Gateway IAG supports the following communication scenarios:

Client to server

Figure 1: Client to server communication

A client node can send requests to a gateway server node to manage database resources and run services.

Server to runner

Figure 2: Server to runner node communication

A gateway server node configured for distributed execution can send service execution requests to runner nodes within the cluster.

Required certificates

Mutual TLS (mTLS) requires the following certificates on each node within the cluster for secure node-to-node communication:

A Certificate Authority (CA) certificate file
A public key signed by the CA
A private key signed by the CA

You can share the CA file among all nodes within the cluster or use multiple CA files signed by the same authority.

For all application modes (client, server, and runner), you can set the CA file's location with the configuration variable GATEWAY_APPLICATION_CA_CERTIFICATE_FILE.

The configuration variables for the public key and private key files are separated by application mode. You can use the following variables to set the public and private certificate files per application mode:

Client

GATEWAY_CLIENT_CERTIFICATE_FILE
GATEWAY_CLIENT_PRIVATE_KEY_FILE

Server

GATEWAY_SERVER_CERTIFICATE_FILE
GATEWAY_SERVER_PRIVATE_KEY_FILE

Runner

GATEWAY_RUNNER_CERTIFICATE_FILE
GATEWAY_RUNNER_PRIVATE_KEY_FILE

Troubleshoot TLS connections

If you experience issues while setting up your TLS certificates, you can enable additional logs to help identify and troubleshoot problems:

GRPC_GO_LOG_SEVERITY_LEVEL=info
GRPC_GO_LOG_VERBOSITY_LEVEL=99

These logs provide additional information about the connections being formed.

Disable TLS

Disabling TLS can be helpful when you first configure your cluster architecture to ensure everything works before placing certificates on your gateway nodes. You can use the following configuration variables to disable TLS depending on the application mode you're using:

GATEWAY_CLIENT_USE_TLS=false
GATEWAY_SERVER_USE_TLS=false
GATEWAY_RUNNER_USE_TLS=false

Important

Itential strongly recommends enabling TLS in production environments.

Example certificate generation

The following walkthrough provides an example certificate generation scenario to help you better understand how to configure mTLS within a gateway cluster. It uses a Docker Compose file that launches a gateway cluster with five containers.

Important

This example is not designed for production environments.

The following example uses the iagctl cert-gen command. IAG provides the cert-gen command for demonstration purposes to help new users familiarize themselves with the certificate generation process and should not be used for production.

This example sets up a gateway cluster with the following architecture:
Figure 3: Example architecture with two runner nodes

Create certificates

Create a directory called /etc/gateway/certificates and set your current user to own this directory:

sudo mkdir -p /etc/gateway/certificates
sudo chown -R $USER /etc/gateway/certificates

Identify the IP addresses and/or domain names for communication between nodes. You must include this information when generating certificates. The following table shows the IP addresses for each node in this example:

Service	Container Name	IP Address	Description
server	gateway-server	172.30.0.2	Gateway server node, handles client connections and manages runners
runner1	gateway-runner-1	172.30.0.3	First gateway runner node
runner2	gateway-runner-2	172.30.0.4	Second gateway runner node
etcd	etcd	172.30.0.5	etcd instance for distributed key-value store

Create the CA certificate to sign all your certificates:

iagctl cert-gen ca --output /etc/gateway/certificates/
2024/09/11 09:57:21 [INFO] generate received request
2024/09/11 09:57:21 [INFO] received CSR
2024/09/11 09:57:21 [INFO] generating key: rsa-2048
2024/09/11 09:57:21 [INFO] encoded CSR
2024/09/11 09:57:21 [INFO] signed certificate with serial number 429864311756949186721815656348929749031883809299
CA generated successfully

Create the server certificates:

iagctl cert-gen server --output /etc/gateway/certificates --cn dev.example.com --name dev --sans "172.30.0.2,127.0.0.1"
2024/09/11 10:09:19 [INFO] generate received request
2024/09/11 10:09:19 [INFO] received CSR
2024/09/11 10:09:19 [INFO] generating key: rsa-2048
2024/09/11 10:09:19 [INFO] encoded CSR
2024/09/11 10:09:19 [INFO] signed certificate with serial number 253141293548862543650208380282680553664590675152
Server certificate generated successfully

Create the client certificates:

iagctl cert-gen client --output /etc/gateway/certificates/ --cn client@gateway.dev --name myclient
2024/09/11 09:58:13 [INFO] generate received request
2024/09/11 09:58:13 [INFO] received CSR
2024/09/11 09:58:13 [INFO] generating key: rsa-2048
2024/09/11 09:58:13 [INFO] encoded CSR
2024/09/11 09:58:13 [INFO] signed certificate with serial number 281114420220717386094877146752125293500708987966
Client certificate generated successfully

Create the certificates for each runner node:

Runner 1:

iagctl cert-gen runner --output /etc/gateway/certificates/ --cn runner1.example.com --name one --sans "172.30.0.3"
2024/09/11 09:59:48 [INFO] generate received request
2024/09/11 09:59:48 [INFO] received CSR
2024/09/11 09:59:48 [INFO] generating key: rsa-2048
2024/09/11 09:59:49 [INFO] encoded CSR
2024/09/11 09:59:49 [INFO] signed certificate with serial number 168253465288282904579572524845520301857724917505
Runner certificate generated successfully

Runner 2:

iagctl cert-gen runner --output /etc/gateway/certificates/ --cn runner2.example.com --name two --sans "172.30.0.4"
2024/09/11 10:00:04 [INFO] generate received request
2024/09/11 10:00:04 [INFO] received CSR
2024/09/11 10:00:04 [INFO] generating key: rsa-2048
2024/09/11 10:00:04 [INFO] encoded CSR
2024/09/11 10:00:04 [INFO] signed certificate with serial number 168753766732205191835823091463389722061299575454
Runner certificate generated successfully

Verify the certificates were created. List the directory contents, excluding the .json files that iagctl cert-gen uses:

ls /etc/gateway/certificates | grep -v json
ca-key.pem
ca.pem
dev-server-key.pem
dev-server.pem
myclient-client-key.pem
myclient-client.pem
one-runner-key.pem
one-runner.pem
three-runner-key.pem
three-runner.pem
two-runner-key.pem
two-runner.pem

Your certificates are now created and ready to be loaded. Each node type has its own certificate and private key for communication within the cluster. To enable proper trust verification, you'll also load the CA at the application level.

Configure environment variables

Set the following environment variables for each node:

Server

GATEWAY_APPLICATION_CA_CERTIFICATE_FILE=/etc/gateway/certificates/ca.pem
GATEWAY_SERVER_CERTIFICATE_FILE=/etc/gateway/certificates/dev-server.pem
GATEWAY_SERVER_PRIVATE_KEY_FILE=/etc/gateway/certificates/dev-server-key.pem

Client

GATEWAY_APPLICATION_CA_CERTIFICATE_FILE=/etc/gateway/certificates/ca.pem
GATEWAY_CLIENT_CERTIFICATE_FILE=/etc/gateway/certificates/myclient-client.pem
GATEWAY_CLIENT_PRIVATE_KEY_FILE=/etc/gateway/certificates/myclient-client-key.pem

Runner 1

GATEWAY_APPLICATION_CA_CERTIFICATE_FILE=/etc/gateway/certificates/ca.pem
GATEWAY_RUNNER_CERTIFICATE_FILE=/etc/gateway/certificates/one-runner.pem
GATEWAY_RUNNER_PRIVATE_KEY_FILE=/etc/gateway/certificates/one-runner-key.pem

Runner 2

GATEWAY_APPLICATION_CA_CERTIFICATE_FILE=/etc/gateway/certificates/ca.pem
GATEWAY_RUNNER_CERTIFICATE_FILE=/etc/gateway/certificates/two-runner.pem
GATEWAY_RUNNER_PRIVATE_KEY_FILE=/etc/gateway/certificates/two-runner-key.pem

Deploy the example cluster with Docker Compose

After setting up your certificates in /etc/gateway/certificates, create a Docker Compose file named docker-compose.yml.

Copy the following example configuration file to your docker-compose.yml file.

Note

This example disables TLS for the example etcd server to limit the scope. We recommend enabling TLS for etcd servers in production environments.

networks:
  my_network:
    ipam:
      config:
        - subnet: 172.30.0.0/16

services:
  server:
    container_name: gateway-server
    image: 497639811223.dkr.ecr.us-east-2.amazonaws.com/automation-gateway5:5.1.0-amd64
    entrypoint:
        - /usr/local/bin/iagctl
        - server
    environment:
      GATEWAY_CONNECT_ENABLED: "false"
      GATEWAY_APPLICATION_CA_CERTIFICATE_FILE: "/etc/gateway/certificates/ca.pem"
      GATEWAY_SERVER_CERTIFICATE_FILE: "/etc/gateway/certificates/dev-server.pem"
      GATEWAY_SERVER_PRIVATE_KEY_FILE: "/etc/gateway/certificates/dev-server-key.pem"
      GATEWAY_SERVER_USE_TLS: "true"
      GATEWAY_STORE_BACKEND: "etcd"
      GATEWAY_APPLICATION_MODE: "server"
      GATEWAY_SERVER_LISTEN_ADDRESS: "0.0.0.0"
      GATEWAY_SERVER_DISTRIBUTED_EXECUTION: "true"
      GATEWAY_STORE_ETCD_HOSTS: "etcd-1:2379"
      GATEWAY_STORE_ETCD_USE_TLS: "false"
      GATEWAY_LOG_LEVEL: "DEBUG"
    networks:
      my_network:
        ipv4_address: 172.30.0.2
    ports:
      - "127.0.0.1:50051:50051"
      - "127.0.0.1:8090:8090"
    volumes:
      - /etc/gateway/certificates:/etc/gateway/certificates
    depends_on:
      etcd:
        condition: service_started

  runner1:
    container_name: gateway-runner-1
    image: 497639811223.dkr.ecr.us-east-2.amazonaws.com/automation-gateway5:5.1.0-amd64
    entrypoint:
      - /usr/local/bin/iagctl
      - runner
    environment:
      GATEWAY_APPLICATION_CA_CERTIFICATE_FILE: "/etc/gateway/certificates/ca.pem"
      GATEWAY_RUNNER_CERTIFICATE_FILE: "/etc/gateway/certificates/one-runner.pem"
      GATEWAY_RUNNER_PRIVATE_KEY_FILE: "/etc/gateway/certificates/one-runner-key.pem"
      GATEWAY_RUNNER_LISTEN_ADDRESS: "0.0.0.0"
      GATEWAY_RUNNER_USE_TLS: "true"
      GATEWAY_STORE_BACKEND: "etcd"
      GATEWAY_APPLICATION_MODE: "runner"
      GATEWAY_STORE_ETCD_HOSTS: "etcd-1:2379"
      GATEWAY_STORE_ETCD_USE_TLS: "false"
      GATEWAY_LOG_LEVEL: "DEBUG"
    networks:
      my_network:
        ipv4_address: 172.30.0.3
    volumes:
      - /etc/gateway/certificates:/etc/gateway/certificates
    depends_on:
      etcd:
        condition: service_started

  runner2:
    container_name: gateway-runner-2
    image: 497639811223.dkr.ecr.us-east-2.amazonaws.com/automation-gateway5:5.1.0-amd64
    entrypoint:
      - /usr/local/bin/iagctl
      - runner
    environment:
      GATEWAY_APPLICATION_CA_CERTIFICATE_FILE: "/etc/gateway/certificates/ca.pem"
      GATEWAY_RUNNER_CERTIFICATE_FILE: "/etc/gateway/certificates/two-runner.pem"
      GATEWAY_RUNNER_PRIVATE_KEY_FILE: "/etc/gateway/certificates/two-runner-key.pem"
      GATEWAY_RUNNER_LISTEN_ADDRESS: "0.0.0.0"
      GATEWAY_RUNNER_USE_TLS: "true"
      GATEWAY_STORE_BACKEND: "etcd"
      GATEWAY_APPLICATION_MODE: "runner"
      GATEWAY_STORE_ETCD_HOSTS: "etcd-1:2379"
      GATEWAY_STORE_ETCD_USE_TLS: "false"
      GATEWAY_LOG_LEVEL: "DEBUG"
    networks:
      my_network:
        ipv4_address: 172.30.0.4
    volumes:
      - /etc/gateway/certificates:/etc/gateway/certificates
    depends_on:
      etcd:
        condition: service_started

  etcd:
    container_name: etcd-1
    image: quay.io/coreos/etcd:v3.5.13
    entrypoint: /usr/local/bin/etcd
    command:
      - '--name=etcd'
      - '--initial-advertise-peer-urls=http://etcd-1:2380'
      - '--listen-peer-urls=http://0.0.0.0:2380'
      - '--listen-client-urls=http://0.0.0.0:2379'
      - '--advertise-client-urls=http://etcd-1:2379'
      - '--initial-cluster=etcd=http://etcd-1:2380'
      - '--initial-cluster-state=new'
      - '--initial-cluster-token=etcd-cluster-1'
    networks:
      my_network:
        ipv4_address: 172.30.0.5
    ports:
      - "127.0.0.1:2379:2379"
    volumes:
      - etcd:/etcd_data

volumes:
  etcd:

Start the cluster

Start the Docker Compose deployment by running the following command in the same directory as your docker-compose.yml file:

docker compose up

Test the cluster by making requests from a client. Ensure you have a gateway client set up on your machine, then open another terminal and run:

export GATEWAY_APPLICATION_CA_CERTIFICATE_FILE=/etc/gateway/certificates/ca.pem
export GATEWAY_CLIENT_CERTIFICATE_FILE=/etc/gateway/certificates/myclient-client.pem
export GATEWAY_CLIENT_PRIVATE_KEY_FILE=/etc/gateway/certificates/myclient-client-key.pem
export GATEWAY_APPLICATION_MODE=client
export GATEWAY_CLIENT_USE_TLS=true

iagctl login admin

You now have a gateway cluster with full TLS configured between all nodes!