Active-standby architecture
Active-standby architecture
Active-standby architecture
The active-standby architecture is Itential’s most resilient validated deployment model. It is designed for organizations that operate automation at a scale where unplanned downtime carries measurable business risk, whether that risk is regulatory, contractual, or operational.
This architecture provides geographic redundancy across three data centers and is capable of surviving the complete loss of any single data center without operator intervention. We recommend deploying this architecture if you have strict uptime requirements and formal business continuity programs.
This architecture requires significant infrastructure investment but delivers high resilience. The sections below describe the component footprint, hardware requirements, failover behavior, and operational expectations in detail so that infrastructure, operations, and security teams can assess readiness and plan accordingly.
An active-standby architecture (ASA) deploys Itential Platform, MongoDB, and Redis across three geographically distributed data centers with full redundancy. This architecture builds on two geographically redundant HA2 installations and uses a larger geographically redundant MongoDB replica set.
The Itential Platform performs frequent reads and writes to the database, so low latency between the active Platform instances and MongoDB is critical. All active components run in the same data center. MongoDB and Redis replication processes replicate data from the primary node in the active data center to secondary data centers. All components require authentication.
The minimum active-standby deployment requires 17 virtual machines (VMs) distributed across three data centers. The table below summarizes the full server inventory.
All servers require solid-state storage (SSD or NVMe) capable of at least 20,000 IOPS and network connectivity of 10 Gbps or higher. For complete hardware specifications, see Server specifications.
Itential Platform instances communicate with one another in an abstract manner via Redis. Adding a new Itential Platform node and configuring it to access the correct MongoDB and Redis instances achieves high availability. As Itential Platform instances are added and configured, they are enabled to perform work.
Configure Itential Platform instances with the following:
Both MongoDB and Redis use a primary/secondary replication model. When a primary node fails, the replica set initiates an election for a new primary. The replica set cannot accept reads and writes until the new primary is selected, typically within a few seconds. Once a new primary is identified, the Itential Platform resumes normal operation. Operators do not need to take action during elections.
MongoDB clusters operate in a primary/secondary model where data written to the primary replicates to secondary nodes. To prevent split-brain scenarios during elections, the architecture requires an odd number of replica set members distributed across three data centers: 2 in the primary region, 2 in the secondary region, and 1 in a tertiary region. When a region is lost, three voting members of the replica set remain. The replica set configuration must enforce a preference to influence the voting in this architecture to guarantee that the primary MongoDB shifts to the secondary region in the case of a disaster.
Configure Itential’s MongoDB cluster with the following requirements:
priority settings to influence voting as follows:Learn more:
To avoid single points of failure, arrange Redis data-bearing nodes in pairs across both data centers as a replica set. Configure Sentinels in 3 data centers to avoid a data center outage from reducing Sentinel availability below a majority. A majority of Sentinels must always be available for failover to work.
Configure Itential’s Redis replica sets with the following requirements:
Redis requires careful latency management. If latency between Redis nodes exceeds 10ms, replication lag and failover issues can occur. Keep all Redis nodes within a single region or use high-bandwidth, low-latency interconnects between regions.
Learn more:
The active-standby architecture is designed so that the loss of any single data center triggers automatic recovery without requiring operator action. The specific behavior depends on which data center is lost.
If Data Center 1 (active) becomes unavailable, the MongoDB replica set detects the loss of its two highest-priority members and holds an election. The two MongoDB nodes in the secondary data center carry sufficient priority to assume the primary role. Redis Sentinel, with members still available in Data Centers 2 and 3, similarly promotes the secondary Redis node to primary. The GTM load balancer then routes traffic to Data Center 2, where a standby set of Platform and Gateway nodes is running and ready to accept work. The Platform nodes in Data Center 2 resume processing once both MongoDB and Redis primaries are established in that data center.
If Data Center 2 becomes unavailable, the primary data center retains a majority in both MongoDB and Redis and continues normal operation without disruption.
Data Center 3 hosts only a Redis Sentinel and a MongoDB arbiter. Losing it does not cause a primary election in either MongoDB or Redis because a majority of voting members (in Data Centers 1 and 2) remain available. No failover occurs and operations continue uninterrupted.
Automatic recovery is handled by MongoDB and Redis election mechanisms. Under normal network conditions, elections complete within a few seconds. During that window the platform may not be able to accept new work or commit data. No manual intervention is required. Once elections complete, the platform resumes automatically.
Define your own Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets based on your specific operational requirements and validate those targets during initial deployment testing. The architecture is designed to support RTOs measured in seconds for component-level failures and minutes for full data center failures, contingent on external load balancer configuration.
Replication provides infrastructure redundancy, not data protection. Backups are essential to guard against logical corruption and accidental deletion.
Geographic replication is not a substitute for backup. Replication propagates all writes, including unintended ones, to all members of the replica set. Implement and test a backup strategy appropriate to your recovery objectives.
MongoDB is the system of record for all platform data including workflows, jobs, inventory, and configuration. Itential recommends a regular backup schedule using MongoDB-native tooling (mongodump or MongoDB Ops Manager) with backups stored outside the primary replica set, preferably in a location not subject to the same failure scenarios as the three data centers. Consider point-in-time recovery using oplog tailing for low RPO requirements.
For more information, see Back up and restore MongoDB.
Redis functions as a transient message and job queue layer. Redis data is reconstructed by the platform on reconnection and does not require the same backup posture as MongoDB. However, if you use Redis persistence (AOF or RDB), ensure that persistence files are included in your broader backup strategy.
Test backup procedures on a scheduled basis, not only at initial deployment.
The answer to how much ongoing operational effort this architecture requires depends on whether normal operations or incident response is being considered.
Under steady-state conditions this architecture requires relatively little hands-on management. The platform, MongoDB, and Redis all handle routine internal events — such as replica lag recovery and leader election — without operator intervention. The primary operational responsibilities are:
Operating this architecture competently requires staff with working knowledge of MongoDB replica set administration, Redis Sentinel configuration, Linux system administration, and your load balancer and network infrastructure. Familiarity with your monitoring stack is also required. Itential Professional Services can assist with initial deployment and knowledge transfer.
Primary elections in MongoDB and Redis are fully automatic. Operators do not need to manually promote a replica to primary during a failover. The platform reconnects to new primaries automatically once elections complete.
Establish monitoring coverage for the following conditions at minimum. Without visibility into these signals, degraded states can go undetected until they compound into an outage.
Monitor replica set member health and replication lag across all five nodes. A lagging secondary that has not caught up with the primary is a risk factor in any subsequent failover. Alert on replication lag exceeding a threshold appropriate to your RPO. Monitor available disk space on all data-bearing nodes, particularly the /var/lib/mongo partition, which holds the data files.
Monitor Sentinel-reported primary/secondary topology to confirm the expected node holds the primary role at all times. Alert on any unexpected role change, which may indicate a silent failover occurred. Monitor replication lag between Redis nodes and connection counts from the platform.
Monitor application logs for job processing errors and connectivity failures to MongoDB or Redis. Elevated error rates are early indicators of a dependency problem.
Monitor network latency between data centers. The platform is sensitive to latency in its MongoDB and Redis connections. Elevated cross-datacenter latency is a leading indicator of replication problems.
The installation process handles setting the appropriate configurations in MongoDB and Redis. There are a few other things to consider in Itential Platform — most importantly, knowing what state the Workflow Engine is in at any given moment. In an ASA, the secondary data center (standby site) contains Itential Platform servers that you must configure to remain passive until a failover event occurs. This configuration prevents both data centers from processing workloads simultaneously.
There are two settings for controlling the state of the Workflow Engine, both found in either the configuration file or in corresponding environment variables. The properties file is typically found at /etc/itential/platform.properties.
Initialize both properties to false in the secondary data center so that it remains passive and does not start or process jobs or tasks. During a failover event, after both MongoDB and Redis have successfully completed their elections, set these to true to activate the secondary workflow engines and resume automations. Manage both properties with a RESTful API call — start the task worker first, then the job worker. Both APIs require a valid session token and must be run against each Itential Platform server individually; they cannot be executed through a load balancer.
After making these requests the secondary data center processes automations. At this point, disable these on the previously active data center:
Itential recommends setting both properties to false in both the active and secondary data centers. Setting all Itential Platform servers to disable jobs and tasks gives you a known state whenever instances stop and restart, making it a deliberate action to enable job and task processing even in the primary data center.
For production environments, all Itential Platform components should be installed on their own individual servers to properly support High Availability (HA). Disk references to pronghorn (seen in older deployments) should be changed to itential.
The following applies to a simple All-In-One implementation of Itential Automation Gateway (IAG). For a more information about alternative IAG architectures, see Choose a deployment architecture.
Processor specification requirements:
Memory specification requirement:
Storage performance requirements in IOPS (16 kiB):
Network speed requirement:
In some instances, adding additional dedicated interfaces that are focused on routing specific traffic to specific external systems can be explored. This routing of traffic would be configured at the OS-level (custom interfaces and routes) and requires the system administrator to manage it. An example would be separating NSO traffic from Redis/MongoDB destined traffic.
These settings are strongly recommended for high load applications of Itential Platform:
Example: Assuming an Itential Platform VM on a server capable of 2.5GHz nominal speed:
Follow hypervisor recommendations when performing CPU reservations. In most cases the total of all CPU reservations for all VMs on a host cannot be more than 90% of the host capacity as 10% is reserved by the host itself.
MongoDB discourages the utilization of Transparent Huge Pages with versions 7 and below. This advice is changed in version 8 which encourages the use of Transparent Huge Pages.
In an environment where components are installed on more than one host, the following network traffic flows need to be allowed. All ports and networking specs are TCP protocol unless otherwise noted. Not all ports will need to be open for every supported architecture. Secure ports are only required when explicitly configured.
The validated designs are opinionated installations of Itential and its dependencies. The following user accounts are required by the dependencies.