High-availability architecture
High-availability architecture
High-availability architecture
A Highly Available Architecture (HA2) is an Itential architecture where all components are redundant and can gracefully tolerate at least one catastrophic failure. This architecture is the recommended architecture for production and testing environments.
The Itential Platform application performs many reads and writes against the database and is sensitive to high latencies. All components must be installed in the same data center and have authentication enabled.
The minimum HA2 architecture is composed of nine VMs:
Itential Platform instances communicate with one another through Redis and share data via MongoDB. Adding a new Itential Platform node and pointing it to the correct MongoDB and Redis is sufficient to achieve high availability. As Itential Platforms are added and configured, they are enabled to perform work.
Itential Platforms must have the following configurations:
MongoDB clusters operate in a primary/secondary model where data written to the primary replicates to the secondary. If a primary MongoDB node fails, the replica set detects this failure and forces an election for a new primary. During this time the replica set may not accept reads and writes until the new primary is selected, usually after a few seconds. Once finished and a new primary is identified, the Itential Platform application resumes normal operation. Operators do not need to take action during this election.
Itential’s MongoDB cluster must have the following requirements:
For more information, see MongoDB Replication documentation.
Redis clusters operate in a primary/secondary model where data written to the primary replicates to the secondary. If a primary Redis node fails, the replica set detects this failure via Redis Sentinels and forces an election for a new primary. During this time the replica set may not accept reads and writes until the new primary is selected, usually after a few seconds. Once finished and a new primary is identified, the Itential Platform application resumes normal operation. Operators do not need to take action during this election.
Itential’s Redis cluster must have the following requirements:
For more information, see Redis Replication documentation.
The validated designs are opinionated installations of Itential and its dependencies. The following user accounts are required by the dependencies.
In an environment where components are installed on more than one host, the following network traffic flows need to be allowed. All ports and networking specs are TCP protocol unless otherwise noted. Not all ports will need to be open for every supported architecture. Secure ports are only required when explicitly configured.
Processor specification requirements:
Memory specification requirement:
Storage performance requirements in IOPS (16 kiB):
Network speed requirement:
In some instances, adding additional dedicated interfaces that are focused on routing specific traffic to specific external systems can be explored. This routing of traffic would be configured at the OS-level (custom interfaces and routes) and requires the system administrator to manage it. An example would be separating NSO traffic from Redis/MongoDB destined traffic.
These settings are strongly recommended for high load applications of Itential Platform:
Example: Assuming an Itential Platform VM on a server capable of 2.5GHz nominal speed:
Follow hypervisor recommendations when performing CPU reservations. In most cases the total of all CPU reservations for all VMs on a host cannot be more than 90% of the host capacity as 10% is reserved by the host itself.
MongoDB discourages the utilization of Transparent Huge Pages.
For production environments, all Itential Platform components should be installed on their own individual servers to properly support High Availability (HA). Disk references to pronghorn (seen in older deployments) should be changed to itential.
Troubleshoot common issues related to high availability architecture.
If task execution fails in an HA Itential Platform environment, you can determine the specific Itential Platform server that attempted to execute the task by referencing the Server ID of the failed task.
Each Itential Platform server in an HA environment has a unique Server ID property that is defined in one of two ways:
serverName property of the properties.json file located in the Itential Platform installation directoryWhen a server attempts to execute a task, its Server ID property is added to the Task Details panel of that task. Verify there are no connection issues affecting the server identified by the Server ID property of a failed task.