Event Deduplication in IAP
The event deduplication feature is designed to handle situations where the same event might occur from multiple sources, especially in the context of highly-available architecture. If multiple downstream resources can report the same event, or if multiple IAP instances are listening to events from the same downstream resource, multiple event messages may be broadcast in the IAP event system for the same downstream occurrence. In this case, actions listening for those events will execute multiple times. In most cases, however, the desired behavior is to respond only once for each distinct occurrence.
For example, event deduplication may be required if an instance of NSO is observed by a cluster of IAP servers. In this scenario, each IAP server will maintain its own individual connection to the NSO server. NETCONF events produced by the NSO server will be observed by all running instances of IAP, resulting in multiple messages being generated in the IAP event system for a single NETCONF event. In this case, IAP event deduplication may be configured to ensure that only one message is emitted to the event bus so that there is only one response to the event from the system as a whole.
In order for the event deduplication feature to work, all involved RabbitMQ instances must be configured beforehand, and the feature itself must be enabled globally through the system profile.
The feature operates using a third-party plugin to RabbitMQ, and has the ability to be configured on a service-by-service basis which event message properties should be used to determine when a message should be dropped.
When event deduplication settings change, IAP must be restarted for those changes to be propagated to RabbitMQ. If IAP is deployed in an HA cluster, all instances of IAP must be restarted in order for the event system to operate. Restarting only one instance will result in undefined behavior due to the fact that all other servers will be running with an outdated profile.
Due to the nature of RabbitMQ, propagating any configuration changes involves deleting and recreating the
iap_event_exchange exchange and
iap_event queue, since RabbitMQ resource properties may not be updated once created. To handle cases where there are unprocessed messages left in the event queue when a recreation occurs, IAP provides the ability to configure an event message preservation strategy. This feature utilizes the
rabbitmq_shovel plugin to move all transient messages to a temporary queue while the
iap_event queue are recreated with new properties. This preservation strategy helps to prevent message loss.