Configure event deduplication

The event deduplication feature handles situations where the same event might occur from multiple sources, especially in highly-available architectures. If multiple downstream resources can report the same event, or if multiple Itential Platform instances listen to events from the same downstream resource, multiple event messages may be broadcast in the Itential Platform event system for the same downstream occurrence. In this case, actions listening for those events execute multiple times. In most cases, however, the desired behavior is to respond only once for each distinct occurrence.

For example, event deduplication may be required if an instance of NSO is observed by a cluster of Itential Platform servers. In this scenario, each Itential Platform server maintains its own individual connection to the NSO server. NETCONF events produced by the NSO server are observed by all running instances of Itential Platform, resulting in multiple messages generated in the Itential Platform event system for a single NETCONF event. In this case, configure Itential Platform event deduplication to ensure that only one message is emitted to the event bus so that there is only one response to the event from the system as a whole.

Configuring Event Deduplication

In the active Itential Platform profile, set the service config of each event producer which requires event deduplication. Once the active profile is configured to enable event deduplication, configure each service for which events are to be deduplicated.

The most important settings shown below are active and uniqueProps, which determine the on/off status of the feature, as well as the set of message properties that will be used to determine which messages are duplicates. The uniqueProps setting lets you select the fields that will be used to evaluate if two events are duplicates. For example, if the values of the three fields specified below ("/value/device", "/value/event", "/value/timestamp") are the same between two events, the event is considered a duplicate. If any of these are different, then the events are not deduplicated.

Below is a sample service config for eventDeduplication settings.

1
{
2
  ...,
3
  "properties" : { ... },
4
  "eventDeduplication": {
5
    "active" : true,
6
    "cacheTtl" : 100,
7
    "algorithm" : "md5",
8
    "uniqueProps" : [
9
      "/value/device",
10
      "/value/event",
11
      "/value/timestamp"
12
    ]
13
  }
14
}

eventDeduplication

Use this reference table to help set eventDeduplication properties for the service config.

algorithm

The algorithm setting is unlikely to be required, but if needed, available options are discoverable via openssl list -digest-algorithms, or openssl list-message-digest-algorithms on older systems. The library used to create a message digest uses the system’s openssl utility to produce a result.

uniqueProps

The uniqueProps setting tells the deduplication system which fields define a unique message. The fields specified in this property extract a subset of the message which is then hashed to provide a unique value to the deduplication system.

For example, if the uniqueProps property specifies the fields device, event and timestamp, the following messages are considered redundant to each other, and two of them are dropped while the first is emitted through the event system.

1
{
2
  "device": "ios1",
3
  "event": "CRASH",
4
  "timestamp": 1592329817,
5
  "remediation": "REBOOT"
6
}
7
8
{
9
  "device": "ios1",
10
  "event": "CRASH",
11
  "timestamp": 1592329817,
12
}
13
14
{
15
  "device": "ios1",
16
  "event": "CRASH",
17
  "timestamp": 1592329817,
18
  "region": "us-east-1"
19
}

When using inclusive configuration (shown below as JSON Pointer Array and Inclusive Mask Object), you must specify uniqueProps explicitly. If none of the specified fields are found in a message, it is considered unique and subsequent messages like it will not be deduplicated.

JSON Pointer Array

When uniqueProps is an array, it must contain a list of JSON pointer expressions. These expressions tell Itential Platform which properties should be used when comparing messages, forming an inclusive list.

1
[
2
  "/value/device",
3
  "/value/event",
4
  "/value/timestamp"
5
]

Inclusive Mask Object

The uniqueProps setting may also be specified as an object with true or false as its leaf values (example below). In this form, uniqueProps acts as a ‘mask’ object. Note that true and false leaf values may not be mixed, as this causes ambiguity in the mask object in some situations.

The example below is equivalent to the JSON Pointer Array example above.

1
{
2
  "value": {
3
    "device": true,
4
    "event": true,
5
    "timestamp": true
6
  }
7
}

The inverse of the above is also possible for use when the messages from an event source are deduplicated based on all but a known set of properties. For example, if the message in question also contains properties for region and severity, the following mask object would be equivalent to both of the above examples:

1
{
2
  "value": {
3
    "region": false,
4
    "severity": false
5
  }
6
}

Email Event Triggers

Under certain conditions, the email adapter creates duplicate jobs when used as an event trigger in Operations Manager. This issue can occur when emails are processed at a high rate and can be remediated by specifying a messageId field in the uniqueProps setting of the service configuration as shown in the example below.

1
{
2
  ...,
3
  "properties" : { ... },
4
  "eventDeduplication": {
5
    "active" : true,
6
    "cacheTtl" : 10000,
7
    "algorithm" : "md5",
8
    "uniqueProps" : [
9
      "/messageId"
10
    ]
11
  }
12
}