Event Service Configuration
  • Dark
    Light
  • PDF

Event Service Configuration

  • Dark
    Light
  • PDF

Once the active IAP profile is configured to enable event deduplication, each service for which events are to be deduplicated must be configured. The most important settings shown below are active and uniqueProps, which determine the on/off status of the feature, as well as the set of message properties that will be used to determine which messages are duplicates. The uniqueProps allow you to handpick the fields that will be used to evaluate if two events are to be regarded as duplicates. For example, if the values of the three fields specified below ("/value/device", "/value/event", "/value/timestamp") are the same between two events, the event is considered as a duplicate. If any of these are different, then the events are not deduplicated.

Example

The following is a sample service config for deduplication settings.

{
  ...,
  "properties" : { ... },
  "eventDeduplication": {
    "active" : true,
    "cacheTtl" : 100,
    "algorithm" : "md5",
    "uniqueProps" : [
      "/value/device",
      "/value/event",
      "/value/timestamp"
    ]
  }
}

eventDeduplication

Use this table to set eventDeduplication properties for the service config.

Property Type Default Description
active Boolean False Turns event deduplication on or off at the component level.
cacheTtl Integer 100 Overrides the global message identifier time-to-live setting for this service.
algorithm String sha512 Algorithm used to hash event messages when producing a message identifier. See below for more info.
uniqueProps Enum
("array",
"object",
null)
Null
(Use all properties.)
Denotes which message fields will be used when producing a message identifier. This setting may be used to exclude fields which may vary slightly between event producers or do not pertain to the uniqueness of the message. See below for more detail.

algorithm

The algorithm setting is unlikely to be required, but in the case that it is, available options are discoverable via openssl list -digest-algorithms, or openssl list-message-digest-algorithms on older systems. The library used to create a message digest uses the system's openssl utility to produce a result.

uniqueProps

The uniqueProps setting is used to tell the deduplication system which fields define a unique message. The fields specified in this property are used to extract a subset of the message which is then hashed to provide a unique value to the deduplication system.

For example, if the uniqueProps property specifies the fields device, event and timestamp, the following messages will be considered as redundant to each other, and two of them will be dropped while the first is emitted through the event system.

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
  "remediation": "REBOOT"
}

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
}

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
  "region": "us-east-1"
}

When using inclusive configuration (shown below as JSON Pointer Array and Inclusive Mask Object), it is mandatory that uniqueProps be specified explicitly. In the event that none of the specified fields are found in a message, it is considered unique and subsequent messages like it will not be deduplicated.

JSON Pointer Array

When uniqueProps is an array, it must contain a list of JSON pointer expressions. These expressions tell IAP which properties should be used when comparing messages, forming an inclusive list.

[
  "/value/device",
  "/value/event",
  "/value/timestamp"
]

Inclusive Mask Object

The uniqueProps setting may also be specified as an object with true or false as its leaf values (example below). In this form, uniqueProps acts as a 'mask' object. Keep in mind, true and false leaf values may not be mixed, as this causes ambiguity in the mask object in some situations.

The example below is equivalent to the JSON Pointer Array example above.

{
  "value": {
    "device": true,
    "event": true,
    "timestamp": true
  }
}

The inverse of the above is also possible for use in case the messages from an event source are deduplicated based on all but a known set of properties. For example, if the message in question also contains properties for region and severity, the following mask object would be equivalent to both of the above examples:

{
  "value": {
    "region": false,
    "severity": false
  }
}