Itential Automation Platform

On this page:

Event Logs

Itential Automation Platform (IAP) provides event logs for system and application administrators, developers, and troubleshooters. These event logs provide an audit trail to understand system activity and diagnose problems.

Log Message Activity

IAP supports the following log severity levels.

Log Level Description
error The most severe level. Indicates a problem, such as a prematurely terminated program.
warn Used for warning conditions. Warn events may or may not indicate a problem.
info Used for informational messages.
debug Used only when debugging problems or transactions. Provides detailed information on messages between IAP adapters and external systems.
trace Trace events have a higher verbosity than debug. Trace events are uncommon.
spam The least severe level. Spam events have the highest verbosity and are rare.

Syslog Settings (Admins)

IAP log settings specify where logs are stored, log rotation behavior, and event severity.

To view log settings in IAP:

  1. Navigate to the Admin Essentials application from the IAP homepage.
  2. Select Profiles from the left navbar to open the accordian menu, and select a profile. In the example below, profile1 was selected.
  3. Click the Configure tab to open the editor.
  4. Click loggerProps from the Edit Profile Properties list.
  5. Scroll to the Syslog property section to configure your log settings.
  6. Click the Save icon at the top to retain your changes. Click the reverse arrow icon to undo all changes.
  7. Restart IAP.

Properties Logging

Logging Settings

Default Logging settings are referenced below.

{
  "description": "Logging",
  "log_max_files": 100,
  "log_max_file_size": 10485760,
  "log_level": "debug",
  "log_directory": "/var/log/pronghorn",
  "log_filename": "pronghorn.log",
  "console_level": "warn"
}

Logging JSON Schema

Below is an example JSON schema describing loggerProps. See JSON Schema to learn more about JSON schema specification.

{
  "$id": "https://www.itential.com/schemas/loggerProps.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "description": {
      "$id": "/properties/description",
      "type": "string",
      "title": "Pronghorn Properties Tab Title",
      "description": "Text shown for the logging tab in Pronghorn
                      Properties view accessed from Pronghorn menu
                      SETTINGS->PROPERTIES.",
      "default": "Logging",
      "examples": [
        "Logging"
      ]
    },
    "log_max_files": {
      "$id": "/properties/log_max_files",
      "type": "integer",
      "title": "Number of Saved Log Files",
      "description": "The number of old log files automatically
                      preserved.",
      "default": 0,
      "minimum": 0,
      "examples": [
        100
      ]
    },
    "log_max_file_size": {
      "$id": "/properties/log_max_file_size",
      "type": "integer",
      "title": "Maximum Log File Size",
      "description": "When Pronghorn's log file reaches this size
                      in bytes, Pronghorn makes a backup and clears
                      the log file's contents.",
      "default": 0,
      "minimum": 0,
      "examples": [
        104857600
      ]
    },
    "log_timezone_offset": {
      "$id": "/properties/log_timezone_offset",
      "type": "number",
      "title": "Timezone Offset",
      "description": "A number specifying the offset from UTC.
                      Pronghorn event logs will write the messages'
                      timestamps using this property for the
                      timezone.",
      "default": 0,
      "examples": [
        -5,
        1,
        2.5
      ]
    },
    "log_level": {
      "$id": "/properties/log_level",
      "enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
      "title": "Minimum Logged Severity Level",
      "description": "Specifies which event messages will be logged. Messages
                      with a severity level equal to or higher than this
                      property are logged. Severity error is the most
                      severe; severity spam is the least sever. Severity
                      error events indicate a problem, such as a
                      prematurely terminated program. Severity warn
                      events are used for warning conditions; they may
                      or may not indicate a problem. Severity info events
                      are informational messages. Severity debug events
                      are for debug-level messages, contain information
                      normally of use only when debugging a program, and
                      provide detailed information on messages between
                      Pronghorn adapters and external systems. Severity
                      trace events offer higher verbosity than debug,
                      and they are uncommon. Severity spam events offer
                      the highest verbosity and are rare.",
      "default": "",
      "examples": [
        "debug"
      ]
    },
    "log_directory": {
      "$id": "/properties/log_directory",
      "type": "string",
      "title": "Log File Directory",
      "description": "Directory, specified as an absolute path, where
                      log files are saved.",
      "default": "",
      "examples": [
        "/var/log/pronghorn"
      ]
    },
    "log_filename": {
      "$id": "/properties/log_filename",
      "type": "string",
      "title": "Log File",
      "description": "Log file filename.",
      "default": "",
      "examples": [
        "pronghorn.log"
      ]
    },
    "console_level": {
      "$id": "/properties/console_level",
      "enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
      "title": "Minimum Console Severity Level",
      "description": "Specifies which event messages will be written
                      to the console. Messages with a severity level
                      equal to or higher than this property are
                      logged. Severity error is the most severe;
                      severity spam is the least sever. Severity
                      error events indicate a problem, such as a
                      prematurely terminated program. Severity warn
                      events are used for warning conditions; they
                      may or may not indicate a problem. Severity
                      info events are informational messages.
                      Severity debug events are for debug-level
                      messages, contain information normally of
                      use only when debugging a program, and provide
                      detailed information on messages between
                      Pronghorn adapters and external systems.
                      Severity trace events offer higher verbosity
                      than debug, and they are uncommon. Severity
                      spam events offer the highest verbosity and
                      are rare.",
      "default": "",
      "examples": [
        "warn"
      ]
    }
  }
}

Severity Thresholds

Production environments should generally have the log_level set to warn or info. Debug, trace, and spam log levels will likely generate a large amount of log data and additional server load. Only configure production servers in debug mode at the request of Itential Support.

For systemd operating systems, the system journal manages console logging. The system journal may also contain application life cycle error messages that cannot be saved to the IAP file logs.

Log File Retention

IAP provides native support for rotation of the pronghorn.log file. Administrators can define the total allocated storage for logs by configuring the maximum file size and maximum number of files. For example, if the maximum file size is 1 MB and the total number of files is 100, the total space consumed by pronghorn.log files will be 100 MB. Make sure the disk that holds the pronghorn logs contains enough space for IAP to exercise a full log rotation cycle.

Consider both IAP file logs and system logs when planning server storage requirements.

Temporal Severity Threshold Settings

Every IAP adapter and application inherits the system log and console severity thresholds when IAP starts. After IAP starts, every adapter and application can be optionally configured with local, temporal log and console severity thresholds. The temporal thresholds are reset to IAP global thresholds when IAP restarts.

  1. Navigate to Admin Essentials from the IAP homepage.
  2. Click Applications or Adapters from the left navbar to open the accordian menu.
  3. Select an application (or adapter) and then click the Logging tab to open the Log Settings view.
  4. Set the File Log level.
  5. Set the Console Log level.
  6. Click the Save icon to retain your changes. Click the reverse arrow icon to undo all changes.

Severity Settings

Troubleshooting

If a problem prevents IAP from loading the global log object from the Logger class, log messages will not reach the IAP log file. Log messages will, however, reach the systemd journal or /var/log/messages on System V (“System Five”) hosts.

Monitor the system journal for errors and warnings using shell command journalctl -f. If many processes are sending message to the system journal, filter log messages to show only IAP logs using shell command journalctl -f -u pronghorn.service.

See Red Hat systemd Journal Documentation for more information.

Monitor IAP logs with a tail follow shell command, such as tail -F /var/log/pronghorn/pronghorn.log.

Reading the Logs

IAP logs are JSON objects that are better for automating log parsing operations compared to simple text.

Log Object Properties

Property Description
@timestamp Automatically added.
@fields.origin The message location: application that created the log message and its line number associated with the event.
@fields.level The event message severity level.
@message An array. Elements can be strings or objects. An array with one string element containing log message text that can be a stack trace, an error received from an external system, or a log message the IAP developer sent with the global log object.

Troubleshooting a Layered Architecture

IAP applications exist in a layered architecture. IAP has multiple layers, and it is one of many autonomous systems that collectively create network automation. The best way to troubleshooting multiple layers is to establish control and eliminate variables.

For example, suppose the application method called from a job task is failing. The task coordinates with an external network controller to affect device changes. Begin by verifying the network controller is able to execute the exact requested action. If the controller has a CLI, that is the best place to verify an action. If the CLI test passes, try calling the action through the controller API. If the controller API test passes, use a web API client, such as Google Postman, to test the IAP application method without the IAP workflow engine. Finally, use the Jobs dashboard to inspect run-time data: passed arguments, data or errors returned.

Global Log Instance (Developers)

Adapter and application developers can log events using the IAP global log instance of the Logger class. The log methods specify the event severity. Each takes an argument, a string to be saved in the logs.

Best Practices

For inside functions:

  • Log non-sensitive passed arguments at debug severity.
  • Log non-sensitive returned data at debug severity.

Examples

Event Method Message
Error Severity log.error() This message will be recorded as an error. It will be sent for all severity levels.
Warning Severity log.warn() This message will be recorded as a warning. It will be sent for severity levels warn, info, debug, trace, and spam.
Informational Severity log.info() This message will be recorded as informational. It will be sent for severity levels info, debug, trace, and spam.
Debug Severity log.debug() This message will be recorded as debug. It will be sent for severity levels debug, trace, and spam.
Trace Severity log.trace() This message will be recorded as a trace. It will be sent for severity levels trace and spam.
Spam Severity log.spam() This message will be recorded as spam. It will only be sent for severity level spam.