Event Logs & Logging

Structure of a Log Message in IAP

This section explains how a log message is structured in IAP for error reporting, including the log statements that write a record to the event log.

Logs are found at: /var/log/pronghorn

Log Entry Example

This log message is an example entry under: /var/log/pronghorn/pronghorn.log

{
 "@message": "[\"Cannot register method 'AutomationGateway.deleteInventoryDevice'. It has already been declared.\"]",
 "@timestamp": "2023-04-26T22:05:07.931Z",
 "@fields": {
  "@origin": "/opt/pronghorn/itential-bundle3_2022.1.5/node_modules/@itential/pronghorn-core/core/Endpoints.bin:28",
  "@level": "warn"
 }
}

Log Statement

The different parts of the log statement are explained below.

Log Levels

Log levels indicate the severity of a log entry. IAP supports the following log levels.

Syslog Settings

IAP log settings specify where logs are stored, log rotation behavior, and event severity.

To view log settings in IAP:

1. Navigate to the Admin Essentials application from the IAP homepage.
2. Select Profiles from the left navbar to open the accordian menu, and select a profile. In the example below, profile1 was selected.
3. Click the Configure tab to open the editor.
4. Click loggerProps from the Edit Profile Properties list.
5. Scroll to the Syslog property section to configure your log settings.
6. Click the Save icon at the top to retain your changes. Click the reverse arrow icon to undo all changes.
7. Restart IAP.

Figure 1: Syslog loggerProps

Default Log Settings

Default Logging settings are referenced below.

{
  "description": "Logging",
  "log_max_files": 100,
  "log_max_file_size": 10485760,
  "log_level": "debug",
  "log_directory": "/var/log/pronghorn",
  "log_filename": "pronghorn.log",
  "console_level": "warn"
}

Global Log Instance (Developers)

Adapter and application developers can log events using the IAP global log instance of the Logger class. The log methods specify the event severity. Each takes an argument, a string to be saved in the logs.

For inside functions:

• Log non-sensitive passed arguments at debug severity.
• Log non-sensitive returned data at debug severity.

Examples

JSON Schema Logger Props

Click the link below to show an example JSON schema describing loggerProps. See JSON Schema to learn more about JSON schema specification.

{
  "$id": "https://www.itential.com/schemas/loggerProps.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "description": {
      "$id": "/properties/description",
      "type": "string",
      "title": "Pronghorn Properties Tab Title",
      "description": "Text shown for the logging tab in Pronghorn
                      Properties view accessed from Pronghorn menu
                      SETTINGS->PROPERTIES.",
      "default": "Logging",
      "examples": [
        "Logging"
      ]
    },
    "log_max_files": {
      "$id": "/properties/log_max_files",
      "type": "integer",
      "title": "Number of Saved Log Files",
      "description": "The number of old log files automatically
                      preserved.",
      "default": 0,
      "minimum": 0,
      "examples": [
        100
      ]
    },
    "log_max_file_size": {
      "$id": "/properties/log_max_file_size",
      "type": "integer",
      "title": "Maximum Log File Size",
      "description": "When Pronghorn's log file reaches this size
                      in bytes, Pronghorn makes a backup and clears
                      the log file's contents.",
      "default": 0,
      "minimum": 0,
      "examples": [
        104857600
      ]
    },
    "log_timezone_offset": {
      "$id": "/properties/log_timezone_offset",
      "type": "number",
      "title": "Timezone Offset",
      "description": "A number specifying the offset from UTC.
                      Pronghorn event logs will write the messages'
                      timestamps using this property for the
                      timezone.",
      "default": 0,
      "examples": [
        -5,
        1,
        2.5
      ]
    },
    "log_level": {
      "$id": "/properties/log_level",
      "enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
      "title": "Minimum Logged Severity Level",
      "description": "Specifies which event messages will be logged. Messages
                      with a severity level equal to or higher than this
                      property are logged. Severity error is the most
                      severe; severity spam is the least sever. Severity
                      error events indicate a problem, such as a
                      prematurely terminated program. Severity warn
                      events are used for warning conditions; they may
                      or may not indicate a problem. Severity info events
                      are informational messages. Severity debug events
                      are for debug-level messages, contain information
                      normally of use only when debugging a program, and
                      provide detailed information on messages between
                      Pronghorn adapters and external systems. Severity
                      trace events offer higher verbosity than debug,
                      and they are uncommon. Severity spam events offer
                      the highest verbosity and are rare.",
      "default": "",
      "examples": [
        "debug"
      ]
    },
    "log_directory": {
      "$id": "/properties/log_directory",
      "type": "string",
      "title": "Log File Directory",
      "description": "Directory, specified as an absolute path, where
                      log files are saved.",
      "default": "",
      "examples": [
        "/var/log/pronghorn"
      ]
    },
    "log_filename": {
      "$id": "/properties/log_filename",
      "type": "string",
      "title": "Log File",
      "description": "Log file filename.",
      "default": "",
      "examples": [
        "pronghorn.log"
      ]
    },
    "console_level": {
      "$id": "/properties/console_level",
      "enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
      "title": "Minimum Console Severity Level",
      "description": "Specifies which event messages will be written
                      to the console. Messages with a severity level
                      equal to or higher than this property are
                      logged. Severity error is the most severe;
                      severity spam is the least sever. Severity
                      error events indicate a problem, such as a
                      prematurely terminated program. Severity warn
                      events are used for warning conditions; they
                      may or may not indicate a problem. Severity
                      info events are informational messages.
                      Severity debug events are for debug-level
                      messages, contain information normally of
                      use only when debugging a program, and provide
                      detailed information on messages between
                      Pronghorn adapters and external systems.
                      Severity trace events offer higher verbosity
                      than debug, and they are uncommon. Severity
                      spam events offer the highest verbosity and
                      are rare.",
      "default": "",
      "examples": [
        "warn"
      ]
    }
  }
}

Log File Retention

IAP provides native support for rotation of the pronghorn.log file. Administrators can define the total allocated storage for logs by configuring the maximum file size and maximum number of files. For example, if the maximum file size is 1 MB and the total number of files is 100, the total space consumed by pronghorn.log files will be 100 MB. Make sure the disk that holds the pronghorn logs contains enough space for IAP to exercise a full log rotation cycle.

Consider both IAP file logs and system logs when planning server storage requirements.

Severity Thresholds

Production environments should generally have the log_level set to warn or info. Debug, trace, and spam log levels will likely generate a large amount of log data and additional server load. Only configure production servers in debug mode at the request of Itential Support.

For systemd operating systems, the system journal manages console logging. The system journal may also contain application life cycle error messages that cannot be saved to the IAP file logs.

Temporal Severity Threshold Settings

Every IAP adapter and application inherits the system log and console severity thresholds when IAP starts. After IAP starts, every adapter and application can be optionally configured with local, temporal log and console severity thresholds. The temporal thresholds are reset to IAP global thresholds when IAP restarts.

1. Navigate to Admin Essentials from the IAP homepage.
2. Click Applications or Adapters from the left navbar to open the accordian menu.
3. Select an application (or adapter) and then click the Logging tab to open the Log Settings view.
4. Set the File Log level.
5. Set the Console Log level.
6. Click the Save icon to retain your changes. Click the reverse arrow icon to undo all changes.

Figure 2: Application Logging

Troubleshooting

If a problem prevents IAP from loading the global log object from the Logger class, log messages will not reach the IAP log file. Log messages will, however, reach the systemd journal or /var/log/messages on System V (“System Five”) hosts.

Monitor the system journal for errors and warnings using shell command journalctl -f. If many processes are sending message to the system journal, filter log messages to show only IAP logs using shell command journalctl -f -u pronghorn.service.

See Red Hat systemd Journal Documentation for more information.

Monitor IAP logs with a tail follow shell command, such as tail -F /var/log/pronghorn/pronghorn.log.

Reading the Logs

IAP logs are JSON objects that are better for automating log parsing operations compared to simple text.

Log Object Properties

Troubleshooting a Layered Architecture

IAP applications exist in a layered architecture. IAP has multiple layers, and it is one of many autonomous systems that collectively create network automation. The best way to troubleshooting multiple layers is to establish control and eliminate variables.

For example, suppose the application method called from a job task is failing. The task coordinates with an external network controller to affect device changes. Begin by verifying the network controller is able to execute the exact requested action. If the controller has a CLI, that is the best place to verify an action. If the CLI test passes, try calling the action through the controller API. If the controller API test passes, use a web API client, such as Google Postman, to test the IAP application method without the IAP workflow engine. Finally, use the Jobs dashboard to inspect run-time data: passed arguments, data or errors returned.