Script Execution Engine

Automation Gateway (AG) provides a Script Execution Engine that allows for the decoration and execution of scripts written in any language including Python, Perl, and Bash. As with Ansible modules, roles, and playbooks, Automation Gateway automatically performs discovery of scripts at startup time and maintains a cache of all managed scripts in memory. Users determine the scripts being managed by providing a list of directory paths in their properties.yml configuration file. A recursive search of the directory paths is performed. Once a script has been discovered, it is available to be decorated. The decoration of scripts within Automation Gateway is similar to that of Ansible playbooks. However, instead of decorating the variables within a playbook, users decorate the command line arguments and environment variables utilized to execute a script. A permanent copy of each script's decoration is stored in a local database that is maintained by Automation Gateway. Scripts that have been decorated can be executed either locally or on a remote set of hosts.

A complete set of REST APIs are available for clients to manage script decoration and execution. See the API Documentation section within the Automation Gateway UI for more information.

Figure: Script Execution (Example)

Decoration

Decorating a script involves creating metadata in a JSON file format that describes the command line arguments and environment variables used to execute a script. The metadata is more specifically known as a JSON schema. Not surprisingly, the format of the flags and arguments that make up a script's command line can vary significantly from script to script. To help users with the decoration process, Automation Gateway provides a low-level schema that serves as a template for creating script specific schemas. The template allows for a wide range of flexibility while also offering some custom features that facilitate the handling of some common syntax patterns found in many scripts. Based on their knowledge of the script, users identify the parameters that will compose the script's schema. At a low level, parameters are JSON objects with type and description fields. All parameters are considered to be either a string or an array of strings. If a script has a minimum set of parameters that are needed for successful execution, the required field can be added to the decoration.

Default Schema

Each script being managed by Automation Gateway has a default schema. The schema consists of the parameters argument_list and env_vars. The argument_list parameter is of type array and allows arguments to be passed to a script as if they were being entered directly on the command line within the Linux shell or a network device's CLI. A description of the env_vars parameter can be found in the Decorating Environment Variables section later on in this document.

The JSON metadata file for the default schema is as follows:

Default Script Decoration

{
  "properties": {
    "argument_list": {
      "type": "array",
      "description": "Array of arguments to be passed to the script",
      "items": {
        "type": "string"
      }
    },
    "env_vars": {
      "type": "object",
      "description": "Object containing environment variables to be passed to script",
      "properties": {
        "env_list": {
          "type": "array",
          "description": "Array of environment variable 'name=value' pairs for script",
          "items": {
            "type": "string"
          }
        }
      }
    }
  },
  "script_argument_order": [
    "argument_list"
  ]
}

Notice the script_argument_order key in the above decoration. Since the order of arguments is often very important for a script to function properly, Automation Gateway requires each parameter that appears within the properties object to be present in the script_argument_order array. During execution, arguments will be processed and inserted on the script's command line in the order that the parameter appears in the array. Parameters missing from the script_argument_order array will not be included on the script's command line. A warning will be returned in the response data indicating the argument has been skipped.

The default decoration can be used to execute a script using the POST {hostname:port}/scripts/{script_name}/execute API found within the Automation Gateway UI API Documentation. Below is an example of how the default decoration schema is used to build the execution arguments for a simple script that front-ends a remote copy operation.

Note: Scripts are allowed to run without any arguments and are only required to have arguments when dictated by the script's schema. Execute a script without arguments by setting the args parameter to an empty object, i.e., args:{}.

Sample Script Execution Parameters

{
  "args":{"argument_list":["--src file1",
                           "--dest sample_host:file2"]},
  "hosts": []
}

The args object in the script execution parameters above will produce the following command line that will be executed on the host that Automation Gateway is running:

/directory_path/scripts/sample_script.sh --src file1 --dest sample_host:file2

A single entry in the argument_list array could have been used to produce the same command line results. Multiple entries were used in the previous sample for clarity and to show the flexibility available from the default schema when using arrays.

Sample Execution Using Single Array Entry

{
  "args":{"argument_list":["--src file1 --dest sample_host:file2"]},
  "hosts": []
}

These examples are meant to demonstrate how the default behavior works. The next section illustrates a more powerful feature for specifying variables to populate for the API call.

Decoration Using String Types

Although an array of strings is provided by the default schema, using simple string types provides more specific descriptions for each command line parameter and provides additional clarity for end users executing scripts. To help simplify and reduce the arguments entered by the end user when executing a script, the Automation Gateway schema supports the use of a prefix (and suffix) key in the script decoration. These fields can be used to indicate flags or characters that always proceed or follow a parameter and will be added by Automation Gateway to the command line automatically.

Note: Remember Proper Spacing. It is important to realize that the spacing between flags and their corresponding arguments varies from script to script and that a prefix or suffix may need to include a space before or after a flag. Note the space following the –src and –dest in the prefix key values shown in the sample below. Also by default, there is always a space added on the command line after each execution argument.

Sample Decoration Using Strings

{
  "properties": {
    "source": {
      "type": "string",
      "description": "Source filename",
      "prefix": "--src "
    },
    "destination": {
      "type": "string",
      "description": "Destination filename",
      "prefix": "--dest ",
      "suffix": " 1"
    }
  },
  "required": [
    "source",
    "destination"
  ],
  "script_argument_order": [
    "source",
    "destination"]
}

The sample above introduces the use of the required key to script schemas. Entries in the required array must be present in the script execution arguments. Any attempts to execute the script without a required parameter will return a failure from Automation Gateway. The script execution arguments below will produce the same command line result as the previous examples.

Sample Execution Using Strings

{
  "args": {"source": "filename1",
           "destination": "filename2"},
  "hosts": []
}

Decoration Using an Array of Strings

The prefix and suffix keys can also be applied to an array of strings. Each command line parameter that is generated by an element in the array will automatically have the designated prefix and/or suffix applied accordingly.

Sample Decoration Using Array of Strings

{
  "properties": {
    "protocol": {
      "type": "string",
      "description": "Protocol type (http, https, ftp, etc.)",
      "prefix": "--prot "
    },
    "domains": {
      "type": "array",
      "description": "List of commercial domain names",
      "items": {
        "type": "string"
      },
      "suffix": ".com"
    },
    "path": {
      "type": "string",
      "description": "Path portion of the URL",
      "prefix": "--path ",
      "suffix": ".html"
    }
  },
  "required": [
    "protocol",
    "domains",
    "path"
  ],
  "script_argument_order": [
    "protocol",
    "path",
    "domains"
  ]
}

Sample Execution Using Array of Strings

{
  "args":{"protocol": "http",
          "domains": ["blue", "red", "green"],
          "path": "/a/path/to/verify"}
}

The args object in the script execution parameters above will produce the following command line that will be executed on the server that Automation Gateway is running:

sample_script.sh --prot http --path /a/path/to/verify.html blue.com red.com green.com

Decoration Using Boolean Types

In addition to string and array types, the Automation Gateway script schema also supports parameters of the type boolean. Two additional keys for boolean parameters are available: value_if_true and value_if_false. These additional keys allow the end user to add command line flags based on the value (true or false) of the parameter. An end-user chooses the name of the conditional parameter and the corresponding flags based on their knowledge of the script. One of the more common use cases is enabling the display of debug information via the command line. A single debug parameter of type boolean can be used to add multiple debug flags to the command line when set to true. The same parameter could also be used to disable or minimize the default output of the command when set to false. The example schema below shows how a boolean type can be utilized.

Sample Decoration Using Boolean

{
  "properties": {
    "source": {
      "type": "string",
      "description": "Source filename",
      "prefix": "--src "
    },
    "destination": {
      "type": "string",
      "description": "Destination filename",
      "prefix": "--dest "
    },
    "debug": {
      "type": "boolean",
      "description": "Save session log data when enabled",
      "value_if_true": "--verbose --save_log",
      "value_if_false": "--quiet"
    }
  },
  "required": [
    "source",
    "destination"
  ],
  "script_argument_order": [
    "source",
    "destination",
    "debug"
  ]
}

Below is an example of the script execution arguments to run the sample remote copy script with debugging enabled.

Sample Execution Using Boolean

{
  "args": {"debug": true,
           "source": "filename1",
           "destination": "sample_host:filename2"},
  "hosts": []
}

The args object in the script execution parameters above will produce the following command line that will be executed on the host Automation Gateway is running:

/directory_path/scripts/sample_script.sh --src filename1 --dest sample_host:filename2 --verbose --save_log

Environment Variables

Many scripts control their flow of execution through the use of environment variables. In Linux, environmental variables are variables that are defined for the current shell and are inherited by any child shells or processes. They are used to pass information into processes that are spawned from the shell.

Automation Gateway passes environment variables to scripts executed on the local host via the script API {hostname:port}/scripts/{script_name}/execute. The contents of the variables are included within an env object in the same way script arguments are included using the args object. Like script arguments, environment variables can be decorated by including them in a script's schema.

Decorating Environment Variables

Each script being managed by Automation Gateway includes an env_vars parameter within its default schema. The env_vars parameter is of type object. All environment variable decorations for a script must be contained within the env_vars object. Any parameter residing outside of env_vars will be treated as a script argument. The default schema also includes the parameter env_list as a member of the env_vars object. The env_list parameter is of type array and allows variables to be passed to a script as if they were being entered directly on the command line within the Linux shell.

Default Script Environment Variables Schema

"env_vars": {
      "type": "object",
      "description": "Object containing environment variables to be passed to script",
      "properties": {
        "env_list": {
          "type": "array",
          "description": "Array of environment variable 'name=value' pairs for script",
          "items": {
            "type": "string"
          }
        }
      }
    }

Below is an example of how a parameter in the default schema is used to construct an env object for the script execution API.

Sample Script Environment Variables Execution

{
  "args":{},
  "env": {"env_list": ["first_env=123", "SECOND_ENV=hello"]}
  "hosts": []
}

The env object above will produce the following command line that will be executed on the host Automation Gateway is running:

first_env=123 SECOND_ENV=hello /path/scripts/sample_script.sh

Note: When using a parameter of type array, each environment variable must be in a separate element within the array and be of the form name=value without any spaces before or after the '='.

Environment Variables as Strings

Although an array of strings is provided by the default schema, using simple string types provides more specific descriptions for each command line parameter and provides additional clarity for end users executing scripts.

Note: Environment variable parameters are always of type string, or an array of strings.

Sample Decoration Using Strings

{
  "properties": {
    "argument_list": {
      "type": "array",
      "description": "Array of arguments to be passed to the script",
      "items": {
        "type": "string"
      }
    },
    "env_vars": {
      "type": "object",
      "description": "Object containing environment variables to be passed to script",
      "properties": {
        "env_list": {
          "type": "array",
          "description": "Array of environment variable 'name=value' pairs for script",
          "items": {
            "type": "string"
          }
        },
        "first_env": {
          "type": "string",
          "description": "Test environment variable"
        },
        "SECOND_ENV": {
          "type": "string",
          "description": "A second test environment variable"
        }
      }
    }
  },
  "script_argument_order": ["argument_list"]
}

Sample Execution Using Environment Variable Strings

{
  "args":{},
  "env": {"first_env": "456", "SECOND_ENV": "testing"}
  "hosts": []
}

The env object above will produce the following command line that will be executed on the host Automation Gateway is running:

first_env=456 SECOND_ENV=testing /path/scripts/sample_script.sh

Local Execution

Scripts are executed by issuing a POST {hostname:port}/scripts/{script_name}/execute call to Automation Gateway with the corresponding arguments and environment variables that match the script's schema. This can be done via the Automation Gateway UI or by a separate application. Scripts can either be executed on the host the AG server is running (local) or on a set of remote hosts. The API returns an array of objects, each containing results for the host upon which the script was executed.

Note: The hosts parameter is not required when executing a script locally. If the hosts parameter is present, it should be empty, i.e., "hosts":[].

By default, scripts are executed from the home directory of the user that owns the Automation Gateway server process. To change the default execution directory, add the working_dir key to the script's schema outside the properties object. The value of working_dir can be an absolute or a relative directory path.

The following fields are contained within the results object:

Below are the results for a basic Perl script that prints messages to stdout and stderr.

Local Script Execution Results

[
    {
        "status": "SUCCESS",
        "stdout": "Hello World\n",
        "stderr": "Hello stderr at /Users/sample/scripts/perl_demo.pl line 7.\n",
        "command": "/Users/sample/scripts/perl_demo.pl",
        "env": [
            "first_env=123",
            "SECOND_ENV=testing"
        ],
        "msg": "",
        "argument_warnings": null,
        "env_warnings": null,
        "working_directory": "/Users/sample",
        "raw_result": {
            "rc": 0
        }
    }
]

Remote Execution

Automation Gateway uses the Ansible shell module to execute scripts on remote hosts. To execute a script remotely, the remote host must be present in Automation Gateway's inventory with the appropriate device connection and credential variables that allow Ansible access to the host. Use the hosts parameter in the script execution API to run a script remotely, i.e., "hosts": ["testhost"].

Note: When executing a script remotely, a copy of the script must reside on the remote host and have the same directory path as the script that resides on the Automation Gateway host.

Sample Remote Node Inventory

{
  {
    "name": "centos1",
    "variables": {
        "ansible_host": "192.168.32.10",
        "ansible_user": "sample",
        "ansible_password": "sample",
        "ansible_connection": "paramiko"
    }
  },
  {
    "name": "centos1-ssh",
    "variables": {
        "ansible_host": "192.168.32.10",
        "ansible_user": "sample",
        "ansible_ssh_private_key_file": "/Users/sample/.ssh/id_rsa"
    }
  }
}

Below are the results for a basic Perl script that prints messages to stdout and stderr. The shell module creates additional data that can be found in the raw_result object. The extra information may be useful when debugging error cases.

Remote Script Execution Results

[
    {
        "status": "SUCCESS",
        "stdout": "Hello World\n",
        "stderr": "Hello stderr at /Users/sample/scripts/perl_demo.pl line 7.",
        "command": "first_env=123 SECOND_ENV=testing /Users/sample/scripts/perl_demo.pl",
        "env": [
            "first_env=123",
            "SECOND_ENV=testing"
        ],
        "msg": "",
        "argument_warnings": null,
        "working_directory": null,
        "raw_result": {
            "module": "shell",
            "task": "command",
            "host": "testhost",
            "status": "SUCCESS",
            "argument_warnings": [],
            "env_warnings": [],
            "results": {
                "changed": true,
                "end": "2019-05-09 17:11:01.942516",
                "stdout": "Hello World\nCounted 0 arguments",
                "cmd": "first_env=123 SECOND_ENV=testing /Users/sample/scripts/perl_demo.pl",
                "rc": 0,
                "start": "2019-05-09 17:11:01.900307",
                "stderr": "Hello stderr at /Users/sample/scripts/perl_demo.pl line 7.",
                "delta": "0:00:00.042209",
                "stdout_lines": [
                    "Hello World",
                    "Counted 0 arguments"
                ],
                "stderr_lines": [
                    "Hello stderr at /Users/sample/scripts/perl_demo.pl line 7."
                ]
            }
        }
    }
]