Login
Contents x
Archiving & Purging
  • 14 Aug 2023
  • Dark
    Light
  • PDF
Contents

Archiving & Purging

  • Updated on 14 Aug 2023
  • Dark
    Light
  • PDF
Article Summary

Retention policies in place for archiving, backing up files and purging should also include your IAP and IAG assets. This guide provides several best practices and suggested methods of complying with such policies for on-premise IAP/IAG installations.

⚠ The Itential platform is not a system of record. Itential strongly recommends that you specify a retention policy that suits your business needs.

Itential Automation Platform (IAP)

IAP uses MongoDB to store application and instance data. Itential recommends that regular backups of the Mongo database are performed and backup data is stored in a separate location for safekeeping and recovery requirements. The specific IAP .bin file used for installation should also be retained to facilitate a re-install to exact specifications should restoration be necessary.

Itential Automation Gateway (IAG)

IAG maintains host files, scripts, playbooks, and custom modules on a Linux file system. Those file locations should be backed up periodically at scheduled times using cron jobs.

Data Storage Size

In the broader context of data preservation and ensuring accessible data backups:

  • Check the configuration and default directory size. The maximum storage size for backup data is often limited.
  • Avoid archiving long running workflows.
  • Make large job documents modular. In MongoDB, GridFS is used for storing files larger than 16 MB.

Log Rotation

Log data should be gathered from a rolling (rollover) log. As a recommended default, a rolling log should be distributed across 99 files over time (this number can be adjusted lower). To keep the logging system within specified space limits, rolled log files can be emptied (purged) whenever the log directory reaches a particular size and space is needed, or whenever the active log file generated by debug and trace has reached its file size threshold.

Checklist

Use the following checklist as an aid to help guide you through the process of archiving and performing backup operations.

Operation Recommendations
☐ Take snapshot and dump MongoDB data. - Can be the entire Mongo database, if no CI/CD process in place to repair workflows, JSTs, or forms.
- Can also be only jobs, tasks, Golden Config compliance collections, and LifeCycle Manager entities (2023.1).
-All data files should be zipped and stored in a separate location.
☐ Set log rolling to 99 files. - Rollover log threshold can be adjusted to a lower file number.
- Configure a time-based policy to purge rollover logs.
☐ Archive IAG assets. - A CI/CD process should be used to manage hosts, scripts, playbooks, etc.
- All assets should be zipped and placed in a storage archive for snapshot purposes.
☐ Backup device inventory ⚠ If using an internal inventory in IAG, backup the SQLite database that holds the device data.
☐ Set a frequency for retention and purging. - 30 days is the minimum standard for purging stale data.
- 60-90 days or longer is the minimum standard for retaining data, depending on storage, organizational policy, and business needs.

Collections

Itential recommends the following list of MongoDB collections be included in a data backup.

Collection Name Description
job_history Number of jobs in the system over a period of time.
job_output Output generated after running a workflow.
jobs Job documents.
wfe_job_metrics Workflow Engine metrics data for workflows.
wfe_task_metrics Workflow Engine performance data on tasks within workflows.
ucm_configs Configuration backups.
ucm_compliance_reports Compliance reports.

Additional Collections

While additional collections from Lifecyle Manager (LCM) can be archived and cleaned (purged), the cadence can vary based on policy and business need. Some will find this type of data is valid for several years, while others may only want to maintain LCM data that covers the last 12-months. Since LCM collections are comprised of integral state data and resource models for network configuration, Itential strongly recommends that you specify what LCM collections to store, where they should be stored and for how long.

Sample Commands

The following is a list of frequently used commands to support your backup strategy and preserve data.

MongoDB

To run mongodump:

mongodump --db=<old_db_name> --collection=<collection_name> --out=data/

To run mongorestore:

mongorestore --db=<new_db_name> --collection=<collection_name> data/<db_name>/<collection_name>.bson

Linux

To enter edit mode for a Cron job:

crontab -e

To backup IAG inventory:

sqlite3 iag.sq3 ".backup 'inventory'"

To restore IAG inventory:

sqlite3 iag.sq3 ".restore 'inventory'"

Itential Pre-builts

This pre-built allows IAP users to archive the jobs and tasks collections:

Was this article helpful?
What's Next
Privacy Policy
Cookie Policy
Terms of Use
EULA
Copyright © 2022 Itential for Developers
Change password!
Changing your password will log you out immediately. Use the new password to log back in.
Change profile
Success!
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.
Logout