> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.itential.com/itential-platform/6/maintain/troubleshoot/network-performance/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.itential.com/_mcp/server. # Troubleshoot network performance > Checklists and database commands for diagnosing and resolving performance issues in Itential Platform. Use the following checklists to help identify and resolve performance issues. Every environment and problem is unique — these checklists provide a baseline to guide you. If you are unable to resolve any problem areas or have questions, contact the Itential [Product Support Team](https://itential.atlassian.net/servicedesk/customer/portal/14). ## Common performance issues Performance issues you may encounter include: * Itential Platform UI loads slowly. * Jobs are running slower than usual. * Apps and their functions are unresponsive or working slowly. * Pronghorn consumes a lot of resources. ## Troubleshooting steps | Step | Suggested action | | :--: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ☐ | For Itential Platform in HA, review all corrective measures on each applicable server instance. Check journalctl, application, MongoDB, and Redis logs to eliminate any known errors. | | ☐ | Inspect available applications and adapters in the Itential Platform profile and ensure all required apps and adapters are "up" and loaded. | | ☐ | Validate system resources (Memory, CPU) on each application, adapter, Redis, and MongoDB instance. Ensure CPU, Memory, and Disk space are adequately available. | | ☐ | Check if any recent changes to apps, adapters, network, or infrastructure triggered the issue. Validate connectivity and network latency between Itential Platform and its dependencies. | | ☐ | Determine the number of jobs triggered for the entire day compared to previous days, especially when jobs are running slower than usual. | | ☐ | Collect the job documents of slow-running jobs from MongoDB and export for Itential to investigate. See [Recovery steps](#recovery-steps). | | ☐ | If the issue is with custom apps, verify the custom app APIs respond correctly from outside Itential Platform using Postman or curl. | | ☐ | Locate any `COLLSCAN` in MongoDB logs indicating missing indexes. Add indexes on collections or queries as required. | | ☐ | Inspect network latency issues using `mtr`. See [My Traceroute (MTR) tool](#my-traceroute-mtr-tool). | | ☐ | Inspect the data payload within tasks to determine if the processed data is too large. | | ☐ | Check the size of the Jobs and Tasks collections and archive them if necessary. | | ☐ | Enable debug logs for investigation when required. | | ☐ | Check the size of the Redis DB and clear keys if they have not been refreshed in a long time. | | ☐ | Check `tcpdump` between two endpoints to identify network-related issues. See [Wireshark tcpdump documentation](https://www.wireshark.org/docs/wsug_html_chunked/AppToolstcpdump.html). | ### My Traceroute (MTR) tool To inspect network latency and packet loss, use `mtr`: ```bash # Install mtr sudo yum -y install mtr # Check packet loss and latency to a destination host mtr --report -T -c 20 192.168.33.25 # Run mtr continuously and check each hop mtr www.google.com ``` Example output: ``` Start: Thu Sep 29 18:43:03 2022 HOST: iap202115 Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.33.25 0.0% 20 2072. 1880. 235.2 2072. 446.7 ``` In this report, packet loss is 0%. Average time to reach the destination is 1880ms, best is 235.2ms, worst is 2072ms. ![](/_fern-img/1058cda45692875b9292827f727a859a4142ad146b5ec0193a35709a6b2ceb67.webp) ## Recovery steps For instances where the cause of a performance issue is unknown: 1. Pause the Task Worker on all Itential Platform instances to ensure in-flight tasks are completed, then restart all instances. 2. If the issue persists, stop Itential Platform and restart all dependencies (MongoDB and Redis). ## Recommendations | Step | Suggested action | | :--: | ----------------------------------------------------------------------------------------------------------------------- | | ☐ | Ensure sufficient resources (RAM, CPU, Disk space) are available on Itential Platform servers. | | ☐ | Increase the number of Itential Platform instances when the number of jobs has significantly increased. | | ☐ | Add more logging to custom apps to identify where they consume the most time. | | ☐ | Use DB queries to identify potential problems such as jobs running in a loop. | | ☐ | Use the `childJob` loop feature as an alternative to the `forEach` task. | | ☐ | Evaluate what the parallel loop type of `childJob` is doing compared to the hardware — it is highly resource intensive. | ## Performance database commands ### Job counts Fetch a count for jobs that have run more than 500 times (Job Metrics in Operations Manager is also available for Itential Platform 2021.2 and higher): ```bash var now = new Date(); var NumberOfDaysAgo = 1; var ReportDuration = 1; var epochday = 24 * 60 * 60 * 1000; var reportStartDate = new Date(now - NumberOfDaysAgo * epochday) - 1; var reportEndDate = new Date(reportStartDate + ReportDuration * epochday) - 1; db.jobs.aggregate([ {"$match": { "metrics.start_time": {$gte: reportStartDate, $lte: reportEndDate} }}, {"$group": { "_id": "$name", "count": {$sum: 1} }}, {"$match": { "count": {$gte: 500} }}, {"$sort": { "count": -1 }} ]); ``` ### Jobs and tasks collection count Fetch the jobs and tasks collection count for the full day plus every hour: ```bash db.getCollection('jobs').find( {$and: [ {"metrics.start_time": {$gte: LOWERBOUNDS}}, {"metrics.start_time": {$lte: UPPERBOUNDS}} ]} ).count() ``` Replace `UPPERBOUNDS` and `LOWERBOUNDS` with epoch time values (in milliseconds) between two date timestamps. ### MongoDB metrics Export job metrics data for analysis. Replace `"JOB-ID-HERE"` with the specific job ID: ```bash mongoexport --db=pronghorn --collection=tasks --fields="_id,metrics,name" --query='{"job._id": "JOB-ID-HERE"}' --out=JOB-ID-HERE.json ``` ### Job velocity Validate job run and completion rates: ```bash db.getCollection('jobs').find({"status": "complete"}).count() db.getCollection('jobs').find({"status": "running"}).count() ``` ## Check core memory usage Use **Admin Essentials** to evaluate core memory usage. ![](/_fern-img/08f4fa87e7a226eaf94698cf0b436ebcc06768bfa8cb3c2929abb84120c3dd59.webp) From the **Profile** view, you can also check memory usage for both Applications and Adapters, and compare it to the memory being used in your local server controls. ![](/_fern-img/9b34f5e02ada78b33079a115b862a7f7f0fb38652e93316c477a2dec4056c3ba.webp) If the memory for an app keeps growing over time without decreasing, there may be a memory leak. Submit an [ISD ticket](https://itential.atlassian.net/servicedesk/customer/portal/14) with Itential for any product apps or adapters showing higher than expected memory use. > Checklists and database commands for diagnosing and resolving performance issues in Itential Platform.