This topic explains how to iteratively adjust rate limits to optimize performance.
For initial configuration, see Configure rate limiting.
Iterative tuning process
1Start with conservative rate limit values based on your system capacity.
2Monitor system resources under typical load: CPU utilization, memory usage, and disk I/O.
3Gradually increase the rate limit if resources allow.
4Test with realistic workflow volumes.
5Adjust based on observed patterns.
Signs to increase rate limit
- Tasks consistently queuing during normal operation
- Queue depth steadily increasing
- System resources underutilized (less than 70% CPU)
- Task workers have idle capacity
Signs to decrease rate limit
- Sustained high CPU utilization (greater than 80%)
- Memory pressure or out-of-memory errors
- Frequent task failures
- Downstream services reporting overload
- System becoming unresponsive
When optimal configuration is achieved
- Workflows complete in acceptable timeframes
- System resources well-utilized (60–80% CPU)
- Queue depths stable or decreasing
- Minimal rate limiting impact during normal operation
- System responsive during peak loads
When to adjust specific parameters
Increase task rate limit
- Resources underutilized and workflows slow
- Queue depths consistently high
- Business requirements not being met
Decrease task rate limit
- System showing signs of overload
- High resource utilization
- Downstream services struggling
Adjust rate limit period
Adjust the period only if you need to rate limit to slower than one task per second:
task_worker_rate_limit=1 with task_worker_rate_limit_period=2 = one task per two secondstask_worker_rate_limit=1 with task_worker_rate_limit_period=10 = one task per 10 seconds
For rates of one task per second or faster, keep the default period of one second.
