This topic explains how to iteratively adjust rate limits to optimize performance.
For initial configuration, see Configure rate limiting.
Iterative tuning process
- Start with conservative rate limit values based on your system capacity
- Monitor system resources under typical load:
- CPU utilization
- Memory usage
- Disk I/O
- Gradually increase rate limit if resources allow
- Test with realistic workflow volumes
- Adjust based on observed patterns
Signs to increase rate limit
- Tasks consistently queuing during normal operation
- Queue depth steadily increasing
- System resources underutilized (<70% CPU)
- Task workers have idle capacity
Signs to decrease rate limit
- Sustained high CPU utilization (>80%)
- Memory pressure or out-of-memory errors
- Frequent task failures
- Downstream services reporting overload
- System becoming unresponsive
When optimal configuration is achieved
- Workflows complete in acceptable timeframes
- System resources well-utilized (60-80% CPU)
- Queue depths stable or decreasing
- Minimal rate limiting impact during normal operation
- System responsive during peak loads
When to adjust specific parameters
Increase task rate limit
- Resources underutilized and workflows slow
- Queue depths consistently high
- Business requirements not being met
Decrease task rate limit
- System showing signs of overload
- High resource utilization
- Downstream services struggling
Adjust number of task workers
- CPU underutilized with single worker
- Task queue depth high but CPU available
- Want better concurrency for task processing
Adjust rate limit period
Adjust the period only if you need to rate limit to slower than 1 task per second:
task_worker_rate_limit=1withtask_worker_rate_limit_period=2= 1 task per 2 secondstask_worker_rate_limit=1withtask_worker_rate_limit_period=10= 1 task per 10 seconds
For rates of 1 task per second or faster, keep the default period of 1 second.