Key Takeaways
- 100ms latency feels instantaneous; 1+ second causes user frustration
- P99 latency matters more than averages - 1% of 1M requests = 10,000 slow requests
- External APIs are typically the biggest latency contributors (50-200ms P50)
- Parallelization can dramatically reduce end-to-end latency
- Always reserve 10-20% buffer for unexpected delays
Understanding API Latency Budgets
In distributed systems and microservices architectures, managing latency is critical for user experience and system reliability. A latency budget is the total time allocated for a request to complete, distributed across all components in the service chain. This guide explains how to plan, allocate, and optimize latency budgets effectively.
Why Latency Budgets Matter
User Experience Impact
- 100ms: Feels instantaneous to users
- 300ms: Noticeable but acceptable
- 1 second: Users notice delay, may lose focus
- 3+ seconds: Significant user frustration, abandonment
Business Impact
Studies show that latency directly affects business metrics:
- Amazon: 100ms latency = 1% sales decrease
- Google: 500ms delay = 20% traffic decrease
- Mobile users are even more sensitive to latency
Understanding Percentiles
What Percentiles Tell You
| Percentile | Meaning | Use Case |
|---|---|---|
| P50 (Median) | 50% of requests faster | Typical user experience |
| P90 | 90% of requests faster | Most users' experience |
| P95 | 95% of requests faster | SLA targets |
| P99 | 99% of requests faster | Tail latency, worst cases |
| P99.9 | 99.9% of requests faster | Extreme outliers |
Why P99 Matters More Than Average
- Averages hide outliers and tail latency
- High-traffic systems have many P99 occurrences daily
- 1% of 1 million requests = 10,000 slow requests
- Slow requests often cascade into bigger problems
Components of API Latency
Network Latency
- Same datacenter: 0.5-2ms per hop
- Cross-region: 20-100ms per hop
- Cross-continent: 100-300ms per hop
- DNS resolution: 10-50ms (uncached)
- TLS handshake: 10-50ms
Data Access
| Operation | Typical P50 | Typical P99 |
|---|---|---|
| Redis/Cache GET | 0.5-1ms | 2-5ms |
| DB indexed query | 2-10ms | 20-100ms |
| DB full scan | 50-200ms | 500ms+ |
| External API | 50-200ms | 500-2000ms |
Optimization Techniques
1. Parallelization
Run independent operations concurrently:
- Parallel database queries
- Concurrent external API calls
- Async processing where possible
- Promise.all() / async.parallel patterns
2. Caching Strategies
- Application-level caching
- Distributed cache (Redis, Memcached)
- CDN for static content
- Database query caching
3. Connection Optimization
- Connection pooling
- Keep-alive connections
- HTTP/2 multiplexing
- gRPC for internal services
Monitoring and Alerting
Key Metrics to Track
- P50, P95, P99 latencies per endpoint
- Error rates correlated with latency
- Upstream dependency latencies
- Queue wait times
Alert Thresholds
| Severity | Trigger | Action |
|---|---|---|
| Warning | P95 > 80% of budget | Investigate trending |
| Error | P95 > 100% of budget | Immediate investigation |
| Critical | P99 > 150% of budget | Incident response |
Conclusion
Effective latency budget management requires understanding your system's components, measuring actual performance, and continuously optimizing. Start by establishing a realistic budget based on user requirements, allocate it thoughtfully across your service chain, and monitor to ensure you're meeting targets. Remember that latency optimization is an ongoing process - as your system evolves, your budgets should be revisited and adjusted.