Designing an API capable of handling 1 million requests per second (RPS) is a classic system design challenge. It tests your understanding of scalability, fault tolerance, performance optimization, and cost efficiency.
At this scale, no single optimization is enough. Success comes from layered design, where each component reduces load, latency, or failure risk. This article walks through the core architectural principles commonly expected in system design interviews and real-world high-scale systems.
Load balancing is the first line of defence against traffic overload.
How it helps:
Distributes incoming traffic evenly across multiple backend servers.
Prevents any single instance from becoming a bottleneck.
Enables horizontal scaling and fault tolerance.
Common approaches:
Layer 4 (TCP/UDP) load balancers for speed.
Layer 7 (HTTP) load balancers for routing based on paths, headers, or regions.
Key considerations:
Health checks to avoid sending traffic to unhealthy nodes.
Sticky sessions only when absolutely necessary.
Handling 1M RPS is not about bigger servers—it’s about more servers.
How it works:
Add more API instances instead of increasing CPU or memory on a single machine.
Stateless services allow any request to be handled by any instance.
Best practices:
Keep APIs stateless (externalize session data).
Use auto-scaling based on CPU, memory, or request rate.
Design for failure—instances should be disposable.
Why it matters:
Horizontal scaling is the foundation that makes high throughput achievable.
Caching dramatically reduces load on databases and backend services.
Where caching fits:
In-memory caches (Redis, Memcached) between API and database.
Cache frequently accessed or computationally expensive data.
Benefits:
Lower latency responses.
Reduced database contention.
Higher throughput at lower cost.
Things to watch out for:
Cache invalidation strategies.
TTL (time-to-live) configuration.
Avoid caching highly volatile data unnecessarily.
A CDN offloads traffic before it ever reaches your API.
What a CDN does:
Serves static and cacheable content from edge locations close to users.
Reduces latency and origin server load.
Typical use cases:
Static assets (images, CSS, JS).
Public API responses that are cacheable.
Edge-side caching for GET requests.
Impact at scale:
A well-configured CDN can absorb a significant percentage of total traffic, making 1M RPS far more achievable.
Not all requests need immediate processing.
How async helps:
API responds quickly after validating and enqueueing a task.
Heavy work is handled by background consumers.
Examples:
Event logging
Notifications
Analytics
Payment processing
Email or webhook delivery
Tools commonly used:
Message queues (SQS, RabbitMQ)
Streaming platforms (Kafka, Pulsar)
Why it matters:
Async processing smooths traffic spikes and prevents slow operations from blocking API responses.
At high scale, protecting the system is as important as serving traffic.
Purpose of rate limiting:
Prevent abuse and accidental overload.
Ensure fair usage across clients.
Protect downstream dependencies.
Common strategies:
Token bucket
Leaky bucket
Fixed or sliding window counters
Where to enforce it:
At the API gateway
At the load balancer
At the CDN edge
Reducing response size has a massive impact at scale.
Why it matters:
Smaller payloads = lower bandwidth usage.
Faster serialization and deserialization.
Improved latency for clients.
Best practices:
Avoid over-fetching.
Use field selection (e.g., GraphQL or query parameters).
Compress responses (gzip, Brotli).
Remove unused metadata.
At 1M RPS, even saving a few bytes per response adds up quickly.
Handling 1 million requests per second is not about a single technology—it’s about systemic efficiency.
Load balancers distribute traffic.
Horizontal scaling ensures capacity and resilience.
Caching layers reduce backend pressure.
CDNs absorb traffic at the edge.
Asynchronous processing prevents blocking and smooths spikes.
Rate limiting protects system stability.
Lean responses improve performance at scale.
In system design interviews, the key is to explain how these pieces work together, how they evolve as traffic grows, and how you balance performance, reliability, and cost.
At scale, architecture isn’t just about handling traffic—it’s about handling it sustainably