In-Process Client-Side Load Balancing: Zalando's 1 Million RPS Architecture

Zalando's engineering team recently published a fascinating architectural deep-dive detailing how they achieved predictable, single-digit-millisecond latency for their Product Read API while handling over 1 million requests per second.

The Problem: Shared Infrastructure and Fan-Out Latency

Zalando's architecture relies on a batch endpoint that fans out a single request into up to 100 parallel calls to individual product pods. Historically, each of these hops transited through Skipper, a shared cluster edge load balancer.

Because Skipper was a shared piece of infrastructure not owned by the product team, diagnosing latency spikes became a massive headache. They could not easily differentiate between actual application latency and network hop overhead within the shared load balancer. At a scale of 100 parallel requests per batch, waiting on the slowest dependency (the 'tail latency' problem) severely degraded the API's overall performance.

The Solution: In-Process Client-Side Load Balancing

To reclaim control over their tail latency and slash infrastructure costs, Zalando moved routing for high fan-out internal traffic in-process using a client-side load balancer. By skipping the shared intermediary for internal pods, they minimized hop overhead and drastically improved visibility into where failures actually originate. Skipper was retained only for edge-facing and single-GET traffic.

SRE Takeaways: Boundary Monitoring and Dependency Isolation

This architectural shift underscores a critical Site Reliability Engineering (SRE) truth: You cannot manage what you cannot isolate. When your application depends on shared internal platforms or third-party cloud infrastructure, monitoring those boundaries is paramount to maintaining tight Service Level Objectives (SLOs).

Here is how Rabbit SaaS helps you implement these exact SRE principles:

• Isolate External Noise with CloudStatusHQ: Just as Zalando struggled to separate Skipper's latency from their application code, your team likely struggles with third-party vendor downtime. CloudStatusHQ acts as your central aggregator for third-party dependency health, ensuring that when AWS, GitHub, or your external APIs experience a hiccup, your team knows immediately—preventing hours of wasted internal debugging.
• Clear Incident Communication with Status Navigator: Major architectural changes, migrations to client-side load balancing, or unexpected platform latency require transparent communication. Status Navigator provides custom-branded incident status pages to keep your internal stakeholders and customers informed, isolating support desks from floodgates of redundant tickets.
• Prevent Silent Background Failures with Cron Rabbit: Running high-throughput data syncs or background caches that power these client-side routing maps? Use Cron Rabbit to ensure your background cron jobs and synchronizations never fail silently in the background, keeping your client-side configurations fresh.

By ensuring deep observability across both internal configurations and external dependencies, your team can achieve Zalando-level resilience at any scale.