The Cascade Effect: What Supabase's 25-Day Incident Teaches Us About SRE and Transparency
In the world of Site Reliability Engineering (SRE), we often talk about cascading failures. But rarely do we see a real-world case study as illustrative—and ultimately as praised for its honesty—as Supabase's recent multi-week incident cycle.
What started as an issue with projects returning 401 errors for valid JSON Web Tokens (JWTs) ballooned into a 25-day saga of rollbacks, stale caches, and environment mismatches. Here is a breakdown of what happened, the SRE lessons we can extract, and how modern tooling helps manage the fallout.
The Incident Timeline: A Chain Reaction
On August 14, Supabase identified an issue causing JWT rejections. To resolve it, they deployed a fix (PostgREST 14.17). However, this fix introduced "unintended performance side effects," forcing them to revert back to version 14.5.
Unfortunately, the act of reverting triggered a second, distinct incident. From August 27 to 31, users experienced elevated latency and 525 errors, which were eventually traced back to a stale time cache. During this same turbulent period, Supabase dealt with:
- GLIBC version mismatches (missing 2.34 and 2.33 on separate days, impacting supautils).
- Six additional incidents between late August and early September, including an eight-hour freeze on project lifecycle operations.
SRE Takeaways: Complex Systems Fail in Complex Ways
1. Rollbacks Are Not Risk-Free
In theory, reverting to a previous "known good" state is the safest recovery path. In practice, stateful systems, dependency trees, and caches can drift during the window of the upgrade. As Supabase discovered, a stale time cache can lie dormant until a system restart or revert exposes it. Always test the rollback path as thoroughly as the deployment path.
2. Guard Against Environment Drift
The recurring GLIBC mismatches suggest variance in the build environments or container base images. Standardizing build pipelines and continuously auditing base image dependencies is crucial to preventing runtime environment failures during emergency redeployments.
3. Radical Transparency Builds Trust
Despite the length of the incident, the SRE community widely praised Supabase. Why? Because instead of dividing the issues into minor, disconnected updates to artificially preserve SLA appearances, they explicitly linked the incidents on their status page. They openly admitted that their fix had failed and caused the subsequent regression.
How Rabbit SaaS Keeps Your Systems—and Communications—Resilient
At Rabbit SaaS, we build tools that empower SREs to navigate complex failures and maintain user trust:
- Status Navigator: Supabase's crisis communication succeeded because of their status page. With Status Navigator, you can deploy beautiful, custom-branded incident status pages that allow you to link related incidents, post transparent post-mortems, and keep customers informed during multi-day events.
- CloudStatusHQ: When core infrastructure providers experience cascading outages, downstream applications suffer. If you depend on third-party cloud services, CloudStatusHQ aggregates external vendor health alerts into a unified dashboard, giving your team instant visibility into downstream failures before your customers start filing tickets.
Maintaining 100% uptime is impossible, but maintaining 100% transparency is a choice. Ensure your team has the tools to monitor dependencies and communicate clearly when things go sideways.