RCA-Lab Open Sourced: A Realistic Sandbox for SRE Failure Simulation

A highly realistic sandbox for root cause analysis (RCA) and observability testing has been open-sourced by the team at Coroot under the Apache 2.0 license. Appropriately named RCA-lab, this platform addresses a long-standing frustration in the SRE community: standard benchmarking tools (like the classic OpenTelemetry demo shop) rely heavily on feature flags to fake errors, rather than producing actual system strain.

What Makes RCA-Lab Different?

Unlike sterile synthetic demos, RCA-lab runs on Kubernetes with actual microservices built in Go, Java, Python, Node.js, Rust, and PHP. It utilizes real databases subjected to real load generators, allowing teams to trigger authentic operational headaches, including:

• Database Locks and Replication Lag
• Memory Leaks
• Bad Code Deployments
• Network Latency and Outages

All services come instrumented with OpenTelemetry SDKs, meaning you can direct the OTel Collector at your own observability stack and test how quickly your team—or your AI-driven observability tools—can diagnose the root cause.

Elevating the Sandbox with Rabbit SaaS

Simulating failures in a sandbox is the first step toward high availability; the second is validating your external communication and background automation stacks during these simulated disasters. This is where Rabbit SaaS bridges the gap:

1. Game-Day Communication with Status Navigator When testing outages inside RCA-lab, incident response training shouldn't stop at the CLI. Connect your alerting pipelines to Status Navigator to practice incident communication. Ensure your custom-branded public status pages update seamlessly, giving internal stakeholders and customers immediate transparency without manual intervention.

2. Catching Silent Background Failures with Cron Rabbit Simulated database locks and memory leaks in RCA-lab often degrade the performance of background workers and cron jobs. While standard APM traces might highlight database sluggishness, they can miss silent background task failures. By integrating Cron Rabbit via curl pings, you can immediately detect when simulated microservice failures prevent crucial background tasks from checking in.