Scheduling cron jobs on a single Linux server is simple: you edit the /etc/crontab file, write a bash script, and you are done.
But as your SaaS grows, a single server turns into an auto-scaled cluster of virtual machines or containers. If you leave your crontab unmodified, your scheduled tasks will run on every single container simultaneously, causing duplicate database updates and high load spikes.
Here is how to design cron infrastructure that scales reliably across any infrastructure.
1. The Distributed Lock Pattern
If your cron job must run on your web servers, you can prevent duplicate execution by acquiring a distributed lock using Redis or Memcached before running the task.
- Start: The cron job triggers on all servers at 12:00 AM.
- Acquire Lock: Each server attempts to set a unique key in Redis:
SET key_name "locked" EX 300 NX(NX makes it write only if it doesn't exist). - Execute: Only the server that successfully set the lock runs the job. The other servers fail to acquire the lock and exit immediately.
2. Centralized Task Scheduling
Instead of scheduling jobs locally on each server, migrate to a centralized task scheduler:
- Cloud Schedulers: Use managed services like AWS EventBridge, Google Cloud Scheduler, or Vercel Crons to send HTTP requests to a secure webhook endpoint on your application.
- Orchestration Tools: Use Kubernetes CronJobs, which spin up a temporary container specifically to run the task and destroy it once complete.
3. Passive Heartbeat Auditing
Centralized schedulers can still fail if the API endpoint returns an error, if the container fails to pull its Docker image, or if the queue broker is saturated.
Always keep an independent external auditor (like CronRabbit) watching the results. By monitoring active heartbeats, you get alerted the moment scheduling logic fails, regardless of whether you're using Kubernetes, serverless API routes, or standard crontabs.