Back to Main
Learn System Design
In a Hurry
Core Concepts
Key Technologies
Patterns
Advanced Topics
Get Premium
Common Problems
Design a Distributed Job Scheduler Like Airflow
Understanding the Problem
There are two key terms worth defining before we jump into solving the problem:
- Task: A task is the abstract concept of work to be done. For example, "send an email". Tasks are reusable and can be executed multiple times by different jobs.
- Job: A job is an instance of a task. It is made up of the task to be executed, the schedule for when the task should be executed, and parameters needed to execute the task. For example, if the task is "send an email" then a job could be "send an email to john@example.com at 10:00 AM Friday".
The main responsibility of a job scheduler is to take a set of jobs and execute them according to the schedule.
Functional Requirements
Core Requirements
- Users should be able to schedule jobs to be executed immediately, at a future date, or on a recurring schedule (ie. "every day at 10:00 AM").
- Users should be able monitor the status of their jobs.
Below the line (out of scope)
- Users should be able to cancel or reschedule jobs.
Non-Functional Requirements
Now is a good time to ask about the scale of the system in your interview. If I were your interviewer, I would explain that the system should be able to execute 10k jobs per second.
The Set Up
Planning the Approach
Defining the Core Entities
The API
Data Flow
High-Level Design
1) Users should be able to schedule jobs to be executed immediately, at a future date, or on a recurring schedule
2) Users should be able monitor the status of their jobs.
Potential Deep Dives
1) How can we ensure the system executes jobs within 2s of their scheduled time?
2) How can we ensure the system is scalable to support up to 10k jobs per second?
3) How can we ensure at-least-once execution of jobs?
What is Expected at Each Level?
Mid-level
Senior
Staff+
Not sure where your gaps are?
Mock interview with an interviewer from your target company. Learn exactly what's standing in between you and your dream job.
Schedule a mock interview
Meet with a FAANG senior+ engineer or manager and learn exactly what it takes to get the job.
© 2025 Optick Labs Inc. All rights reserved.
Loading comments...