links: [[System Design MOC]] --- # Requirements ## Functional - Should be able to take 1000 rps - It should persist the jobs and run during the scheduled time - The system should be read heavy - Results of the job should stored and can be queried ## Non Functional - Scalability: Thousands or even millions of jobs can be scheduled and run per day - Durability: Jobs must not get lost -> we need to persist jobs - Reliability: Jobs must not be executed much later than expected or dropped -> we need a fault-tolerant system - Availability: It should always be possible to schedule and execute jobs -> (dynamical) horizontal scaling - Jobs must not be executed multiple times (or such occurences should be kept to a minimum) # Domain Analysis: *Concepts* Job: - Represents a Job to be executed - Properties - Id, name, JobExecutorClass, Priority, Running, LastStartTime, LastEndTime, LastExecutor, Data(Params) Trigger (based on the concept Quartz Scheduler uses): - Defines when a job is executed - We can define different triggers like One Time or Cron Time - Based on the type we have properties like - id, type, start time, end time, one time, cron job, interval Executor: - is a single job executor/worker node - can have properties like: - id, last heartbeat High-Level Design  **Microservices that want to schedule a non-/recuring Job**_: Can send a Message (or produce in Kafka terminology) to the corresponding Kafka Queue (precisely a Topic). **Job Scheduler Service:** Consumes the Jobs from the Kafka Queue and puts in the Database **RDBMS**: Save the Jobs here **Job Executor Service**: - On Startup picks the pending jobs to execute - --- tags: #job-processing #cron #asynchronous #scheduled-tasks sources: - [LeetCode Distributed Job Scheduler](https://leetcode.com/discuss/general-discussion/1082786/System-Design:-Designing-a-distributed-Job-Scheduler-or-Many-interesting-concepts-to-learn) - [Async Tasks Framework](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox) - [Job Scheduling System](https://towardsdatascience.com/ace-the-system-design-interview-job-scheduling-system-b25693817950) -