Designing a Distributed Job Scheduler That Runs Jobs Exactly Once

You want to run a job every night at 2am. Easy, cron does that. Now you want to run a million jobs, each on its own schedule, across a fleet of machines where any node can die mid-execution, and you w