In my past job, we were running about 80 crons (hourly, daily and weekly) across a cluster of 100 machines.We ended up using an external system like Jenkins to schedule and execute the script. The code for the script is a normal function/API call. On a given schedule, Jenkins either SSH's into a particular machine and executes the cron script or simply hits an API (obviously with authentication). If the cron fails, Jenkins sends out an e-mail with the failure details.

I prefer this approach over messing around with a crontab because:

• The notification system is generalized. I configure it once on Jenkins and it remains the same for any cron that we write in the future. I hate Postfix or Sendmail with vengeance. :D
• If you have a distributed cluster, running a cron requires maintaining a distributed cron system. While Dkron exists, I'm not familiar with it and we already use Jenkins as a build system. No harm doubling it as a cron scheduler.
• A ready web interface to look at the logs, history & statistics of past cron jobs. How long did it take to run? Is it taking more time today than a month back? Do we need to optimize our code?
• A ready web interface to manage the cron system. Don't need a cron? Simply disable the build job. Need it on a different schedule? Just change the scheduler on the web. Takes the management pain away from me.

Hope this helps you design a better cron system.

Depends on the job.

For some, manually. Every now and then, I look at the logs. Either that, or someone tells me something isn't working right, and I immediately know the scheduled job failed. Rarely happens, thankfully.

For others, I have emails that come to me daily that go into success or fail folders. For example, jobs that have to connect to other servers, such as via SFTP... I want to know if they fail, so I have the job email me either way (success or fail). I rarely get fails, but I'm sure glad when they come in.

Cronjobs have one task - note time and inform! Fail they will if there is an external factor. Mostly crons are the most rigidly written piece of code after lots of refactoring and testing. They rarely fail, and if they do, it's mostly due to external reasons.

It best suits you to make a custom wrapper suited to your needs to get notifications or record success and failures of the system to keep track of how to improve your success rates of the jobs.

I usually log everything into redis for short term and then use a script to log all the failures into a mongodb collection for later reviewing at the end of the day. Mostly, I see a good lonely 0 count most days. The days I get errors, i anyhow know since the the people who needed to be 'informed' do inform me before hand that they were not informed.

My 2 paise.