nightly cron stopped running 23 days ago and nothing told me Cron
Nightly job builds a usage digest and emails about 300 accounts. A customer wrote in asking why she had not had one "in a while". It last ran 23 days ago.
There is no failure. No red X, no error, nothing in the run history at all after that date. My error tracking has zero events, because zero events is exactly what happens when nothing runs.
It is a scheduled GitHub Actions workflow. How do people find out about this without a customer telling them?
@merino_max · 3h ago · 3 replies
Check the Actions tab for a banner saying the workflow was disabled. Scheduled workflows get switched off automatically on repositories with no activity for 60 days, and the notification goes to the repo owner by email, where it is read by nobody. I would put money on that being your 23 days.
But the cause almost does not matter, because the fix is the same and it is the lesson everybody learns exactly once: monitor absence, not failure.
The job pings a heartbeat URL as its last successful statement. The monitor alerts when the ping does not arrive inside a window. Failures announce themselves; not running is silent by construction, and no amount of error tracking will ever catch it.
Reply
Report
@lunchboxrotation · 3h ago
It is not only the disabling either. Scheduled runs can be delayed or dropped under load, so "runs most nights, sometimes 40 minutes late" is the normal state of that scheduler. Fine for a cleanup chore, not fine for something a customer expects in their inbox.
Reply
Report
@spreadsheet_tabs · 21h ago
The disabled-workflow email goes to the repo owner, so on an org repo it can land on someone who left the company eighteen months ago. Worth checking who that actually is.
Reply
Report