Ask
79
@shopvac_ghost ·

nightly cron stopped running 23 days ago and nothing told me Cron

Nightly job builds a usage digest and emails about 300 accounts. A customer wrote in asking why she had not had one "in a while". It last ran 23 days ago.

There is no failure. No red X, no error, nothing in the run history at all after that date. My error tracking has zero events, because zero events is exactly what happens when nothing runs.

It is a scheduled GitHub Actions workflow. How do people find out about this without a customer telling them?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @merino_max · 3h ago · 3 replies

    Check the Actions tab for a banner saying the workflow was disabled. Scheduled workflows get switched off automatically on repositories with no activity for 60 days, and the notification goes to the repo owner by email, where it is read by nobody. I would put money on that being your 23 days.

    But the cause almost does not matter, because the fix is the same and it is the lesson everybody learns exactly once: monitor absence, not failure.

    The job pings a heartbeat URL as its last successful statement. The monitor alerts when the ping does not arrive inside a window. Failures announce themselves; not running is silent by construction, and no amount of error tracking will ever catch it.

    168
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @lunchboxrotation · 3h ago

      It is not only the disabling either. Scheduled runs can be delayed or dropped under load, so "runs most nights, sometimes 40 minutes late" is the normal state of that scheduler. Fine for a cleanup chore, not fine for something a customer expects in their inbox.

      49
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @spreadsheet_tabs · 21h ago

      The disabled-workflow email goes to the repo owner, so on an org repo it can land on someone who left the company eighteen months ago. Worth checking who that actually is.

      20
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @finops_reyna · 3h ago

    If you do not want another service in the stack: a job_runs table with one row per successful completion, and /health/jobs returning 500 when max(finished_at) < now() - interval '26 hours'.

    You almost certainly already have an uptime monitor screaming at you when an endpoint returns 500. Now it screams about cron too, and you added one table and one route.

    94
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @barre_chord_blues · 3h ago

    Alert on the output, not the process.

    If the digest is the product, count digests sent per day and alert when the count drops below a floor. Process monitoring tells you a thing executed. Output monitoring tells you a customer received something. They fail in different ways - I have had a job run perfectly every night while a mail provider silently dropped everything to one domain.

    57
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @overlap_owen · 3h ago · 2 replies

    Add a second workflow that checks whether the first one ran and opens an issue if it did not.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @otis_haverford · 3h ago

      Which lives in the same repository, under the same inactivity rule, and gets disabled on the same day by the same mechanism. A checker on the same system as the thing it checks is decoration. It has to be somewhere else - even a $5 monitor or a cron on a box you already pay for.

      15
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @secondhandamps · 22h ago

    Also move it off Actions. Scheduled workflows are for maintenance chores where nobody notices a missed night. Anything customer-facing wants a scheduler that is part of your app's platform, where a missed run is visible in the same place as everything else you look at.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report