a runaway job ran overnight and the 80% budget alert landed in my inbox the next morning, by which point it had been going for nine hours. the budget alerting is clearly not built for detecting things fast, it's built for reporting. what do people actually run that catches a spike inside an hour or two, and what does it cost you to run it?
budgets are a monthly reporting tool and the underlying cost data they read is only refreshed a few times a day, so no amount of tuning gets you an hourly alarm out of them. two things actually page us. anomaly detection catches the shape changes but it's usually next-day, so it's a safety net rather than a first responder. the thing that actually wakes someone up is a scheduled query against the hourly cost and usage report in athena, running every hour, comparing the last hour's spend per service against the trailing median. it's about fifteen lines of sql and a lambda.
the CUR in parquet with hourly granularity is the single best thing you can turn on and most teams leave it off because it looks like a data engineering project. it isn't. one bucket, one glue table, and the athena scan cost for an hourly query over a month of data is measured in cents.
alerting is treating the symptom. an alert tells you money is on fire, it doesn't stop the fire, and at 3am there's an average of twenty minutes between the page and anyone doing anything. spend the same afternoon on guardrails instead: service quotas set deliberately low, reserved concurrency caps on anything that can scale itself, per-project accounts under an organisation so one project cannot bankrupt another, and hard resource limits wherever the provider offers them. on aws specifically there is no true hard spend cap, which means the architecture has to be the cap.
this is why my personal projects live on providers that offer an actual spend limit that stops serving. for a hobby project i'd genuinely rather the site goes down at $20 than stays up at $2,000, and the big clouds are philosophically incapable of offering me that.
my contribution is stupid and common: my budget alerts worked perfectly and went to an address nobody reads. i found four of them sitting in the billing inbox describing the problem in order, hours before i noticed. whatever you build, send it to the same place your production pages go and test it by setting the threshold to a value you'll definitely breach today.
hourly CUR into athena, alert on deviation from the trailing median, page the on-call rotation. quotas so the worst case is bounded while you sleep.
@ines_marchetti · last mo.
the important design decision isn't the tool, it's the threshold. a monthly budget percentage is useless for detection because early in the month everything is under budget and late in the month everything is over. we alert on the trailing seven day median hourly spend per service, and page at four times that for two consecutive hours. that catches a runaway within roughly two hours regardless of where you are in the month, and it has fired three times in eighteen months — twice legitimately, once because a colleague started a genuinely large batch job on a friday without telling anyone.
Reply
Report