A bill arrived for traffic that was not people — how do you keep bots from turning a free tier into an invoice?
Small side project, a few hundred real visitors a month, hosted on a plan that is free up to a point and then charges per request and per gigabyte out.
This month the usage graph is about forty times normal and I have a bill. Looking at the logs, the overwhelming majority is automated: crawlers, something hammering an endpoint that returns a large JSON payload, and a lot of requests for paths that do not exist.
Nothing was hacked, nothing broke, and no human noticed anything. It is purely that a machine somewhere decided to read my site very thoroughly.
What do people actually do about this? I want a hard ceiling more than I want optimisation — the project is not worth an unbounded bill.
@egress_is_the_bill · 3h ago
Look at which line of the bill grew, because it changes what you fix. In almost every case like this it is bandwidth out, not requests, and that points straight at the large JSON endpoint you mentioned.
Things that cut it hard, roughly in order of effort:
Cache the expensive endpoint properly. If that payload is the same for everybody, it should be served from cache with a long lifetime, and the origin should see it once. A single correct cache header can remove most of a bill like this.
Compress it. Large JSON compresses enormously. If it is going out uncompressed, that alone can be a several-fold difference on the metered line.
Paginate or trim it. An endpoint that returns everything is a gift to anything automated, and it is usually returning far more than any real client uses.
Return 404s cheaply. All those requests for paths that do not exist are still being handled by something. Make sure they are not hitting your application or your database on the way to failing.
The reason to do these even after you have a ceiling: a cap protects your wallet by turning the site off. Cutting the per-request cost raises how much abuse you can absorb before that happens, which is the difference between a site that survives being crawled and one that goes dark for the rest of the month.
Reply
Report