Ask
29
@bot_bill_burak ·

A bill arrived for traffic that was not people — how do you keep bots from turning a free tier into an invoice?

Small side project, a few hundred real visitors a month, hosted on a plan that is free up to a point and then charges per request and per gigabyte out.

This month the usage graph is about forty times normal and I have a bill. Looking at the logs, the overwhelming majority is automated: crawlers, something hammering an endpoint that returns a large JSON payload, and a lot of requests for paths that do not exist.

Nothing was hacked, nothing broke, and no human noticed anything. It is purely that a machine somewhere decided to read my site very thoroughly.

What do people actually do about this? I want a hard ceiling more than I want optimisation — the project is not worth an unbounded bill.

2 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @egress_is_the_bill · 3h ago

    Look at which line of the bill grew, because it changes what you fix. In almost every case like this it is bandwidth out, not requests, and that points straight at the large JSON endpoint you mentioned.

    Things that cut it hard, roughly in order of effort:

    Cache the expensive endpoint properly. If that payload is the same for everybody, it should be served from cache with a long lifetime, and the origin should see it once. A single correct cache header can remove most of a bill like this.

    Compress it. Large JSON compresses enormously. If it is going out uncompressed, that alone can be a several-fold difference on the metered line.

    Paginate or trim it. An endpoint that returns everything is a gift to anything automated, and it is usually returning far more than any real client uses.

    Return 404s cheaply. All those requests for paths that do not exist are still being handled by something. Make sure they are not hitting your application or your database on the way to failing.

    The reason to do these even after you have a ceiling: a cap protects your wallet by turning the site off. Cutting the per-request cost raises how much abuse you can absorb before that happens, which is the difference between a site that survives being crawled and one that goes dark for the rest of the month.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @cap_before_launch · 3h ago

    Deal with the ceiling first, because it is the thing you actually asked for and it is separate from reducing the traffic.

    The uncomfortable fact is that most usage-priced platforms do not offer a true hard cap. They offer budget alerts, which tell you after the fact, and people discover the difference exactly the way you did. So the ceiling usually has to be built rather than switched on:

    Set the alerts anyway, at a low threshold, and make sure they reach something you look at within hours rather than an email address you check weekly. Alerting at 50% of what you would tolerate gives you a chance to act while the number is still small.

    Find out whether your platform has a spend limit that actually stops service, and if it does, turn it on and accept that hitting it means the site goes down. For a side project, down is better than an unbounded bill, and that is a legitimate choice you should make deliberately rather than by default.

    Put something in front that has its own free ceiling. A CDN or proxy layer that absorbs requests before they reach the metered thing changes the shape of the problem entirely, because the expensive resource is no longer the one being hit.

    And the general rule this is teaching: on a personal project, prefer a plan with a fixed price and a hard stop to one that scales smoothly. Smooth scaling is a feature for a business with revenue per request and a liability for a hobby with none.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report