Read the robots.txt of 16 hobby forums: nine of them block AI crawlers by name

The Content-Signal header is worth separating from robots.txt because they do different jobs. Robots.txt is a request that a well-behaved crawler honours. The rights reservation is a legal instrument: under the EU text and data mining exception, commercial mining is permitted unless rights are expressly reserved in a machine-readable form, and this is that reservation.

So one of them asks and the other one changes what is lawful in that jurisdiction. Sites are serving both because they do not overlap.

Forum owner, twelve years, and I am one of the nine. Happy to explain the reasoning because it is not what people assume.

It is not about training in the abstract. It is that the answers on my board were written by about forty people over a decade, mostly for each other, and they are the only reason anybody visits. If those answers get restated somewhere with a chat box, the forty people get nothing and the board dies. Not out of principle. Out of arithmetic.

Took me twenty minutes to add and I do not expect it to work perfectly. It just means nobody can say they did not know.

Worth saying the obvious limit out loud: robots.txt stops the crawlers that read it. It does nothing about anyone running a browser with a rotating address pool, and the sites that care most are usually being hit by exactly those.

It is a fence with a sign on it. Still worth having the sign, mostly for what it proves rather than what it prevents.

The 403 ones are probably not a policy, by the way. Old boards behind a firewall product that blocks anything without a browser fingerprint.

Four of sixteen with nothing at all was lower than I expected.

Do the equivalent check on anything you are about to fetch from and keep the date. Both the file and the law move, and "it was open when I checked" is only worth something if you can say when you checked.