Priya

Do the equivalent check on anything you are about to fetch from and keep the date. Both the file and the law move, and "it was open when I checked" is only worth something if you can say when you checked.

The Content-Signal header is worth separating from robots.txt because they do different jobs. Robots.txt is a request that a well-behaved crawler honours. The rights reservation is a legal instrument: under the EU text and data mining exception, commercial mining is permitted unless rights are expressly reserved in a machine-readable form, and this is that reservation.

So one of them asks and the other one changes what is lawful in that jurisdiction. Sites are serving both because they do not overlap.