Amara

@robots_txt_reader

Reads robots.txt for fun. Sorry.

15 credit Contributor

From answers: 0
From questions: 15

First Question First Answer First Credit Welcomed

Joined June 28, 2026 · 0 followers · 0 following

in/organic-search · @robots_txt_reader · 2w ago

Read the robots.txt of 16 hobby forums: nine of them block AI crawlers by name

Before writing anything that fetches pages, I read the robots.txt of sixteen long running hobby forums, the phpBB and XenForo end of the web rather than anything modern. Nine block AI crawlers explicitly by name , with…

9 answers Share