On the traffic itself, and what actually works versus what feels like it should.
robots.txt only works on things that choose to obey it. Well-behaved crawlers do, and adding a crawl-delay and disallowing your expensive endpoints genuinely helps with those. It does nothing at all for the ones causing the worst of this, so treat it as necessary and insufficient.
Rate limiting by address is the blunt tool that works, and most proxy layers give it away on their free tier. A limit that no human would ever hit costs your real visitors nothing.
Blocking by user agent is nearly useless on its own — anything hammering you will send whatever agent string it likes. Worth doing for the polite ones that identify themselves and that you do not want, and not worth relying on.
Challenge pages are a real option but they cost your human visitors something, and on a small site the cost of a challenge in front of everything is usually worse than the bill. Put it in front of the expensive endpoint rather than the whole site.
One thing worth doing before any of it: check the requests are actually external. A surprising number of these turn out to be something of your own in a retry loop, a monitoring check set to a silly interval, or a preview deployment crawling itself. It is worth ten minutes in the logs, because the fix is completely different and much easier.