

do they just want everything to be crawled
Yes. Web crawling has been a normal and vital part of the web from day 1. We’d have no search engines without crawlers.
The web is user-centric by design. I’m sick of tech companies trying to flip the script and hoard information, most of which is not theirs to begin with (e.g. Google, Reddit, Twitter, Facebook, etc.).
Kind of funny how quickly we’ve flipped from “you should treat LLM output like advice from a random stranger” to “you should treat advice from a random stranger like LLM output”.
Either way, it’s the right idea. If you can’t understand what you’re doing but you do it anyway, you’re going to create all kinds of problems for yourself.