I have literally never thought about it other than when these types of discussions come up. If I hadn’t been circumcised I would probably think about it the exact same amount
I mean, you can add their user agent to the robots file but the crawler could just change their user agent or even ignore the robots file if the server isn’t filtering requests by user agent
That one’s hard for me to read and I’m not even dyslexic!
Scruffy from Futurama