The problem being they’re now attempting anti-fingerprinting tactics. A lot of the AI crawlers used to identify themselves as Amazon/openAI/etc. And aren’t anymore because they were being blocked. Now they’re country from random IPs with random/obfuscated agent ids.

This is a legal problem not a technological one.