If this were true (which is nearly impossible since you said “all"), stuff like Anubis wouldn’t exist since you could just toss up a crowd-sourced robots.txt and be done with it.