You can set a trap that isn’t or is forbidden in robots.txt. Anyone who ignores your robots.txt is not friendly.
Comment on Running web sites makes me not want to - Lowkey Party
hperrin@lemmy.ca 5 days ago
I host tons of open source projects on forge.sciactive.com
Recently, it was getting absolutely hounded with tons of meaningless requests. I’m assuming they are just results of AI.
So, I put Anubis in front of it. Now images hosted there won’t load on other sites, people have to wait forever to see the code, and the experience on phones is even worse. Just because AI doesn’t respect anything, and the people who run it are too stupid to realize that.
This is why we can’t have nice things.
monkdervierte@lemmy.zip@lemmy.zip 5 days ago
Jason2357@lemmy.ca 5 days ago
This makes a difference on Forgejo: github.com/go-gitea/gitea/pull/34024
Also, obviously, a caching reverse proxy. Forges and wikis are hard because the bots drill down into the thousands of version control pages that get generated on demand and cant be cached.
MrMoopne@lemmus.org 4 days ago
It’s odd, I see so many people complain about Anubis taking a significant amount of time, but I’ve only ever had it 1-2 seconds on my devices
diaphragmwp@discuss.tchncs.de 4 days ago
Then let’s ship your devices.
AHemlocksLie@lemmy.zip 5 days ago
They’re perfectly capable of realizing it. How do you think it got built that way? They don’t respect anything, either.
hperrin@lemmy.ca 5 days ago
True.