As someone with a public facing website, there are significant volumes of scraping still happening. But largely this appears to come out of South East Asia and South America and they take steps to hide who they are so it’s not clear who is doing it or why, but like you say it doesn’t appear to be OpenAI, Google, etc.

It doesn’t appear to be web search indexing, the scraping is aggressive and the volume will bring down a Lemmy server no matter how powerful the hardware.