Comment on Bing has been revamped to prioritize AI search results – whether you like it or not

• UnderpantsWeevil@lemmy.world ⁨1⁩ ⁨month⁩ ago

  Crawling the web is expensive, and plenty of bigger sites have proprietary deals (Reddit serving exclusive results to Google for instance). Also, since actively hosting data costs money, lots of sites have archived or compressed their offerings. Others have set up higher and higher paywalls, to limit what anyone without a subscription can see.

  The end result is a treasure trove of data that is inaccessible to modern crawlers and scrapers. If you’re not tapping into one of the big search engine catalogs, you’re going to miss a lot of the more attractive results.

  Then there’s the problem of AI crap filling up lots of the spaces that used to be mineable for search results. This isn’t a problem unique to Bing. AI contamination is everywhere and crawlers can’t avoid it easily. What’s a modern search engine to do?

  source