ColinHayhurst
@ColinHayhurst@lemmy.world
- Comment on Google is no longer asking — feed the AI or you’re not in search results 2 months ago:
Yes.
- Comment on Google is no longer asking — feed the AI or you’re not in search results 2 months ago:
Some discussion on that here: lemmy.world/comment/11859761
- Comment on Google is no longer asking — feed the AI or you’re not in search results 2 months ago:
Where is your evidence for that? It used to be Bing and Yandex, but now it’s just Bing. They use other non search engine APIs and do a small amount of crawling AFAIK. Details of who uses what here: seirdy.one/…/search-engines-with-own-indexes/
- Comment on Google is no longer asking — feed the AI or you’re not in search results 2 months ago:
Put should put these entries into your robots.txt file.
To block the Google search crawler use for all of your site:
User-agent: Googlebot Disallow: /
To block the Google AI crawler use:
User-agent: Google-Advanced Disallow: /
- Comment on Any “small-web” search engines? 2 months ago:
Yes, it was. Matt Wells closed it down just over one year ago.
- Comment on Any “small-web” search engines? 2 months ago:
yep, in footer “© 2024 Infospace Holdings LLC, A System1 Company”
- Comment on Any “small-web” search engines? 2 months ago:
system1.com adtech company syndicating Bing and/or Google
- Comment on Any “small-web” search engines? 2 months ago:
We’d love to build a distributed search engine, but it would be too slow I think. When you send us a query we go and search 8 billion+ pages, and bring back the top 10, 20…up to 1,000 results. For a good service we need to do that in 200ms, and thus one needs to cenetralise the index. It took years, several iterations and our carefully designed algos & architecture to make something so fast. No doubt Google, Bing, Yandex & Baidu went through similar hoops.