Comment

Comment on OpenAI finally admitted they're crawling the web to profit off of GPT. Block it from your sites using robots.txt.

<- View Parent

empireOfLove@lemmy.one ⁨1⁩ ⁨year⁩ ago

There’s a significant difference in the purpose of the scraping.

Google et.al run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue. People view content directly from those who created it, meaning those creators (regardless of whoever they are) get full credit. Yes, Google makes money too, but site owners are not left in the cold.

ChatGPT and other LLM’s works by combing its huge database of known content its “learned” to cook up an answer through fast math magic. Content it scrapes to populate this database can be regurgitated at any time, only now its been completely processed and obfuscated to an insane degree. Any attribution of content is completely stripped in the final product, even if it ends up being a word-for-word reproduction. Everything OpenAI charges for its LLM goes directly to OpenAI, and those who have created content to train it will never even know it was used without their consent.

Essentially, LLM’s operate like a huge middle school plagiarism machine, only now they’re making billions off said plagiarism. It’s a huge ethical conundrum and one I heavily disagree with.

source

Sort:hotnew top

shadowspirit@lemmy.world ⁨1⁩ ⁨year⁩ ago
And pretty sure this is the catalyst for reddit’s API changes. Other companies are getting rich off of them and they want a piece of the pie.

source
Spectacle8011@lemmy.comfysnug.space ⁨1⁩ ⁨year⁩ ago

Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue.

This is not necessarily true. Google’s instant answers are designed to use the content from websites to answer searcher’s questions without actually leading them to the website. Whether you’re trying to find the definition for the word, the year a movie came out, or a recipe, Google will take the information they’ve scraped from a website and present it on their page with a link to the website. Their hope is that the information will be useful enough that the searcher never needs to leave the search engine.

This might be useful for searchers, but it doesn’t help the sites much. This is one of the reasons news companies attempted to take action against Google a few years ago. I think a search engine should provide some useful utilities, but not try to replace the sites they’re ostensibly attempting to connect users to. Not all search engines are like this, but Google is.

source