good, that means it’s working
I’m gonna be frustrated (though not surprised) if the response is anything other than this.
Perplexity AI is complaining their plagiarism bot machine cannot bypass Cloudflare's firewall
Submitted 5 hours ago by Davriellelouna@lemmy.world to technology@lemmy.world
Comments
gravitas_deficiency@sh.itjust.works 40 minutes ago
Glitchvid@lemmy.world 4 hours ago
When a firm outright admits to bypassing or trying to bypass measures taken to keep them out, you think that would be a slam dunk case of unauthorized access under the CFAA with felony enhancements.
GamingChairModel@lemmy.world 3 hours ago
Fuck that. I don’t need prosecutors and the courts to rule that accessing publicly available information in a way that the website owner doesn’t want is literally a crime. That logic would extend to ad blockers and editing HTML/js in an “inspect element” tag.
kibiz0r@midwest.social 11 minutes ago
They already prosecute people under the unauthorized access provision. They just don’t prosecute rich people under it.
EncryptKeeper@lemmy.world 3 hours ago
That logic would not extend to ad blockers, as the point of concern is gaining unauthorized access to a computer system or asset. Blocking ads would not be considered gaining unauthorized access to anything. In fact it would be the opposite of that.
ubergeek@lemmy.today 1 hour ago
Good. I went through my CF panel, and blocked some of those “AI Assistants” that by default were open, including Perplexity’s.
peoplebeproblems@midwest.social 1 hour ago
Well… Good.
floquant@lemmy.dbzer0.com 5 hours ago
It’s difficult to be a shittier company than OpenAI, but Perplexity seems to be trying hard.
Brunbrun6766@lemmy.world 2 hours ago
Step 1, SOMEHOW find a more punchable face than Altman
fossilesque@mander.xyz 1 hour ago
I hate that these bots ruin my read it later app. :(
cupcakezealot@piefed.blahaj.zone 3 hours ago
rare cloudflare w
boonhet@sopuli.xyz 1 hour ago
As far as security is concerned, their w’s are pretty common tbh. It’s just the whole centralization issue.
JeeBaiChow@lemmy.world 5 hours ago
Uh… good?
wosat@lemmy.world 2 hours ago
This is why companies like Perplexity and OpenAI are creating browsers.
iAvicenna@lemmy.world 3 hours ago
ask AI how to do it?
sylver_dragon@lemmy.world 5 hours ago
You’d think that a competent technology company, with their own AI would be able to figure out a way to spoof Cloudflare’s checks. I’d still think that.
spankmonkey@lemmy.world 4 hours ago
Or find a more efficient way to manage data, since their current approach is basically DDOSing the internet for training data and for responding to user interactions.
Quill7513@slrpnk.net 4 hours ago
see, but they’re not competent. further, they don’t care. most of these ai companies are snake oil. they’re selling you a solution that doesn’t meaningfully solve a problem. their main way of surviving is saying “this is what it can do now, just imagine what it can do if you invest money in my company.”
they’re scammers, the lot of them, running ponzi schemes with our money. if the planet dies for it, that’s no concern of theirs. ponzi schemes require the schemer to have no long term plan, just a line of credit that they can keep drawing from until they skip town before the tax collector comes
lemmyng@piefed.ca 4 hours ago
Perplexity: "But that would cost us moneeyyyy!"
GissaMittJobb@lemmy.ml 2 hours ago
Skill issue. Cope and seethe
sol6_vi@lemmy.makearmy.io 1 hour ago
this made me lol
BaroqueInMind@piefed.social 4 hours ago
Cry more, Perplexity.
kescusay@lemmy.world 3 hours ago
I set up a WAF for my company’s publicly facing developer portal to block out bot traffic from assholes like these guys. It reduced bot traffic to the site by something like - I kid you not - 99.999%.
Fucking data vultures.
FauxLiving@lemmy.world 4 hours ago
The amount of people just reacting to the headline in the comments on these kinds of articles is always surprising.
Your browser acts as an agent too, you don’t manually visit every script link, image source and CSS file. Everyone has experienced how annoying it is to have your browser be targeted by Cloudflare.
There’s a pretty major difference between a human user loading a page and having it summarized and a bot that is scraping 1500 pages/second.
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation. But a user initiated operation isn’t the same as a bot.
Which is the point of the article and the article’s title.
It isn’t clear why OP had to alter the headline,p to bait the anti-ai crowd.
OmgItBurns@discuss.online 11 minutes ago
I think part of the issue is that it does act more like a search engine crawler than a traditional user. A lot of sites rely on real human traffic for revenue (serving ads, requests to sign up for Patreon, using affiliate links, etc) that gets bypassed by these bots. Hell in some cases the people running the sites are just looking for interaction. So while there is a spike in traffic, and potentially cost, the people running these sites aren’t getting the benefit of that traffic.
Basically these have the same issues as the summaries that Google does in their search results but, potentially, have much larger impact on the host’s bandwidth
unpossum@sh.itjust.works 1 hour ago
Thank you for trying to fight the irrational anti-AI brainrot on lemmy! It’s probably a lost cause, but your efforts are appreciated :)
FauxLiving@lemmy.world 42 minutes ago
It’s an uphill battle. Lots of motivated reasoning and bad faith arguments
spankmonkey@lemmy.world 3 hours ago
But a user initiated operation isn’t the same as a bot.
Oh fuck off with that AI company propaganda.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It’s the same fucking thing.
Web crawlers for search engines don’t scrape pages every time a user searches like AI does. Both web crawlers and scrapers are bots, and how a human initiates their operation, scheduled or not, doesn’t matter as much as the fact that they do things very differently and only one of the two respects robots.txt.
FauxLiving@lemmy.world 3 hours ago
There’s no difference in server load between a user looking at a page and a user using an AI tool to summarize the page.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It’s the same fucking thing.
You either didn’t read the article or are deliberately making bad faith arguments. The entire point of the article is that the traffic that they’re referring to is initiated by a user, just like when you type an address into your browser’s address bar.
This traffic, initiated by a user, creates the same server load as that same user loading the page in a browser.
Yes, mass scraping of web pages creates a bunch of server load. This was the case before AI was even a thing.
This situation is like Cloudflare presenting was a captcha in order to load each individual image, css or JavaScript asset into a web browser because bot traffic pretends to be a browser.
I don’t think it’s too hard to understand that a bot pretending to be a browser and a human operated browser are two completely different things and classifying them as the same (and captchaing them) would be a classification error.
This is exactly the same kind of error. Even if you personally believe that users using AI tools should be blocked, not everyone has the same opinion. If Cloudflare can’t distinguish between bot requests and human requests then their customers can’t opt out and allow their users to use AI tools even if they want to.
_cryptagion@lemmy.dbzer0.com 3 hours ago
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation.
Well I suppose it’s a good thing then that the anti-AI shield is opt-in, and Cloudflare isn’t making any decisions for anyone on whether or not AI scrapers get to visit their pages. That little bit of context makes your entire argument fall apart.
FauxLiving@lemmy.world 2 hours ago
It isn’t opt in.
You can block all bot page scraping, and also block user initiated AI tools or you can block no traffic.
There isn’t an option to block bot page scraping but allow user initiated AI tools.
Because, as the article points out, Cloudflare is not able to distinguish between the two
HarkMahlberg@kbin.earth 3 hours ago
In a better timeline, we wouldn't need to cheer the victory of one megacorporation over another, they would both be the losers. But also people are still capable of holding two thoughts simultaneously.
For instance, we'd all be happy to see Apple lose the Epic Games lawsuit and be forced out of their monopoly on app stores on iOS. But those same people are aware it would allow Epic to continue being a disgusting company.
bait the anti-ai crowd
Oh I see lol
FauxLiving@lemmy.world 2 hours ago
What does any of that have to do with the fact that Cloudflare isn’t able to classify traffic in order to distinguish between human user generated traffic and mass scraping bot traffic?
If they’re incapable of distinguishing the two, then their customers are having legitimate user requests blocked by Cloudflare with no ability to opt out.
Oh I see lol
Yeah, I think people who’re unable to think rationally about a problem because they made up their mind before knowing any of the details are intellectually lazy.
Ermiar@lemmy.world 4 hours ago
EncryptKeeper@lemmy.world 3 hours ago
I can’t get over their CEO that looks like a nine year old. Not sure what it is about him
Darkenfolk@sh.itjust.works 14 minutes ago
I think it’s the beard, it makes his cheeks look puffed up a bit. His whole expression kinda looks like a grouchy toddler.
Ekybio@lemmy.world 5 hours ago
Can someone with more knowledge shine a bit more light on this while situation? Im out of the loop on the technical details
spankmonkey@lemmy.world 5 hours ago
AI crawlers tend to overwhelm websites by doing the least efficient scraping of data possible, basically DDOSing a huge portion of the internet. Perplexity already scraped the net for training data and is now hammering it inefficiently for searches.
Cloudflare is just trying to keep the bots from overwhelming everything.
panda_abyss@lemmy.ca 5 hours ago
Cloudflare runs as a CDN/cache/gateway service in front of a ton of websites. Their service is to help protect against DDOS and malicious traffic.
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can do, so it actually becomes a revenue source for them.
This is a response to that from Perplexity who run an AI search company. I don’t actually know how their service works, but they were specifically called out in the announcement and Cloudflare accused them of “stealth scraping” and ignoring robots.txt and other things.
very_well_lost@lemmy.world 4 hours ago
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can use, so it actually becomes a revenue source for them.
I think it’s also worth pointing out that all of the big AI companies are burning through cash at an absolutely astonishing rate, and none of them are anywhere close to being profitable. So pay-walling the data they use is probably gonna be painful for their already-tortured bottom line (good).
_cryptagion@lemmy.dbzer0.com 3 hours ago
It should be pointed out that Cloudflare didn’t say they were going to block AI traffic, they give you the option to. The service is a free opt-in for people who want it.
nutsack@lemmy.dbzer0.com 3 hours ago
they don’t outright block ai crawlers. but they added some new tools and options for managing or blocking ai bot traffic which the cloudflare customer can choose to use or to not use.
im running a free educational resource and i let the crawlers hit my site all they want because its useful knowledge and it’s served to them from cloudflare’s free tier cache.
RogueBanana@piefed.zip 4 hours ago
But the website owner can still choose to continue blocking them right? Without using additional stuff like Anubis that is.
BetaDoggo_@lemmy.world 4 hours ago
Perplexity (an “AI search engine” company with 500 million in funding) can’t bypass cloudflare’s anti-bot checks. For each search Perplexity scrapes the top results and summarizes them for the user. Cloudflare intentionally blocks perplexity’s scrapers because they consider them to be malicious traffic. Perplexity argues that their scraping is acceptable because it’s user initiated.
Personally I think cloudflare is in the right here. The scraped sites get 0 revenue from Perplexity searches (unless the user decides to go through the sources section and click the links) and Perplexity’s scraping is unnecessarily traffic intensive since they don’t cache the scraped data.
lividweasel@lemmy.world 2 hours ago
…and Perplexity’s scraping is unnecessarily traffic intensive since they don’t cache the scraped data.
That seems almost maliciously stupid. We need to train a new model. Hey, where’d the data go? Oh well, let’s just go scrape it all again. Wait, did we already scrape this site? No idea, let’s scrape it again just to be sure.
interdimensionalmeme@lemmy.ml 4 hours ago
Just buy cloudflare duh
_cryptagion@lemmy.dbzer0.com 3 hours ago
The anti-AI shield and bot-fight mode are free, you don’t need to pay anything to use them.
interdimensionalmeme@lemmy.ml 2 hours ago
No I’m telling Perplexity, they can just buy their obstacle
People who use the things you have described, for free are themselves the products being sold
this is implied in the price
panda_abyss@lemmy.ca 5 hours ago
I actually agree with them
This feels like cloudflare trying to collect rent from both sides instead of doing what’s best for the website owners.
There is a problem with AI crawlers, but these technologies are essentially doing a search, fetching a several pages, scanning/summarizing them, then presenting the findings to the user.
I don’t really think that’s wrong, it’s just a faster version of rummaging through the SEO shit you do when you Google something.
drspod@lemmy.ml 5 hours ago
What’s best for the website owners is to have people actually visit and interact with their website. Blocking AI tools is consistent with that.
panda_abyss@lemmy.ca 5 hours ago
For a lot of AI search I actually end up reading the pages, so I don’t know hope much this stops that
kopasz7@sh.itjust.works 5 hours ago
Search engines been going relatively fine for decades now. But the crawlers from AI companies basically DDOS hosts in comparison, sending so many requests in such a short interval. Crawling dynamic links as well that are expensive to render compared to a static page, ignoring the robots.txt entirely, or even using it discover unlinked pages.
Servers have finite resources, especially self hosted sites, while AI companies have disproportinately more at their disposal, easily grinding other systems to a halt by overwhelming them with requests.
pr06lefs@lemmy.ml 4 hours ago
If a neighborhood is beset by roving bands of thieves, sooner or later strangers will be greeted by a shotgun rather than an invitation to tea, regardless of their intentions. Them’s the breaks. Bots are going to take a hit now and their operators are just going to have to deal with it. Sucks when people don’t play nice, but this is what you get.
FauxLiving@lemmy.world 3 hours ago
I’m sure people that are attempting to drive to their house in a new vehicle wouldn’t appreciate being riddled with bullets because the neighborhood watch makes no attempt to distinguish between thieves and homeowners.
ordnance_qf_17_pounder@reddthat.com 4 hours ago
josefo@leminal.space 2 hours ago
I really hope Cloudflare doesn’t eventually evolve into a shitty ass company, so far I like them very much, and all this massive L for AI only improves my opinion on them.
starchylemming@lemmy.world 2 hours ago
next step: cloudflare sends hit squads to blow up the source of these slimy data grabber attacks
dzajew@piefed.social 4 hours ago
Cry me a river
xxce2AAb@feddit.dk 4 hours ago
Ooh, that’s though sweetheart. If the owners of those servers want you to visit, they’ll just choose another WAF than CF’s.
All zero of them.
EtherWhack@lemmy.world 57 minutes ago
Image