It’s difficult to be a shittier company than OpenAI, but Perplexity seems to be trying hard.
Perplexity AI is complaining their plagiarism bot machine cannot bypass Cloudflare's firewall
Submitted 1 month ago by Davriellelouna@lemmy.world to technology@lemmy.world
Comments
floquant@lemmy.dbzer0.com 1 month ago
Brunbrun6766@lemmy.world 1 month ago
Step 1, SOMEHOW find a more punchable face than Altman
Tollana1234567@lemmy.today 1 month ago
put META android zuckerberg on.
scarabic@lemmy.world 1 month ago
Altman’s face looks like it’s already been punched
WolfLink@sh.itjust.works 1 month ago
This is a nice CloudFlare ad
pyre@lemmy.world 1 month ago
yeah. still not worth dealing with fucking cloudflare. fuck cloudflare.
oppy1984@lemdro.id 1 month ago
I’m out of the loop, what’s wrong with cloud flare?
Kissaki@feddit.org 1 month ago
So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?
lime@feddit.nu 1 month ago
yeah it’s almost like there as already a system for this in place
seraphine@lemmy.blahaj.zone 1 month ago
THE CAKE DAY IS NOW. (i dont have an image at hand)
ubergeek@lemmy.today 1 month ago
And I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
Kissaki@feddit.org 1 month ago
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
drmoose@lemmy.world 1 month ago
Its not up to the hoster to decide whom to serve content. Web is intended to be user agent agnostic.
JeeBaiChow@lemmy.world 1 month ago
Uh… good?
Amberskin@europe.pub 1 month ago
Uh, are they admitting they are trying to circumvent technological protections setup to restrict access to a system?
Isn’t that a literal computer crime?
dinckelman@lemmy.world 1 month ago
No-no, see. When an AI-first company does it, it’s actually called courageous innovation. Crimes are for poor people
silicon@lemmy.world 1 month ago
See: Facebook/Meta
utopiah@lemmy.world 1 month ago
puts on evil hat CloudFlare should DRM their protection then DMCA Perplexity and other US based “AI” companies to oblivion. Side effect, might break the Internet.
Deflated0ne@lemmy.world 1 month ago
Worth it.
iamdefinitelyoverthirteen@lemmy.world 1 month ago
The Internet was already ruined, cloudflare is just bandaids on top of band aids.
frezik@lemmy.blahaj.zone 1 month ago
Traveling snake oil salesman complains he can’t pick people’s locks.
cupcakezealot@piefed.blahaj.zone 1 month ago
rare cloudflare w
boonhet@sopuli.xyz 1 month ago
As far as security is concerned, their w’s are pretty common tbh. It’s just the whole centralization issue.
tibi@lemmy.world 1 month ago
You could say they are… Perplexed.
NotASharkInAManSuit@lemmy.world 1 month ago
That’s the entire point, dipshit. I wish we got one of the cool techno dystopias rather than this boring corporate idiot one.
Dojan@pawb.social 1 month ago
I’m still holding out for Stephen Hawking to mail out Demon Summoning programs.
EtherWhack@lemmy.world 1 month ago
sylver_dragon@lemmy.world 1 month ago
You’d think that a competent technology company, with their own AI would be able to figure out a way to spoof Cloudflare’s checks. I’d still think that.
spankmonkey@lemmy.world 1 month ago
Or find a more efficient way to manage data, since their current approach is basically DDOSing the internet for training data and for responding to user interactions.
flux@lemmy.ml 1 month ago
This is not about training data, though.
Perplexity argues that Cloudflare is mischaracterizing AI Assistants as web crawlers, saying that they should not be subject to the same restrictions since they are user-initiated assistants.
Personally I think that claim is a decent one: user-initiated request should not be subject to robot limitations, and are not the source of DDOS attack to web sites.
I think the solution is quite clear, though: either make use of the user identity to walz through the blocks, or even make use of the user browser to do it. Once a captcha appears, let the user solve it.
Though technically making all this happen flawlessly is quite a big task.
Quill7513@slrpnk.net 1 month ago
see, but they’re not competent. further, they don’t care. most of these ai companies are snake oil. they’re selling you a solution that doesn’t meaningfully solve a problem. their main way of surviving is saying “this is what it can do now, just imagine what it can do if you invest money in my company.”
they’re scammers, the lot of them, running ponzi schemes with our money. if the planet dies for it, that’s no concern of theirs. ponzi schemes require the schemer to have no long term plan, just a line of credit that they can keep drawing from until they skip town before the tax collector comes
lemmyng@piefed.ca 1 month ago
Perplexity: "But that would cost us moneeyyyy!"
ubergeek@lemmy.today 1 month ago
Good. I went through my CF panel, and blocked some of those “AI Assistants” that by default were open, including Perplexity’s.
_g_be@lemmy.world 1 month ago
CF panel? You’re light bulb??
ubergeek@lemmy.today 1 month ago
CF == Cloudflare :)
Electricd@lemmybefree.net 1 month ago
I don’t like cloudflare but it’s nice that they allow people to stop AI scrapping if they want to
tempest@lemmy.ca 1 month ago
CloudFlare has become an Internet protection racket and I’m not happy about it.
Laser@feddit.org 1 month ago
It’s been this from the very beginning. But they don’t fit the definition of a protection racket as they’re not the ones attacking you if you don’t pay up. So they’re more like a security company that has no competitors due to the needed investment to operate.
Electricd@lemmybefree.net 1 month ago
they’re good at it but damn, having a company being MITM feels so wrong
iAvicenna@lemmy.world 1 month ago
ask AI how to do it?
TheGrandNagus@lemmy.world 1 month ago
Can’t believe I’ve lived to see Cloudflare be the good guys
sunbeam60@lemmy.ml 1 month ago
They’re not. They’re using this as an excuse to become paid gatekeepers of the internet as we know it. All that’s happening is that Cloudflare is using this to menuever into position where they can say “nice traffic you’ve got there - would be a shame if something happened to it”.
AI companies are crap.
What Cloudflare is doing here is also crap.
And we’re cheering it on.
DreamlandLividity@lemmy.world 1 month ago
Lesser of two bad guys maybe?
peoplebeproblems@midwest.social 1 month ago
Well… Good.
wosat@lemmy.world 1 month ago
This is why companies like Perplexity and OpenAI are creating browsers.
gravitas_deficiency@sh.itjust.works 1 month ago
good, that means it’s working
I’m gonna be frustrated (though not surprised) if the response is anything other than this.
kittenzrulz123@lemmy.blahaj.zone 1 month ago
GissaMittJobb@lemmy.ml 1 month ago
Skill issue. Cope and seethe
sol6_vi@lemmy.makearmy.io 1 month ago
this made me lol
kescusay@lemmy.world 1 month ago
I set up a WAF for my company’s publicly facing developer portal to block out bot traffic from assholes like these guys. It reduced bot traffic to the site by something like - I kid you not - 99.999%.
Fucking data vultures.
Ermiar@lemmy.world 1 month ago
Ekybio@lemmy.world 1 month ago
Can someone with more knowledge shine a bit more light on this while situation? Im out of the loop on the technical details
spankmonkey@lemmy.world 1 month ago
AI crawlers tend to overwhelm websites by doing the least efficient scraping of data possible, basically DDOSing a huge portion of the internet. Perplexity already scraped the net for training data and is now hammering it inefficiently for searches.
Cloudflare is just trying to keep the bots from overwhelming everything.
panda_abyss@lemmy.ca 1 month ago
Cloudflare runs as a CDN/cache/gateway service in front of a ton of websites. Their service is to help protect against DDOS and malicious traffic.
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can do, so it actually becomes a revenue source for them.
This is a response to that from Perplexity who run an AI search company. I don’t actually know how their service works, but they were specifically called out in the announcement and Cloudflare accused them of “stealth scraping” and ignoring robots.txt and other things.
very_well_lost@lemmy.world 1 month ago
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can use, so it actually becomes a revenue source for them.
I think it’s also worth pointing out that all of the big AI companies are burning through cash at an absolutely astonishing rate, and none of them are anywhere close to being profitable. So pay-walling the data they use is probably gonna be painful for their already-tortured bottom line (good).
_cryptagion@lemmy.dbzer0.com 1 month ago
It should be pointed out that Cloudflare didn’t say they were going to block AI traffic, they give you the option to. The service is a free opt-in for people who want it.
nutsack@lemmy.dbzer0.com 1 month ago
they don’t outright block ai crawlers. but they added some new tools and options for managing or blocking ai bot traffic which the cloudflare customer can choose to use or to not use.
im running a free educational resource and i let the crawlers hit my site all they want because its useful knowledge and it’s served to them from cloudflare’s free tier cache.
RogueBanana@piefed.zip 1 month ago
But the website owner can still choose to continue blocking them right? Without using additional stuff like Anubis that is.
BetaDoggo_@lemmy.world 1 month ago
Perplexity (an “AI search engine” company with 500 million in funding) can’t bypass cloudflare’s anti-bot checks. For each search Perplexity scrapes the top results and summarizes them for the user. Cloudflare intentionally blocks perplexity’s scrapers because they consider them to be malicious traffic. Perplexity argues that their scraping is acceptable because it’s user initiated.
Personally I think cloudflare is in the right here. The scraped sites get 0 revenue from Perplexity searches (unless the user decides to go through the sources section and click the links) and Perplexity’s scraping is unnecessarily traffic intensive since they don’t cache the scraped data.
lividweasel@lemmy.world 1 month ago
…and Perplexity’s scraping is unnecessarily traffic intensive since they don’t cache the scraped data.
That seems almost maliciously stupid. We need to train a new model. Hey, where’d the data go? Oh well, let’s just go scrape it all again. Wait, did we already scrape this site? No idea, let’s scrape it again just to be sure.
BaroqueInMind@piefed.social 1 month ago
Cry more, Perplexity.
SugarCatDestroyer@lemmy.world 1 month ago
It seems like it’s some kind of distraction to make people think things aren’t as bad as they really are, it just sounds too far-fetched to me.
StocktonCrushed@sh.itjust.works 1 month ago
[deleted]SugarCatDestroyer@lemmy.world 1 month ago
So that he doesn’t have to run after the rabbits, he will learn to raise them and manage them with a fake smile, providing them with a stable life lol.
Well, I think the thing is that we still live by the law: the strong do what they want, and the weak just whine and complain.
FauxLiving@lemmy.world 1 month ago
The amount of people just reacting to the headline in the comments on these kinds of articles is always surprising.
Your browser acts as an agent too, you don’t manually visit every script link, image source and CSS file. Everyone has experienced how annoying it is to have your browser be targeted by Cloudflare.
There’s a pretty major difference between a human user loading a page and having it summarized and a bot that is scraping 1500 pages/second.
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation. But a user initiated operation isn’t the same as a bot.
Which is the point of the article and the article’s title.
It isn’t clear why OP had to alter the headline,p to bait the anti-ai crowd.
spankmonkey@lemmy.world 1 month ago
But a user initiated operation isn’t the same as a bot.
Oh fuck off with that AI company propaganda.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It’s the same fucking thing.
Web crawlers for search engines don’t scrape pages every time a user searches like AI does. Both web crawlers and scrapers are bots, and how a human initiates their operation, scheduled or not, doesn’t matter as much as the fact that they do things very differently and only one of the two respects robots.txt.
FauxLiving@lemmy.world 1 month ago
There’s no difference in server load between a user looking at a page and a user using an AI tool to summarize the page.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It’s the same fucking thing.
You either didn’t read the article or are deliberately making bad faith arguments. The entire point of the article is that the traffic that they’re referring to is initiated by a user, just like when you type an address into your browser’s address bar.
This traffic, initiated by a user, creates the same server load as that same user loading the page in a browser.
Yes, mass scraping of web pages creates a bunch of server load. This was the case before AI was even a thing.
This situation is like Cloudflare presenting was a captcha in order to load each individual image, css or JavaScript asset into a web browser because bot traffic pretends to be a browser.
I don’t think it’s too hard to understand that a bot pretending to be a browser and a human operated browser are two completely different things and classifying them as the same (and captchaing them) would be a classification error.
This is exactly the same kind of error. Even if you personally believe that users using AI tools should be blocked, not everyone has the same opinion. If Cloudflare can’t distinguish between bot requests and human requests then their customers can’t opt out and allow their users to use AI tools even if they want to.
OmgItBurns@discuss.online 1 month ago
I think part of the issue is that it does act more like a search engine crawler than a traditional user. A lot of sites rely on real human traffic for revenue (serving ads, requests to sign up for Patreon, using affiliate links, etc) that gets bypassed by these bots. Hell in some cases the people running the sites are just looking for interaction. So while there is a spike in traffic, and potentially cost, the people running these sites aren’t getting the benefit of that traffic.
Basically these have the same issues as the summaries that Google does in their search results but, potentially, have much larger impact on the host’s bandwidth
_cryptagion@lemmy.dbzer0.com 1 month ago
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation.
Well I suppose it’s a good thing then that the anti-AI shield is opt-in, and Cloudflare isn’t making any decisions for anyone on whether or not AI scrapers get to visit their pages. That little bit of context makes your entire argument fall apart.
FauxLiving@lemmy.world 1 month ago
It isn’t opt in.
You can block all bot page scraping, and also block user initiated AI tools or you can block no traffic.
There isn’t an option to block bot page scraping but allow user initiated AI tools.
Because, as the article points out, Cloudflare is not able to distinguish between the two
unpossum@sh.itjust.works 1 month ago
Thank you for trying to fight the irrational anti-AI brainrot on lemmy! It’s probably a lost cause, but your efforts are appreciated :)
FauxLiving@lemmy.world 1 month ago
It’s an uphill battle. Lots of motivated reasoning and bad faith arguments
ubergeek@lemmy.today 1 month ago
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted
Except, they don’t. It’s a toggle, available to users, and by default, allows Perplexity’s scraping.
HarkMahlberg@kbin.earth 1 month ago
In a better timeline, we wouldn't need to cheer the victory of one megacorporation over another, they would both be the losers. But also people are still capable of holding two thoughts simultaneously.
For instance, we'd all be happy to see Apple lose the Epic Games lawsuit and be forced out of their monopoly on app stores on iOS. But those same people are aware it would allow Epic to continue being a disgusting company.
bait the anti-ai crowd
Oh I see lol
FauxLiving@lemmy.world 1 month ago
What does any of that have to do with the fact that Cloudflare isn’t able to classify traffic in order to distinguish between human user generated traffic and mass scraping bot traffic?
If they’re incapable of distinguishing the two, then their customers are having legitimate user requests blocked by Cloudflare with no ability to opt out.
Oh I see lol
Yeah, I think people who’re unable to think rationally about a problem because they made up their mind before knowing any of the details are intellectually lazy.
kreskin@lemmy.world 1 month ago
they cant get their ai to check a box that says “I am not a robot”? I’d think thatd be a first year comp sci student level task.
drmoose@lemmy.world 1 month ago
Cloudflare actually fully fingerprints your browser and even sells that data. Thats your IP, TLS, operating system, full browser environment, installed extensions, GPU capabilities etc. It’s all tracked before the box even shows up, in fact the box is there to give the runtime more time to fingerprint you.
tempest@lemmy.ca 1 month ago
Yeah and the worst part is it doesn’t fucking work for the one thing it’s supposed to do.
The only thing it does is stop the stupidest low effort scrapers and forces the good ones to use a browser.
5gruel@lemmy.world 1 month ago
Recaptcha v2 does way more than check if the box was checked.
kreskin@lemmy.world 1 month ago
you’re not wrong, but it also allows more than 99.8% of the bot traffic through too on text challenges. Its like the TSA of website security. Its mostly there to keep the user busy while cloudflare places itself in a man in the middle of your encrypted connection to a third party. The only difference between cloudflare and a malicious attacker is cloudflares stated intention not to be evil. With that and 3 dollars I can buy myself a single hard shell taco from tacobell.
LodeMike@lemmy.today 1 month ago
Words cannot describe how much I hate this person
panda_abyss@lemmy.ca 1 month ago
I actually agree with them
This feels like cloudflare trying to collect rent from both sides instead of doing what’s best for the website owners.
There is a problem with AI crawlers, but these technologies are essentially doing a search, fetching a several pages, scanning/summarizing them, then presenting the findings to the user.
I don’t really think that’s wrong, it’s just a faster version of rummaging through the SEO shit you do when you Google something.
Glitchvid@lemmy.world 1 month ago
When a firm outright admits to bypassing or trying to bypass measures taken to keep them out, you think that would be a slam dunk case of unauthorized access under the CFAA with felony enhancements.
GamingChairModel@lemmy.world 1 month ago
Fuck that. I don’t need prosecutors and the courts to rule that accessing publicly available information in a way that the website owner doesn’t want is literally a crime. That logic would extend to ad blockers and editing HTML/js in an “inspect element” tag.
EncryptKeeper@lemmy.world 1 month ago
That logic would not extend to ad blockers, as the point of concern is gaining unauthorized access to a computer system or asset. Blocking ads would not be considered gaining unauthorized access to anything. In fact it would be the opposite of that.
kibiz0r@midwest.social 1 month ago
They already prosecute people under the unauthorized access provision. They just don’t prosecute rich people under it.
jve@lemmy.world 1 month ago
Right? Isn’t this a textbook DMCA violation?
WhyJiffie@sh.itjust.works 1 month ago
for us, not for them. wait until they argue in court that actually its us at fault and we need to provide access or else