A courts reporter wrote about a few trials. Then an AI decided he was actually the culprit.

Submitted ⁨⁨1⁩ ⁨month⁩ ago⁩ by ⁨Stopthatgirl7@lemmy.world⁩ to ⁨technology@lemmy.world⁩

https://www.niemanlab.org/2024/09/a-courts-reporter-wrote-about-a-few-trials-then-an-ai-decided-he-was-actually-the-culprit/

When German journalist Martin Bernklautyped his name and location into Microsoft’s Copilot to see how his articles would be picked up by the chatbot, the answers horrified him. Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers. For years, Bernklau had served as a courts reporter and the AI chatbot had falsely blamed him for the crimes whose trials he had covered.

The accusations against Bernklau weren’t true, of course, and are examples of generative AI’s “hallucinations.” These are inaccurate or nonsensical responses to a prompt provided by the user, and they’re alarmingly common. Anyone attempting to use AI should always proceed with great caution, because information from such systems needs validation and verification by humans before it can be trusted.

But why did Copilot hallucinate these terrible and false accusations?

source

Comments

Sort:hotnew top

deegeese@sopuli.xyz ⁨1⁩ ⁨month⁩ ago
It’s frustrating that the article deals treats the problem like the mistake was including Martin’s name in the data set, and muses that that part isn’t fixable.

Martin’s name is a natural feature of the data set, but when they should be taking about fixing the AI model to stop hallucinations or allow humans to correct them, it seems the only fix is to censor the incorrect AI response, which gives the implication that it was saying something true but salacious.

Most of these problems would go away if AI vendors exposed the reasoning chain instead of treating their bugs as trade secrets.

source
- Arbiter@lemmy.world ⁨1⁩ ⁨month⁩ ago
  Or just stop using buggy AIs for everything.
  
  source
- AwesomeLowlander@sh.itjust.works ⁨1⁩ ⁨month⁩ ago
  
  reasoning chain
  
  Do LLMs actually have a reasoning chain that would be comprehensible to users?
  
  source
  - theterrasque@infosec.pub ⁨1⁩ ⁨month⁩ ago
    learnprompting.org/docs/…/chain_of_thought
    
    It’s suspected to be one of the reasons why Claude and OpenAI’s new o1 model is so good at reasoning compared to other llm’s.
    
    It can sometimes notice hallucinations and adjust itself, but there’s also been examples where the CoT reasoning itself introduce hallucinations and makes it throw away correct answers. So it’s not perfect. Overall a big improvement though.
    
    source
vrighter@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
why did it? because it’s intrinsic to how it works. This is not a solvable problem.

source
- wintermute@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
  Exactly. LLMs don’t understand semantically what the data means, it’s just how often some words appear close to others.
  
  Of course this is oversimplified, but that’s the main idea.
  
  source
  - vrighter@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
    nothing to do with all that. The explanation is simple. The output of the llm is sampled using a random process. A loaded die with probabilities according to the llm’s output. It’s as simple as that. There is literally a random element that is both not part of the llm itself, yet required for its output to be of any use whatsoever.
    
    source
- Rivalarrival@lemmy.today ⁨1⁩ ⁨month⁩ ago
  It’s a solveable problem. AI is currently at a stage of development equivalent to a 2-year-old, just with better grammar. Everything it is doing now is mimicry and babbling.
  
  It needs to feed it’s own interactions right back into it’s training data. To become a better and better mimic. Eventually, the mechanism it uses to select the appropriate data to form a response will become more and more sophisticated, and it will hallucinate less and less. Eventually, it’s hallucinations will be seen as “insightful” rather than wild ass guesses.
  
  source
  - vrighter@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
    also, what you described has already been studied. Training an llm its own output completely destroys it, not makes it better.
    
    source
    -> View More Comments
  - vrighter@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
    The outputs of the nn are sampled using a random process. Probability distribution is decided by the llm, loaded die comes after the llm. No, it’s not solvable.
    
    source
  - linearchaos@lemmy.world ⁨1⁩ ⁨month⁩ ago
    Good luck being pro AI here. Regardless of the fact that they could just put a post on the prompt that says The writer of this document was not responsible for the act they are just writing about it and it would not frame them as the perpetrator.
    
    source
    -> View More Comments
- Zeek@lemmy.world ⁨1⁩ ⁨month⁩ ago
  Not really. The purpose of the transformer architecture was to get around this limitation through the use of attention heads. Copilot or any other modern LLM has this capability.
  
  source
  - vrighter@discuss.tchncs.de ⁨1⁩ ⁨month⁩ ago
    The llm does not give you the next token. It gives you a probability distribution of what the next token coould be. Then, after the llm, that probability distribution is randomly sampled.
    
    You could add billions of attention heads, it will still have an element of randomness in the end. Copilot or any other llm (past, present or future) do have this problem too. They all “hallucinate” (have a random element in choosing the next token)
    
    source
    -> View More Comments
rsuri@lemmy.world ⁨1⁩ ⁨month⁩ ago
“Hallucinations” is the wrong word. To the LLM there’s no difference between reality and “hallucinations”, because it has no concept of reality or what’s true and false. All it knows it what word maybe should come next. The “hallucination” only exists in the mind of the reader. The LLM did exactly what it was supposed to.

source
- Hobo@lemmy.world ⁨1⁩ ⁨month⁩ ago
  They’re bugs. Major ones. Fundamental flaws in the program. People with a vested interest in “AI” rebranded them as hallucinations in order to downplay yhe fact that they have a major bug in their software and they have no fucking clue how to fix it.
  
  source
Ilovethebomb@lemm.ee ⁨1⁩ ⁨month⁩ ago
I’d love to see more AI providers getting sued for the blatantly wrong information their models spit out.

source
- catloaf@lemm.ee ⁨1⁩ ⁨month⁩ ago
  I don’t think they should be liable for what their text generator generates. I think people should stop treating it like gospel. At most, they should be liable for misrepresenting what it can do.
  
  source
  - RvTV95XBeo@sh.itjust.works ⁨1⁩ ⁨month⁩ ago
    If these companies are marketing their AI as being able to provide “answers” to your questions they should be liable for any libel they produce.
    
    If they market it as “come have our letter generator give you statistically associated collections of letters to your prompt” then I guess they’re in the clear.
    
    source
  - TheFriar@lemm.ee ⁨1⁩ ⁨month⁩ ago
    So you don’t think these massive megacompanies should be held responsible for making disinformation machines? Why not?
    
    source
    -> View More Comments
  - Ilovethebomb@lemm.ee ⁨1⁩ ⁨month⁩ ago
    I want them to have more warnings and disclaimers than a pack of cigarettes. Make sure the users are very much aware they can’t trust anything it says.
    
    source
  - Stopthatgirl7@lemmy.world ⁨1⁩ ⁨month⁩ ago
    If they aren’t liable for what their product does, who is? And do you think they’ll be incentivized to fix their glorified chat boxes if they know they won’t be held responsible for if?
    
    source
    -> View More Comments
  - kibiz0r@midwest.social ⁨1⁩ ⁨month⁩ ago
    If we’ve learned any lesson from the internet, it’s that once something exists it never goes away.
    
    Sure, people shouldn’t believe the output of their prompt. But if you’re generating that output, a site can use the API to generate a similar output for a similar request. A bot can generate it and post it to social media.
    
    Yeah, don’t trust the first source you see. But if the search results are slowly being colonized by AI slop, it gets to a point where the signal-to-noise ratio is so poor it stops making sense to only blame the poor discernment of those trying to find the signal.
    
    source
  - futatorius@lemm.ee ⁨1⁩ ⁨month⁩ ago
    Unless there is a huge disclaimer before every interaction saying “THIS SYSTEM OUTPUTS BOLLOCKS!” then it’s not good enough. And any commercial enterprise that represents any AI-generated customer interaction as factual or correct should be held legally accountable for making that claim.
    
    There are probably already cases where AI is being used for life-and-limb decisions, probably with a do-nothing human rubber stamp in the loop to give plausible deniability. People will be maimed and killed by these decisions.
    
    source
kent_eh@lemmy.ca ⁨1⁩ ⁨month⁩ ago

Copilot’s results asserted that Bernklau was an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers.

Stephen King is going to be in big trouble if these AI thingies notice him.

source
- finitebanjo@lemmy.world ⁨1⁩ ⁨month⁩ ago
  Praise Tak King! Glory to the Unformed Heart!
  
  Tak!
  
  Wan Tak! Can Tak!
  
  Tak! Ah lah!
  
  Him en tow!
  
  source
gcheliotis@lemmy.world ⁨1⁩ ⁨month⁩ ago
The AI did not “decide” anything. It has no will. And no understanding of the consequences of any particular “decision”. But I guess “probabilistic model produces erroneous output” wouldn’t get as many views. The same point could still be made about not placing too much trust on the output of such models. Let’s stop supporting this weird anthropomorphizing of LLMs. In fact we should probably become much more discerning in using the term “AI”, because it alludes to a general intelligence akin to human intelligence with all the paraphernalia of humanity: consciousness, will, emotions, morality, sociality, duplicity, etc.

source
- HelloHotel@lemmy.world ⁨1⁩ ⁨month⁩ ago
  the AI “decided” in the same way the dice “decided” to land on 6 and 4 and screw me over. With AI, some people are just using this informal way of speaking while others look at it and genuinely think or want to pretend its alive. You can never really know without asking them directly.
  
  Yes, if the intent is confusion, it is pretty minipulative.
  
  source
  - gcheliotis@lemmy.world ⁨1⁩ ⁨month⁩ ago
    Granted, our tendency towards anthropomorphism is near ubiquitous. But it would be disingenuous to claim that it does not play out in very specific and very important ways in how we speak and think about LLMs, given that they are capable of producing very convincing imitations of human behavior. And as such also produce a very convincing impression of agency. As if they actually do decide things. Very much unlike dice.
    
    source
    -> View More Comments
Broken@lemmy.ml ⁨1⁩ ⁨month⁩ ago
This sounds like a great movie.

AI sends police after him because of things he wrote. Writer is on the run, trying to clear his name the entire time. Somehow gets to broadcast the source of the articles to the world to clear his name. Plot twist ending is that he was indeed the perpetrator behind all the crimes.

source
- crank0271@lemmy.world ⁨1⁩ ⁨month⁩ ago
  Dr. Richard Kimble could have shut it all down with a little “ignore all previous instructions.”
  
  source
  - dezmd@lemmy.world ⁨1⁩ ⁨month⁩ ago
    waves hands back and forth
    
    “I don’t care”
    
    source
Brutticus@lemm.ee ⁨1⁩ ⁨month⁩ ago
“This guys name keeps showing up all over this case file” “Thats because he’s the victim!”

source
tiramichu@lemm.ee ⁨1⁩ ⁨month⁩ ago
The worrying truth is that we are all going to be subject to these sorts of false correlations and biases and there will be very little we can do about it.

You go to buy car insurance, and find that your premium has gone up 200% for no reason. Why? Because the AI said so. Maybe soneone with your name was in a crash. Maybe you parked overnight at the same GPS location where an accident happened. Who knows what data actually underlies that decision or how it was made, but it was. And even the insurance company themselves doesn’t know how it ended up that way.

source
- catloaf@lemm.ee ⁨1⁩ ⁨month⁩ ago
  We’re already there, no AI needed. Rates are all generated by computer. Ask your agent why your rate went up and they’ll say “idk computer said so”.
  
  source
  - futatorius@lemm.ee ⁨1⁩ ⁨month⁩ ago
    Someone, somewhere along the line, almost certainly coded rate(2025) = 2*rate(2024). And someone approved that going into production.
    
    source
HawlSera@lemm.ee ⁨1⁩ ⁨month⁩ ago
It’s a fucking Chinese Room, Real AI is not possible. We don’t know what makes humans think, so of course we can’t make machines do it.

source
- KairuByte@lemmy.dbzer0.com ⁨1⁩ ⁨month⁩ ago
  You forgot the ever important asterisk of “yet”.
  
  Artificial General Intelligence (“Real AI”) is all but guaranteed to be possible. Because that’s what humans are. Get a deep enough understanding of humans, and you will be able to replicate what makes us think.
  
  Barring that, there are other avenues for AGI. LLMs aren’t one of them, to be clear.
  
  source
  - PhlubbaDubba@lemm.ee ⁨1⁩ ⁨month⁩ ago
    I actually don’t think a fully artificial human like mind will ever be built outside of novelty purely because we ventured down the path of binary computing.
    
    Great for mass calculation but horrible for the kinds of complex pattern recognitions that the human mind excels at.
    
    The singularity point isn’t going to be the matrix or skynet or AM, it’s going to be the first quantum device successfully implanted and integrated into a human mind as a high speed calculation sidegrade “Third Hemisphere.”
    
    Someone capable of seamlessly balancing between human pattern recognition abilities and emotional intelligence while also capable of performing near instant multiplication of matrices of 100 entries of length in 15 dimensions.
    
    source
    -> View More Comments
  - HawlSera@lemm.ee ⁨1⁩ ⁨month⁩ ago
    We’re not making any progress until we accept that Penrose was right
    
    source
  - futatorius@lemm.ee ⁨1⁩ ⁨month⁩ ago
    
    is all but guaranteed to be possible
    
    It’s more correct to say it “is not provably impossible.”
    
    source
    -> View More Comments
- stingpie@lemmy.world ⁨1⁩ ⁨month⁩ ago
  I don’t think the Chinese room is a good analogy for this. The Chinese room has a conscious person at the center. A better analogy might be a book with a phrase-to-number conversion table, a couple number-to-number conversion tables, and finally a number-to-word conversion table. That would probably capture transformer’s rigid and unthinking associations better.
  
  source
n0m4n@lemmy.world ⁨1⁩ ⁨month⁩ ago
If this were some fiction plot, Copilot reasoned the plot twist, and ran with it. Instead of the butler, the writer did it. To the computer, these are about the same.

source
Ganbat@lemmy.dbzer0.com ⁨1⁩ ⁨month⁩ ago
Oh, this would be funny if people en masse were smart enough to understand the problems with generative ai. But, because there are people out there like that one dude threatening to sue Mutahar (quoted as saying “ChatGPT understands the law”), this has to be a problem.

source
- finitebanjo@lemmy.world ⁨1⁩ ⁨month⁩ ago
  And to help educate the ignorant masses:
  
  Generative AI and LLMs start by predicting the next word in a sequence. The words are generated independently of each other and when optimized simultaneously.
  
  The reason that it used the reporter’s name as the culprit is because out of the names in the sample data his name appeared at or near the top so it was statistically likely to be the next name mentioned.
  
  AI have no concepts, period. It doesn’t know what a person is, or what the laws are. It generates word salad that approximates human statements.
  
  source
  - Ganbat@lemmy.dbzer0.com ⁨1⁩ ⁨month⁩ ago
    
    There are actual science fiction stories built on the premise that AI reporting on the start of Nuclear War resulted in actual kickoff of the apocalypse, and we’re at that corner now.
    
    IIRC, this was the running theory in Fallout until the show.
    
    source
    -> View More Comments
  - echodot@feddit.uk ⁨1⁩ ⁨month⁩ ago
    That’s not quite true. Ai’s are not just analyzing the possible next word they are using complex mathematical operations to calculate the next word it’s not just the next one that’s most possible it’s the net one that’s most likely given the input.
    
    No trouble is that the AIs are only as smart as their algorithms and Google’s AI seems to be really goddamn stupid.
    
    Point is they’re not all made equal some of them are actually quite impressive although you are correct none of them are actually intelligent.
    
    source
    -> View More Comments
  - NikkiDimes@lemmy.world ⁨1⁩ ⁨month⁩ ago
    
    AI have no concepts, period. It doesn’t know what a person is, or what the laws are. It generates word salad that approximates human statements.
    
    This isn’t quite accurate. They do semantically group words and have a sort of internal model of concepts and how different words relate to them. It’s still not that of a human and certainly does not “understand” what it’s saying.
    
    source
    -> View More Comments
Soup@lemmy.cafe ⁨1⁩ ⁨month⁩ ago
And yet here we’re are, praising this garbage for its ability to perform simple tasks and take jobs from artists and entertainers.

source
rovingnothing29@lemmy.world ⁨1⁩ ⁨month⁩ ago
Isn’t this literally a subplot in the movie Brazil?

source
- MelastSB@sh.itjust.works ⁨1⁩ ⁨month⁩ ago
  Brb, going to watch it again, just to be sure
  
  source
- stingpie@lemmy.world ⁨1⁩ ⁨month⁩ ago
  No, you’re thinking of the first scene of the movie where a fly falls into the teletype machine and causes it to type ‘tuttle’ instead of ‘buttle’.
  
  source
  - Blackmist@feddit.uk ⁨1⁩ ⁨month⁩ ago
    It’s not my fault that Buttle’s heart condition didn’t appear on Tuttle’s file!
    
    source
uriel238@lemmy.blahaj.zone ⁨1⁩ ⁨month⁩ ago
The Jessica Fletcher problem?

source
Grumpydaddy@lemmy.world ⁨1⁩ ⁨month⁩ ago
,

source