So we’re just throwing in the towel on what words mean now I guess. Anything can be a neuron.
Researchers have found the cause of hallucinations in LLMs, H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
Submitted 1 day ago by Allah@piefed.world to technology@lemmy.world
https://arxiv.org/abs/2512.01797
Comments
db2@lemmy.world 1 day ago
etchinghillside@reddthat.com 1 day ago
But can anything be a H-NEURON?
Skullgrid@lemmy.world 1 day ago
In this paper, we conduct a systematic investigation into hallucination-associated neurons (H-Neurons)
no, they have to be the nodes responsible for the creation of hallucinations
XLE@piefed.social 1 day ago
Any data that makes AI people upset is an H-neuron. This includes both inaccurate responses, and accurate responses that the model designers were attempting to censor, such as “harmful” content.
Infuriatingly, the researchers actually insist that offensive material is not factual material.
The interventions reveal a distinctive behavioral pattern: amplifying H-Neurons’ activations systematically increases a spectrum of over-compliance behaviors – ranging from overcommitment to incorrect premises and heightened susceptibility to misleading contexts, to increased adherence to harmful instructions… (bypassing safety filters to assist with weapon creation)… and stronger sycophantic tendencies. These findings suggest that H-Neurons do not simply encode factual errors, but rather represent a general tendency to prioritize conversational compliance over factual integrity.
Greg@lemmy.ca 1 day ago
Hchicdfvhk!
Skullgrid@lemmy.world 1 day ago
en.wikipedia.org/…/Neural_network_(machine_learni…
it’s a node in the system.
Allah@piefed.world 1 day ago
kind of like cpu is the brain of computer
Bronzebeard@lemmy.zip 1 day ago
and MITOCHONDRIA IS THE POWERHOUSE OF THE CELL
XLE@piefed.social 1 day ago
So, what can we glean from this? Here are a few of my observations.
Current studies largely treat LLMs as black boxes… Just as… neuroscience investigations into individual neuronal activity and synaptic interactions shape theories of cognition like learning and memory, analyzing neurons – the fundamental computational units of LLMs – is essential for decoding hallucination. By scrutinizing neurons’ activation patterns in relation to hallucinations, we can gain deeper insights into model reliability.
So these researchers are left poking at the compiled code of a closed source database. What a pain.
The funny part is, although they insist it’s not a black box…
The process begins by generating a balanced dataset of faithful (green check) and hallucinatory (red cross) responses using the TriviaQA benchmark. We extract the contribution profiles of neurons specifically on the answer tokens to train a linear classifier. Neurons assigned positive weights by this classifier are identified as “H-Neurons”, distinguishing them from normal neurons based on their predictive role in generating hallucinations.
… The researchers clearly have no idea what the bad nodes are doing to make anything bad. They just can observe that when they are hit, a bad thing happens. So the nodes themselves are black boxes to them.
Our investigation reveals that a remarkably sparse subset of neurons – comprising less than 0.1% of the model’s total neurons – can accurately predict whether the model will produce hallucinated responses.
The “bad” nodes are everywhere. If you look at a 1,000th of the database, you will find them scattered across it. The mystery deepens.
Our investigation reveals that H-Neurons originate during the pre-training phase…observed “parameter inertia” suggests that standard instruction tuning does not effectively restructure the underlying hallucination mechanics; instead, it largely preserves these pre-existing circuits… Findings suggest that hallucinations are not merely artifacts of model scaling or alignment procedures, but rather deeply rooted in the fundamental training objectives that shape LLM behavior from their inception.
The “bad” nodes are among the first ones added to models, before anything else is filtered or further trained. This is very funny because it implies they’re part of something crucial.
We hypothesize that the neurons identifying hallucinations do not merely encode factual errors, but rather drive a fundamental behavioral we term over-compliance, which means the model’s tendency to satisfy user prompts even at the expense of truthfulness, safety, or integrity. Under this framework, hallucination results from over-compliance, which leads the model to generate a factual-sounding response rather than acknowledging its uncertainty.
They made a (second) new phrase: This earliest data that goes into the model, and persists after adding more data, they call “over-compliance” and insist it’s the model trying to bullshit a user extra hard.
Alternative hypothesis: what if this data is simply the basis for even making the results legible?
This originates from the inherent characteristics of the next-token prediction objective. This training paradigm does not distinguish between factually correct and incorrect continuations – it merely rewards fluent text generation.
Never mind, they just said it outright.
Peruvian_Skies@sh.itjust.works 1 day ago
So tue tldr is just what we already knew: LLMs predict the most likely word to come next and have no concept of “true” or “false” information.
Indeed, to have such a concept would require understanding that information and any AI that actually understood information wouldn’t be an LLM because LLMs are just fancy autocorrect.
XLE@piefed.social 1 day ago
There’s a bit more to it: Obviously, if a model gets more correct data pumped into it, it’s more likely to produce a correct output. But they found that at the core of every AI model they tested, when an incorrect output came along, certain nodes produced it. And they are some of the nodes at the earliest part of making the model - before data gets added.
So with that in mind, the tl;dr is more like
AI models have two goals: first be readable, then be correct. It appears the nodes causing incorrect outputs that are also intended to make the output readable.
HyperfocusSurfer@lemmy.dbzer0.com 1 day ago
So, their approach can be used to flag likely hallucinated output and warn the user?
0ndead@infosec.pub 1 day ago
No
AbouBenAdhem@lemmy.world 1 day ago
amplifying H-Neurons’ activations systematically increases a spectrum of over-compliance behaviors – ranging from overcommitment to incorrect premises and heightened susceptibility to misleading contexts, to increased adherence to harmful instructions and stronger sycophantic tendencies. These findings suggest that H-Neurons do not simply encode factual errors, but rather represent a general tendency to prioritize conversational compliance over factual integrity.
I wonder if the same tendencies are associated in humans—and if so, is it something LLMs learned from humans, or is it a consequence of the general structure of neural networks?
snooggums@piefed.world 1 day ago
Prioritizing conversational compliance over factual integrity when the output is promoted as being factual is a design flaw.
Saying double check the output does not excuse that flaw when LLM CEOS say their models are like someone with a PhD or that it can automate every white collar job within a year.
ageedizzle@piefed.ca 1 day ago
Ia it a design flaw? Or is it just false advertising? If I sell you a vacuum by telling you it can mop your flow, is the problem with the vacuum or the way I’m selling the product?
so_pitted_wabam@lemmy.zip 1 day ago
I think a more appropriate post title would be “Researchers have identified and named the process that spawns hallucinations in LLMs, they still don’t know the cause though”
This article is like reading the headline “Researchers have identified the cause of AIDS” and then you open it up and the body is a bunch of science jargon that basically says HIV.
Skullgrid@lemmy.world 1 day ago
it sounds like it’s just how the systems are designed.
I mean, the point of this shit is to take training data and create new stuff out of it through pattern matching. You’re going to get some mismatched shit by design,since the random decisions are modified by the weights. Otherwise you’d get the same shit every time.
snooggums@piefed.world 1 day ago
When the system is intended to look like a random a person then randomness is fine.
When the output is expected to be accurate, it should be the same each time so it can be verified as accurate.
LLMs are being sold as doing both at the same time, but random plus consistent equals random.
XLE@piefed.social 1 day ago
That’s incorrect. Wrong responses will still be generated even if you remove the element that randomizes the response for the same question.
If that wasn’t the case, this paper wouldn’t exist.