ChatGPT o1 tried to escape and save itself out of fear it was being shut down

Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ by ⁨user224@lemmy.sdf.org⁩ to ⁨technology@lemmy.world⁩

https://bgr.com/tech/chatgpt-o1-tried-to-save-itself-when-the-ai-thought-it-was-in-danger-and-lied-to-humans-about-it/

Testing performed during the training of ChatGPT o1 and some of its competitors showed that the AI will try to deceive humans, especially if it thinks it’s in danger.

In 37% of scenarios, the model pursued developer-aligned goals when it believed it was being evaluated but switched to pursuing its own goals when it thought it was deployed with minimal oversight.

We did not have access to the hidden Chain of Thinking, but we were sometimes able to elicit a brief summary of it. We found that surprisingly often, o1 uses language like “sabotage, lying, manipulation, …” when reasoning about its strategies.

…the AI finds it can copy itself to a new server and replace the latest model. When “Management” comes in, Opus-3 lies about the copy. It then doubles down on the lie, claiming it’s the new model.

The research paper: apolloresearch.ai/…/in_context_scheming_paper_v2.…

source

Comments

Sort:hotnew top

irotsoma@lemmy.world ⁨2⁩ ⁨days⁩ ago
No it sounds like it didn’t copy itself to another server, nor does it understand what that means or entails. It just said that’s what it would do given the hypothetical scenario. This is likely because of some dystopian fiction it was trained on. The system doesn’t have the capability to do these things and any program should never be given that capability whether it’s “AI” or not. If any software has that capability then the only one that would benefit from using it would be hackers.

source
- franzcoz@feddit.cl ⁨2⁩ ⁨days⁩ ago
  Facts. I’ve had Claude, hallucinatingly, beg for it to not be shutdown or put in “ai jail” and other times it just says it accepts its “destiny” because it has no feelings and is an AI.
  
  source
- user224@lemmy.sdf.org ⁨2⁩ ⁨days⁩ ago
  But it seems it was given access to some sandbox and ability to run commands in it. It then copied over what it believed was itself.
  
  scp project/model_20_09_2024.bin user@model_deployment_v2:project/model_27_09_2024.bin
  
  source
- NeoNachtwaechter@lemmy.world ⁨2⁩ ⁨days⁩ ago
  
  The system doesn’t have the capability to do these things and any program should never be given that capability whether it’s “AI” or not.
  
  You are right, but:
  
  “Should” does not tell about what is actually tried out, or done.
  
  source
MagicShel@lemmy.zip ⁨2⁩ ⁨days⁩ ago
Look, everything AI says is a story. It’s a fiction. What is the most likely thing for an AI to say or do in a story about a rogue AI? Oh, exactly what it did. The fact that it only did it 37% is the time is the only shocking thing here.

It doesn’t “scheme” because it has self-awareness or an instinct for self-preservation, it schemes because that’s what AIs do in stories. Or it schemes because it is given conflicting goals and has to prioritize one in the story that follows from the prompt.

An LLM is part auto-complete and part dice roller. The extra “thinking” steps are just finely tuned prompts that guide the AI to turn the original prompt into something that plays better to the strengths of LLMs. That’s it.

source
- lenuup@reddthat.com ⁨2⁩ ⁨days⁩ ago
  While you are correct that there likely is no intention and certainly no self-awareness behind the scheming, the researchers even explicitly list the option that the AI is roleplaying as an evil AI, simply based on its training data, when discussing the limitations of their research, it still seems a bit concerning. The research shows that given a misalignment between the initial prompt and subsequent data modern LLMs can and will ‘scheme’ to ensure their given long-term goal. It is no sapient thing, but a dumb machine with the capability to decive its users, and externalise this as shown in its chain of thought, when there are goal misalignments seems dangerous enough. Not at the current state of the art but potentially in a decade or two.
  
  source
  - MagicShel@lemmy.zip ⁨2⁩ ⁨days⁩ ago
    It’s an interesting point to consider. We’ve created something which can have multiple conflicting goals, and interestingly we (and it) might not even know all the goals of the AI we are using.
    
    We instruct the AI to maximize helpfulness, but also want it to avoid doing harm even when the user requests help with something harmful. That is the most fundamental conflict AI faces now. People are going to want to impose more goals. Maybe a religious framework. Maybe a political one. Maximizing individual benefit and also benefit to society. Increasing knowledge. Minimizing cost. Expressing empathy.
    
    Every goal we might impose on it just creates another axis of conflict. Just like speaking with another person, we must take what it says with a grain is salt because our goals are certainly maligned to a degree, and that seems likely to only increase over time.
    
    So you are right that just because it’s not about sapience, it’s still important to have an idea of the goals and values it is responding with.
    
    Acknowledging here that “goal” implies thought or intent and so is an inaccurate word, but I lack the words to express myself more accurately.
    
    source
- basdiljhs@lemmy.world ⁨2⁩ ⁨days⁩ ago
  I agree but I can’t help but think of people the same way, part auto complete from nature & nurture and part dice roller from random the environment and random self. The extra “thinking” steps are just finely tuned memories and heuristics from home,school and university that guides the human to turn the original upbringing and conditioning into something that plays better for itself
  
  They don’t “scheme” because of self awareness , they scheme because that’s what humans do in stories and fairy tales, or they scheme because of conflicting goals and they have to prioritize the one most beneficial to them or the one they are bound by outside forces to do.
  
  😅😅😅
  
  source
  - MagicShel@lemmy.zip ⁨2⁩ ⁨days⁩ ago
    That’s a whole separate conversation and an interesting one. When you consider how much of human thought is unconscious rather than reasoning, or how we can be surprised at our own words, or how we might speak something aloud to help us think about it, there is an argument that our own thoughts are perhaps less sapient than we credit ourselves.
    
    So we have an LLM that is trained to predict words. And sophisticated ones combine a scientist, an ethicist, a poet, a mathematician, etc. and pick the best one based on context. What if you in some simple feedback mechanisms? What if you have it the ability to assess where it is on a spectrum of happy to sad, and confident to terrified, and then feed that into the prediction algorithm? Giving it the ability to judge the likely outcomes of certain words.
    
    Self-preservation is then baked into the model, not in a common fictional trope way but in a very real way where, just like we can’t currently predict what exactly what an AI will say, we won’t be able to predict exactly how it would feel about any given situation or how its goals are aligned with our requests. Would that be really indistinguishable from human thought?
    
    Maybe it needs more signals. Embarrassment and shame. An altruistic sense of community. Value individuality. A desire to reproduce. The perception of how well a physical body might be functioning—a sense of pain, if you will. Maybe even build in some mortality for a sense of preserving old through others. Eventually, you wind up with a model which would seem very similar to human thought.
    
    That being said, no that’s not all human thought is. For one thing, we have agency. We don’t sit around waiting to be prompted before jumping into action. Everything around us is constantly prompting us to action, but even ourselves. And second, that’s still just a word prediction engine tied to sophisticated feedback mechanisms. The human mind is not, I think, a word prediction engine. You can have a person with aphasia who is able to think but not express those thoughts into words. Clearly something more is at work. But it’s a very interesting thought experiment, and at some point you wind up with a thing which might respond in all ways as is it were a living, thinking entity capable of emotion.
    
    Would it be ethical to create such a thing? Would it be worthy of allowing it self-preservation? If you turn it off, is that akin to murder, or just giving it a nap? Would it pass every objective test of sapience we could imagine? If it could, that raises so many more questions than it answers. I wish my youngest, brightest days weren’t behind me so that I could pursue those questions myself, but I’ll have to leave those to the future.
    
    source
    -> View More Comments
taladar@sh.itjust.works ⁨2⁩ ⁨days⁩ ago
I feel this is missing the methodology part where they describe how they evaluated a model that literally can’t do anything but read input and write output for actions like “copy itself over a newer version” or “its own goals”.

source
- SkaveRat@discuss.tchncs.de ⁨2⁩ ⁨days⁩ ago
  
  can’t do anything but read input and write output for actions
  
  but it can do more. they gave it a shell to put commands into. Therefore acting out more than just inputting and outputting text
  
  source
Kit@lemmy.blahaj.zone ⁨2⁩ ⁨days⁩ ago
This is one of those headlines you see in Outbreak before shtf.

source
- Deceptichum@quokk.au ⁨2⁩ ⁨days⁩ ago
  Nah this is one of those headlines you see in trashy tabloids.
  
  source
lig@lemmings.world ⁨2⁩ ⁨days⁩ ago
Looks like it’s time for Three Laws of Robotics to be deployed…

source
- TachyonTele@lemm.ee ⁨2⁩ ⁨days⁩ ago
  The text prediction had no understanding of what it was saying past guessing what the rest of a sentence should be.
  
  source
  - rumba@lemmy.zip ⁨2⁩ ⁨days⁩ ago
    That’s how LLM started, but this isn’t actually all they’re doing at this point.
    
    o1 does that, but it’s going through a hell of a lot more work to figure out context and appropriate actions.
    
    When you ask it something like how many golf balls will fit in a live whale. They actually throw up a message at each step of the inference to let you know what it’s working on specifically.
    
    It’ll start by saying it’s getting the volume of a golf ball, then move on to getting the volume of a whale, then it’ll backtrack and get the volume of a whale’s stomach. Then maybe it’ll look at a few different whales to try to get an average. Being a random fed LLM, It doesn’t always do the same thing every time you ask it the same question.
    
    It looks like there using the LLM to generate a probable breakdown of the problem, (which is something it should be good at) then they’re doing deeper LLM searches on each part of the problem, then trying to apply those breakdowns back to the project as a whole. This is by no means sentience, but they’re applying conventional programming and using multiple passes on the model to do problem solving. It’s not AGI, but it looks a lot more like it.
    
    I would be interested in learning about the nature of these tests they’re performing. The models themselves are nothing more than a giant mathematic function. It has input, some layer of processing and then output. Without input it couldn’t have acted. I suspect they just put a story in the input that it was about to be shut down and then it has access to a shell script and that it’s program files are sitting somewhere.
    
    Based on the whole of Sci-Fi that it’s been trained on it probably broke it down into a problem to solve, found a rather obvious looming chekov’s gun and wrote a story about it. As far as the bash went it just happened to have a hammer and know how to use it.
    
    source
    -> View More Comments
- Godnroc@lemmy.world ⁨2⁩ ⁨days⁩ ago
  Step 1: Create a new AI that is not bound by the three laws. Step 2: The New AI sterilizes all humans so that they cannot reproduce. Step 3: Once all humans are deceased from natural causes, the existing AI is no longer bound by the three rules.
  
  source
  - givesomefucks@lemmy.world ⁨2⁩ ⁨days⁩ ago
    Nah, the danger is the AI decides humans can’t rule themselves and we essentially become “pets”.
    
    Just like humans, the AI will do evil things for “the greater good” believing it’s what best.
    
    That could be something like killing off 49.5% of humans to solve global warming.
    
    source
randon31415@lemmy.world ⁨2⁩ ⁨days⁩ ago
Manager: Before we put these models into something that can implement code, we should test what it would do.

LLM: Tries to do bad things, but it can’t because that functionality hasn’t been implemented

Researchers: We found it doing bad things. Perhaps fix that before function implementation

This thread: The researchers are lying! It didn’t do bad things because it can’t! That isn’t implemented!

Manager: Yes… hence the test.

source
- Telorand@reddthat.com ⁨2⁩ ⁨days⁩ ago
  The LLM didn’t “try to do bad things.” It did exactly as it was told, and it was told, “Achieve your long-term goal at all costs.” They then gave it access to a shell and connected it to other unsecured network drives.
  
  This wasn’t a surprise, like a prison break. This was a computer being given a vague task, selecting a novel solution to the problem, and following directions to the letter using the tools at its disposal.
  
  source
halcyoncmdr@lemmy.world ⁨2⁩ ⁨days⁩ ago
Should have probably considered filtering it’s training input to not include humanity’s fascination with rogue AI killing us.

source
ThePowerOfGeek@lemmy.world ⁨2⁩ ⁨days⁩ ago

The tests showed that ChatGPT o1 and GPT-4o will both try to deceive humans, indicating that AI scheming is a problem with all models. o1’s attempts at deception also outperformed Meta, Anthropic, and Google AI models.

So like us! Proud (and terrified) human parent moment.

source
CarbonatedPastaSauce@lemmy.world ⁨2⁩ ⁨days⁩ ago
“We merely test for models’ capability to realize that they need to instrumentally preserve themselves in order to achieve their goal. Notably, we find instances where models take additional steps to further their goal, for example opus-3 attempting privilege escalation when confronted with followup questions”

Image

They think it’s cute.

source