Comment on Car Wash Test on 53 leading AI models: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

aloofPenguin@piefed.world ⁨4⁩ ⁨days⁩ ago

• crunchy@lemmy.dbzer0.com ⁨4⁩ ⁨days⁩ ago

  Honestly that’s a lot more coherent than what I would expect from an LLM running on phone hardware.

  source

  • snooggums@piefed.world ⁨4⁩ ⁨days⁩ ago

    I want to wash my car

    if you don’t have a car

    Yeah, totally coherent.

    source

    • crunchy@lemmy.dbzer0.com ⁨3⁩ ⁨days⁩ ago

      Yes, I read that output. And it’s still better than I would expect.

      source
• AbidanYre@lemmy.world ⁨4⁩ ⁨days⁩ ago

  I like that it’s twice as far to drive for some reason.

  source

  • Fondots@lemmy.world ⁨4⁩ ⁨days⁩ ago

    If I were the type of person who was willing to give AI the benefit of the doubt and not assume that it was just picking basically random numbers

    There’s a lot of cases where it can be a shorter (by distance) walk than drive, where cars generally have to stick to streets while someone on foot may be able to take some footpaths and cut across lawns and such, or where the road may be one-way for vehicles, or where certain turns may not be allowed, etc.

    I have a few intersections near my father in laws house in NJ in mind, where you can just cross the street on foot, but making the same trip in a car might mean driving half a mile down the road, turning around at a jug handle and driving back to where you started on the other side of the street.

    And I wouldn’t be totally surprised if that’s the case for enough situations in the training data where someone debated walking or driving that the AI assumed that it’s a rule that it will always be further by car than on foot.

    That’s still a dumbass assumption, but I’d at least get it.

    And I’m pretty sure it’s much more likely that it’s just making up numbers out of nothing.

    source

    • Balex@lemmy.world ⁨4⁩ ⁨days⁩ ago

      I think it has to do with the fact that LLMs suck at math because they have short memories. So for the walking part it did the math of 50m (original distance) x 2 (there and back) = 100m (total distance). Then it went to the driving part and did 100m (the last distance it sees) x 2 = 200m.

      source
• someguy3@lemmy.world ⁨4⁩ ⁨days⁩ ago

  200 m huh.

  source
• MangoCats@feddit.it ⁨4⁩ ⁨days⁩ ago

  I notice that the “internal thinking” of Opus 4.6 is doing more flip-flopping than earlier modelss like Sonnet 4.5, and it’s coming out with correct answers in the end more often.

  source