Comment on Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

• floquant@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago

  Is this a gotcha? Not sure where you got the “visual” from, but yes it is best human performance vs best LLM performance

  source

  • lath@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

    I don’t know why you assume there has to be a gotcha, maybe it’s the competitive background… Anyway, it’s visual because you look at it to see it. And it’s not the best human performance vs best LLM performance, it’s best controlled performance because the testing is limited to a set of parameters.

    source

    • floquant@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago

      That’s what games are? I really don’t see how it is an unfair comparison to you. How would you change it?

      source

      • lath@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

        Stress test it. Low, average, high, impairment conditions, safeguards off, order, chaos and everything in between.

        source

        • -> View More Comments