Comment on Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

• lath@lemmy.world ⁨9⁩ ⁨hours⁩ ago

  I don’t know why you assume there has to be a gotcha, maybe it’s the competitive background… Anyway, it’s visual because you look at it to see it. And it’s not the best human performance vs best LLM performance, it’s best controlled performance because the testing is limited to a set of parameters.

  source

  • floquant@lemmy.dbzer0.com ⁨9⁩ ⁨hours⁩ ago

    That’s what games are? I really don’t see how it is an unfair comparison to you. How would you change it?

    source

    • lath@lemmy.world ⁨7⁩ ⁨hours⁩ ago

      Stress test it. Low, average, high, impairment conditions, safeguards off, order, chaos and everything in between.

      source

      • gnufuu@infosec.pub ⁨6⁩ ⁨hours⁩ ago

        I haven’t read all of their Benchmark introduction and Technical Documentation. I assume you have and didn’t find any of the tests you’re asking for?

        source