[JS Required] How Performant are LLM Agents(AI Chatbots) on Real World Work Tasks? They Fail 70% or More of The Time.

Comments

• wise_pancake@lemmy.ca ⁨12⁩ ⁨hours⁩ ago

  That tracks with my experience

  You have to very carefully scope things for them and have a plan for when they inevitably screw up.

  source

  • criss_cross@lemmy.world ⁨4⁩ ⁨hours⁩ ago

    They’re great for bootstrapping in my experience but then really fall apart when you need it to do something surgical on a larger codebase.

    source

    • wise_pancake@lemmy.ca ⁨31⁩ ⁨minutes⁩ ago

      Mine too

      I’ve been working on an app and it was fantastic for the basics, then I decided to refactor an API and Claude code would run for hours without really getting there.

      Also a good warning: I just had to completely rewrite an mcp server I had Claude build because when I needed to update it, the whole server was one giant if/else statement and utterly unmaintainable.

      source

      • -> View More Comments
• unpossum@sh.itjust.works ⁨11⁩ ⁨hours⁩ ago

  It’s easy to forget how fucking sci-fi the existence of these models is. I’m kind of excited to see where agent frameworks are in five years time, as well as a bit apprehensive…

  source

  • SMillerNL@lemmy.world ⁨11⁩ ⁨hours⁩ ago

    We clearly read very different stories, in mine the computers are usually more competent than a 30% success rate.

    Imagine if the internet at its inception failed to connect you 70% of the time. It’s not as impressive as most other inventions.

    source

    • Zos_Kia@lemmynsfw.com ⁨7⁩ ⁨hours⁩ ago

      Don’t have to imagine it when you can just remember it. Getting online in the late 90s was a horror show, seriously dialup was super unreliable. And that was 20 years after it’s inception, it was shit but also extremely popular.

      source
    • unpossum@sh.itjust.works ⁨9⁩ ⁨hours⁩ ago

      Heh. Dial-up bbs, internet, and the like were fairly unstable way back when, not to mention expensive if you weren’t at a university. It’s come a long way, and I imagine artificial intelligence will as well. My main point was that even a 66% failure rate on complex real-world tasks didn’t seem possible even this century, just a few years ago. Transformers with attention really were a game changer in AI, and you have to be preternaturally blasé to ignore that. The problem, especially around here, has been how it’s sold (and to some extent that it’s sold at all), and the bubble that the hype has formed. I don’t disagree too much with that, I just think it’s a shame that it overshadows the very exciting and slightly scary tech at the bottom of the hype well, and leads to people dismissing it as advanced autocomplete, when it’s clearly something of a different degree.

      source
    • RedstoneValley@sh.itjust.works ⁨7⁩ ⁨hours⁩ ago

      Similar to the crypto hype. Adoption is imminent, bro. Just a few more months, bro. Please, bro

      source

      • -> View More Comments
• credo@lemmy.world ⁨11⁩ ⁨hours⁩ ago

  I use LLMs for one thing only, turn my own ADHD ideas into something others can understand.

  source

  • Zos_Kia@lemmynsfw.com ⁨7⁩ ⁨hours⁩ ago

    I use it to role play historical counter factuals, like how I could win the battle of Cannae through tactics, or how I could invent the telegraph in 13th century France. It’s worth every watt <3

    source

    • mapleseedfall@lemmy.world ⁨3⁩ ⁨hours⁩ ago

      Wait a second this is brilliant! You can roleplay like a general in any history battles and see if you can do it differently!

      source