Comment on Show top LLMs buggy code and they'll finish off the mistakes rather than fix them.

nectar45@lemmy.zip ⁨1⁩ ⁨month⁩ ago

• Thorry84@feddit.nl ⁨1⁩ ⁨month⁩ ago

  In my experience it will write three paragraphs about the mistake, what went wrong and how to fix it. Only to then output the exact same code, or very close to it, with the same bug. And once you get into that infinite loop, it’s basically impossible to get out of it.

  source

  • spooky2092@lemmy.blahaj.zone ⁨1⁩ ⁨month⁩ ago

    And once you get into that infinite loop, it’s basically impossible to get out of it.

    The easiest way I found to get out of that loop, is to get mad at the AI so it hangs up on you.

    source
  • nectar45@lemmy.zip ⁨1⁩ ⁨month⁩ ago

    I was in that problem too but you can at least occationally tell it to “change the code further” and it can work.

    Often YOU have to try and fix its code though its far from perfect…

    source
• Luccus@feddit.org ⁨1⁩ ⁨month⁩ ago

  I’ve only used an LLM (you can guess which one) once to write code. Mostly because I didn’t feel like writing down some numbers and making a little drawing for myself to solve the problem.

  And because a friend insisted that it writes code just fine.

  But it didn’t. It confidently didn’t. Instead, it made up something weird and kept telling me that it had now “fixed” the problem, when in reality it was trying random fixes that were related to the error message but had nothing to do with the actual core problem. It just guessed and prayed.

  In the end, I solved the problem in 10 minutes with a small scribble and a pen. And most of the time was spend drawing small boxes, because my solution relied on a coordinate system, I needed to visualize.

  source

  • jj4211@lemmy.world ⁨1⁩ ⁨month⁩ ago

    And because a friend insisted that it writes code just fine.

    It’s so weird, I feel like I’m being gaslit from all over the place. People talking about “vibe coding” to generate thousands of lines of code without ever having to actually read any of it and swearing it can work fine.

    I’ve repeatedly given LLMs a shot and always the experience is very similar. If I don’t know how to do it, neither does it, but it will spit out code confidently, hallucinating function names or REST urls as needed to fit the narrative that would have been convenient. If I can’t spot the logic issue with some code that isn’t acting correct, it will also fail to generate useful text that would describe the problem.

    If the query is within reach of copy/paste of the top stack overflow answer, then it can generate the code. The nature of LLM integration with IDEs makes the workflow easier to pull in than stack overflow answers, but you need to be vigilant as it’s impossible to tell a viable result from junk, as both are presented with equal confidence and certainty. It can also do a better job of spotting issues within things like key values that are strings with typo than traditional code analysis, and by extension errors in less structured languages like Javascript and Python (where ‘everything is a hash/dictionary’ design prevails).

    So far I can’t say I’ve seen improvements, I see how it could be seen as valuable, but the resulting babysitting carries a cost that has been more annoying than the theoretical time saves. Maybe for more boilerplate tasks, but generally speaking those are highly wrapped by libraries already, and when I have to create significant volume of code, it’s because there’s no library and if there’s no library, it’s niche enough that the LLMs can’t generate either.

    I think the most credible time save was a report of refreshing an old codebase that used a lot of deprecated function and changing most of the calls to the new method without explicit human intervention. Better than tools like ‘2to3’ for python, but still not magical either.

    source

    • Reliant1087@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

      Thank you! That’s been my experience as well. It just gives you insidious junk that you have to be vigilant to catch. That is so much more stressful for me than having to deal with repetitive stuff.

      I can’t fathom how people are claiming it generated whole projects for them.

      source

      • jj4211@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

        I assume there’s a large amount of people who do nothing but write pretty boilerplate projects that have already been done a thousand times, maybe with some very milquetoast variations like branding or styling. Like a web form doing one to one manipulations of some database from user input.

        And/or a large number of people who think they need to be seen as “with it” and claim success because they see everyone else claim success. This is super common with any hype phase, where there’s a desperate need for people to claim affinity with the “hot thing”.

        source

        • -> View More Comments
  • barsoap@lemm.ee ⁨1⁩ ⁨month⁩ ago

    Paper + pencil are still a programmer’s best friend. YMMV when it comes to graphics tablets, but no fancy software doing anything but precisely emulating paper + pencil gives you that raw braindump interface, when you have to think about how to squeeze things into syntax you have a bottleneck that chokes everything.

    source