Show top LLMs buggy code and they'll finish off the mistakes rather than fix them.

Comments

• taladar@sh.itjust.works ⁨5⁩ ⁨weeks⁩ ago

  I don’t see why anyone would expect anything else out of a “what is the most likely way to continue this” algorithm.

  source

  • skip0110@lemm.ee ⁨5⁩ ⁨weeks⁩ ago

    To be fair, if you give me a shit code base and expect me to add features with no time to fix the existing ones, I will also just add more shit on the pile. Because obviously that’s how you want your codebase to look.

    source

    • sugar_in_your_tea@sh.itjust.works ⁨5⁩ ⁨weeks⁩ ago

      And if you do that without saying you want to factor, I likely won’t stand up for you on the next round of layoffs. If I wanted to make the codebase worse, I’d use AI.

      source

      • -> View More Comments
  • xthexder@l.sw0.com ⁨5⁩ ⁨weeks⁩ ago

    It doesn’t help that the AI also has no ability to go backwards or edit code, it can only append. The best it can do is write it all out again with changes made, but even then, the chance of it losing the plot while doing that is pretty high.

    source

    • Lemminary@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

      Yeah, that’s what the canvas feature is for with ChatGPT. And you guessed it, it’s behind a paywall. :)

      source
  • Goun@lemmy.ml ⁨5⁩ ⁨weeks⁩ ago

    This is what I was thinking, if you give the code to a person and ask them to finish it, they would do the same.

    If you rather ask the LLM to give some insights about the code, it might tell you what’s wrong with it.

    source
• LovableSidekick@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

  As a software developer I’ve never used AI to write code, but several of my friends use it daily and they say it really helps them in their jobs. To explain this to non-programmers, they don’t tell it “Write some code” and then watch TV while it does their job. Coding involves a lot of very routine busy work that’s little more than typing. AI can generate approximately what they want, which they then edit, and according to them this helps them work a lot faster.

  AI isn’t binary good/bad. Neither is a hammer - which can’t build a building by itself and is really shitty as a drill. AI is a tool with good uses.

  source

  • bpev@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

    100%. As a solo dev who used to work corporate, I compare it to having a jr engineer who completes every task instantly. If you give it something well-documented and not too complex, it’ll be perfect. If you give it something more complex or newer tech, it could work, but may have some mistakes or unadvised shortcuts.

    I’ve also found it pretty good for when a dependency I’m evaluating has shit documentation. Not always correct, but sometimes it’ll spit out some apis I didn’t notice.

    source

    • Reliant1087@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

      I’ve found it okay to get a general feel for stuff but I’ve been given insidiously bad code. Functions and data structures that look similar enough to real stuff but are deeply wrong or non+existent.

      source

      • -> View More Comments
  • Lemminary@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

    Coding involves a lot of very routine busy work that’s little more than typing.

    That’s right. You watch it type it out and right where it gets to the important part you realize that’s not what you meant at all, so you hit the stop button. Then you modify the prompt and repeat that one more time. That’s when you realize there are so many things it’s not even considering which gives you the satisfaction that your job is still secure. Then you write a more focused prompt for one aspect of them problem and take whatever good enough bullshit it spewed as a starting point for you to do the manual work. Rinse and repeat.

    source

    • Excrubulent@slrpnk.net ⁨5⁩ ⁨weeks⁩ ago

      That sounds exhausting to me.

      Like seriously what busywork is so routine and so basic that you need an AI to do it but couldn’t make a template for it? And how is it less work to read what it gave yout to check for errors? That’s always the harder part of coding in my experience.

      I would love to know the specifics of where this supposedly saves time.

      I suspect the energy you’re putting into learning this tool could go into becoming a better typist, and you wouldn’t need to cook the planet to do it.

      source
  • sugar_in_your_tea@sh.itjust.works ⁨5⁩ ⁨weeks⁩ ago

    Exactly. I have a coworker use it effectively.

    Personally, I’ve been around the block so it’s usually faster for me to just do the busy work myself. I have lots of tricks for manipulating text quickly (I’m quite proficient with vim), so it’s not a big deal to automate turning JSON into a serializer class or copy and modify a function a bunch of times to build out a bunch of controllers or something. What takes others on my team 30 min I can sometimes get done in 5 through the power of regex or macros.

    But at the end of the day, it doesn’t really matter what tools you use because you’re not being paid for your typing speed or ability to do mundane work quickly, you’re being paid to design and support complex software.

    source
  • IphtashuFitz@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

    We have a handful of Python tools that we require to adhere to PEP8 formatting, and have Jenkins pipeline jobs to validate it and block merge requests if any of the code isn’t properly formatted. I haven’t personally tried it yet, but I wonder if these AI’s might be good for fixing up this sort of formatting lint.

    source

    • witten@lemmy.world ⁨4⁩ ⁨weeks⁩ ago

      Why bother with AI for that? black.readthedocs.io/en/stable/index.html

      source
• etchinghillside@reddthat.com ⁨5⁩ ⁨weeks⁩ ago

  I also do that.

  source

  • filcuk@lemmy.zip ⁨5⁩ ⁨weeks⁩ ago

    Can you select all crosswalks and press ok:

    source
• nectar45@lemmy.zip ⁨5⁩ ⁨weeks⁩ ago

  If you ask the llm for code it will often give you a buggy code but if you run it get an error annd then tell the ai what error you had it will often fix the error so that is cool.

  Wont always work though…

  source

  • Thorry84@feddit.nl ⁨5⁩ ⁨weeks⁩ ago

    In my experience it will write three paragraphs about the mistake, what went wrong and how to fix it. Only to then output the exact same code, or very close to it, with the same bug. And once you get into that infinite loop, it’s basically impossible to get out of it.

    source

    • spooky2092@lemmy.blahaj.zone ⁨5⁩ ⁨weeks⁩ ago

      And once you get into that infinite loop, it’s basically impossible to get out of it.

      The easiest way I found to get out of that loop, is to get mad at the AI so it hangs up on you.

      source
    • nectar45@lemmy.zip ⁨5⁩ ⁨weeks⁩ ago

      I was in that problem too but you can at least occationally tell it to “change the code further” and it can work.

      Often YOU have to try and fix its code though its far from perfect…

      source
  • Luccus@feddit.org ⁨5⁩ ⁨weeks⁩ ago

    I’ve only used an LLM (you can guess which one) once to write code. Mostly because I didn’t feel like writing down some numbers and making a little drawing for myself to solve the problem.

    And because a friend insisted that it writes code just fine.

    But it didn’t. It confidently didn’t. Instead, it made up something weird and kept telling me that it had now “fixed” the problem, when in reality it was trying random fixes that were related to the error message but had nothing to do with the actual core problem. It just guessed and prayed.

    In the end, I solved the problem in 10 minutes with a small scribble and a pen. And most of the time was spend drawing small boxes, because my solution relied on a coordinate system, I needed to visualize.

    source

    • jj4211@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

      And because a friend insisted that it writes code just fine.

      It’s so weird, I feel like I’m being gaslit from all over the place. People talking about “vibe coding” to generate thousands of lines of code without ever having to actually read any of it and swearing it can work fine.

      I’ve repeatedly given LLMs a shot and always the experience is very similar. If I don’t know how to do it, neither does it, but it will spit out code confidently, hallucinating function names or REST urls as needed to fit the narrative that would have been convenient. If I can’t spot the logic issue with some code that isn’t acting correct, it will also fail to generate useful text that would describe the problem.

      If the query is within reach of copy/paste of the top stack overflow answer, then it can generate the code. The nature of LLM integration with IDEs makes the workflow easier to pull in than stack overflow answers, but you need to be vigilant as it’s impossible to tell a viable result from junk, as both are presented with equal confidence and certainty. It can also do a better job of spotting issues within things like key values that are strings with typo than traditional code analysis, and by extension errors in less structured languages like Javascript and Python (where ‘everything is a hash/dictionary’ design prevails).

      So far I can’t say I’ve seen improvements, I see how it could be seen as valuable, but the resulting babysitting carries a cost that has been more annoying than the theoretical time saves. Maybe for more boilerplate tasks, but generally speaking those are highly wrapped by libraries already, and when I have to create significant volume of code, it’s because there’s no library and if there’s no library, it’s niche enough that the LLMs can’t generate either.

      I think the most credible time save was a report of refreshing an old codebase that used a lot of deprecated function and changing most of the calls to the new method without explicit human intervention. Better than tools like ‘2to3’ for python, but still not magical either.

      source

      • -> View More Comments
    • barsoap@lemm.ee ⁨5⁩ ⁨weeks⁩ ago

      Paper + pencil are still a programmer’s best friend. YMMV when it comes to graphics tablets, but no fancy software doing anything but precisely emulating paper + pencil gives you that raw braindump interface, when you have to think about how to squeeze things into syntax you have a bottleneck that chokes everything.

      source
• aliser@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

  fancy autocomplete autocompletes whatever it given. tech bros: "surprised Pikachu*.

  source
• Not_mikey@lemmy.dbzer0.com ⁨5⁩ ⁨weeks⁩ ago

  I guess that’s one advantage of stack overflow, sometimes you need a guy to tell you the entire basis of your question is dumb and wrong.

  source
• msage@programming.dev ⁨5⁩ ⁨weeks⁩ ago

  It’s that time again… for LLMentalist.

  Seriously, it should be linked to every mention of LLM anywhere.

  source
• masterspace@lemmy.ca ⁨5⁩ ⁨weeks⁩ ago

  What a waste of time. Both the article and the researchers.

  Literally by the time their research was published, it was using irrelevant models, on top of the fact that, yeah, that’s how LLMs work. That would be obvious from 5m of using them.

  source
• tordenflesk@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

  AutoComplete 2, Wasted Electricity Boogaloo!

  source
• xia@lemmy.sdf.org ⁨4⁩ ⁨weeks⁩ ago

  I noticed they aggresively build out todo comments, even if it’s not possible to do in place they take a wild stab at it.

  source
• tordenflesk@lemmy.world ⁨5⁩ ⁨weeks⁩ ago

  AutoComplete 2, Wasted Electricity Boogaloo!

  source