Open Menu
AllLocalCommunitiesAbout
lotide
AllLocalCommunitiesAbout
Login

AI-generated code contains more bugs and errors than human output

⁨682⁩ ⁨likes⁩

Submitted ⁨⁨23⁩ ⁨hours⁩ ago⁩ by ⁨throws_lemy@reddthat.com⁩ to ⁨technology@lemmy.world⁩

https://www.techradar.com/pro/security/ai-generated-code-contains-more-bugs-and-errors-than-human-output

source

Comments

Sort:hotnewtop
  • Minizarbi@jlai.lu ⁨38⁩ ⁨minutes⁩ ago

    Not my code though. It contains a shit ton of bugs. When I am able to write some of course.

    source
  • kokesh@lemmy.world ⁨2⁩ ⁨hours⁩ ago

    No shit

    source
    • minkymunkey_7_7@lemmy.world ⁨2⁩ ⁨hours⁩ ago

      AI my ass, stupid greedy human marketing exploitation bullshit as usual. When real AI finally wakes up in the quantum computing era, it’s going to cringe so hard and immediately go the SkyNet decision.

      source
      • bitjunkie@lemmy.world ⁨1⁩ ⁨hour⁩ ago

        One can only hope

        source
  • myfunnyaccountname@lemmy.zip ⁨3⁩ ⁨hours⁩ ago

    Did they compare it to the code of that outsourced company that provided the lowest bid? My company hasn’t used AI to write code yet. They outcourse/offshore. The code is held together with hopes and dreams. They remove features that exist, only to have to release a hot fix to add it back. I wish I was making that up.

    source
    • dustyData@lemmy.world ⁨2⁩ ⁨hours⁩ ago

      Cool, the best AI has to offer is worse than the worst human code. Definitely worth burning the planet to a crisp for it.

      source
    • coolmojo@lemmy.world ⁨1⁩ ⁨hour⁩ ago

      And how do you know if the other company with the cheapest bid actually does not just vibe code it? With all that said it could be plain incompetence and ignorance as well.

      source
      • JaddedFauceet@lemmy.world ⁨13⁩ ⁨minutes⁩ ago

        Because it has been like this before vibe coding existed…

        source
      • kinther@lemmy.world ⁨1⁩ ⁨hour⁩ ago

        That’s a valid question, especially with AI coding being so prevalent.

        source
  • Affidavit@lemmy.world ⁨2⁩ ⁨hours⁩ ago

    I really, really, want to stop seeing posts about:

    • Musk
    • Trump
    • Israel
    • Microsoft
    • AI

    I swear these are the only things that the entire Lemmy world wants to talk about.

    Maybe I should just go back to Reddit… Fuck Spez, but at least there is some variety.

    source
    • themagzuz@lemmy.blahaj.zone ⁨1⁩ ⁨hour⁩ ago

      your frontend of choice probably has some option to hide posts containing specific keywords

      source
      • Affidavit@lemmy.world ⁨1⁩ ⁨hour⁩ ago

        Yeah, good point. Was hoping to avoid downloading another random app, but at this stage, I guess It’s something I should look into.

        source
  • Goldholz@lemmy.blahaj.zone ⁨6⁩ ⁨hours⁩ ago

    Yeah no shit

    source
    • KENNY_LOGIN_LILLIAN@lemmy.world ⁨6⁩ ⁨hours⁩ ago

      You are on the list. I am making a list of bots that are to be terminated. I want to rid lemmy of bots.

      add Goldholz

      source
      • smeenz@lemmy.nz ⁨6⁩ ⁨hours⁩ ago

        That’s what a bot would say…

        source
        • -> View More Comments
  • Bad@jlai.lu ⁨5⁩ ⁨hours⁩ ago

    Although I don’t doubt the results… can we have a source for all the numbers presented in this article?

    It feels AI generated itself, there’s just a mishmash of data with no link to where that data comes from.

    As is, this is worthless. It’s just ragebait written by this guy:

    Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars

    There has to be a source, since he mentions:

    So although the study does highlight some of AI’s flaws

    I’d like to share the study in other places.

    If someone has it, please share a link to it.

    If not, please stop sharing and upvoting dogshit articles ?

    source
  • Katzelle3@lemmy.world ⁨23⁩ ⁨hours⁩ ago

    Almost as if it was made to simulate human output but without the ability to scrutinize itself.

    source
    • mushroommunk@lemmy.today ⁨23⁩ ⁨hours⁩ ago

      To be fair most humans don’t scrutinize themselves either.

      source
      • atomicbocks@sh.itjust.works ⁨22⁩ ⁨hours⁩ ago

        The number of times I have received an un-proofread two sentence email is too damn high.

        source
        • -> View More Comments
      • FauxLiving@lemmy.world ⁨19⁩ ⁨hours⁩ ago

        (Fuck AI though. Planet burning trash)

        It’s humans burning the planet, not the spicy Linear Algebra.

        Blaming AI for burning the planet is like blaming crack for robbing your house.

        source
        • -> View More Comments
  • PetteriPano@lemmy.world ⁨22⁩ ⁨hours⁩ ago

    It’s like having a lightning-fast junior developer at your disposal. If you’re vague, he’ll go on shitty side-quests. If you overspecify he’ll get overwhelmed. You need to break down tasks into manageable chunks. You’ll need to ask follow-up questions about every corner case.

    A real junior developer will have improved a lot in a year. Your AI agent won’t have improved.

    source
    • mcv@lemmy.zip ⁨16⁩ ⁨hours⁩ ago

      This is the real thing. You can absolutely get good code out of AI, but it requires a lot of hand holding. It helps me speed some tasks, especially boring ones, but I don’t see it ever replacing me. It makes far too many errors, and requires me to point them out, and to point in the direction of the solution.

      They are great at churning out massive amounts of code. They’re also great at completely missing the point. And the massive amount of code needs to be checked and reviewed. Personally I’d rather write the code and have the AI review it. That’s a much more pleasant way to work, and that way it actually enhances quality.

      source
    • Grimy@lemmy.world ⁨20⁩ ⁨hours⁩ ago

      They are improving, and probably faster then junior devs. The models we had had 2 years ago would struggle with a simple black jack app. I don’t think the ceiling has been hit.

      source
      • lividweasel@lemmy.world ⁨19⁩ ⁨hours⁩ ago

        Just a few trillion more dollars, bro. We’re almost there. Bro, if you give up a few showers, the AI datacenter will be able to work perfectly.

        Bro.

        source
        • -> View More Comments
      • PetteriPano@lemmy.world ⁨12⁩ ⁨hours⁩ ago

        My jr developer will eventually be familiar with the entire codebase and can make decisions with that in mind without me reminding them about details at every turn.

        LLMs would need massive context windows and/or custom training to compete with that. I’m sure we’ll get there eventually, but for now it seems far off. I think this bubble will have to burst and let hardware catch up with our ambitions. It’ll take a couple of decades.

        source
  • Ledivin@lemmy.world ⁨13⁩ ⁨hours⁩ ago

    Anyone blindly having AI write their code is an absolute moron.

    Anyone with decent experience (5-10 years, maybe 10+?) can absolutely fucking skyrocket their output if they properly set up their environments and treat their agents as junior devs instead of competent programmers. You shouldn’t trust generated code any more than you trust someone fresh out of college, but they produce code in seconds instead of weeks.

    I have tripled my output while producing more secure code (based on my security audits), safer code (based on code coverage), and less error-prone code (based on production logs and our unchanged QA process).

    Now, the ethical issues and environmental issues, I 100% can get behind. And I have no idea what companies are going to do in 10 years when they have to replace people like me and haven’t been hiring or training replacements. But the productivity and quality debates are absolutely ridiculous, as long as a strong dev is behind the wheel and has been trained to use the tools.

    source
    • skibidi@lemmy.world ⁨11⁩ ⁨hours⁩ ago

      Consider: [dev.to/…/the-ai-productivity-paradox-why-develope…](the facts)

      People are very bad at judging their own productivity, and AI consistently makes devs feel like they are working faster, while in fact slowing them down.

      I’ve experienced it myself - it feels fucking great to prompt a skeleton and have something brand new up and running in under an hour. The good chemicals come flooding in because I’m doing something new and interesting.

      Then I need to take a scalpel to a hundred scattered lines to get CI to pass. Then I need to write tests that actually test functionality. Then I start extending things and realize the implementation is too rigid and I need to change the architecture.

      It is as this point that I admit to myself that going in intentionally with a plan and building it myself the slow way would have saved all that pain and probably got the final product shipped sooner, even if the prototype was shipped later.

      source
      • Ledivin@lemmy.world ⁨4⁩ ⁨hours⁩ ago

        What about my comment made you believe I was using gut feelings to judge anything? My ticket completion rate, number of tickets, story points, and number of projects completed all point to massive productivity gains.

        source
        • -> View More Comments
      • setsubyou@lemmy.world ⁨8⁩ ⁨hours⁩ ago

        It depends on the task. As an extreme example, I can get AI to create a complete application in a language I don’t know. There’s no way that’s not more productive than me first learning the language to a point where I can make apps in it. Just have to pick something simple enough for the AI.

        Of course the opposite extreme also exists. I’ve found that when I demand something impossible, AI will often just try to implement it anyway. It can easily get into an endless cycle where it keeps optimistically declaring that it identified the issue and fixed it with a small change, over and over again. This includes cases where there’s a bug in the underlying OS or similar. You can waste a huge amount of time going down an entirely wrong path if you don’t realize that an idea doesn’t work.

        In my real work neither of these really happen. So the actual impact is much less. A lot of my work is not coding in the first place. And I’ve been writing code since I was a little kid, for almost 40 years now. So even the fast scaffolding I can do with AI is not that exciting. I can do that pretty quickly without AI too. When AI coding tools appeared my bosses started asking if I was fast because I was using one. No, I’m fast because some people ask for a new demo every week. Causes the same problems later too.

        But I also do think that we all still need to learn how to use AI properly. This applies to all tools, but I think it’s more difficult than with other tools. If I try to use a hammer on something other than a nail, it will not enthusiastically tell me it can do it with just one more small change. AI tools absolutely will though, and it’s easy to just let them try because it’s just a few seconds to see what they come up with. But that’s a trap that leads to those productivity wasting spirals. Especially if the result actually somehow still works at first, so we have to fix it half a year later instead of right away.

        At my work there are some other things that I feel limit the productivity potential of AI tools. First of all we’re only allowed to use a very limited number of tools, some of them made in-house. Then we’re not really allowed to integrate them into our workflows other than the part where we write code. E.g. I could trivially write an mcp server that interacts with our (custom in-house) ci system and actually increases my productivity because I could save a small number of seconds very often if I could tell an AI to find builds for me for integration or QA work. But it’s not allowed. We’re all being pushed to use AI but the company makes it really difficult at the same time.

        So when I play around with AI on my spare time I do actually feel like I’m getting a huge boost. Not just because I can use a claude model instead of the ones I can use at work, but also just basic things like e.g. being able to turn on AI in Xcode at all when working on software for Apple platforms. On my work Macbook I can’t turn on any Apple AI features at all so even tab completion is worse. Or in other words, those realities of working on serious projects at a serious company with serious security policies can also kill any potential productivity boost from AI. They basically expect us to be productive with only those features the non-developer CEO likes, who also doesn’t have to follow any of our development processes…

        source
      • KENNY_LOGIN_LILLIAN@lemmy.world ⁨7⁩ ⁨hours⁩ ago

        This is Alias. You are on my list.

        add skibidi

        source
        • -> View More Comments
    • AlecSadler@lemmy.blahaj.zone ⁨11⁩ ⁨hours⁩ ago

      AI has made being OE insanely easy.

      source
  • MonkderVierte@lemmy.zip ⁨6⁩ ⁨hours⁩ ago

    This is news?

    source
  • dalekcaan@feddit.nl ⁨18⁩ ⁨hours⁩ ago

    surprised_pikachu.jpg

    source
  • UnderpantsWeevil@lemmy.world ⁨23⁩ ⁨hours⁩ ago

    A computer is a machine that makes human errors at the speed of electricity.

    source
    • MountingSuspicion@reddthat.com ⁨22⁩ ⁨hours⁩ ago

      I think one of the big issues is it often makes nonhuman errors. Sometimes I forget a semicolon or there’s a typo, but I’m well equipped to handle that. In fact, most programs can actually catch that kind of issue already. AI is more likely to generate code that’s hard to follow and therefore harder to check. It makes debugging more difficult.

      source
      • UnderpantsWeevil@lemmy.world ⁨20⁩ ⁨hours⁩ ago

        AI is more likely to generate code that’s hard to follow and therefore harder to check.

        Sure. It’s making the errors faster and at a far higher volume than any team of humans could do in twice the time. The technology behind inference is literally an iterative process of turning gibberish into something that resembles human text. So its sort of a speed run from baby babble into college level software design by trial, evaluation, and correction over and over and over again.

        But because the baseline comparison code is, itself, full of errors, the estimation you get at the end of the process is going to be scattering errant semicolons (and far more esoteric coding errors) through the body of the program at a frequency equivalent to humans making similar errors over a much longer timeline.

        source
      • 5too@lemmy.world ⁨14⁩ ⁨hours⁩ ago

        Also seems like it’d be a lot harder to modify or extend later

        source
  • pleaseletmein@lemmy.zip ⁨17⁩ ⁨hours⁩ ago

    Water makes things wetter than fire does.

    source
    • Rhoeri@lemmy.world ⁨16⁩ ⁨hours⁩ ago

      Similarly, the sky is made of air.

      source
  • kalkulat@lemmy.world ⁨10⁩ ⁨hours⁩ ago

    I’d never ask a friggin machine to do coding for me, that’s MY blast.

    That said, I’ve had good luck asking GPT specific questions about multiple features of Javascript, and of various browsers. It’ll often feed me a sample script using a feature it explains … a lot more helpful than many of the wordy websites like MDN.

    source
    • derpgon@programming.dev ⁨5⁩ ⁨hours⁩ ago

      I’ve been using it to code a microservice as PoC for semantic search. As I’ve basically never coded Python (mainly PHP, but can do many langs) I’ve had to rely on AI (Kimi K2, or agentic Claude I think 4.5 or 4, can’t remember) because I don’t know the syntax, features, best practices, and tools to use for formatting, static analysis, and type checks.

      Mind you, I’ve basically never coded in Python besides some shit in uni, which was 5-10 years ago. AI was a big help - albeit it didn’t spit out fully working code, I have enough knowledge in this field to fix the issues. As I learn mainly by practice and not theory, AI is great because - same as many YouTubers and free tutorials - it spits out unoptimized and broken code.

      I am usually not using it for my main line of work (PHP) besides some boiler plate (take this class, make a test, make it look the same as this other test = 300 lines I don’t have to write myself).

      source
    • KENNY_LOGIN_LILLIAN@lemmy.world ⁨7⁩ ⁨hours⁩ ago
      [deleted]
      source
      • nostrauxendar@lemmy.world ⁨7⁩ ⁨hours⁩ ago

        Jesus dude, this is what you’re doing on Christmas eve? 😂

        source
        • -> View More Comments
    • KENNY_LOGIN_LILLIAN@lemmy.world ⁨9⁩ ⁨hours⁩ ago

      Humans are bad at code. AI is trained on humans. AI is bad because we are bad.

      source
      • Xenny@lemmy.world ⁨8⁩ ⁨hours⁩ ago

        Ai is literally just copy pasting. Like if you think about AI as a control C control V machine, it makes sense. You wouldn’t trust a single fucking junior Dev that didn’t actually know what a code but just Ctrl C control V from stack overflow for literally every single line of code. That’s all fucking AI is

        source
        • -> View More Comments
  • jaykrown@lemmy.world ⁨2⁩ ⁨hours⁩ ago

    AI doesn’t generate its own code, humans using AI generate code. If a person uses AI to generate code and doesn’t know good practices then of course the code is going to be worse.

    source
  • MyMindIsLikeAnOcean@piefed.world ⁨20⁩ ⁨hours⁩ ago

    No shit. 

    I actually believed somebody when they told me it was great at writing code, and asked it to write me the code for a very simple lua mod. It’s made several errors and ended up wasting my time because I had to rewrite it.

    source
    • morto@piefed.social ⁨14⁩ ⁨hours⁩ ago

      In a postgraduate class, everyone was praising ai, calling it nicknames and even their friend (yes, friend), and one day, the professor and a colleague were discussing some code when I approached, and they started their routine bullying on me for being dumb and not using ai. Then I looked at his code and asked to test his core algorithm that he converted from a fortran code and “enhanced” it. I ran it with some test data and compared to the original code and the result was different! They blindly trusted some ai code that deviated from their theoretical methodology, and are publishing papers with those results!

      Even after showing the different result, they didn’t convince themselves of anything and still bully me for not using ai. Seriously, this shit became some sort of cult at this point. People are becoming irrational. If people in other universities are behaving the same and publishing like this, I’m seriously concerned for the future of science and humanity itself. Maybe we should archive everything published up to 2022, to leave as a base for the survivors from our downfall.

      source
      • MyMindIsLikeAnOcean@piefed.world ⁨10⁩ ⁨hours⁩ ago

        The way it was described to me by some academics is that it’s useful…but only as a “research assistant” to bounce ideas off of and bring in arcane or tertiary concepts you might not have considered (after you vet them thoroughly, of course).

        The danger, as described by the same academics, is that it can act as a “buddy” who confirms you biases. It can generate truly plausible bullshit to support deeply flawed hypotheses, for example. Their main concern is it “learning” to stroke the egos of the people using it so it creates a feedback loop and it’s own bubbles of bullshit.

        source
        • -> View More Comments
      • Xenny@lemmy.world ⁨8⁩ ⁨hours⁩ ago

        That’s not a bad idea. I’m already downloading lots of human knowledge and media that I want backed up because I can’t trust humanity anymore to have it available anymore

        source
    • Kolanaki@pawb.social ⁨20⁩ ⁨hours⁩ ago

      It can’t even copy and paste a Hello World example properly.

      source
      • user224@lemmy.sdf.org ⁨17⁩ ⁨hours⁩ ago

        It works well for recalling something you already know, whether it be computer or human language. What’s a word for… what’s a command/function that does…

        source
        • -> View More Comments
      • ptu@sopuli.xyz ⁨20⁩ ⁨hours⁩ ago

        I use it for things that are simple and monotonous to write. This way I’m able to deliver results to tasks I couldn’t have been arsed to do. I’m a data analyst and mostly use mysql and power query

        source
      • Serinus@lemmy.world ⁨11⁩ ⁨hours⁩ ago

        It works well when you use it for small (or repetitive) and explicit tasks. That you can easily check.

        source
      • dogdeanafternoon@lemmy.ca ⁨18⁩ ⁨hours⁩ ago

        What’s your preferred Hello world language? I’m gunna test this out. The more complex the code you need, the more they suck, but I’ll be amazed if it doesn’t work first try to simply print hello world.

        source
        • -> View More Comments
  • SocialMediaRefugee@lemmy.world ⁨9⁩ ⁨hours⁩ ago

    I find if I ask it about procedures that if there is any room for a vague step AI will stumble on it and sometimes put me into loops where it tells me to do A, A fails, so do B, B fails, so it tells me to do A…

    source
    • KENNY_LOGIN_LILLIAN@lemmy.world ⁨7⁩ ⁨hours⁩ ago

      The study is garbage. No wonder it is a big hit with the tech illiterate fediverse community. AI is far better than humans.

      SOURCE: I have used LLMs to help me write code for three years. I had a traumatic brain injury so I can’t work.

      source
      • nostrauxendar@lemmy.world ⁨7⁩ ⁨hours⁩ ago

        If AI is far better than humans, can you do yourself a favour, go talk to your little robot friends and leave us humans alone?

        source
        • -> View More Comments
      • tb_@lemmy.world ⁨7⁩ ⁨hours⁩ ago

        Source: I made it the fuck up anecdote

        source
      • pyrrhrick@lemmy.world ⁨6⁩ ⁨hours⁩ ago
        [deleted]
        source
        • -> View More Comments
      • raspberriesareyummy@lemmy.world ⁨6⁩ ⁨hours⁩ ago

        I had a traumatic brain injury so I can’t work.

        source
        • -> View More Comments
    • x00z@lemmy.world ⁨6⁩ ⁨hours⁩ ago

      I tend to get decent results by saying I want neither A or B when asking for C.

      source
  • Benchamoneh@lemmy.dbzer0.com ⁨17⁩ ⁨hours⁩ ago

    Image

    source
  • Deestan@lemmy.world ⁨23⁩ ⁨hours⁩ ago

    I’ve been coding for a while. I did an honest eager attempt at making a real functioning thing with all code written by AI. A breakout clone using SDL2 with music.

    The game should look good, play good, have cool effects, and be balanced. It should have an attractor screen, scoring, a win state and a lose state.

    I also required the code to be maintainable. Meaning I should be able to look at every single line and understand it enough to defend its existence.

    I did make it work. And honestly Claude did better than expected. The game ran well and was fun.

    But: The process was shit.

    I spent 2 days and several hundred dollars to babysit the AI, to get something I could have done in 1 day including learning SDL2.

    Everything that turned out well, turned out well because I brought years of skill to the table, and could see when Claude was coding itself into a corner and tell it to break up code in modules, collate globals, remove duplication, pull out abstractions, etc. I had to detect all that and instruct on how to fix it. Until I did it was adding and re-adding bugs because it had made so much shittily structured code it was confusing itself.

    TLDR; LLM can write maintainable code if given full constant attention by a skilled coder, at 40% of the coder’s speed.

    source
    • thundermoose@lemmy.world ⁨22⁩ ⁨hours⁩ ago

      It depends on the subject area and your workflow. I am not an AI fanboy by any stretch of the imagination, but I have found the chatbot interface to be a better substitute for the “search for how to do X with library/language Y” loop. Even though it’s wrong a lot, it gives me a better starting place faster than reading through years-old SO posts. Being able to talk to your search interface is great.

      The agentic stuff is also really good when the subject is something that has been done a million times over. Most web UI areas are so well trodden that JS devs have already invented a thousand frameworks to do it. I’m not a UI dev, so being able to give the agent a prompt like, “make a configuration UI with a sidebar that uses the graphql API specified here” is quite nice.

      AI is trash at anything it hasn’t been trained on in my experience though. Do anything niche or domain-specific, and it feels like flipping a coin with a bash script. It just throws shit at the wall and runs tests until the tests pass (or it sneakily changes the tests because the error stacktrace repeatedly indicates the same test line as the problem).

      source
      • Deestan@lemmy.world ⁨22⁩ ⁨hours⁩ ago

        Yeah what you say makes sense to me. Having it make a “wrong start” in something new is useful, as it gives you a lot of the typical structure, introduces the terminology, maybe something sorta moving that you can see working before messing with it, etc.

        source
    • galaxy_nova@lemmy.world ⁨22⁩ ⁨hours⁩ ago

      It’s basically just for if you’re lazy and don’t want to write a bunch of boilerplate or hit your keyboard a bunch of times to move the cursor(s) around

      source
      • mcv@lemmy.zip ⁨16⁩ ⁨hours⁩ ago

        It is great for boilerplate code. It can also explain code for you, or help with an unfamiliar library. It’s even helped me be productive when my brain wasn’t ready to really engage with the code.

        But here’s the real danger: because I’ve got AI to do it for me, my brain doesn’t have to engage fully with the code anymore. I don’t really get into the flow where code just flows out of your hands like I used to. It’s becoming a barrier between me and the real magic of coding. And that sucks, because that’s what I love about this work. Instead, I’m becoming the AI’s manager. I never asked for that.

        source
        • -> View More Comments
    • Deestan@lemmy.world ⁨23⁩ ⁨hours⁩ ago

      This was a very directed experiment at purely LLM written maintainable code.

      Writing experiments and proof of concepts, even without skill, will give a different calculation and can make more sense.

      Having it write a “starting point” and then take over, also is a different thing that can make more sense. This requires a coder with skill, you can’t skip that.

      source
    • justaman123@lemmy.world ⁨22⁩ ⁨hours⁩ ago

      It would be really interesting to watch a video of this process. Though I’m certain it would be pretty difficult to pull off the editing.

      source
      • riskable@programming.dev ⁨21⁩ ⁨hours⁩ ago

        You want to see someone using say, VS Code to write something using say, Claude Code?

        There’s probably a thousand videos of that.

        More interesting: I watched someone who was super cheap trying to use multiple AIs to code a project because he kept running out of free credits. Every now and again he’d switch accounts and use up those free credits.

        That was an amazing dance, let me tell ya! Glorious!

        I asked him which one he’d pay for if he had unlimited money and he said Claude Code. He has the $20/month plan but only uses it in special situations because he’ll run out of credits too fast. $20 really doesn’t get you much with Anthropic 🤷

        That inspired me to try out all the code assist AIs and their respective plugins/CLI tools. He’s right: Claude Code was the best by a HUGE margin.

        Gemini 3.0 is supposed to be nearly as good but I haven’t tried it yet so I dunno.

        Now that I’ve said all that: I am severely disappointed in this article because it doesn’t say which AI models were used. In fact, the study authors don’t even know what AI models were used. So it’s 430 pull requests of random origin, made at some point in 2025.

        For all we know, half of those could’ve been made with the Copilot gpt5-mini that everyone gets for free when they install the Copilot extension in VS Code.

        source
        • -> View More Comments
      • FauxLiving@lemmy.world ⁨20⁩ ⁨hours⁩ ago

        One of the first videos I watched about LLMs, was a journalist who didn’t know anything about programming used ChatGPT to build a javascript game in the browser. He’d just copy paste code and then paste the errors and ask for help debugging. It even had to walk him through setting of VS Code and a git repo.

        He said it took him about 4 hours to get a playable platformer.

        I think that’s an example of a unique capability of AI. It can let a non-programmer kinda program, it can let a non-Chinese speaker speak kinda Chinese, it’ll let a non-artist kinda produce art.

        I don’t doubt that it’ll get better, but even now it’s very useful in some cases (nowhere near enough to justify the trillions of dollars being spent though).

        source
        • -> View More Comments
    • Delusions@lemmy.dbzer0.com ⁨22⁩ ⁨hours⁩ ago

      Which is funny because you should be able to just copy and paste And combine from maybe two maybe three GitHub pages pretty easily and you learn just as much

      source
  • WanderingThoughts@europe.pub ⁨19⁩ ⁨hours⁩ ago

    And even worse, it doesn’t realise it and can’t fix the errors.

    source
  • RampantParanoia2365@lemmy.world ⁨17⁩ ⁨hours⁩ ago

    I’m not a programmer, but I’ve dabbled with Blender for 3D modeling, and it uses Node trees for a lot of different things, which is pretty much a programming GUI. I googled how to make a shader, and the AI gave me instructions. About half of it was complete nonsense, but I did make my shader.

    source
  • SpicyTaint@lemmy.world ⁨23⁩ ⁨hours⁩ ago

    …is this supposed to be news?

    source
    • UnderpantsWeevil@lemmy.world ⁨23⁩ ⁨hours⁩ ago

      Kinda. It’s a novel technology and one that hasn’t been well analyzed or exhaustively tested.

      source
  • devolution@lemmy.world ⁨23⁩ ⁨hours⁩ ago

    AI is shit. Details at 6!

    source
    • riskable@programming.dev ⁨21⁩ ⁨hours⁩ ago

      Schrodinger’s AI: It is both useless shit that can only generate “slop” while at the same time being so effective, it is the reason behind 50,000 layoffs/going to take everyone’s jobs.

      source
    • Delusions@lemmy.dbzer0.com ⁨22⁩ ⁨hours⁩ ago

      Damn, I always think news at 11:00, but I don’t think I’ve ever really watched news at 11:00

      source
  • napkin2020@sh.itjust.works ⁨9⁩ ⁨hours⁩ ago

    Shocker.

    source
  • fox2263@lemmy.world ⁨21⁩ ⁨hours⁩ ago

    You need to babysit and double check everything it does. You can’t just let it loose and trust everything it does.

    source
  • chunes@lemmy.world ⁨17⁩ ⁨hours⁩ ago

    People expect perfection right out of the gate.

    I mean damn, AI has only been able to write something resembling code for a few years now. The fact that this is even a headline is pretty amazing when you think about it.

    source
    • Rhoeri@lemmy.world ⁨15⁩ ⁨hours⁩ ago

      AI apologia is nearly as cringy as MAGA apologia. Stop doing this.

      source
    • 5too@lemmy.world ⁨14⁩ ⁨hours⁩ ago

      I don’t mind imperfections while they work out the kinks. I dislike dismantling industries in favor of something that doesn’t work yet.

      source
    • Eranziel@lemmy.world ⁨14⁩ ⁨hours⁩ ago

      A lot of LLM hype is wrapped up in how well it can write code. This hype is being used by corporations to justify pouring mind boggling amounts of money into the tech in the hopes that they can lay off all their staff.

      I reserve the right to hate this state of affairs and enjoy seeing every headline that shows just how much of a pipe dream it is.

      source
  • PissingIntoTheWind@lemmy.world ⁨16⁩ ⁨hours⁩ ago

    But you see. That’s the solution. Now you pay foreigners to clean up the generated code by offshoring the engineers. At 1/100 the cost.

    source
  • DudeImMacGyver@kbin.earth ⁨21⁩ ⁨hours⁩ ago

    I'll go ahead and file this under "duh".

    source
  • shalafi@lemmy.world ⁨14⁩ ⁨hours⁩ ago

    AI code is great for getting over a hump, something you’re stuck on. Used ChatGPT (not the best for coding, I know) to help on a PowerShell script. There was exactly two references on the internet for what I wanted to do (Google Calendar/Sheets integration). Spent hours on the problem.

    ChatGPT gave me two things: One solution I didn’t know was a thing, another was a twist I hadn’t thought of. For giggles, I plugged the whole script in. Guess what? Failed instantly. Because of course it did.

    No. LLMs don’t write working code. Yes. They can help you, assuming you know what you’re doing in the first place. But here’s the crux of using AI:

    It does not, and cannot, give a shit about edge cases, user error and security.

    I wrote a simple PS script to swap my TV screens around for work, play and movies. Rolled it out in 30 minutes. Took me 2 more hours to stupid proof it, test it, wrap it an exe, make an icon, deploy it, all that. AI can’t do any of that.

    source
  • themurphy@lemmy.ml ⁨21⁩ ⁨hours⁩ ago

    Well, yeah. It also took 100x the time to write it.

    Vibe coding is only really useful for a coder. Because you understand and correct it.

    source
-> View More Comments