Submitted 1 day ago by SparroHawc@lemmy.zip to technology@lemmy.world
https://futurism.com/artificial-intelligence/new-findings-ai-coding-overhyped
“No Duh,” say senior developers everywhere.
The article explains that vibe code often is close, but not quite, functional, requiring developers to go in and find where the problems are - resulting in a net slowdown of development rather than productivity gains.
EVERYTHING about AI is overhyped. Its people trying to cash in on the latest buzz/trend.
It turns every prototyping exercise into a debugging exercise. Even talented coders often suck ass at debugging.
The biggest value I get from AI in this space is when I get handed a pile of spagehtti and ask for an initial overview.
I thought that as well and got some code from someone that left the company and asked it to comment it.
It did the obvious “x= 5 // assign 5 to x” crap comments and then it got to the actually confusing part and just skipped that mess entirely…
I am jack’s complete lack of surprise.
No shit sherlock!
I’d be inclined to try using it if it was smart enough to write my unit tests properly, but it’s great at double inserting the same mock and have 0 working unit tests.
I might try using it to generate some javadoc though… then when my org inevitably starts polling how much ai I use I won’t be in the gutter lol
I personally think unit tests are the worst application of AI. Tests are there to ensure the code is correct, so ideally the dev would write the tests to verify that the AI-generated code is correct.
I personally don’t use AI to write code, since writing code is the easiest and quickest part of my job. I instead use it to generate examples of using a new library, give me comparisons of different options, etc, and then I write the code after that. Basically, I use it as a replacement for a search engine/blog posts.
Ideally, there are requirements before anything, and some TDD types argue that the tests should come before the code as well.
Ideally, the customer is well represented during requirements development - ideally, not by the code developer.
Ideally, the code developer is not the same person that develops the unit tests.
Ideally, someone other than the test developer reviews the tests to assure that the tests do in-fact provide requirements coverage.
Ideally, the modules that come together to make the system function have similarly tight requirements and unit-tests and reviews, and the whole thing runs CI/CD to notify developers of any regressions/bugs within minutes of code check in.
In reality, some portion of that process (often, most of it) is short-cut for one or many reasons. Replacing the missing bits with AI is better than not having them at all.
To preface I don’t actually use ai for anything at my job, which might be a bad metric but my workflow is 10x slower if i even try using ai
That said, I want AI to be able to do unit tests in the sense that I can write some starting ones, then it be able to infer what branches aren’t covered and help me fill the rest
Obviously it’s not smart enough, and honestly I highly doubt it will ever be because that’s the nature of llm, but my peeve with unit test is that testing branches usually entail just copying the exact same test but changing one field to be an invalid value, or a dependency to throw. It’s not hard, just tedious. Branching coverage is already enforced, so you should know when you forgot to test a case.
I also think you should treat ai code as a pull request and actually review what it writes. My coworkers that do use it don’t really proofread, so it ends up having some bad practices and code smells.
The reason tests are a good candidate is that there is a lot of boilerplate and no complicated business logic. It can be quite a time saver. You probably know some untested code in some project - you could get an llm to write some tests that would at least poke some key code paths, which is better than nothing. If the tests are wrong, it’s barely worse than having no tests.
What model are you using? I’ve had such a radically different experience but I’ve only bothered with the latest models. The old ones weren’t even worth trying with
I’ve seen it generate working unit tests plenty. In the sense that they pass.
…they do not actually test the functionality.
Of course that function returns what you’re asserting - you overwrote its actual output and checked against that!
One of the guys at my old job submitted a PR with tests that basically just mocked everything, tested nothing. Like,
with patch("something.whatever", return_value=True): assert whatever(0) is True assert whatever(1) is True
Except for a few dozen lines, with names that made it look like they were doing useful.
He used AI to generate them, of course. Pretty useless.
We have had guys submit tests like that, long before AI was a thing.
True, I do feel mocked by this code.
Oh wow. No shit. Anyway!
AI companies and investors are absolutely overhyping its capabilities, but if you haven’t tried it before I’d strongly recommend doing so. For simple bash scripts and Python it almost always gets something workable first try, genuinely saving time.
AI LLMs are pretty terrible for nearly every other task I’ve tried. I suspect it’s because the same amount of quality training data just doesn’t exist for other fields.
Actually typing out code has literally never been the bottleneck. It’s a vanishingly small amount of what we do. An experienced engineer can type out bash or Python scripts without so much as blinking. And better yet, they can do it without completely fabricating commands and library functions.
The hard part is truly understanding what it is you’re trying to do in the first place, and that fundamentally requires a level of semantic comprehension that LLMs do not in any way possess.
It’s very much like the “no code” solutions of yesteryear. They sound great on paper until you’re faced with the reality of the buggy, unmaintainable nightmare pile of spaghetti code that they vomit into your repo.
LLMs are truly a complete joke for software development tasks. I remain among the top 3-4 developers in terms of speed and output at my workplace (and all of the fastest people refuse to use LLMs as well), and I don’t create MRs chock full of bullshit that has to be ripped out (fucking sick of telling people to delete absolutely useless tests that do nothing but slow down our CI pipeline). The slowest people are those that keep banging their head against the LLM for “efficiency” when it’s anything but.
It’s the fucking stupidest trend I’ve seen in my career and I can’t wait until people finally wake up and realize it’s both incredibly inefficient and incredibly wasteful.
Oh god, please don't use it for Bash. LLM-generated Bash is such a fucking pot of horse shit bad practices. Regular people have a hard enough time writing good Bash, and something trained on all the fucking crap on StackOverflow and GitHub is inevitably going to be so bad...
Signed, a senior dev who is the "Bash guy" for a very large team.
The problem isn’t the tool, it’s the user: they don’t know if they’re getting good code or not, therefore they cannot make the prompt to improve it.
In my view the problems occur when using AI to do something you don’t already know how to do.
I’ve found it’s pretty good at refactoring existing code to use a different but well-supported and well documented library. It’s absolutely terrible for a new and poorly documented library.
I recently tried using Copilot with Claude to implement something in a fairly young library, and did get the basics working, including a long repetitive string of “that doesn’t work, I’m getting error msg [error]”. Seven times of that, and suddenly it worked! I was quite amazed, though it failed me in many other ways with that library (imagining functions and options that don’t exist). But then redoing the same thing in the older, better supported library, it got it right on the first try.
But maybe the biggest advantage of AI coding is that it allows me to code when my brain isn’t fully engaged. Of course the risk there is that my brain might not fully engage because of the AI.
I always need to laugh when I read “Agentic AI”
I would say absolutely in the general sense nost people, and the salesmen, frame them in.
When I was invited to assist with the GDC development, I got a chance to partner with a few AI developers and see the development process firsthand, try my hand at it myself, and get my hands on a few low parameter models for my own personal use. It’s really interesting just how capable some models are in their specific use-cases. However, even high param. models easily become useless at the drop of a hat.
I found the best case, one that’s rarely done mind you, is integrate the model into a program that has the ability to call a known database. With a properly trained model to format output in both natural language and use a given database for context calls, and concrete information, the qualitative performance leaps ahead by bounds. Problem is, that requires so much customization it pretty much ends up being something a capable hobbyist would do, it’s just not economically sound for a business to adopt.
The good news is: AI is a lot less impressive than it seemed at first.
The bad news is: so are a lot of jobs.
Wait, it was hyped? Not just ridiculed?
A VP and his taking head sycophants at my work has not shut up about it. They went through some trouble to automatically measure employee use of the AI and made it a performance measure. So now we have it generate code so we don’t get fired and mostly throw it away.
Oh, I did not know hell was hiring.
Im not super surprised, but AI has been really useful when it comes to learning or giving me a direction to look into something more directly.
Im not really an advocate for AI, but there are some really nice things AI can do. And i like to test the code quality of the models i have access to.
I always ask for a ftp server and dns server, to check what it can do and they work surprisingly well most of the time.
I can’t even get Copilot to write Vitest files for React without making a mountain of junk code that describes dribble.
Were you trying to say “drivel?”
Yes. I’ll go fix that.
Usefulness really comes down to which model is being used. I’ve noticed most developers choose GPT for Copilot because that’s what they are familiar with (or they often don’t have a choice due to company policy). I recommend to try Claude Sonnet. How it works is true magic.
But I agree, repetitive tasks is what it should be used for. Planning is still programmer’s job
GPT has been quite hit and miss for me, but Claude is usually quite solid.
It needs micromanaging, otherwise it will do bad design decisions and go off on unrelated side quests. When micromanaged it’ll get you to that MVP very fast.
The trap is that you need to be able to find the errors it makes, or at least call them out immediately. Trying to have co-pilot fix it’s own mistakes is usually a neverending prompt-cycle.
It can summarise big code bases fast, and find how things fit together a lot faster than me. It’s been very useful when being thrown in head first into a new project.
It amplifies a person. If you smart, it will help. If you dumb it will make you dumber.
I used to work with a pretty great coder.
Everything he’s done in the last 6 months has been trash
Big code assist energy
Claude Sonnet? so you enjoy being lied to on a daily basis because that’s all you’re getting from Claude. Claude Code is alright but using Claude.ai you might as well just throw a dart at wall to get a solution on something.
LLMs/“Vibe Coding” is probably a little bit more useful than the average intern with some tasks bumping up to an early career hire (what would historically be a Junior Engineer before title inflation/stagnation).
As in: it can generate code that might do what you want. But you need (actual) senior engineers to review the code thoroughly. And… how do people get the experience they need to do that?
Which basically results in turning everyone into a manager. Except your reports aren’t humans and you don’t get more pay. Instead your reports are vscode plugins. Which… sounds like absolute hell but I can get why the (wannabe) management class loves that.
Even that description is vastly overselling it’s usefulness. Every time someone says it’s like a junior dev I just sigh, because literally the only reason I like junior devs is because they turn into not junior devs. Never once has assigning something to a junior dev made my job easier. The entire goal is to train them to the point they make PRs that I don’t have to walk them through reworking.
Nowhere close to any junior ime. Grads learn very quickly. Interns only job is to understand. Code academy career switchers understand requirements and will ask questions. Subservient AI does fuck all of any of those things
They are more akin to yet another Rapid Application Development wave imo. Go see how the previous iterations have done. Lots are still with us (rails ftw!). I’ll bet most will outlive LLMs
Sounds exactly like my experience with Vibe Coding.
Industry? Yes, industry hires people who know how to do things needed by industry and who do nothing besides those things.
Programmers outside “industry” more often find themselves writing using the libraries they see for the first time and using languages they never thought to use. AI helps a lot here.
Except LLMs are absolutely terrible at working with a new, poorly documented library. Commonly-used, well-defined libraries? Sure! Working in an obscure language or an obscure framework? Good luck.
LLMs can surface information. It’s perhaps the one place they’re actually useful. They cannot reason in the same way a human programmer can, and all the big tech companies are trying to sell them on that basis.
I’ve found success using more powerful LLMs to help me create applications using the Rust programming language. If you use a weak LLM and ask it to do something very difficult you’ll get bad results. You still need to have a fundamental understanding of good coding practices. Using an LLM to code doesn’t replace the decision making.
Based on my experience with claude sonnet and gpt4/5… It’s a little useful but generally annoying and fails more often than works.
I do think moderate use still comes out ahead, as it saves a bunch of typing when it does work, but I still get annoyed at the blatantly stupid suggestions I keep having to decline.
I remember GPT 4 being useless and constantly giving wrong information. Now with newer models they’ve become significantly more useful, especially when prompted to be extremely careful and to always double check to ensure the best response.
shocked_pikachu_face.jpg
Feyd@programming.dev 1 day ago
“Devs are slower with and don’t trust LLM based tools. Surely, letting these tools off the leash will somehow manifest their value instead of exacerbating their problems.”
Absolute madness.
Draces@lemmy.world 21 hours ago
How are you interpreting it that way. Did you miss a sentence or something in the quote?
SparroHawc@lemmy.zip 1 hour ago
It’s not interpretation, it’s extrapolation.