What’s funny is this guy has 25 years of experience as a software developer. But three months was all it took to make it worthless.
Comment on I Went All-In on AI. The MIT Study Is Right.
CarbonatedPastaSauce@lemmy.world 1 day ago
Something any (real, trained, educated) developer who has even touched AI in their career could have told you.
AutistoMephisto@lemmy.world 1 day ago
felbane@lemmy.world 22 hours ago
As someone who has been shoved in the direction of using AI for coding by my superiors, that’s been my experience as well. It’s fine at cranking out stackoverflow-level code regurgitation and mostly connecting things in a sane way if the concept is simple enough. The real breakthrough would be if the corrections you make would persist longer than a turn or two. As soon as your “fix-it prompt” is out of the context window, you’re effectively back to square one. If you’re expecting it to “learn” you’re gonna have a bad time. If you’re not constantly double checking its output, you’re gonna have a bad time.
oscarjiminy@aus.social 16 hours ago
@felbane @AutistoMephisto i don't have a cs degree (and am more than willing to accept the conclusions of this piece) but how is it not viable to audit code as it's produced so as it's both vetted and understood in sequence?
felbane@lemmy.world 16 hours ago
Auditing the code it produces is basically the only effective way to use coding LLMs at this point.
You’re basically playing the role of senior dev code reviewing and editing a junior dev’s code, except in this case the junior dev randomly writes an amalgamation of mostly valid, extremely wonky, and/or complete bullshit code. It has no concept of best practices, or fitness for purpose, or anything you’d expect a junior dev to learn as they gain experience.
Now given the above, you might ask yourself: “Self, what if I myself don’t have the skills or experience of a senior dev?” This is where vibe coding gets sketchy or downright dangerous: if you don’t notice the problems in generated code, you’re doomed to fail sooner or later. If you’re lucky, you end up having to do a big refactoring when you realize the code is brittle. If you’re unlucky, your backend is compromised and your CTO is having to decide whether to pay off the ransomware demands or just take a chance on restoring the latest backup.
If you’re just trying to slap together a quick and dirty proof of concept or bang out a one-shot script to accomplish a task, it’s fairly useful. If you’re trying to implement anything moderately complex or that you intend to support for months/years, you’re better off just writing it yourself as you’ll end up with something stylistically cohesive and more easily maintainable.
ctrl_alt_esc@lemmy.ml 1 day ago
It’s still useful to have an actual “study” (I’d rather call it a POC) with hard data you can point to, rather than just “trust me bro”.
prole@lemmy.blahaj.zone 2 hours ago
Like the MIT study that the author refers to? The one that already existed before they decided they need to do it themself?
Dojan@pawb.social 1 day ago
I was in charge of an AI pilot project two years back at my company. That was my conclusion, among others.
some_designer_dude@lemmy.world 1 day ago
Untrained dev here, but the trend I’m seeing is spec-driven development where AI generates the specs with a human, then implements the specs. Humans can modify the specs, and AI can modify the implementation.
This approach seems like it can get us to 99%, maybe.
CaptDust@sh.itjust.works 1 day ago
Trained dev with a decade of professional experience, humans routinely fail to get me workable specs without hours of back and forth meetings. I’d say a solid 25% of my work day is spent understanding what the stakeholders are asking for and how contort the requirements to fit into the system.
If these humans can’t be explict enough with me, a living thinking human that understands my architecture better than any LLM, what chance does an LLM have?
Dojan@pawb.social 23 hours ago
Thus you get a piece of software that no one really knows shit about the inner workings of. Sure you have a bunch of spec sheets but no one was there doing the grunt work so when something inevitably breaks during production there’s no one on the team saying “oh, that might be related to this system I set up over here.”
Piatro@programming.dev 1 day ago
How is what you’re describing different to what the author is talking about? Isn’t it essentially the same as “AI do this thing for me”, “no not like that”, “ok that’s better”? The trouble the author describes, ie the solution being difficult to change, or having no confidence that it can be safely changed, is still the same.
some_designer_dude@lemmy.world 21 hours ago
This poster calckey.world/notes/afzolhb0xk is more articulate than my post.
The difference between this “spec-driven” approach is that the entire process is repeatable by AI once you’ve gotten the spec sorted. So you no longer work on the code, you just work on the spec, which can be a collection of files, files in folders, whatever — but the goal is some kind of determinism, I think.
I use it on a much smaller scale and haven’t really cared much for the “spec as truth” approach myself, at this level. I also work almost exclusively on NextJS apps with the usual Tailwind + etc stack. I would certainly not trust a developer without experience with that stack to generate “correct” code from an AI, but it’s sort of remarkable how I can slowly document the patterns of my own codebase and just auto-include it as context on every prompt (or however Cursor does it) so that everything the LLMs suggest gets LLM-reviewed against my human-written “specs”. And doubly neat is that the resulting documentation of patterns turns out to be really helpful to developers who join or inherit the codebase.
I think the author / developer in the article might not have been experienced enough to direct the LLMs to build good stuff, but these tools like React, NextJS, Tailwind, and so on are all about patterns that make us all build better stuff. The LLMs are like “8 year olds” (someone else in this thread) except now they’re more like somewhat insightful 14 year olds, and where they’ll be in another 5 years… Who knows.
Anyway, just saying. They’re here to stay, and they’re going to get much better.
ChunkMcHorkle@lemmy.world 18 hours ago
They’re here to stay
Eh, probably. At least for as long as there is corporate will to shove them down the rest of our throats. But right now, in terms of sheer numbers, humans still rule, and LLMs are pissing off more and more of us every day while its makers are finding it increasingly harder to forge ahead in spite of us, which they are having to do ever more frequently.
and they’re going to get much better.
They’re already getting so much worse, with what is essentially the digital equivalent of kuru, that I’d be willing to bet they’ve already jumped the shark.
If their makers and funders had been patient, and worked the present nightmares out privately, they’d have a far better chance than they do right now, IMO.
Simply put, LLMs/“AI” were released far too soon, and with far too much “I Have a Dream!” fairly tale promotion that the reality never came close to living up to, and then shoved with brute corporate force down too many throats.
As a result, now you have more and more people across every walk of society pushed into cleaning up the excesses of a product they never wanted in the first place: being forced to share their communities AND energy bills with datacenters, depleted water reserves, privacy violations, EXCESSIVE copyright violations and theft of creative property, having to seek non-AI operating systems just to avoid it, right down to the subject of this thread, the corruption of even the most basic video search.
Can LLMs figure out how to override an angry mob, or resolve a situation wherein the vast majority of the masses are against the current iteration of AI even though the makers of it need us all to be avid, ignorant consumers of AI for it to succeed? Because that’s where we’re going, and we’re already farther down that road than the makers ever foresaw, apparently having no idea just how thin the appeal is getting on the ground for the rest of us.
So yeah, I could be wrong, and you might be right. But at this point, unless something very significant changes, I’d put money on you being mostly wrong.
floofloof@lemmy.ca 21 hours ago
Even more efficient: humans do the specs and the implementation. AI has nothing to contribute to specs, and is worse at implementation than an experienced human. The process you describe, with current AIs, offers no advantages.
AI can write boilerplate code and implement simple small-scale features when given very clear and specific requests, sometimes. It’s basically an assistant to type out stuff you know exactly how to do and review. It can also make suggestions, which are sometimes informative and often wrong.
pelespirit@sh.itjust.works 23 hours ago
Have you used any AI to try and get it to do something? It learns generally, not specifically. So you give it instructions and then it goes, “How about this?” You tell it that it’s not quite right and to fix these things and it goes off on a completely different tangent in other areas. It’s like working with an 8 year old who has access to the greatest stuff around.
SpaceNoodle@lemmy.world 19 hours ago
It doesn’t even actually learn, though.
prole@lemmy.blahaj.zone 2 hours ago
Also, what MIT also told them. Literally MIT lol