[deleted]
AI chatbots were tasked to run a tech company. They built software in under seven minutes — for less than $1.
Submitted 1 year ago by shish_mish@lemmy.world to technology@lemmy.world
Comments
Melco@lemmy.world 1 year ago
Nougat@kbin.social 1 year ago
I've tried to have ChatGPT help me out with some Powershell, and it consistently wanted me to use cmdlets which do not exist for on premise Exchange. I told it as much, it apologized, and wanted me to use cmdlets that don't exist at all.
Large Language Models are not Artificial Intelligence.
amanneedsamaid@sopuli.xyz 1 year ago
Its glorified autocorrect trying to figure out how words string together coherently.
dojan@lemmy.world 1 year ago
I had a weird XAML error I didn’t quite get, and the LLM gave me BS solutions before giving me back my original code.
Lmaydev@programming.dev 1 year ago
There are by definition artificial intelligence.
lilShalom@lemmy.basedcount.com 1 year ago
Ive had google bard supply me code to use with a google api url that doesnt exist.
thorbot@lemmy.world 1 year ago
This also completely glosses over the fact that AI capable of writing this had huge R&D costs to get to that point and also have ongoing costs associated with running them. This whole article is a fucking joke, probably written by AI
aard@kyu.de 1 year ago
You meant to say “a competent human”, which a lot of programmers are not.
While I’d expect this to be of rather low quality I’d bet money on having seen worse projects done by actual humans in the last 25 years.
flamekhan@lemmy.world 1 year ago
“We asked a Chat Bot to solve a problem that already has a solution and it did ok.”
merc@sh.itjust.works 1 year ago
to solve a problem that already has a solution
And whose solution was part of its training set…
variaatio@sopuli.xyz 1 year ago
half the time hallucinating something crazy in the in the mix.
Another funny: Yeah, it’s perfect we just need to solve this small problem of it hallucinating.
Ahemm… solving halloconating is the “no it actually has to understand what it is doing” part aka the actual intelligence. The actually big and hard problem. The actual understanding of what it is asked to do and what solutions to that ask are sane, rational and workable. Understanding the problem and understanding the answer, excluding wrong answers. Actual analysis, understanding and intelligence.
doublejay1999@lemmy.world 1 year ago
Plot twist - the AI just cut and paste from stack overflow like real devs.
Melco@lemmy.world 1 year ago
[deleted]frokie@lemmy.world 1 year ago
It should generate its own acceptance tests and keep asking itself to fix it until they all pass
CmdrShepard@lemmy.one 1 year ago
What if it just kept going and created a brand new language and IDE?
derpgon@programming.dev 1 year ago
Still valid Python code ^/s^
breadsmasher@lemmy.world 1 year ago
It cost less than a dollar to run all those chatbots?
Doubt
dustyData@lemmy.world 1 year ago
Please ignore the hundreds of thousands of dollars and the corresponding electricity that was required to run the servers and infrastructure required to train and use this models, please. Or the master cracks the whip again, please, just say you’ll invest in our startup, please!
igorlogius@lemmy.world 1 year ago
Do Managment next!
mrginger@lemmy.world 1 year ago
This is who will get replaced first, and they don’t want to see it. They’re the most important, valuable part of the company in their own mind, yet that was the one thing the AI got right, the management part. It still needed the creative mind of a human programmer to do the code properly.
thanks_shakey_snake@lemmy.ca 1 year ago
They did do management-- They modeled the whole company as individual “staff” communicating with each other: CEO-bot communicates a product direction to the CTO-bot who communicates technical requirements to the developer-bot who asks for a “beautiful user interface” (lol) from the “art designer” (lol).
It’s all super rudimentary and goofy, but management was definitely part of the experiment.
igorlogius@lemmy.world 1 year ago
Sorry, my mistake i kind of misunderstood … but now I wonder which part of the “company” was most easy to replace and where the most and least failure rate/processing was located/necessary.
Pistcow@lemm.ee 1 year ago
But did it work?
ArbiterXero@lemmy.world 1 year ago
As someone that uses ChatGPT daily for boilerplate code because it’s super helpful…
I call complete bullshite
The program here will be “hello world” or something like that.
LazaroFilm@lemmy.world 1 year ago
Absolutely I can create a code for your app.
void myApp(void) { // add the code for your app here; Return true;
You may need to change the code above to fit your needs. Make sure you replace the comment with the proper code for your app to work.
Ertebolle@kbin.social 1 year ago
OTOH, if you take that hello world program and ask it to compose a themed cocktail menu around it, it'll cheerfully do that for you.
Semi-Hemi-Demigod@kbin.social 1 year ago
It's great for things like "How do I write this kind of loop in this language" but when I asked it for something more complex like a class or a big-ish function it hallucinates. But it makes for a very fast way to get up to speed in a new language
kitonthenet@kbin.social 1 year ago
I can totally see the use case for boilerplate, but I’m also very very rarely writing new classes from scratch or whatever
KoboldCoterie@pawb.social 1 year ago
The study said 86.66% of the generated software systems were “executed flawlessly.”
But…
Nevertheless, the study isn’t perfect: Researchers identified limitations, such as errors and biases in the language models, that could cause issues in the creation of software. Still, the researchers said the findings “may potentially help junior programmers or engineers in the real world” down the line.
m_r_butts@kbin.social 1 year ago
Ah, there's the dream they're chasing. Chatbots write the code for free, and junior devs play janitor to the robots and fix it up. No more expensive senior devs! The savings can be used as an executive bonus.
scarabic@lemmy.world 1 year ago
So… they failed 13.34% of their own unit tests?
radix@lemmy.world 1 year ago
🎵🎵 99 little bugs in the code, 99 bugs in the code, Fix one bug, compile it again, 101 little bugs in the code. 101 little bugs in the code, 101 bugs in the code, Fix one bug, compile it again, 103 little bugs in the code. 🎵🎵
scarabic@lemmy.world 1 year ago
And how long did it take to compose the “assignments?” Humans can work with less precise instructions than machines, usually, and improvise or solve problems along the way or at least sense when a problem should be flagged for escalation and review.
scarabic@lemmy.world 1 year ago
A test that doesn’t include a real commercial trial or A/B test with real human customers means nothing. Put their game in the App Store and tell us how it performs. We don’t care that it shat out code that compiled successfully. Did it produce something real and usable or just gibberish that passed 86% of its own internal unit tests, which were also gibberish?
m_r_butts@kbin.social 1 year ago
Every company I've been at follows this cycle: offshore to Cognizant for pennies, C-suite gets a bonus for saving money. In about two years, fire Cognizant because they suck and your code is a disaster, onshore, get a bonus for solving a huge problem. In about two years, offshore to Cognizant and get a bonus for saving money. Repeat forever.
This will follow the same rhythm but with different actors: the cheap labor is always there, and sometimes senior devs come in to replace the chatbots because the bots are failing in ways offshore can't make up for: either fundamental design problems that shouldn't have been used as a roadmap, or incompetently generated code that offshore assumes is correct because it compiles. This will all get built up and built around until it's both a broken design AND deeply embedded in your stack. The new role of a senior dev will be contract work slicing these Gordian knots.
BombOmOm@lemmy.world 1 year ago
The new role of a senior dev will be contract work slicing these Gordian knots.
The amount of money wasted building and destroying these knots is immeasurable. Getting things right the first time takes experienced individuals who know the product well and can anticipate future pain points. Nothing is as expensive as cheap code.
m_r_butts@kbin.social 1 year ago
Yes. And nothing is as short-sighted as greed.
kitonthenet@kbin.social 1 year ago
At the designing stage, the CEO asked the CTO to "propose a concrete programming language" that would "satisfy the new user's demand," to which the CTO responded with Python. In turn, the CEO said, "Great!" and explained that the programming language's "simplicity and readability make it a popular choice for beginners and experienced developers alike."
I find it extremely funny that project managers are the ones chatbots have learned to immitate perfectly
thanks_shakey_snake@lemmy.ca 1 year ago
What does it even mean for a programming language to “satisfy the new user’s demand?” Like when has the user ever cared whether your app is built in Python or Ruby or Common Lisp?
It’s like “what notebook do I need to buy to pass my exams,” or “what kind of car do I need to make sure I get to work on time?”
Yet I’m 100% certain that real human executives have had equivalent conversations.
realharo@lemm.ee 1 year ago
And ironically Python (with Pygame which they also used) is a terrible choice for this kind of game - they ended up making a desktop game that the user would have to download. Not playable on the web, not usable for a mobile app.
Knusper@feddit.de 1 year ago
the CTO responded with Python. In turn, the CEO said, “Great!” and explained that the programming language’s “simplicity and readability make it a popular choice for beginners and experienced developers alike.”
Yep, that does sound like my CEO.
blazera@kbin.social 1 year ago
Researchers, for example, tasked ChatDev to "design a basic Gomoku game," an abstract strategy board game also known as "Five in a Row."
What tech company is making Connect Four as their business model?
realharo@lemm.ee 1 year ago
This is also the kind of task you would expect it to be great at - tutorial-friendly project for which there are tons of examples and articles written online.
Other things like that include TODO lists (which is even used as a task for framework comparisons), tile-based platformer games, recipe books and other basic CRUD apps.
gencha@feddit.de 1 year ago
What a load of bullshit. If you have a group of researchers provide “minimal human input” to a bunch of LLMs to produce a laughable program like tic-tac-toe, then please just STFU or at least don’t tell us it cost $1. This doesn’t even have the efficiency of a Google search. This AI hype needs to die quick
atzanteol@sh.itjust.works 1 year ago
This research seems to be more focused on whether the bots would interoperate in different roles to coordinate on a task than about creating the actual software. The idea is to reduce “halucinations” by providing each bot a more specific task.
The paper goes into more about this:
Similar to hallucinations encountered when using LLMs for natural language querying, directly generating entire software systems using LLMs can result in severe code hallucinations, such as incomplete implementation, missing dependencies, and undiscovered bugs. These hallucinations may stem from the lack of specificity in the task and the absence of cross-examination in decision- making. To address these limitations, as Figure 1 shows, we establish a virtual chat -powered software tech nology company – CHATDEV, which comprises of recruited agents from diverse social identities, such as chief officers, professional programmers, test engineers, and art designers. When presented with a task, the diverse agents at CHATDEV collaborate to develop a required software, including an executable system, environmental guidelines, and user manuals. This paradigm revolves around leveraging large language models as the core thinking component, enabling the agents to simulate the entire software development process, circumventing the need for additional model training and mitigating undesirable code hallucinations to some extent.
turmacar@kbin.social 1 year ago
I assume the endgame of this is the boardroom suggestion
guybot asking "is this based on real facts? / does this actually function?"
autotldr@lemmings.world [bot] 1 year ago
This is the best summary I could come up with:
AI chatbots like OpenAI’s ChatGPT can operate a software company in a quick, cost-effective manner with minimal human intervention, a new study has found.
Based on the waterfall model — a sequential approach to creating software — the company was broken down into four different stages, in chronological order: designing, coding, testing, and documenting.
After assigning ChatDev 70 different tasks, the study found that the AI-powered company was able to complete the full software development process “in under seven minutes at a cost of less than one dollar,” on average — all while identifying and troubleshooting “potential vulnerabilities” through its “memory” and “self-reflection” capabilities.
“Our experimental results demonstrate the efficiency and cost-effectiveness of the automated software development process driven by CHATDEV,” the researchers wrote in the paper.
The study’s findings highlight one of the many ways powerful generative AI technologies like ChatGPT can perform specific job functions.
Nevertheless, the study isn’t perfect: Researchers identified limitations, such as errors and biases in the language models, that could cause issues in the creation of software.
The original article contains 639 words, the summary contains 172 words. Saved 73%. I’m a bot and I’m open source!
jabberati@social.anoxinon.de 1 year ago
@shish_mish Sometimes if it sounds too good to be true, it is.
BombOmOm@lemmy.world 1 year ago
The difficult part of software development has always been the continuing support. Did the chatbot setup a versioning system, a build system, a backup system, a ticketing system, unit tests, and help docs for users. Did it get a conflicting request from two different customers and intelligently resolve them? Was it given a vague problem description that it then had to get on a call with the customer to figure out and hunt down what the customer actually wanted before devising/implementing a solution?
This is the expensive part of software development. Hiring an outsourced, low-tier programmer for almost nothing has always been possible, the low-tier programmer being slightly cheaper doesn’t change the game in any meaningful way.
Puzzle_Sluts_4Ever@lemmy.world 1 year ago
While I do agree that management is genuinely important in software dev:
If you can rewrite the codebase quickly enough, versioning matters a lot less. Its the idea of “is it faster to just rewrite this function/package than to debug it?” but at a much larger scale. And while I would be concerned about regressions from full rewrites of the code… have you ever used software? Regressions happen near constantly even with proper version control and testing…
As for testing and documentation: This is actually what AI-enhanced tools are good for today. These are the simple tasks you give to junior staff.
Conflicting requests and iterating on descriptions: Have you ever futzed around with chatgpt? That is what it lives off of. Ask a question, then ask a follow up question, and so forth.
I am still skeptical of having no humans in the loop. But all of this is very plausible even with today’s technology and training sets.
Vlyn@lemmy.zip 1 year ago
If you just let it do a full rewrite again and again, what protects against breaking changes in the API? Software doesn’t exist in a vacuum, there might be other businesses or people using a certain API and relying on it. A breaking change could be as simple as the same endpoint now being named slightly differently.
So if you now start to mark every API method as “please no breaking changes for this” at what point do you need a full software developer again to take care of the AI?
I’ve also never seen AI modify an existing code base, it’s always new code getting spit out (80% correct or so, it likes to hallucinate functions that don’t even exist). Sure, for run of the mill templates you can use it, but even a developer who told me on here they rely heavily on ChatGPT said they need to verify all the code it spits out, because sometimes it’s garbage.
In the end it’s a damn language model that uses probability on what the next word should be. It’s fantastic for what it does, but it has no consistent internal logic and the way it works it never will.
Knusper@feddit.de 1 year ago
Yeah, I’m already quite content, if I know upfront that our customer’s goal does not violate the laws of physics.
Obviously, there’s also devs who code more run-of-the-mill stuff, like yet another business webpage, but those are still coded anew (and not just copy-pasted), because customers have different and complex requirements. So, even those are still quite a bit more complex than designing just any Gomoku game.
NoRodent@lemmy.world 1 year ago
Haha, this is so true and I don’t even work in IT. For me there’s bonus points if the customer’s initial idea is solvable within Euclidean geometry.
Corkyskog@sh.itjust.works 1 year ago
Now I am curious what the most outlandish request or goal has been so far?
akrot@lemmy.world 1 year ago
Absolutely true, but many direction into implementing those solution with AIs.
doublejay1999@lemmy.world 1 year ago
Which is why plenty of companies merely pay lip service to it, or don’t do it at all and outsource it to ‘communities’