Was this around the time right after “custom GPTs” was introduced? I’ve seen posts since basically the beginning of ChatGPT claming it got stupid and thinking it was just confirmation bias. But somewhere around that point I felt a shift myself in GPT4:s ability to program; where it before found clever solutions to difficult problems, it now often struggles with basics.
Comment on OpenAI says it is investigating reports ChatGPT has become ‘lazy’
rtfm_modular@lemmy.world 11 months ago
Yep, I spent a month refactoring a few thousand lines of code using GPT4 and I felt like I was working with the best senior developer with infinite patience and availability.
I could vaguely describe what I was after and it would identify the established programming patterns and provide examples based on all the code snippets I fed it. It was amazing and a little terrifying what an LLM is capable of. It didn’t write the code for me but it increased my productivity 2 fold… I’m a developer now a getting rusty being 5 years into management rather than delivering functional code, so just having that copilot was invaluable.
Then one day it just stopped. It lost all context for my project. I asked what it thought what we were working on and it replied with something to do with TCP relays instead of my little Lua pet project dealing with music sequencing and MIDI processing… not even close to the fucking ballpark’s overflow lot.
It’s like my trusty senior developer got smashed in the head with a brick. And as described, would just give me nonsense hand wavy answers.
backgroundcow@lemmy.world 11 months ago
Linkerbaan@lemmy.world 11 months ago
Maybe they’re crippling it so when GPT5 releases it looks better. Like Apple did with cpu throttling of older iphones
tagliatelle@lemmy.world 11 months ago
They probably have to scale down the resources used for each query as they can’t acale up their infrastructure to handle the load.
backgroundcow@lemmy.world 11 months ago
This is my guess as well. They have been limiting new signups for the paid service for a long time, which must mean they are overloaded; and then it makes a lot of sense to just degrade the quality of GPT-4 so they can serve all paying users. I just wish there was a way to know the “quality level” the service is operating at.
monkeyslikebananas2@lemmy.world 11 months ago
This is most likely the answer. Management saw the revenue and cost and said, “whoa! Turn all that unnecessary stuff off!”
Meowoem@sh.itjust.works 11 months ago
I do think part of it is expectation creep but also that it’s got better at some harder elements which aren’t as noticeable - it used to invent functions which should exist but don’t, I haven’t seen it do that in a while but it does seem to have limited the scope it can work with. I think it’s probably like how with images you can have it make great images OR strictly obey the prompt but the more you want it to do one the less it can do the other.
I’ve been using 3.5 to help code and it’s incredibly useful for things it’s good at like reminding me what a certain function call does and what my options are with it, it’s got much better at that and tiny scripts like ‘a python script that reads all the files in a folder and sorts the big images into a separate folder’ or something like that. Getting it to handle anything with more complexity it’s got worse at, it was never great at it tbh so I think maybe it’s getting to s block where now it knows it can’t do it so rejects the answers with critical failures (like when it makes up function of a standard library because it’d be useful) and settles on a weaker but less wrong one - a lot of the making up functions errors were easy to fix because you could just say ‘pil doesn’t have a function to do that can you write one’
So yeah I don’t think it’s really getting worse but there are tradeoffs - if only openAI lived by any of the principles they claimed when setting up and naming themselves then we’d be able to experiment and explore different usage methods for different tasks just like people do with stable diffusion. But capitalists are going to lie, cheat, and try to monopolize so we’re stuck guessing.
BleatingZombie@lemmy.world 11 months ago
“ChatGPT Caught Faking On-Site Injury for L&I”