Seems like a situation where more energy needs to be put into preventing screwups than would have been spent just doing the job in the first place. If I hired a human assistant who made such a blatant mistake, and they then turned around and did the thing I just now explicitly told them not to, I would be thinking that they are clearly not cut out for that line of work, and continuing to employ them as an assistant would be foolish. But an “AI” agent does this, and it’s just “Oh, let’s keep trying. It just needs a better model. It just needs better prompts. You just need to watch out for its mistakes.” No thanks.
I mean, you sound like you’re happy using it, and I don’t want to tell you you’re wrong. From a technology design viewpoint, it is a fascinating use case. I just think that for a majority computer users this will only make computers even more frustrating and confusing than they already were.