On smarter models and the work of understanding what they produce
A business can now ask an AI system to undertake work that once needed the time of several colleagues. If a developer uses it to build a backend, there are ways to test whether the system does what it should. In knowledge work, a sound result also leaves a further question: who understands how the team arrived there?
A plan can look convincing when it is presented. Months later, a customer may challenge one of its assumptions, or events may show that the plan needs to change. The people responsible will need to explain the choices behind it and know where to look again. If that understanding has remained with the system, where does the company's own knowledge reside?
The thought that would not stay put
At Octaflow, our head developer can now build an application for the app store in a week that previously might have taken a team of developers three months, with fewer mistakes along the way. Browser-based tools can help us find contacts for businesses in days, where the same work might once have taken months of effort.
Then there is the experience of working through a basic written argument with Claude or ChatGPT. A point slips out of the agents’ view. The same evidence has to be examined again, only for the answers to change. Time returns to the process in a different form: trying to understand why a thought that was clear at the beginning has become so difficult to hold together.
The world inside the score
The unevenness deserves attention. The decisions that go into training a model can improve some capabilities more than others. There are also serious efforts to measure what models can do with knowledge work. OpenAI’s GDPval, for example, draws on tasks created by experienced professionals across 44 occupations. Its first version asks a model to complete each task in a single attempt, with a defined brief and reference material.[1]
A winner in whose hands?
Researchers in Japan approached the question from inside an insurance business. Using the company’s own data, they compared three models on two tasks: answering questions about documents and turning image content into text. One model led on the first task; another led on the second. The researchers found that their results differed from those of general-purpose benchmarks.[2]
If you work in insurance, or in any business with its own particular body of knowledge, this gives you a more useful question to ask of a model’s score: what did it have to do to earn it, and how close is that to the work in front of you? Even within one company, the answer may depend on the task.
The work before the question
The way a task is presented can make an enormous difference. In Argentina, researchers gave 1,174 adults a business problem, with text, charts and tables to examine. Participants had to identify the cause of a difficulty and write a practical response. Those given access to GPT-4.1 performed better across both educational groups in the study; the gap between the groups narrowed substantially.[3]
That result complicates the frustration at Octaflow. People used AI to reason through information and write a better response. The problem and the material had been selected for them. In a company, somebody may first need to decide which problem deserves attention, which documents belong in the enquiry, and whose knowledge is missing. Those choices shape what the system has an opportunity to find.
When confidence does the thinking
In a study of 319 knowledge workers, researchers found that greater confidence in AI was associated with less self-reported critical-thinking effort on the tasks people described. The study captures how people reported working, rather than measuring a lasting loss of ability.[4]
That distinction matters. A system may produce an answer so coherent that the wrong turn becomes harder to see. As the models improve, the person using them may need to become sharper too: able to recognise which detail deserves a second look, and clear enough about the intended outcome to notice when the work is drifting away from it.
The mistake that keeps its company
A finance case in DAYJOB, an experimental benchmark for professional work, illustrates another difficulty. An agent identified two pricing exceptions in a set of reports and recommended holding them. It missed a separate error: a share price had been entered in the wrong currency unit, making a position appear much larger than it was.[5]
The agent had found genuine reasons for caution. The missed unit changed the apparent size of the exposure. This single benchmark case cannot tell us how often such errors occur at work. It shows how an enquiry can catch something important while another detail needed for the decision remains overlooked. As systems become better at finding mistakes, how will we know which mistakes we have stopped looking for?
Agreement can share a blind spot
Imagine separate agents investigating a new market: one examines demand, another the financial case, a third the risks. If all three were given the same estimate of the market, however, their agreement could leave the most consequential number untouched. The person directing them might ask another agent to trace that estimate to its source, or to look for evidence that would make the opportunity appear smaller.
The amount of information we can give these systems keeps growing. The harder work is choosing what matters, recognising when the person supplying it has become the bottleneck, and noticing when an agent is carrying the work around in circles. A better prompt may help; so may changing the order of the work, bringing in a different source, or stopping to reconsider the outcome altogether. How do we learn to make those choices well, while the systems we are learning to direct keep changing?
Rajan Anand
Co-founder, Octaflow Labs
Sources
- OpenAI, Measuring the performance of our models on real-world tasks. The first version of GDPval is a one-shot assessment spanning 44 occupations.
- Tamao Shimizu, Hibiki Bannai and Yoshiaki Onishi, Performance Evaluation of Multimodal LLM with Life Insurance Business Data, 2025. Findings described from the accessible abstract.
- Guillermo Cruces and colleagues, ¿La IA iguala o amplía las brechas de productividad entre trabajadores?, Sur Futuro, 2026.
- Hao-Ping Lee and colleagues, The Impact of Generative AI on Critical Thinking, CHI 2025.
- Stephanie Finley and colleagues, DAYJOB: A Benchmark for Long-Horizon Professional Work, preprint, 1 October 2026, section 6.