Link: http://demandingchange.blogspot.com/2025/07/how-soon-might-humans-be-replaced-at.html
From Systems Thinking for Demanding Change
As noted by Thomas Claburn in The Register, there seems to be a
contradiction between two pieces of research relating to the development
and use of AI in business organizations.
On the one hand, teams
of researchers have developed benchmarks to study the effectiveness of AI, and have found success rates
between 25% and 40%, depending on the situation.
On the other hand, Gartner reports that business executives are expecting a success rate nearer to 60% – if we interpret not-being-cancelled as a marker for success. More
than 40 percent of agentic AI projects will be cancelled by the end of
2027 due to rising costs, unclear business value, or insufficient risk
controls.
History tells us that the adoption of
technology to perform work is only partially dependent on the quality of
the work, and can often be driven more by cost. The original Luddites
protested at the adoption of machines to replace textile workers, but
their argument was largely based on the inferior quality of the textiles
produced by the machines. It was only later that this label was
attached to anyone who resisted technology on principle.
Around
ten years ago, I attended a debate on artificial intelligence sponsored
by the Chartered Institute of Patent Agents. In my commentary on this
debate (How Soon Might Humans Be Replaced At Work?) I noted that decision-makers may easily be tempted by short-term
cost savings from automation, even if the poor quality of the work
results in higher costs and risks in the longer term.
In their look at the labour market potential of AI, Tyna Eloundou et al note that
A key determinant of
their utility is the level of confidence humans place in them and how humans adapt their habits. For instance,
in the legal profession, the models’ usefulness depends on whether legal professionals can trust model
outputs without verifying original documents or conducting independent research. … Consequently, a
comprehensive understanding of the adoption and use of LLMs by workers and firms requires a more in-depth
exploration of these intricacies.
However, while levels of confidence and trust can be assessed by surveying people’s opinions, such surveys cannot assess whether these levels of confidence and trust are justified. Graham Neubig told The Register that this was what prompted the development of a more objective benchmark for AI effectiveness.
Thomas Claburn, AI agents get office tasks wrong around 70% of the time, and a lot of them aren’t AI at all (The Register, 29 June 2025)
Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock, GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (August 2023)
Wikipedia: Luddite
Related Posts: How Soon Might Humans Be Replaced At Work? November
2015, RPA – Real Value or Painful Experimentation? (August 2019)
