Scientists in China are exploring what it would take to develop "the last AI built by humans" — a powerful, self-improving artificial intelligence (AI) system capable of creating its own successors.

In a new paper, researchers examine how approaches to recursive self-improvement (RSI) could help tackle the rising financial and computational costs of developing ever-more advanced AI models.

RSI refers to the idea that future AI models could direct their own training and development, using what they learn from each round of improvements to guide subsequent rounds of development.

This recursive feedback loop could eventually enable AI systems to develop newer, more powerful generations of models with minimal human input.

The study, which was uploaded to the preprint database arXiv Sept. 10 but has not been peer-reviewed, proposes a framework for classifying AI systems based on their self-improvement capabilities, from Level 1 (lowest) to Level 5 (highest).

At the lower levels, development remains largely human-led. As AI systems move up the scale, they become increasingly capable of identifying their weaknesses and deciding how to address them, eventually gaining the ability to make their own “meta-improvements” — improving how they improve themselves.

Is self-improving AI 'nonsense'?

Whether AI could ever achieve this level of sustained self-improvement remains an open question. Previous experiments have, however, demonstrated more limited forms of RSI.

In one study earlier this year, an AI system called A-Evolve-Training was tasked with improving the performance of a 30 billion-parameter large language model (LLM).

Over four rounds, A-Evolve-Training modestly improved the LLM's benchmark performance by testing different training techniques and using the results to change how it approached subsequent experiments.

Peter Bentley, a professor of digital biology at University College London, questioned whether such experiments offer a credible route to RSI.

"This idea that some AI is magically going to start generating better and better AIs by themselves [is] nonsense," Bentley told Live Science in an email.

Bentley pointed out that self-improving computer systems have been around for decades. Evolutionary computation and reinforcement learning, for instance, are both based on the principle of systems refining their performance over time in response to feedback.

The much greater challenge, he said, is developing AI systems capable of autonomously conducting the research needed to create more powerful successors — not to mention overcoming their dependence on human expertise and physical computing infrastructure.

"You can't 'recursively self-improve' unless you can also enhance your own training, enhance your hardware, and test it exhaustively in real world settings," Bentley added.

The challenge of true RSI

The study's authors acknowledged some of these challenges. Developing RSI-capable systems could require enormous investments in computing power, as well as human supervision to ensure successive improvements delivered meaningful gains. All of this becomes effectively redundant if the resources needed to develop and test subsequent AI models outweigh any benefits they offer, the authors noted.

Scientists would also need a reliable way to determine whether an AI system was genuinely becoming more capable over iterations, rather than exploiting weaknesses in tests used to measure its capabilities. This is a known problem in AI development, where models can achieve higher benchmark scores by exploiting testing methods or cherry-picking results.

Kayla Blomquist, an AI researcher at the University of Oxford and co-founder of the Oxford China Policy Lab, said that, while the paper doesn't provide a practical road map to RSI, it could help developers and policymakers understand the technical challenges and safety risks associated with increasingly autonomous AI systems.

She also questioned whether there would ever be a clear point at which AI became completely independent of human development. "The paper describes a gradual handover of responsibility rather than a single moment, so there may never be an obvious 'last' model built by humans," Blomquist added.

Help us improve Live Science Pro: We're always trying to make our content better. Leave us feedback about Pro here.