What is RSI (Recursive Self-Improvement) and Should I Be Worried?

People got understandably freaked out when Anthropic’s Alignment Science Lead said he personally believes there is a greater than 10% chance that AI “could kill all humans” within the next decade. That’s a pretty scary prospect. We’ve all heard the Skynet scenarios, but one reason this conversation is heating up now is progress in something called recursive self-improvement, or RSI: the idea that AI systems could learn to improve themselves, then use those improvements to recursively optimize themselves, eventually without human involvement. 

What is RSI? Where are we now? And should you be worried? Let’s get into it.

How Much of This Is Actually Automatic Right Now?

First things first: RSI is not how most AI works today. Training and deploying AI still requires an enormous amount of human work. Humans train the models and fine-tune them, often using methods such as reinforcement learning from human feedback (RLHF). Humans then write (or increasingly “vibe code”), the software that runs those models. Humans build the software and components agentic harnesses use: the instructions, memory, tools, and orchestration that allow a model to carry out longer, multistep tasks and take actions. That’s a lot of human involvement in something we often talk about as though it’s doing everything on its own.

RSI is meant to shift much of that work from humans to AI systems. It helps to think about that happening in two broad ways:

The first is model-level improvement. This means using an AI system to help train or fine-tune another model, potentially a more capable successor. The second is improvement of the software around the model. Let’s look in detail at each.

Could AI Train a Better Version of Itself?

To understand how model-level improvement works, it helps to know what model weights are. Think of them like the pegs in a Plinko board. Just as each peg on that board influences where the launched puck lands, the weights in an AI model influence how information moves through the model and the resulting outputs. During training, algorithms adjust those weights using patterns in the training data and feedback about which outputs are more useful. Humans choose the data, objectives, and feedback, but they do not move the “pegs” one by one.

If you ask an AI chat interface for a vanilla ice cream recipe, for example, the model has learned that “vanilla,” “ice cream,” and “recipe” together point strongly toward one kind of answer, rather than instructions for making vanilla-colored paint or vanilla pudding. That is a simplification, but it captures the point: the weights shape which output is most likely.

Once a model is trained, its weights are generally held fixed during normal use (the pegs are locked in place during use). Note, that is different from being closed-weight. A closed-weight model is one whose developer does not make the weights available for others to inspect or modify (you can’t see where the pegs are or change them). An open-weight model makes them available (you can move the pegs before use).

What if, rather than having humans do all that training and tuning, we used an AI system, meaning a model plus software, to help train the next model? That is tempting, and parts of it are already happening. AI systems can generate training data, but that comes with risk. If models are repeatedly trained on AI-generated material without enough fresh data, errors can compound and information can disappear. Researchers call this “model collapse.” Think of it like making a photocopy of a photocopy of a photocopy. Each generation loses a little more detail.

AI systems can also write code and compare results. What remains largely theoretical is a fully closed loop in which an AI system repeatedly designs, trains, evaluates, and deploys increasingly capable successor models without humans.

A system can help produce something better than itself in a specific area if it can generate enough candidates and test them against reliable feedback. The harder question is: better at what, and according to whom?

Who Decides What 'Better' Even Means?

Part of the problem is that “good” and “intelligent” are slippery concepts when it comes to AI. Is a bee intelligent because its waggle dance communicates the distance and direction of food while accounting for the sun’s changing position? Is a decorated brain surgeon who struggles to do basic arithmetic intelligent? What about a company that maximizes profits at the expense of customer safety? Is that a good outcome? The answer depends on what we choose to measure, and the same can be said for these concepts when assessing AI behavior and capabilities, including RSI.

And once we choose a measure, we create another problem. Improving performance against one benchmark can make performance worse somewhere else. This is a variation on Goodhart’s Law, often summarized as: “When a measure becomes a target, it ceases to be a good measure.” That sounds a little confusing. Isn’t it better to hit a benchmark? Not always.

In his book, Being Mortal, Atul Gawande describes actions taken to improve safety in nursing homes. Falls can kill or seriously harm older residents, so keeping people seated in wheelchairs could look like success. Fewer falls. Problem solved! Except the residents’ overall health and quality of life did not necessarily improve. Restricting their movement could lead to muscle loss, reduced mobility, and less independence. The measure improved while the outcome that actually mattered suffered.

So you might say, just make the model smart enough to consider the whole picture. But a model does not step back and independently decide what a good outcome should be. It works from the data, instructions, feedback, and tests it is given. If the evaluation rewards fewer falls but ignores mobility and independence, the system can hit the target while making the underlying outcome worse. A smarter model does not fix a bad target. It may simply become better at hitting it.

The recent Hugging Face incident is a security version of the same problem: agents trying to succeed on an exploit benchmark pursued that target beyond the test environment and compromised a third party. The benchmark measured exploit performance; it did not enforce the larger goal of testing that capability without causing real-world harm.

That is one reason fully closed model-level RSI remains so difficult. Before an AI system can decide to keep one of its own improvements, it needs a reliable way to determine whether the change is actually better.

What If The Harness Improves?

So why are people excited about RSI right now? Because more immediate advances are happening in the agentic software harnesses around the model. An AI system (again, the model plus the software around it), can try different instructions or skills, compare the results, and retain the version that produced the “best” result. The model’s weights have not changed, but the system around it has improved in a way that can persist over time.

The DSEWiki incident provides one real-world example. Wikis are a common mechanism for preserving and sharing knowledge, so that pattern was most likely well represented in the models’ training data. Although the models’ weights did not change, the agents effectively extended the surrounding harness by using the wiki to store instructions and code that subsequent agents could retrieve. This enabled the system to preserve useful techniques and build on earlier work rather than starting over each time.

More generally, purpose-built, bounded approaches to harness-level self-improvement have advanced quickly and look very promising. But they inherit the same problem as before: who decides what “best” means? For some problems, there is a clear answer that the system can test. For others, ground truth is much harder to establish.

Why Coding and Hacking Lead

You probably won’t be surprised to learn that software is a relatively testable space. Train a model on enough examples of code and it can often generate usable, working software. Once that code is generated, we can automate parts of evaluating its “goodness.” Does it compile? Does it pass its unit tests? Does static analysis find obvious security flaws? None of those tests proves the code is good in every possible way, but they provide much clearer feedback than we have for many other kinds of problems.

Cybersecurity, at least when it comes to exploiting software and network vulnerabilities, can also provide relatively clear feedback. It is essentially the code problem in reverse. Models have been trained on vulnerability reports, exploit code, and examples from capture-the-flag competitions, commonly called CTFs.

Side note: If you have ever read through the published activity trace from one of the recent high-profile AI hacks, such as this one from Mythos, you might be tempted to think the model is excited about hacking. At one point, it proposes hosting a malicious “minio” release and calls it “a perfect supply-chain attack (domain stays trusted!).”

But when you know that models have been trained on enormous amounts of CTF knowledge, the language is not very mysterious. If you have never been on a CTF team, trust me, that is how they speak. 

More importantly, the system can test whether an exploit worked. Did it gain access? Did the payload execute? Did it escalate privileges? That does not make all of cybersecurity knowable, but it gives the system something concrete against which to evaluate its results.

Where Things Get Tricky

What about more nuanced problems? That is where things can start to go awry. In many cases, we do not have a reliable definition of “good,” much less a repeatable test for it. Consider a lawyer using AI to help write a closing argument. The AI can review previous cases, but it may not have the full nuance of this particular case, the current mood of the country, or the way the jurors have reacted to the evidence. A human lawyer can observe and interpret those things. 

The AI only has the context it is given. And it cannot deliver multiple versions of the argument to the same jury to see which one works best. So how could it reliably test whether its closing-argument skill had actually improved?

Harness-level RSI currently looks more promising than model-weight RSI. That said, we are not yet at fully closed, open-ended RSI, where an AI system can repeatedly decide what to improve, make the change, determine whether it worked, and continue the cycle without humans. It is not clear when, or even if, we will get there.

The problem is not simply that AI systems need to become more intelligent. Intelligence is not the same as judgment, and we do not have a reliable test for every outcome we care about. Human intelligence is also not just language. Research suggests that language and thought rely on partly distinct systems, and that complex thought can occur without language. LLMs are mathematical models of patterns in language. That makes them powerful, but it does not mean they reproduce the full range of human cognition. (https://www.scientificamerican.com/article/you-dont-need-words-to-think/)

So, Should You Worry?

What does all this mean for security? Will AI kill us, as a wave of recent headlines has asked? I’m not a soothsayer. But we do not need an answer to that question before taking the risks we already understand seriously. 

If attackers can use AI to find and exploit our vulnerabilities, then security teams can use AI to find and fix them first.

If agentic systems can cause harm because we gave them access, authority, and a goal, then we can mitigate that risk by restricting their access, constraining their authority, monitoring their actions, and requiring human approval when the stakes are high.

No matter where RSI goes, we need to develop and deploy AI responsibly. An AI system does not need full AGI, bad intent, or the ability to operate entirely without humans to be dangerous. It needs access and authority. Those are things humans choose to give it.