Essay
AI's Self-Improvement Bubble & The Need for Anchored Loops
Improvement only means something when it's measured by something the loop can't reach.
Debate rages on about if AI is a “bubble” in the financial sense. I’m not writing here to take sides on that topic directly, but to explore a key property of AI development that may well hold the keys to the economic bubble question.
A financial bubble happens when prices start rising for a self-referential reason: because prices are rising. People have a lemming-like tendency toward following what others are doing or what they believe. Socially-buffeted hype inflates until it meets cruel reality and things bottom out spectacularly, from tulips in 1637 to petfood websites in the early aughts.
In essence, a bubble is a loop whose buoyancy has been shielded from critical external facts.
AI as an industry undoubtedly has some bubble-like character, even if there’s a consensus of huge value at its nucleus. But there’s another dimension of AI as a bubble that is equally interesting. A self-reinforcing bubble of development: AI systems that get “better” by feeding on their own outputs, with less and less contact with anything it didn’t make itself.
As the gains plateau from training on pre-AI-era data (with nearly all available human-produced data having been gobbled up), the internet is increasingly full of AI-generated content (50% of new articles by some estimates) to the dismay of a reader whose attention switches off the moment they sniff inauthenticity. When the cost of content creation is essentially free, humans will tend to take the easiest path to whatever their content goal may be. When we use AI content to help improve AI models, researchers label the typical negative outcome as “model collapse.” AI content has neatly sanded edges, and each training pass over-samples what’s common and skips a little of what’s rare. The unusual nuances, the incongruous facts slide away a sliver at a time as the model settles toward a confident, fluent average. Things get blander and more confident while drifting away from the actual, imperfect shape of the world. You can’t reverse engineer the signal.
Others have started calling this degenerative AI, and the word earns its medical weight. Degenerative conditions creep forward silently from the inside, a slow narrowing of function you can miss until it’s far enough along. In image models you can actually watch it happen in real time - grotesque distortions compounding with every generation until the original signal is completely eaten by the average (Jathan Sadowski famously dubbed systems like this as “Habsburg AI” after the inbred historical dynasty).
The shared corpus we all draw from is turning into a hall of mirrors. People are already voting with their eyeballs, retreating into group chats, paid media and newsletters, tacking “reddit” onto their searches in a small daily act of looking for a real human pulse.
We’ve all been living inside one of these loops for years in the form of our algorithmic recommendation engines - self-reinforcing bubbles pointed at a single person. You engage with what it shows you, it learns the narrower you and serves that back. Over time, your inputs shrink toward whatever your past self rewarded. Since attention is the currency, and nothing holds attention like outrage, this pattern sharpens us into a touchier, more reactive version of ourselves, and trains on that. We called this a filter bubble fifteen years ago and mostly stopped worrying about it. It’s worth worrying again, because the same machinery now runs the data layer underneath many models.
My team builds a lot of AI agents in healthcare, and often does so with heavy human inputs into custom tools we’ve built to manage prompts, multi-agent reviews, and knowledge bases. The level of effort is real, so we are building agents-creating-agents loops to take some manual efforts off our hands. One agent invents test cases, including the nasty edge ones we’d forget. Another runs the agent we’re trying to improve. Another scores the results and says what went wrong. A final agent rewrites the prompt or re-wires agent architecture to improve it. Then the whole thing goes around again, faster than we could iterate as human bottlenecks.
Critical to this process is separating the models and agents doing the work. An early first run using a single agent to do multiple jobs looked spectacular, with quality scores nearly doubling in just a few loops. Then we realized the now-obvious fallacy behind the single-agent results. One model had quietly been playing every role: author, worker, and judge, grading its own homework and admiring the results. So we swapped fully separate agents that only ever saw their part of the process, and the story flipped. True quality had barely moved in the few loops (in fact, the first output was pretty good, where improvements were hard-won), and the big jump we observed previously was just a single system scoring its own work and flattering itself toward the narrative it expected we wanted.
Improvement only means something when it’s measured by something the loop can’t reach.
This isn’t a localized hazard for mid-sized engineering teams, it is the central tension at the frontier of AI. In June 2026, the Anthropic Institute published a landmark report revealing that over 80% of their internal codebase is now authored autonomously by Claude. Their autonomous agents are shipping code and fixing bugs at a pace that compresses what they estimated as four years of human labor into a single run.
Yet, the labs are running into the exact same structural wall we found in our sandbox. Anthropic openly acknowledged that their internal engineering bottleneck has shifted entirely to human code review. They are illustrating an important lesson in machine introspection: a system cannot safely judge the ultimate intent and safety of its own design. Recursive self-improvement and model collapse are fundamentally the same operation - a system consuming its own tail - distinguished only by whether an unyielding, outside anchor is present to validate the output.
This is where the two meanings of the word collapse into one.
A financial bubble bursts when value is detached from reality. A self-reinforcing bubble is the mechanism that unhooks it. As more evals are established, systems are taught the test, like students in high school preparing for state exams, where gains may not be highly generalizable to real scenarios. Or in healthcare. Models acing the licensing exam (best models are 96% on the USMLE question bank) still falter with a real person: diagnostic accuracy that sits at 82% on tidy case vignettes falls to 63% once it’s a multi-turn conversation with an actual patient. Barely 5% of medical-AI studies test against real patient data.
Even inside labs, people now admit that self-reported AI gains tend to run well ahead of the third-party-measured kind. Scale that across an industry and you’ve built the exact conditions for the financial kind of bubble.
So is AI a bubble? It becomes a financial bubble to exactly the degree it becomes a purely self-reinforcing one. It’s a failure to stay properly anchored, and the anchor can be bought back.
The role for AI here is real but it lives under one rule: the loop has to stay bolted to an external verdict. Where that rule holds, AI improving AI genuinely compounds. It’s why AlphaFold earned a Nobel - by being validated against structures confirmed in the lab. It’s why “Claude writes 80% of the code” is believable rather than self-flattery: code compiles and passes tests, or it doesn’t.
The research is oddly reassuring, and the industry isn’t passive about any of this. Models don’t collapse when real data keeps getting mixed back into training; only when synthetic data is allowed to fully replace it. The labs know this, so they’re filtering data, generating better synthetic data, standing up verifier models to check the work, holding back evaluation sets the optimizer never gets to see (similar to how we manage multi-agent improvement loops at our company).
The same shape shows up well outside the data center. The “cognitive debt” researchers keep measuring (a real dip in people’s own thinking and persistence when they lean too hard on these tools) appears under passive use, where you simply take the output as it appears. It mostly disappears under active, effortful use of AI, where you stay in the loop, challenge, and co-develop instead of letting AI replace your thought.
In a 2025 Lancet study, experienced endoscopists, with thousands of procedures behind them, got measurably worse at spotting precancerous polyps on their own after a few months leaning on an AI detector: their unassisted detection rate fell from 28% to 22%. The first hard evidence that the tool, used passively, de-skills even experts.
Synthetic data is cheap. Basically infinite. Real human data is slow and costly and there’s only so much of it. In terms of human effort expenditure, passive acceptance of AI outputs is free, while staying engaged costs you the very time you were trying to save in the first place.
Keeping the loop open - to your own mind - is a cost you have to keep deciding to pay.
I’ve written before about the friction we keep removing, the messy and inefficient collision of real perspectives that understanding actually requires. This is that same argument from the other side. The thing that keeps a loop from curdling into a bubble is contact with what it can’t author: real data in the mix, a number the machine wasn’t allowed to pick, a colleague who tells you you’re wrong, a walk outside, the world that is. The dumbest-sounding instruction on the internet - “touch grass” - turns out to be the best advice we’ve got for us as people, but also for our AI training. Also, the most expensive.
The future of AI isn’t a bubble because it has to be; it’s a bubble only if we let it seal. Generative AI is destined to eat itself if we don’t allow its loops the chance to breathe. There is a vital role for machines in the process of improving machines, but we have to keep the brains separate from their tasks.
We have to let human data reign supreme in our test cases, and we must willingly incur the friction of human oversight. If we don’t, the macro-economy will face the same reckoning Anthropic hit in its codebase, where human review became the bottleneck. Our minds will suffer the same fate as expert professionals, losing the ability to see things clearly when AI is turned off.
The job in front of us, technically and economically and personally, is to keep AI as a semi-permeable membrane.
We have to keep letting the outside in faster than the loop can close around us.