稍早一点，在 2007 年，Yudkowsky 还在《Levels of Organization in General Intelligence》里定义了与之相关的"种子 AI"：
种子 AI 是为自我理解、自我修改和递归自我改进而设计的 AI。这对"达成原始智能所需的功能架构"有影响，也对"如果并且当这个 AI 的整体式自我理解开始提升之后，它会如何发展"有影响。种子 AI 不是一条靠从无智能内核自举、从而绕开通用智能这道难题的捷径；种子 AI 只有在已经有一定程度可利用的智能之后，才开始产出好处。种子 AI 的后续后果（比如真正的递归自我改进），只有在这个 AI 已经获得相当的整体式理解和通用智能之后才会显现。
考虑到今天的模型有多通用、多有用，认为我们正处在这个起点上是合理的。
一般来说，RSI 可以这样概括：当 AI 能改进自己，改进后的版本又能更高效地继续改进，形成一个闭合的放大循环，导向智能爆炸，也就是常说的奇点。这里面有几个假设。要让 RSI 发生，必须满足：
循环是闭合的。模型能不断改进自己，并生出更多的模型。
循环是自放大的。下一代模型带来的改进会比这一代更大。
循环在持续运转时不损失效率。没有额外的摩擦把那条指数拦腰砍成一个早熟的 sigmoid。
虽然我同意未来几年持续的 AI 进步会带来重大的、动摇社会的变化，但我预期，当我们回头看时，那条进展曲线会更接近线性而不是指数。取代递归自我改进的，将是有损的自我改进（LSI）——模型成为开发循环的核心，但摩擦会打破 RSI 的每一条核心假设。你往一个问题上扔越多的算力和 agent，冒出来的损耗和重复就越多。
02
能自动化的研究太窄
以 Paul Allen 的「复杂度刹车」打底：agent 擅长优化单一指标，而最好的研究，是让许多可扩展的想法一起起作用。
Interconnects AI 是一份靠读者支持的刊物。欢迎考虑成为订阅者。
我依然相信，复杂度刹车会成为一股强力的反向平衡——尽管现实是，在我们造一个领先 AI 模型所要拼起来的每一项窄任务上，AI 模型都在明显变好。2025 年 4 月回应 AI 2027 时，我引用过下面这段。
微软联合创始人 Paul Allen 提出了与加速回报相反的说法：复杂度刹车——科学对智能的理解越往前推进，再往前推进就越难。一项关于专利数量的研究显示，人类创造力并不呈现加速回报，反而如 Joseph Tainter 在《复杂社会的崩溃》里所指出的，是一条递减回报律。每千人的专利数在 1850 至 1900 年间见顶，此后一直在下降。复杂度的增长最终会自我设限，并导致大范围的"一般系统性崩溃"。
首先，很清楚的是，今年的语言模型已经会是优化局部任务的有用工具，比如把模型的 test loss 降下来。Andrey Karpathy 最近发布的 autoresearch 就把这件事做火了。它让 AI agent 直接在 GPU 上开干，瞄准"降低测试集损失"这类任务。这条路在窄域里有效，也就是只有一个总的 test loss 或者一个总的 reward 的时候。问题在于，"纸面上更准的模型"和"用户觉得更有生产力的模型"之间，一直存在一道长期的缺口。最刺激的例子是预训练，围绕 scaling law 的讨论已经谈得很多了：scaling law 告诉我们 loss 会继续往下走，但我们不知道那是不是在经济上更值钱。
在后训练里，强化学习算法至少和具体的性能提升绑得更直接，因为大多数 RL 训练环境可以直接拿来当评测用。但我仍然担心泛化，以及它能不能接回到"模型更擅长改进自己"这件具体的事上。从"模型在某些事情上变好"到"这必然意味着模型更擅长造自己、设计实验"，是一大跳。我们见过很多 AI 能力在某个人类品味的水平上就差不多饱和了，比如写作质量。AI 研究在这里有点不一样，因为它的天花板高得多。写作之所以会饱和，是因为偏好本身有内在张力；而模型在研究上会饱和，是因为搜索空间和优化目标太宽了。
这个话题我能一直说下去。另一个例子来自我读博期间（2017-2022），当时有个叫"AutoML"的领域炒得极热，想用贝叶斯优化之类的技术去找新的架构和参数。那股热度从没改变过我的工作。语言模型能做的比这多，但短期内还不足以抢走顶尖 AI 研究者的饭碗。研究者的核心货币，仍然是直觉和对复杂度的管理，而不是具体的优化和实现。
在 AI 里这一点应该更容易讲清楚，因为计算机底层的运作细节相当神秘。设想一位 AI 研究者，从手写代码，到用 AI 自动补全辅助，再到用自主的编码 agent。这每一步都是巨大的收益。我们接着往下走。现在这位研究者同时用 3-4 个 agent 处理手头问题的不同子任务或不同路线。这仍然是很大的收益。现在再设想这一位研究者，每天要组织 30-40 个 agent 的任务。有些人能从这个量级里榨出更多价值，但不多。
你觉得有多少人能每天给 AI agent 想出 300-400 个任务？不多。这个问题很快也会撞到 AI 模型自己头上。
3. 资源瓶颈与政治
根本上，所有 AI 公司都在走一条钢丝：拿到大笔资本，靠足够的需求把新的算力转化成收入，然后重复这个过程，同时在研究上砸下极高的开销。在这种资源规模下，永远会有"谁拿到资源、押注什么"的政治瓶颈。在这一层，研究领导层坐在 AI 和研究者之上。即便模型继续变好，这一处摩擦也永远不会被移除。它算不上一处很大的摩擦，但 AI 模型从根本上是在"人是资源瓶颈"的组织里运转的。
语言模型带来的早期规模的改进，是那些局部优化，所用资源每天成本不到一百万美元。结合我对 AI 各种摩擦的其他看法，这一条本身对改进速度的影响非常小；但对那些担心快速起飞、RSI 和对 AI 失控的人来说，有一点应该是显然的：几十亿美元的研究算力，不太可能被完全隔离出来，交给 AI 做端到端的自我实验。
04
每条 S 曲线的底部都像指数
工程自动化和基础研究自动化都是真的，但下一个阶段没有现成的规模可加，爬山式的加速换不来范式跃迁。
这里的结论是：因为我们正处在"自主地、大规模地用 AI 辅助 AI 开发"的早期，我们正集体发现 AI 能怎样大规模地帮到我们。我们都在用这些工具去摘看得见的低垂果实，我们的工作确实正在变得节奏更快、产出更多。问题是，所有这些轴上都有明确的人的、政治的或技术的复杂度瓶颈。
这仍然可能跨过 AGI 那个通俗意义上的门槛，也就是成为大多数远程工作者的直接替代品，那会是一个了不起的里程碑。关于"我们未来几年会不会达到 AGI"这场争论，难点在于 AI 模型是锯齿状的，聪明的方式和人不一样，所以它们看起来不会像远程工作者的直接替代品；但在很多情况下，直接用 AI 会比试着跟一个人合作有效得多。它正在重塑"工作"是什么。
我们来把手上的几种情形过一遍。
工程正在被自动化。人的产出高得多，模型能更快地把复杂的基础设施部署跑通，GPU 利用率更高，等等。基础设施上的收益，会变成实验的速率和规模上的固定改进——而实验正是 AI 进展的基本单位。
基础的 AI 模型研究与优化会被自动化。AI 模型的作用范围在扩大——从写 kernel 转到决定架构。这是从"改进实验工具箱"走向"自己跑一些小实验"。配置、超参数这些，会变成 AI 助手的地盘。
这两件事都是真的。问题在于，第三个时代没有一个简单的、可以跳上去的标度。在 AI 模型能靠综合与执行创造知识之后，下一跳要求的是驾驭数千个 agent，或者让模型做出更新颖的发现——比如解锁推理时 scaling 之后的下一个范式。AI 带来的下游改进会让整个行业在爬山这件事上被极大增强，但我担心这不会带来新的 AI 范畴所需要的范式转变——持续学习、世界模型，随你喜欢哪一味。
AI self-improvement is real but lossy: friction breaks every assumption of recursive self-improvement, and in hindsight it will look linear, not exponential.
Indigo's conclusion
In the argument over whether AI takes off fast, this is the clearest voice for slower: AI wins where problems are narrow and machine-checkable, which is his “too narrow” point; real paradigm shifts still won't grow out of hill-climbing.
How to read this A clear-headed technical case for a slower path, with no commercial angle and a consistent view. It is the opposite of Pachocki's “I strongly expect recursive self-improvement”: a credible outside researcher says it is lossy, not recursive. Mind the date: it was published six months ago; below it is checked against September's math results.
What to remember
Lossy, not recursive: self-improvement is real, but friction breaks the closed loop, the speed-up and the no-loss assumptions.
The three frictions make a checklist: one narrow metric or many broad ones? More agents or more human intuition? Are resources stuck in politics?
“The bottom of every S-curve looks exponential”: 2023, 2025 and 2026 each rode one, and each was only the bottom of an S-curve.
Don't price in a fast takeoff. This is a bounded exponential: capability keeps improving for real, but “it takes over everything next year” needs cold water.
Breakdown · 4 steps
01
Friction breaks the three assumptions of recursive self-improvement
He defines recursive self-improvement and its three assumptions: the loop closes, each round improves faster, nothing is lost. Then he proposes lossy self-improvement instead. Read this part →
02
The research that can be automated is too narrow
Built on Paul Allen's “complexity brake”: agents are good at optimizing a single metric, while the best research makes many scalable ideas work together. Read this part →
03
More agents saturate; people are the last bottleneck
Amdahl's law: even 10,000 remote workers in a data center can't all be pointed at one problem. Above that sits the politics of who gets resources and what they bet on. Read this part →
04
The bottom of every S-curve looks exponential
Automating engineering and automating basic research are both real, but the next phase has no ready scale to climb. Faster hill-climbing won't buy a paradigm shift. Read this part →
What it means for Rewired Index
The baseline for pricing AI's capability curve is a bounded exponential: hill-climbing on fast-forward, not a sudden takeoff. Until a paradigm shift arrives, value concentrates in narrow, checkable domains. No direct names.
What would change my mind
a paradigm shift such as continual learning or world models growing straight out of step-by-step automated research.
How to read this
A clear-headed technical case for a slower path, with no commercial angle and a consistent view. It is the opposite of Pachocki's “I strongly expect recursive self-improvement”: a credible outside researcher says it is lossy, not recursive. Mind the date: it was published six months ago; below it is checked against September's math results.
Friction breaks the three assumptions of recursive self-improvement
He defines recursive self-improvement and its three assumptions: the loop closes, each round improves faster, nothing is lost. Then he proposes lossy self-improvement instead.
Fast takeoff, the singularity, and recursive self-improvement (RSI) are all top of mind in AI circles these days. There are elements of truth to them in what’s happening in the AI industry. Two, maybe three, labs are consolidating as an oligopoly with access to the best AI models (and the resources to build the next ones). The AI tools of today are abruptly transforming engineering and research jobs.
AI research is becoming much easier in many ways. The technical problems that need to be solved to scale training large language models even further are formidable. Super-human coding assistants making these approachable is breaking a lot of former claims of what building these things entailed. Together this is setting us up for a year (or more) of rapid progress at the cutting edge of AI.
We’re also at a time where language models are already extremely good. They’re in fact good enough for plenty of extremely valuable knowledge-work tasks. Language models taking another big step is hard to imagine — it’s unclear which tasks they’re going to master this year outside of code and CLI-based computer-use. There will be some new ones! These capabilities unlock new styles of working that’ll send more ripples through the economy.
These dramatic changes almost make it seem like a foregone conclusion that language models can then just keep accelerating progress on their own. The popular language for this is a recursive self-improvement loop. Early writing on the topic dates back to the 2000s, such as the blog post entirely on the topic from 2008:
Recursion is the sort of thing that happens when you hand the AI the object-level problem of “redesign your own cognitive algorithms”.
And slightly earlier, in 2007, Yudkowsky also defined the related idea of a Seed AI in Levels of Organization in General Intelligence:
A seed AI is an AI designed for self-understanding, self-modification, and recursive self-improvement. This has implications both for the functional architectures needed to achieve primitive intelligence, and for the later development of the AI if and when its holonic self-understanding begins to improve. Seed AI is not a workaround that avoids the challenge of general intelligence by bootstrapping from an unintelligent core; seed AI only begins to yield benefits once there is some degree of available intelligence to be utilized. The later consequences of seed AI (such as true recursive self-improvement) only show up after the AI has achieved significant holonic understanding and general intelligence.
It’s reasonable to think we’re at the start here, with how general and useful today’s models are.
Generally, RSI can be summarized as when AI can improve itself, the improved version can improve even more efficiently, creating a closed amplification loop that leads to an intelligence explosion, often referred to as the singularity. There are a few assumptions in this. For RSI to occur, it needs to be that:
The loop is closed. Models can keep improving on themselves and beget more models.
The loop is self-amplifying. The next models will yield even bigger improvements than the current ones.
The loop continues to run without losing efficiency. There are not added pieces of friction that make the exponential knee-capped as an early sigmoid.
While I agree that momentous, socially destabilizing changes are coming in the next few years from sustained AI improvements, I expect the trend line of progress to be more linear than exponential when we reflect back. Instead of recursive self-improvement, it will be lossy self-improvement (LSI) – the models become core to the development loop but friction breaks down all the core assumptions of RSI. The more compute and agents you throw at a problem, the more loss and repetition shows up.
02
The research that can be automated is too narrow
Built on Paul Allen's “complexity brake”: agents are good at optimizing a single metric, while the best research makes many scalable ideas work together.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.
I’m still a believer that the complexity brake on advanced systems will be a strong counterbalance to the reality that AI models are getting substantially better at every narrow task we need to compose together in making a leading AI model. I quoted this previously in April of 2025 in response to AI 2027.
Microsoft co-founder Paul Allen argued the opposite of accelerating returns, the complexity brake: the more progress science makes towards understanding intelligence, the more difficult it becomes to make additional progress. A study of the number of patents shows that human creativity does not show accelerating returns, but in fact, as suggested by Joseph Tainter in his The Collapse of Complex Societies, a law of diminishing returns. The number of patents per thousand peaked in the period from 1850 to 1900, and has been declining since. The growth of complexity eventually becomes self-limiting, and leads to a widespread “general systems collapse”.
There are plenty of examples in how models are already trained, the deep intuitions we need to get them right, and the organizations that build them that show where the losses will come from. Building leading language models is incredibly complex, and only becoming more-so. There are a few core frictions in my mind.
1. Automatable research is too narrow
First, it is clear that language models this year will already be useful tools at optimizing localized tasks like lowering the test loss of a model. Andrey Karpathy recently launched his autoresearch that popularized doing just this. This allows AI agents to play directly on GPUs to target tasks like lowering the loss on the test set. This approach works in narrow domains, i.e. one general test loss or one overall reward. The problem is that there’s a long-standing gap between an on-paper more accurate model and models that users find more productive. The most provocative case is for pretraining, which was discussed more at length around scaling laws. Scaling laws show us that the loss will continue going down, but we don’t know if that’ll be economically more valuable.
In post-training, reinforcement learning algorithms are at least more directly tied to specific performance gains as most RL training environments can be used directly as an evaluation. Still, I worry about generalization and tying back to models that are better at the specific task of improving themselves. It’s a big leap from models get better at some things to that necessarily translating to models that are better at building themselves and designing experiments. We’ve seen many AI capabilities sort of saturate at certain levels of human taste, such as writing quality. AI research is a bit different here, as there is a very high ceiling to climb up to. Where models mostly saturate on writing because there’s inherent tension in preferences, models will saturate on research because the search space and optimization target is too wide.
The early benchmarks for measuring this sort of ability all fall prey to the same problem – narrow scope. Agents will do well at optimizing single metrics, but the leap required to navigate many metrics at once is a very different skill set. That is actually what the best researchers do — they make many scalable ideas work together.
The most related benchmark we have to measure this is PostTrainBench, which is quite fun, but progress will very rapidly get distorted on this. Over 90% of the challenge in doing post-training well is getting the last 1-3% of performance, especially without cooking the model in out-of-domain tasks. Post-training a general, leading model is extremely complex, and only getting more complex.
I could go on and on about this. Another example is from during my Ph.D. (2017-2022), when there was immense hype around a field called “AutoML” which aimed to use techniques like Bayesian Optimization to find new architectures and parameters for models. The hype never translated into changing my job. Language models will do more than this, but not enough to take jobs away from top AI researchers any time soon. The core currency of researchers is still intuition and managing complexity, rather than specific optimization and implementation.
03
More agents saturate; people are the last bottleneck
Amdahl's law: even 10,000 remote workers in a data center can't all be pointed at one problem. Above that sits the politics of who gets resources and what they bet on.
2. Diminishing returns of more AI agents in parallel
The biggest problem for rapid improvement in AI is that even though we’ll have 10,000 remote workers in a datacenter, it’ll be nearly impossible to channel all of them at one problem. Inherently, especially when the models are still so similar, they’re sampling from the same distribution of solutions and capabilities while being bottlenecked by human supervision. Adding more agents will have a strict saturation in the amount of marginal performance that can be added – the intuition of the best few researchers (and time to run experiments) will be the final bottleneck.
A common idea to illustrate this is Amdahl’s law, which is taken from computer architecture and shows that a given task can only generate a fixed speedup proportional to how much can be parallelized and how many parallel workers exist. An illustration is below:
In AI this should be relatively easier to convey, as the low-level operating details of computers are fairly mysterious. Consider an AI researcher on the transition from writing code by hand to using AI autocomplete assistance to now using autonomous coding agents. These are all massive gains. Let us continue. Now this researcher uses 3-4 agents working on different sub-tasks or approaches to the problem at hand. This is still a large gain. Now consider this single researcher trying to organize 30-40 agents with tasks to do every day. Some people can get more value out of this scale, but not many.
How many people do you think could come up with 300-400 tasks for AI agents every day? Not many. This problem will hit the AI models soon enough as well.
3. Resource bottlenecks and politics
Fundamentally, all the AI companies are walking a fine line of acquiring substantial capital, converting new compute resources to revenue via sufficient demand, and repeating the process all-the-while spending an extreme amount on research. With the scale of resources here, there will always be political bottlenecks on who gets resources and what gets bet on. In this layer, research leadership sits above the AIs and the researchers. Even as models continue to improve, this source of friction will never get removed. It isn’t a substantial friction, but the AI models are fundamentally operating in organizations where humans are the bottleneck on resources.
The early scale of improvements with language models is local optimizations, where the resources used cost <$1M per day. With my other views on the frictions of AI, this is on its own a very minor impact on the rate of improvement, but for those with worries of fast take-off, RSI, and loss of control to AIs, it should be obvious that billions of dollars of compute resources for research are unlikely to be totally isolated for end-to-end experimentation of AI models.
04
The bottom of every S-curve looks exponential
Automating engineering and automating basic research are both real, but the next phase has no ready scale to climb. Faster hill-climbing won't buy a paradigm shift.
The conclusion here is that because we’re at the early stages of using AI assistance, autonomously and at scale for AI-development, we’re collectively discovering the ways that AI can help us massively. We’re all applying these tools to capture the low-hanging fruit we see and our jobs are literally changing to be higher paced and more productive. The problem is that all of these axes have clear human, political, or technical complexity bottlenecks.
The bottom of every sigmoid feels like an exponential. We’ve ridden multiple exponentials in the era of language models, in 2023 we scaled to huge models and GPT-4 felt like magic, by 2025 we added inference-time scaling with o1 and reasoning models — they let us “solve” math and coding, now we’re going to take a big step by polishing the entire AI workflow (all the while scaling training compute massively). 2026 will feel like a huge step, but it doesn’t have a fundamental change convincing me that progress will begin to take off.
This could still cross the colloquial threshold for AGI, which is a drop-in replacement for most remote workers, which would be an incredible milestone. Much of the challenge in the debate of if we hit AGI in the coming years is that AI models are jagged and smart in different ways than humans, so they won’t look like drop-in replacements for remote workers, but in many cases just using AI will be far more effective than trying to work with a human. It’s reshaping what jobs are.
Let us consider the scenarios we’re working through.
Engineering is becoming automated today. Humans are way more productive, models can scale through complex infrastructure deployments much faster, run with higher GPU utilization, etc. Infrastructure gains become fixed improvements in the rate and scale of experimentation, the fundamental units of progress in AI.
Basic AI model research and optimization will be automated. The AI models are expanding in scope – they transition from writing kernels to deciding on architectures. This is moving from improving the experimentation toolkit to running minor experiments themselves. Configs, hyperparameters, etc. become the domain of the AI assistants.
These are both real. The problem is that a third era doesn’t have a simple scale to jump to. Where the AI models can create knowledge by synthesis and execution, the next jump requires harnessing thousands of agents or having models make more novel discoveries – like unlocking the next paradigm after inference time scaling. The improvements downstream of AI are going to make the industry supercharged at hill climbing, but I worry that this won’t bring paradigm shifts that are needed for new categories of AI – continual learning, world models, whatever your drug of choice is.
All together, the models are becoming core to the development loop and that’s worth being excited (and worried) about. The models are performing self-improvement. They’re not transforming the approach. We are scaling up the compute we spend on our own research practices and tools. There are diminishing returns. Agents are going to start being autonomous entities we work with. They feel like a cross between a genius and a 5 year old. We will be in this era of lossy self-improvement (LSI) for a few years, but it is not enough for a fast takeoff.
Where Indigo landsFurther
Indigo's conclusion
In the argument over whether AI takes off fast, this is the clearest voice for slower: AI wins where problems are narrow and machine-checkable, which is his “too narrow” point; real paradigm shifts still won't grow out of hill-climbing.
What to remember
Lossy, not recursive: self-improvement is real, but friction breaks the closed loop, the speed-up and the no-loss assumptions.
The three frictions make a checklist: one narrow metric or many broad ones? More agents or more human intuition? Are resources stuck in politics?
“The bottom of every S-curve looks exponential”: 2023, 2025 and 2026 each rode one, and each was only the bottom of an S-curve.
Don't price in a fast takeoff. This is a bounded exponential: capability keeps improving for real, but “it takes over everything next year” needs cold water.
Claims you can check later
Claim
Who
When we will know
How firm
In hindsight it is linear and bounded, not an exponential takeoff (lossy, not recursive)
Lambert
Next few years
First-hand; a directional bet
Returns to more parallel agents saturate; human intuition and time to run experiments are the last bottleneck
Lambert
Ongoing
First-hand; Amdahl's mechanism
Automated research saturates where search is wide and metrics are many; it surges only in narrow domains
Lambert
Ongoing
First-hand judgment
Hill-climbing speeds up enormously but brings no paradigm shift like continual learning or world models
Lambert
Next few years
First-hand; the most falsifiable
2026 will feel like a big step, with no fundamental change that makes progress take off
Lambert
2026
First-hand (six months on: largely holding)
Back on the long-running theses
adds to
Continual learning ends in the weights He names continual learning and world models as paradigm shifts that won't come from hill-climbing: one more point for this view.
adds to
The shape of demand: bounded vs. unbounded Amdahl's friction plus “narrow goals automate best” echo this thread: what can be sped up is the narrow, checkable part.
conflicts
Jakub Pachocki, An Alien Mind Two poles on one question: one rests on belief built from internal results, the other on three frictions you can point to.
Ryan Greenblatt: misalignment isn't Skynet, it's a sloppocalypse “Lossy” and “sloppocalypse” are close cousins: automated AI research piles up losses rather than amplifying cleanly.
confirms
fin, AI semiconductor endgame III: two-dimensional S-curve relay “The bottom of every S-curve looks exponential” is the relay of S-curves seen from the knowledge side; both use Amdahl's law.
What it means for Rewired Index
The baseline for pricing AI's capability curve is a bounded exponential: hill-climbing on fast-forward, not a sudden takeoff. Until a paradigm shift arrives, value concentrates in narrow, checkable domains. No direct names.
What would change my mind
a paradigm shift such as continual learning or world models growing straight out of step-by-step automated research.
Finished. Indigo's take on this piece is in two places: