Economics of Continual Learning

2026-03-11

Dwarkesh thinks continual learning is a bottleneck to AGI. Regardless of the technical difficulties, I don't understand how it's ever supposed to be viable for a business.

As I understand it, the definition of continual learning is that the model updates its weights in order to preserve new information (as opposed to keeping memories in an external database - whether that's memory.md or RAG). The idea is that every user would have a model that would learn how to work for that user and would get better over time. That implies that every user has to use a model with personalized weights. This seems impossible to me!

First, there are the obvious storage costs. OpenAI has ~1 billion users and 50 million paying users¹. OpenAI would have to store a 1-10 TB model for each one of those users (if they all get access). That comes out to more than $10B a month. Less obviously, all inference providers rely on a trick called batching, which means that instead of running one query through all the layers of the model end-to-end, many queries are processed per layer before moving on to the next layer. This saves time on loading parameters into the GPU. This would be impossible if everyone's models have different weights and would make inference prohibitively expensive.

Yes, a lab can offer this model to a few high paying customers but is that enough of an incentive to spend R&D on? The labs don't even release models RL'd for specific tasks (outside of coding) which would have far more users.