Economics of Continual Learning

2026-03-11

Dwarkesh thinks continual learning is a bottleneck to AGI. Regardless of the technical difficulties, I don't understand how it's ever supposed to be viable for a business.

As I understand it, the definition of continual learning is that the model updates its weights in order to preserve new information (as opposed to keeping memories in an external database - whether that's memory.md or RAG). The idea is that every user would have a model that would learn how to work for that user and would get better over time. That implies that every user has to use a model with personalized weights. This seems impossible to me!

First, there are the obvious storage costs. OpenAI has ~1 billion users and 50 million paying users¹. OpenAI would have to store a 1-10 TB model for each one of those users (if they all get access). That comes out to more than $10B a month. Less obviously, all inference providers rely on a trick called batching, which means that instead of running one query through all the layers of the model end-to-end, many queries are processed per layer before moving on to the next layer. This saves time on loading parameters into the GPU. This would be impossible if everyone's models have different weights and would make inference prohibitively expensive.

Yes, a lab can offer this model to a few high paying customers but is that enough of an incentive to spend R&D on? The labs don't even release models RL'd for specific tasks (outside of coding) which would have far more users.

Tmux brackted paste issue

2026-03-02

While perfecting the perfect shade of blue for links, which mostly entailed copy/pasting the values from the default google color picker I came across a (sort of) bug in tmux and simple fix.

When pasting the color hex into nvim inside a tmux window, the hex trailed off with ^[[201~. Weirder was I couldn't do anything anymore, every character I entered was instead represented by the utf code. Even Ctrl-b was hijacked, I couldn't exit (n)vim (that brought back bad memories...), or close the pane. By pure luck, I discovered the copy/pasting something again into nvim gave me back full control.

I found this issue which explains the cause. tmux waits escape-time long after an ESC to decide if the following chars are part of a longer key sequence. Due to lag (or possibly Windows Terminal) and a short escape-time tmux no longer interprets [201~] as "End bracketed paste", it just takes the raw code and passes it through to the open program (nvim).

The solution (as mentioned in the issue comments) is to set a higher escape-time. There is a trade off - esc to normal mode being less snappy - so I'm still experimenting with the right time, but 50ms seems good so far.

I still don't understand why it was messing up my terminal that badly, or why pasting again fixed that...

Further reading

What's the effect of escape-time in Tmux? tmux manual, escape-time