How to Iterate and Improve
A few days ago I came across a tweet by Andrej Karpathy on “how to become an expert in a field.” It had three points. First, find a concrete project to work on, and look things up as you hit things you don’t understand—learn in an “output-driven” way. Second, don’t go “bottom-up”: don’t insist, like you did in school, on finishing all the foundational theory before you start. Third, only compare yourself to your past self, not to others.
This reminded me of a line from Feynman: if you can’t explain something clearly to someone who knows nothing about it, it means you don’t really understand it yourself.
Both are saying the same thing: use output to check input. You think you understand, but only when you actually do something or open your mouth do you find out whether you really do. This piece is about making clear why this approach works, why it’s hard, and how to keep at it.
Parallels with Deep Learning
Training a neural network is a loop of four steps:
- Forward (forward propagation): the network produces an output based on its current parameters.
- Compute the Loss: compare the output to the correct answer and calculate the gap.
- Backward (backpropagation): based on the gap, work backward to figure out which direction each parameter should be adjusted.
- Gradient descent: nudge the parameters a little in that direction.
Then go back to step one and repeat thousands of times, and the output gets more and more accurate.
Human learning is almost identical: “do a project first” is the Forward pass; the gap between what you produce and the ideal result is the Loss; and your review of “what went wrong, how to do it better next time” is Backward and gradient descent. When Karpathy says “put in enough reps and you’ll become an expert,” he’s essentially saying this is an iterative algorithm. There’s no one-shot solution; all the power comes from small, numerous, sustained repetition.
The Limits of Being Human
The biggest difference between people and neural networks is this: a network feels nothing in the face of a large Loss, but a person hurts.
And there’s no getting around the pain. The reasoning is simple: to improve, you have to produce output in an area you haven’t yet mastered, and along the way you’ll inevitably make frequent mistakes and keep bumping into your own ignorance. Conversely, practicing something you’ve already fully mastered yields no new progress. Real growth always happens at the edge of your ability, and the edge is, by definition, uncomfortable.
So the real difficulty of this approach isn’t intelligence—it’s how to endure the discomfort of the process.
Why do people hurt while networks don’t? Because people tend to conflate “how I did” with “what kind of person I am.” Once your self-worth is built on “I’m capable,” every mistake becomes a negation of who you are, and so you instinctively avoid every situation that might expose a shortcoming—which means avoiding all growth.
To face this pain, the key is to separate two things: you can criticize your “behavior” mercilessly, but don’t wound your “worth.”
Think of it through the neural network: gradient descent adjusts the parameters, which correspond to your behavior, skills, and methods—things that are supposed to be corrected over and over. The architecture, by contrast, is something deeper that shouldn’t be messed with carelessly. You should place your self-worth at the architecture level, not the parameter level. That way mistakes only change your behavior; they don’t shake “who you are.” If you do it the other way around and let every Loss strike the architecture itself—equivalent to tearing down and rebuilding the entire network every time you make a mistake—training won’t converge at all; it will just oscillate violently and then collapse. That is the state of “totally negating yourself the moment you slip up.” Stable training requires that the Loss only updates the parameters and never blows up the architecture.
Ray Dalio’s Principles
Everything above has, in fact, long since been organized into a full, operational system—Principles, written by Ray Dalio, the founder of Bridgewater Associates.[1] The book is a “decision-making manual” he assembled by writing down, one by one, the mistakes he made over several decades and the practices he distilled from them. What’s striking is that what he worked out independently from investing and management practice corresponds almost point for point with the conclusions we derived above from the neural network.
His most famous formula is: Pain + Reflection = Progress. This is the same thing as “people hurt, and the pain is unavoidable” from earlier—pain itself isn’t a signal to dodge but raw material for reflection; pain without reflection is pure waste. In training terms, a large Loss isn’t the problem; the problem is not running Backward.
Dalio repeatedly stresses embracing reality and confronting pain: see reality as a machine that runs on cause and effect, and your job is to figure out how it works, not to complain about how it “ought to be.” Making a mistake isn’t shameful; learning nothing from the mistake is—which maps exactly onto the earlier point about “criticizing behavior without negating worth”: mistakes are data for updating parameters, not a verdict on who you are.
He distills “how to solve problems” into a five-step process:
- Set clear goals;
- Identify the problems standing between you and the goals, and don’t shy away from them;
- Diagnose the problems to get at the root causes, rather than just treating surface symptoms;
- Design a plan that gets around or solves the problems;
- Execute the plan resolutely.
This is really the human version of the training loop: the goal is the ideal output, identifying problems is computing the Loss, diagnosing root causes is Backward—tracing the gap back to “which step needs to change”—and designing and executing the plan is gradient descent. Dalio specifically notes that these five steps should be taken one at a time, not blurred together, the same reason Forward, Loss, Backward, and update must keep their order straight in training.
What underpins this process is the radical truth and radical transparency he practices at Bridgewater: put problems on the table, don’t hide them out of embarrassment, and welcome others pointing out your shortcomings without mercy. On a personal level, this is that earlier “criticize behavior mercilessly”—you have to let the real Loss be computed and seen before reflection has anything to work from; a person who hides their mistakes is feeding themselves fake data.
The most crucial thing Dalio did was to write these lessons down and systematize them: each time he ran into a class of problem, he distilled a principle for handling it, and the next time a similar situation came up he simply invoked it; he even wrote the principles as explicit decision criteria, letting them run automatically like a “machine” and iterating on them in practice. This is exactly the same as the “write your mistakes down serially” that follows—he just took it further: not only recording, but refining the record into reusable decision rules. He also stresses distinguishing first-order from second-order consequences—many right choices are painful first-order (exercise, say, or refusing short-term temptation) and only good second-order; fixating on first-order comfort leads you to make systematically wrong decisions.
That a man running a hedge fund and a metaphor about training a neural network converge on the same method is itself telling: this iterative logic of “small steps, sustained, updating yourself from your mistakes” is probably the universal form of progress for any complex system.
Two Tricks
Trick one: write your mistakes down serially.
Every time you hit a mistake, a gap in your knowledge, or something you want to improve, record it in a log or notebook. There’s a wonderful role model for this: the Meditations of the Roman emperor Marcus Aurelius, written as private notes “to himself.” Even as an emperor, he recorded and reviewed his own shortcomings every day.
Why does writing it down help? When pain stays in your head it’s a tangle that loops around—many thoughts churning at once, none of them settling—and that’s exactly its most tormenting state. The act of writing forces you to line it up into a single thread: you can only write one sentence at a time, and one sentence can only convey one meaning clearly. This “serialization” forces the brain into a rational state, where you think the matter through calmly instead of letting emotions race around in your head and drain your confidence.
Trick two: as soon as you finish writing, start the next round.
Once the log is written, this round’s Backward and gradient descent are done. The next step is not to linger in the pain but to go straight into the next Forward pass—commit to a new action. There’s a basis for this too: when training, you don’t run Backward on the same batch of data ten times in a row, because that would make the network overfit to that single sample. People are the same: chewing over the same mistake again and again only traps you in the details of this one failure, and you fail to extract a lesson you can use next time. “Thinking about it more only brings more pain” is, in essence, “running Backward more only causes overfitting.”
These two tricks go together, and there’s an order to them: the reason you can dare to “forget” is precisely that you’ve already “written it down.” Once the matter is clearly filed away, the alarm in your head that says “you have to keep remembering this or you’ll pay for it” can be switched off; if you need to recall it later, you just check the record. Writing the log closes out the previous round; acting immediately kicks off the next one.
References
- Ray Dalio. Principles: Life and Work. principles.com