Not long ago, I watched an AI agent fix a failing test in an unfamiliar open-source codebase.
Within minutes, it produced a polished patch. The names were sensible. It added a regression test, wrote confident comments, and provided a clean explanation. Everything passed.
Then I asked two questions: What invariant was actually violated? And why was this the right place to fix it?
The explanation fell apart.
The agent had optimized around the visible failure. It made the symptom disappear without understanding the broader ownership and lifecycle model behind it. When I explicitly asked it to reconsider the design, it proposed a substantially different solution.
That experience stayed with me because the patch looked trustworthy. It carried all the familiar signals: passing tests, clean formatting, detailed comments, and a persuasive summary. But AI had made those signals surprisingly cheap. None of them guaranteed that the change reflected a sound understanding of the system.
AI had not merely made coding faster. It had separated the production of convincing code from the judgment required to trust it.
The bottleneck has moved
Writing code is no longer the primary constraint it once was.
Before capable AI coding tools, implementation imposed a natural speed limit. Writing the code forced an engineer to spend time inside the problem. Along the way, you absorbed context, encountered awkward abstractions, discovered hidden constraints, and sometimes reconsidered the design entirely.
That process was inefficient in useful ways.
Today, an AI agent can skip much of it and produce a large, plausible, test-passing change almost instantly. Teams can generate code much faster than they can genuinely understand, review, and own it.
The scarce resource is shifting from implementation to judgment.
The important questions are increasingly not “Can we produce this code?” or even “Does this patch pass its tests?” They are: Should this implementation exist? Does it fit the architecture? What assumptions does it make? And could someone safely debug it at three in the morning?
A test can demonstrate that code works in the cases we imagined. It cannot prove that the ownership model, concurrency strategy, or overall design makes sense.
Code is not the whole program
This distinction reminded me of Peter Naur’s classic essay, “Programming as Theory Building”. Naur argued that a program is not merely its source code and documentation. The people who build it develop a theory of the system: an understanding of the problem, the constraints that shaped the design, the alternatives that were rejected, and the kinds of changes that belong within it.
That theory lives, imperfectly, in people’s minds.
Rachel Thomas recently highlighted Pol Alvarez Vecino’s application of Naur’s argument to AI-generated code. An AI can measure lines of code, identify duplication, and optimize conventional complexity metrics. But genuine simplicity depends on information that may never appear in the repository: business priorities, operational history, expected changes, and judgments developed through experience.
This helps explain why an AI can propose locally elegant code that is globally wrong. It sees the artifact more readily than the theory that produced it.
Providing more context certainly helps. But documents, tickets, and chat histories are themselves incomplete artifacts of the team’s understanding. Context windows can contain a great deal of information without containing the lived judgment required to interpret it.
The old irony of new automation
This is not an entirely new problem.
In her 1983 paper “Ironies of Automation”, cognitive psychologist Lisanne Bainbridge observed that automation often removes routine work while leaving humans responsible for the rarest and most difficult situations. Yet by removing people from everyday operation, automation also deprives them of the practice and feedback needed to intervene successfully.
The better the automation works, the less prepared the operator may become when it fails.
Software engineering now faces a version of this irony. As agents perform more implementation, engineers may spend less time developing an intimate understanding of their systems. But they remain responsible when generated code behaves unexpectedly, when requirements change, or when a production failure falls outside the cases anticipated by the tests.
Oversight without understanding is not meaningful control.
Recent research offers early evidence of the tension. In a randomized study of experienced open-source developers working in repositories they knew well, AI tools increased measured completion time by 19 percent, even though participants believed the tools had made them faster. Another randomized study found that developers using AI performed worse on a subsequent assessment of concepts they had just used. Importantly, the outcome depended on how they worked: people who asked conceptual questions and examined the generated code learned more than those who delegated the task wholesale. The authors’ conclusion was not that AI necessarily prevents learning, but that productivity is not a shortcut to competence.
These studies examine particular tools and tasks, not universal laws. AI capabilities and development practices are changing quickly. But the results should make us skeptical of equating generated output—or even perceived speed—with durable progress.
AI as an amplifier
I remain deeply excited about AI-assisted software development.
AI has made curiosity much cheaper.
Questions that once required too much time or setup can now become immediate experiments: How does this system really work? What assumption is this design making? What would an alternative approach look like?
While the curiosity is still fresh, AI can help turn those questions into prototypes, tests, diagrams, and concrete comparisons. It can help us explore unfamiliar systems, test competing explanations, and investigate ideas we might otherwise abandon.
The answers still require verification. But the path toward understanding can become far more interactive and accessible.
This is the possibility I find most compelling. At its best, AI does not replace understanding; it creates more opportunities to reach it.
Jeremy Howard advocates using AI through small, interactive steps with immediate feedback, rather than asking it to produce a large artifact all at once. In his recent conversation with Machine Learning Street Talk, he draws an important distinction between coding and software engineering. AI may perform much of the typing, while forming abstractions, developing intuition, and deciding how a system should evolve remain fundamentally different challenges. I highly recommend listening to the full episode, “Vibe Coding Is a Slot Machine”.
Jeremy’s approach also resembles how I increasingly work. I keep a Jupyter notebook open inside VS Code and test code interactively as I write it, including code generated with AI. I inspect intermediate results and use a separate AI window to ask questions about the implementation, understand unexpected behavior, debug failures, and improve the design. The goal is not simply to arrive at working code. It is to remain engaged with the system while building it.
That perspective feels more constructive than either dismissing AI coding or surrendering engineering judgment to it.
Keeping understanding in the loop
In my own work, three practices have become increasingly important.
First, I ask an agent to explain the problem before fixing it. What invariant is being violated? Where should that invariant live? What alternative explanations or designs did it consider? Agents are remarkably effective at making visible symptoms disappear. They need pressure to distinguish a root fix from a plausible bandage.
Second, I decompose changes around decisions. Each change should answer one reviewable question and provide evidence for its claims. “The tests pass” is not enough. I want to know what behavior changed, which assumptions were made, what was not tested, and how the change could be reversed.
Third, the human author must own the result. My rule is simple: do not ship code you could not explain, debug, and revise without the agent. Read the code. Challenge suspicious choices. Ask the agent to attack its own design rather than merely confirm that it works.
AI can perform much of the typing and investigation. Accountability cannot be delegated.
What progress should mean
The same principle applies to organizations.
Lines of generated code, completed tickets, merged pull requests, and short cycle times are visible and easy to count. With AI, they are also becoming easy to inflate. A team can appear dramatically faster while accumulating software that nobody fully understands or owns.
An organization’s knowledge of its systems is a real asset. It enables engineers to make sound trade-offs, diagnose failures, evolve an architecture, and recover when something goes wrong. If AI increases visible output while eroding that knowledge, the organization may be borrowing speed from its future.
We should care instead about durable outcomes: problems genuinely solved, changes that remain understandable, low rework and incident rates, strong evidence, and clear human ownership. We should ask whether AI is helping engineers learn their systems and improve their designs—not merely helping them generate more changes.
Speed should mean reducing the time required to deliver a trustworthy outcome, not increasing the volume of code moving through the system.
Code is becoming abundant. Understanding is not.
The future of software engineering will depend on how carefully we protect—and expand—the latter.