Is AI Replacing Programmers or Boosting Them?

Justified Posteriors reads “Writing Code vs. Shipping Code” by Mert Demirer, Leon Musolff, and Liyuan Yang In this week’s episode of Justified Posteriors, we update our beliefs with evidence from an ambitious new paper estimating the impact of AI on software production productivity. Demirer (friend of the show), Musolff, and Yang combine public GitHub records for over 100,000 developers with confidential Microsoft data to trace the effect of distinct generations of AI coding tools — autocomplete, sync agents, and async agents — on the code production hierarchy: lines of code, files, commits, pull requests, projects, and releases. The main empirical finding is attenuation of the effect of AI at each step. Enormous gains of 1000% productivity increases or more at the top of the chain translate into about a 30% increase in shipped releases. The model they use to explain this result is closely connected to Kremer’s O-ring logic, which regular listeners will recognize as from a few episodes back. Seth likes the spirit of the model, but feels it is overcomplicated for this context, for reasons he explains. In addition to discussing the data analysis, Andrey and Seth have a good back-and-forth about what we can conclude from it and extrapolate to the economy more generally. The implication that grabbed Seth’s attention is a sentence in this paper’s abstract. Nested in the summary of the careful empirical exercise is an estimated elasticity of substitution of 0.25 between AI and human effort. Big if true! Seth points out the enormous long-run implications of humans being complements to AI in what seems to be the most AI-friendly of tasks: as AI gets cheaper, the human share of income goes up, wages skyrocket, and ultimately AI boosts jobs instead of taking them. That’s a huge real-world hook. Seth and Andrey discuss whether, and if so how much, we update our beliefs in this direction, with Andrey being careful to point out the difficulties of extrapolating from a partial equilibrium elasticity to long-run macro consequences. This episode is sponsored by Revelio Labs — a great source of labor economics data for academics and firms. Now available on WRDS. Priors → Posteriors Prior 1: Does access to AI coding tools boost lines of code written by more than 100%? * Seth: 95% → 99%. Seth came in confident and left more so. Great to see giant numbers. * Andrey: 80% → 95%. A high yes, hedged because “which developers” and “which tools” do a lot of work in that sentence. Prior 2: Does AI boost economic value by 50% or less of the factor by which it boosts lines of code? * Seth: 95% → 97.5%. I have personally produced a great deal of economically worthless code lately. * Andrey: 85% → 95%. Prior 3: Are AI coding tools a gross complement to human labor? Seth’s answer depends on the level of aggregation:The average normie programmer 20%→20% (unchanged)A human engineering department 33%→40%A software company / open source project 60%→85–90%The economy as a whole 33%→33% (unchanged) Andrey: 75% complement at the sectoral level, and he’d put it as low as the programming department — because right now the code that comes out is not shippable without substantial human input. Posterior: still a complement, mildly supported. He declines, on the record and repeatedly, to extrapolate to the macroeconomy. No fun! References The paper under review * Mert Demirer, Leon Musolff & Liyuan Yang, “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools,” NBER Working Paper 35275 (May 2026). * The authors’ own summary: “Writing code versus shipping code”, VoxEU, June 2026. Prior work by the same team * Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng & Tobias Salz, “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers,” Management Science (2026). 4,867 developers, roughly a 26% increase in completed tasks, larger gains for the less experienced. Related Research and Prior Episodes * METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” — the RCT Andrey refers to, in which 16 experienced developers took 19% longer with AI while believing they were 20% faster. arXiv version. * METR’s own update, “We are Changing our Developer Productivity Experiment Design” (Feb 2026) — developers increasingly refuse to be randomized into working without AI, which biases the estimated speedup downward. * Michael Kremer, “The O-Ring Theory of Economic Development,” Quarterly Journal of Economics 108(3), 1993. Our episode on it: Weak Links, Strong Predictions: Kremer’s O-Ring at 30. * Josh Gans & Avi Goldfarb, “O-Ring Automation,” NBER Working Paper 34639 — and our conversation with Avi: Avi Goldfarb on Prediction Machines, O-Ring Tasks, and How AI is Reshaping Economics. * Aaron Chatterji, Tom Cunningham, David Deming, Zoë Hitzig, Christopher Ong, Carl Shan & Kevin Wadman, “How People Use ChatGPT,” NBER Working Paper 34255 — a reference for Andrey’s point about substitution toward home production. Non-work usage grew to over 70% of messages; computer programming is a small share. * Our episode on the same phenomenon in a different market: The Economics of Book Slop — more books, not obviously more valuable books. More apps, not obviously more downloads. Rhymes. * The universal token budget: Alex Imas — Demand Collapse, Bargaining with Machines, and Behavioral AI Economics and Seb Krier on AGI, the Coasean Singularity, and EDM. Table of Contents * Introduction and Today’s Paper — [00:00] * Priors: [02:58] * The Evidence: Ideal and Possible Experiments— [18:34] * The Evidence: Adoption Events and the Attenuating Waterfall — [24:56] * The Evidence: Model and Simulation Results — [42:56] * Aggregates, App Stores, and Posteriors — [56:00] Transcript Introduction and Today’s Paper [00:00] Seth: The abstract of the paper has the following sentence: blah, blah, blah, based on the results in the model, there’s an estimated elasticity of substitution of 0.25, so high complementarity between AI and human effort, which indicates strong complementarities. Wow. As the people say, big if true. Humans and AI, 0.25 complements. Everybody worried about AI taking all our jobs — wrong. All labor share to 100%. Welcome to the Justified Posteriors podcast, the podcast that updates beliefs about the economics of AI and technology. I’m Seth Benzell, with a 0% productivity impact on my code writing, as measured by my podcast release schedule, coming to you from the Pocono Mountains of eastern Pennsylvania. Andrey: And I’m Andrey Fradkin, coming to you from San Francisco, California. Justified Posteriors is sponsored by the fine folks at Revelio Labs, and please do sign up to our podcast and our Substack whenever you get the chance. Seth: Today we’re talking about a really interesting empirical study investigating the impact of AI tool use — autocomplete, synchronous agents, asynchronous agents — on people’s productivity in writing code. This is that kind of hard empirical data that maybe has the potential to move our beliefs. So I’m cautiously optimistic that I’m going to learn a lot from this one. Andrey: It’s the big question, in many ways. We have these tools. We’re using them. What do we get out of them? Are we really that much more productive? That is the question on everyone’s mind, especially since so many of these tools are very costly. There are people token maxing under the belief that the more tokens that are used, the more valuable the output will be. Seth: People are going broke over the tokens. We discussed a universal token budget when Alex Imas was on — or maybe that was with Seb. What are people getting when they’re actually paying for them? The paper we read to look into this is called “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools,” from Demirer, Leon Musolff, and Liyuan Yang. So, butchered every name, as is common for us. Andrey: Mert is a co-author of the pod, so very excited to be reading a paper of his. Seth: Despite being friends of the show, no punches pulled. Andrey: We never pull any punches. As listeners may know, now is the time for our priors. So what do we think about this topic before we read the paper? Priors: Three Claims, One Spicy [02:58] Seth: Reading this abstract, it seems to make two pretty narrow claims and then one claim that’s a really big spicy one. The first claim is that AI is really productive for helping you write lines of code. The next, more detailed claim is that the translation function going from lines of code to subsequently more advanced stages of production has an attenuation effect. You write more code, then you get more economic stuff out of that code — but the benefit attenuates. And then finally, using a model and some simulation, the paper goes on to argue that this can tell us something about the degree of complementarity between human coders and AIs. So let’s hit those one by one. First off, let me ask you this prior, Andrey. Do you think access to AI coding tools boosts the lines of code written by programmers by more than 100%? Andrey: Yes. I put my prior at 80%. Seth: 80%? That’s not a “yes, shut up.” Why only 80%? Andrey: That’s a pretty high yes. I’m a good Bayesian. Like any empirical question, there are sub-questions about which developers we’re talking about and which specific agentic tools we’re talking about. I’m sure the answer varies by those. It probably increases lines of code written by non-programmers by a ton. Seth: In percentage terms — from, you know, infinity. Andrey: You start with zero and then you go to something. It’s a pretty big percentage increase. It’s really important what samples are being used. But what’s your prior, Seth? Seth: Maybe I’m not a good Bayesian, but I came in pretty confident. I’ve got in mind all those people who have written one line of code in their lives now writing thousands of lines of code. If we look at the population as a whole, I’d be 95% confident going in that the amount of coding is going up by more than 100%. Andrey: And just to be clear, my interpretation was not people who previously did not code. Seth: Right. What we’re going to see is people who were using GitHub in the ex ante period, which raises you above some very basic level of coding in the pre-period. But even still, I’m pretty confident in this one. Andrey: [06:08] Let me ask you a related question. If you just plotted the lines of code produced by developers over time, in aggregate, not per developer — would that increase by more than 100%? Seth: Right, because now the guy who’s going from zero to 1,000 is a lot less important than the guy going from 1,000 to 10,000. If we’re saying the total amount, I would say yes — my guess is that it would double — but the confidence goes down. Probably that brings me to the 80 to 85% range. [06:21] All right, prior number two. Now it’s this idea that maybe the benefits attenuate. We’re not actually getting 100% more program even if we’re getting more than 100% more code. So let me ask you, Andrey: would you say that AI boosts economic output or economic value by 50% or less of the factor by which it boosts the lines of code written? Andrey: It boosts it by less than 50%. I’m very confident of that. I’ll even say 85%. Seth: I again come in pretty confident, because I’ve seen some real economically worthless code written by AIs recently. I’m a producer of a lot of it myself. I come in in the 90 to 95% range. In my head I’m also thinking about the fact that when we’re talking about lines of code, we’re probably also talking about comments. AI is really good at commenting and leaving descriptive comments if you let it. That’s one way the code might end up less dense. But also just my experience of writing my own code versus seeing what the AI writes — when you say “go do A, B, and C,” you end up with a whole bunch of stuff that I would never have done myself because I’m lazy. So that’s another reason I’m confident: 95%. Seth: [08:07] Big question, Andrey. The abstract of the paper has the following sentence, which I love: based on the results in the model, there’s an estimated elasticity of substitution of 0.25, so high complementarity between AI and human effort, which indicates strong complementarities. Wow. Big if true. Humans and AI, 0.25 complements. Everybody worried about AI taking all our jobs — wrong. All labor share to 100%, wages skyrocket. What’s your prior on AI coding’s complementarity to humans? Andrey: In the prepared priors, you made the comment that this is a prior about gross complements. Seth: [09:08] The way I think about complementarity — this is a good way to think about it if you’re not constantly dealing with elasticities of substitution — is that the elasticity in the middle is Cobb-Douglas. What Cobb-Douglas gives you is that no matter how cheap capital or labor is, you always spend a fixed share of your income on capital and a fixed share on labor. So put gross complements on one side of that and gross substitutes on the other. If you’re gross substitutes, then as your factor gets cheaper, you spend more of the budget on it. So if AI and humans were gross substitutes, that would mean that as AI gets better, the AI share of income goes up and the human share goes down. That’s the robots-take-our-jobs universe. Gross complements would mean the opposite. As AI gets cheaper, we’ve got all of this jelly, and the human peanut butter becomes more and more valuable. The share of income going to humans would go up. So: Cobb-Douglas in the middle, and what this paper argues is that in their data, in their sample, AI coders and human coders are very much on the peanut butter and jelly side. Andrey: To be clear, this paper is not about gross complements or substitutes. I view this paper as about holding all else equal, no readjustment — are they complements or substitutes? Which is a different story from what you’re talking about, which involves reallocation of resources. Seth: Fair enough. But the argument would be that they’re measuring an underlying parameter such that if there were reallocation, you would get the effects I’m talking about. That’s my reading. Andrey: This is not a macro model, Seth. Seth: Why do you care about estimating micro parameters except to plug them into a macro model? Andrey: My reading of the paper is that, given the way we’re currently organizing code production, this is their attempt at estimating the production function. Seth: And then I extrapolated out into infinity. Andrey: I don’t view this as a macro paper about a reallocation of labor and capital in the macroeconomy, if that makes sense. Seth: Andrey, your comment is so sympathetic to the authors that I worry you may have been replaced by some sort of alien invader. But feel free to address it just on the basis of an estimate of a parameter in a production function. You don’t have to extrapolate anything from that. Andrey: [12:15] The question is how much this paper informs our beliefs about the macro versus the micro, which we’ll talk about. But is AI coding a gross complement or substitute for human labor? My prior is that it is a complement, at the moment, with the current technology — which is a big caveat. The AIs, or at least how we’re using AIs, are not yet autonomous enough that humans can be taken out of the loop. As a result, humans still have a huge role to play in being in the loop on the coding production process. They’re essentially the bottleneck. That said, it really depends on the human, and this goes in many directions. Now we have a lot of people who were previously not coding, coding, and presumably that code would never have been written had this technology not existed. I would venture a guess that these humans in their roles are becoming more productive — they’re highly complementary, since literally the code would not have existed without them. There are some programmers who are not very good programmers, who were writing slop code even before AI tools. I imagine those coders are substitutes. And then there are other coders who understand the business logic, architecture, and so on, who are at the moment highly complementary to the AI. How that all nets out — I think it’s still on the complements side in the aggregate for the sector at current technological levels. Seth: But you can zoom out even more. Andrey: I’m not a macroeconomist, so I don’t want to extrapolate to macro from just coding tools. There’s a lot more to the economy. Seth: Ask a computer programmer. They say you get infinite code, and then the economy goes brr. Andrey: Our singularitarian friends notwithstanding, I’d rather not speculate about that in this prior. And I didn’t give you a number. I need to give you a number, right? I would say it’s a complement with a prior of 75%. Seth: And that’s at the sectoral level? Andrey: Yes, at the sectoral level. And I have to be very precise — I’m not even including Fable in here, because we haven’t seen Fable in the wild. Seth: [14:49] Well, beloved listeners, while you have one host who will not extrapolate wildly, I am always here for you. I agree with Andrey that the question depends on what level of the economy we’re looking at, so let me build up from narrow to aggregate. If we’re thinking about the level of mediocre coders — the average coder — I would be very surprised if AI were a complement to the average computer programmer at this point. I’d say a 20% chance that AI is a complement to the average normie computer programmer. Then if we move up to the level of a human engineering department, where maybe you’ve got one really sophisticated computer science architect who can complement the slop code being made, my probability of complementarity goes up to maybe one-third. But then we hit the level Andrey was focused on, the software company. At the software company, it seems clear that being able to churn out slop code and have a software product that is useful for people involves complementarity with people who have business expertise, people who can figure out how to sell the thing — which is not being automated when we’re talking about pure software. So at the level of the software company, maybe a 60% chance of complementarity between the AI and the entire software company. Andrey: I do want to clarify — you mischaracterized what I said. I was saying complements not at the software company level, but even at the programming department. Seth: But you said more substitutable for the lower quality coders. Andrey: Yes. But I’m not even talking about salespeople. I’m just talking about the fact that literally right now, if you’re coding, the code that is produced is not good enough to ship — Fable notwithstanding — without substantial human input. Seth: Okay. So that’s the human engineering department level, and you’d say even at the individual programmer level there are complements. Possible. And then I zoom out one more time, from the sectoral level to the economy-wide level. What do people do with software, Andrey? They automate things, and they consume digital products that don’t have large labor inputs. So at the economy-wide level I’d say AI coding tools are a substitute for human labor — a 33% chance they’re a complement. That’s the hierarchy I’d split out. Any response? Andrey: So let’s say you have someone at the insurance company reading claims. To this date there hasn’t been software that really automates them, but now it’s finally there, and so those insurance readers are donezo. Seth: That would be my argument. Andrey: In the short run. Seth: As software gets better generally because of this, that’s automative, that’s a gross substitute for humans. Andrey: Interesting. Seth: So now we will see if this paper has a chance to change our views. The Experiment We’d Run If We Could Clone the Earth [18:34] Andrey: Before we get to that — if we could design an optimal experiment, how would we study this question? Seth: I would make a clone of the planet Earth. Actually, I would make a thousand clones of the planet Earth, and in half of them Sam Altman would be born, and in half of them alt-Sam Altman would be born. Help me out here, Andrey. What’s the correct answer? Andrey: Seth wants to do a macroeconomics experiment, which is a nice thought experiment, where we forget about the existence of — where the transformer does not get invented. Is that the counterfactual? I’m not even sure. Seth: I go back in time, I assassinate Geoffrey Hinton. Ilya goes and sells ice cream somewhere. Andrey: And then we see how macroeconomic productivity and labor shares change in those worlds where LLMs don’t exist versus the ones in which they do. That’s an interesting thought experiment. A hard one to approximate, obviously. I was thinking of something more mundane. We have a bunch of firms, and half of those firms are randomly not allowed to use LLM-based tools — and half of them are imperfect anyway because of business stealing, but let’s forget that for a second. And then we see whether they’re more productive when they have LLMs, and how they change their labor shares. What do you think about that experiment? Seth: You pointed out the main concern I’d have, which is business stealing effects. So maybe these are in two separate countries where you do these splits. Andrey: What about the following experiment? What if we gave some developers LLMs, and some developers were not allowed to use LLMs? What do we learn from that experiment? Seth: The issue is that you can’t force anyone to use the LLMs. So you’d find the treatment effect on the group that chooses to adopt when they’re available. Andrey: Isn’t that always true, though? You can’t force a company to use LLMs. Seth: In medical settings you can inject somebody with something. They don’t have to be given a choice. Andrey: Sure. But relative to our other thought experiments — why isn’t that satisfactory for what we’re interested in? Seth: The reason it’s not satisfactory is that first you have to think about whether the randomization is at the level of the company or at the level of the programmer. You’d imagine those are different effects. If you’re randomizing at the level of the programmer, one issue is that when people can build up a whole new workflow at the company, that might have a different effect than a single person adopting. Andrey: What about just the fact that you can’t get labor substitution at all? Seth: If we’re doing it at the programmer level, you can’t fire a treated programmer. Andrey: Exactly. You can’t study that reallocation of labor. Seth: Right. If you’re randomly assigning tools to the As and not the Bs, and you think the long-run effect is that some of the As should actually not be programmers, then obviously that’s a mechanism you’re not going to capture. Andrey: [22:50] So it does have limitations, but you can still at least see how much more productive, in this narrow sense, the treated are versus the control programmers. Now, METR famously tried to do this experiment about a year ago and made the claim that AI tools as of a year ago — I think this was Cursor — did not make programmers more productive. It made them less productive. Much respect to our friends at METR, but most of us did not believe that experiment at all. Seth: What was the productivity outcome in that paper? Andrey: Speed. How long it took them to solve some issues. Seth: Speed is frequency inverted. Andrey: Great insight, Seth. Seth: Frequency is our productivity measure here today. Sorry. Andrey: Designing and implementing such an experiment in a credible way — people have thought pretty hard about how to do it, and it’s essentially impossible. It’s just hard to get the validity of true programming tasks, true randomization, preventing some people from using LLMs in all ways. And then most importantly, maybe, statistical power. Maybe with millions of dollars you could do this, but to have validity you need so many observations. So it doesn’t seem like a fruitful way to study this. We’re kind of left with the tools of observational causal inference, where we have to find something in the data that looks like an experiment if you squint at it and use the right econometric techniques. Seth: If only there were a social science field with a bucket of techniques for inferring causality from observational data. Andrey: It’s true. So that’s what the authors set out to do in this frankly really ambitious, impressive paper, in my opinion. Seth: Are you ready to go to the evidence, Andrey? Andrey: Sure. The Evidence: Adoption Events and the Attenuating Waterfall [24:56] Seth: Maybe I can quiz you through the different elements. Let’s start with the setting. What’s the population we’re going to evaluate, and what’s the natural experiment? Andrey: At a high level, we’re looking at contributors to open source projects on GitHub. The authors have other data as well — some proprietary data from Microsoft, which is pretty important for this empirical study — and they have some side data on non-open-source development projects. But broadly, that’s the main population. And then the thought experiment. Credit to the authors, because this is a mistake a lot of people make: a lot of people will write a paper like, “what is the effect of AI?” Seth: Oh my gosh, I am so frustrated by these papers that say “what’s the effect of AI.” I got a restaurant to use AI to write ads once. Andrey: [26:03] What we’re hinting at is that there is no one AI. It’s not the one ring to rule them all. You have the elven rings, which are the coding agents. You have the dwarven rings, which are the slop generators. You have the human-shaped rings, which are— Seth: Well, they were the men doomed to die, so that’s the AI for health research. Andrey: But seriously — over the past two years, AI has been many, many different things depending on whether you’re at the frontier or not and what tools you’re using. That includes the quality of the models. It includes the tooling, and how autonomous the tooling is. The authors do a very careful job of thinking about different ways people have been able to code via AI over this period, including just autosuggest. Remember back in the day when you were typing in an IDE such as VS Code, and it filled it out for you because what you were doing was predictable. Seth: I think the dwarven rings are the AIs for finance, because they used them to get rich. Andrey: And the one ring is the backdoor that Sam Altman has to all the GPT models. Seth: All right. Back to the paper. Andrey: So then you have what they call sync agents. I understand why they made this distinction, but it’s a pretty strange one in my opinion. A sync agent is essentially what we think of as an agent. You have Claude Code open, you’re typing something into Claude Code, Claude Code does something and gives you something back, you iterate with it, and then in the end you tell it when to push to the repository. Seth: Seems pretty agent-y to me, Andrey. Andrey: And then they also have async agents, which are autonomously sitting on a repo, refactoring the code, implementing things, and creating pull requests. Maybe it’s useful at this point to think a bit about the production hierarchy of code, which I found fascinating. Seth: [28:40] It’s provocative. I don’t know if I agree with every step, but it’s an interesting way of thinking about it. So what is the production hierarchy? We go from lines of code on one end, and then question mark question mark question mark, and then money at the other end. What’s in between? Andrey: Well, I don’t think open source projects are supposed to make money, Seth, but just FYI. So: there’s lines of code. There’s how many distinct files are touched. Obviously those are fungible in many ways — you can write one really long script, or a bunch of short scripts. You have commits, which are what programmers consider a discrete change to the code base worthy of noting. Once again, something that’s fungible and perhaps arbitrary. Then you have a pull request, which is when you submit this into the production code base, and usually someone else is going to review that in a professionally managed project. Then you have repos touched, so how many distinct repositories you’re contributing to — different repositories are distinct software projects. And then lastly, releases of those projects. Seth: And they’re going to tell us the effect of each of these different kinds of agents on each of these different outcomes? Andrey: Before we get to the empirics, do you want to go through the theory model? Seth: No. I know that’s how the paper lays it out, but I think it’s kind of silly the way they give you some theory, then some empirics, then some simulation. If I wrote this paper, I would do all of the empirics and then all of the model and simulations. I’d just as soon do it that way. Andrey: [30:29] So you have this dataset. It has the GitHub data. It has data on what they call adoption events — when specific GitHub users start using GitHub autocomplete, which is obviously an earlier period, back in 2022 and 2023. Then there’s the sync agents. Importantly, GitHub Copilot is a main one here — not always considered the best agent by many, sorry to our friends at Microsoft. But they also have ways of figuring out whether someone ostensibly is using Claude Code. For example, there might be a claude.md file in the repository, suggesting someone is using Claude somewhere. And then for the async agents, in particular Codex, OpenAI’s tool actually has a prefix saying that Codex generated a particular set of changes in the repository. Seth: And a human would never delete that before uploading code, would they? Andrey: I don’t think they would. Extremely low probability. I have very many concerns about what these adoption events mean, and in particular the intensity of adoption is very tricky to understand. You have a claude.md file. That could be because you’re just playing around with Claude Code. It could be because you’re very actively using Claude Code. Seth: You have a partner who’s using it. Andrey: Did your collaborator use Claude and you didn’t? This is a very long paper, so maybe I missed something, but some of these are imperfect proxies. That’s one way to put it. Seth: You could summarize that about a lot of this paper. Andrey: But again, I really do want to defend the paper: this is kind of the only thing we’ve got. We can’t do an RCT. I think it’s quite implausible. Seth: And the other things I’ve seen are country-level shocks, where the problems get twice as bad. Andrey: [33:07] Now, what is the control group here? This is really tricky. The obvious causal inference issue is that when you adopt a tool, you might be adopting it for reasons related to your future expected need to use the tool to produce code. Seth: But Andrey, they’re matched on observables. Andrey: This is the only exercise we can do, so we’re going to have to roll with it. I’m just explaining what the main challenge is. I’m not trying to snipe at it like you are already. Seth: I’m not sniping. I’m having fun, Andrey. Andrey: They need to find people who didn’t adopt the tool but otherwise would have attempted essentially similar things had they adopted. One way to think about it: if this async agent did not come out, both people would be doing the same exact thing. It did come out. One of them chose to use it, one of them chose not to, in a completely random way. That’s what they’re trying to get to. Seth: Ideally they’re working the same number of hours on a project of the same difficulty level. Andrey: Even that’s not true. If they were in the control group, they would be working on the same project, because one of the things the coding tools allow you to do is work on more projects. So you can’t condition on the project. Now, the authors are justifiably hesitant to literally match the behavior of someone who adopted to someone who didn’t adopt in the same time period. Why? It’s a reasonable assumption that many software developers have adopted these tools, and the measurement is quite imperfect for whether someone adopted or not. So the control group might have also been treated. What they do instead is find people they don’t observe adopting, and then look at their behavior literally a year before. That’s the behavior they compare to the behavior of the adopting developers. It’s not a strategy I’ve ever seen before. It’s quite non-standard. Usually in this sort of design you look at literally the same time period. But I understand why they do it, because of these very serious measurement error issues. What do you think about that, Seth? Seth: I think there is some logic there. The reason you’d want to use the same chronological time period is if you were worried there was some calendar-time shock to people’s writing of code. If in 2023 everybody’s writing more code than in 2022, that’s when you’d want to match up calendar dates. I don’t know if they do anything to deal with seasonality. Andrey: Seasonality is not the issue. It’s more that maybe the previous year was COVID, which would be a serious problem as a comparison year. Seth: A weird year. But given that coding is kind of in a steady state between 2022 and 2023, or 2023 and 2024, it does not seem that implausible to me. Andrey: I think it’s reasonable. The authors have sanity checks. They look at other adoption events like Docker, which is not an AI tool, and see whether they detect that it matters — a placebo test. And they don’t. So they do reasonable checks that this empirical strategy doesn’t always just pick up effects. And the magnitudes involved — it’s always easier to study something— Seth: When the magnitude is massive. Andrey: Which we know has massive effects. It would be crazy not to think it has massive effects, as our priors suggest. Seth: This is not a case where they’re picking up significance because they have so many observations. They find big numbers here. Andrey: [37:12] Let’s go through the outcomes very quickly. Lines of code: they find a 17x increase in lines of code after the adoption of an asynchronous agent. They find a slightly lower one, about 10x, for the sync agent, and about 2x for just using autocomplete. So just massive increases in lines of code. Files touched increases by about 4x with the agents, but only 0.5x with autocomplete. And then we see this pattern of diminishing numbers down the production hierarchy— Seth: An attenuation. Andrey: —to commits, pull requests. By the time we get to releases, the adoption of either type of agent only increases releases by 30%, even though it increased lines of code by 17 times. Seth: Right. So 30% from the agents, and only 10% from autocomplete. Andrey: [38:16] The other really interesting empirical pattern is what happens over time after adoption. As all of us know, we can spend some time playing around with these tools. Seth: We can make a lot of commits real fast. Andrey: They find a pattern that’s true across all three tool types: you get an initial spike in activity, whether it’s commits or lines of code, and then it diminishes a bit over time, although sometimes it rises again toward the end of the period. This is one of these very tricky things for them to disentangle. Seth: How big is the spike compared to the long-term effect? Andrey: For commits, the spike is about a 175% increase, and then it diminishes to about a 100% increase, and then it goes back up a little after that. That’s the pattern for the sync agent. Seth: So why do we think that is? Andrey: There’s people exploring, or they had a specific need for it and that’s why they adopted. And one thing to think about is that a job like ours — sometimes we’re coding a lot, sometimes we’re not coding a lot. We’re not coding all the time, and that’s probably true for a lot of people, especially open source developers, who might have other jobs in addition to their open source contributions. So I view that as an activity bias. There are also calendar effects that are really relevant. If Codex comes out in January, and everyone who adopts adopts in January for the most part, then you track them over time — and let’s say a new frontier model comes out in March that’s really, really good. Once you get to March, everyone’s going to start using the new frontier model. That’s going to increase your output. Now it looks like there’s a much larger effect later on, but that’s confounding model quality with the tool release. Which goes back to the point that there’s no one AI. What AI is is constantly changing. Seth: Right. So there’s a tension here. You’d want to take the short-term impact more seriously because there’s less margin for parallel trends to have a problem in the short term than in the long term. But on the other hand, we think there are all these short-term effects from adoption — either experimenting, or “I adopted the tool because I need it right now.” They focus on the long-term impact rather than the short-term impact, which is interesting, because usually in these difference-in-differences papers you see the opposite. They’ll say, “the effect in the short term is where I’ve really got the natural experiment, and then the long term, who knows?” Andrey: I want to push back on that. It just depends on which AI we’re interested in. Let’s say the parallel trends hold and we just have bigger effects once we have Opus 4.5. We might be more interested in the effects of agents with 4.5 than in agents with 4.0. They’re just different objects. And the key parallel trends are from the previous year, remember. There’s no sense in which Opus 4.5 being released in 2025 affects the parallel trend assumption for 2024. What would affect the parallel trend assumption for 2024 is if the people in 2024 started using some new tool at the end of 2024 that wasn’t being captured by the adoption measure. Seth: Right, but we already mentioned that concern, so it’s not an additional concern. It’s just unmeasured use again. Andrey: [42:07] And then the final empirical thing: they split out the causal effects by your pre-period activity. So how much you coded— Seth: We were talking about the effect on power users versus normies earlier. Andrey: And you see the biggest effects by far for people who have not been coding much according to these measures— Seth: And that’s in percentage terms. Andrey: In percentage terms, exactly. But it really should affect our interpretation, because presumably here we care about professional programmers more than non-programmers. All right, Seth, why don’t you tell us about the model the authors propose? The Model: A Seven-Layer CES Cake [42:56] Seth: The model is a fascinating one. It does a lot of what you would hope from a micro model of how code gets produced and economic value gets created. It’s got a lot of the elements you’d want. As we go, I’m going to point out that maybe there are a few too many elements, given that they’re only going to be able to bring the model to the data in a pretty superficial way. So what’s the model? One of our favorite functional forms: a constant elasticity of substitution production function at each level of production. Starting from lines of code and then to files and commits and pulls and repos and releases, they think about each of those levels as having its own production function that takes a little bit from the level above and combines it with what you’re adding at the current level. So there’s an elasticity of substitution between lines of code and some new effort brought in at the files level, and they’re combined to produce an input that goes down to the commit level. A seven-layer nested constant elasticity of substitution production function, for people following along at home. The key parameters they’re interested in estimating are the elasticities of substitution — how much the previous level’s inputs combine with the current level’s inputs. And then they’re also interested in what I would call the share term: what share of input at each level comes from the level above versus the current level. In a Cobb-Douglas production function those would just be the power terms. But they’re not going to interpret it that way. They’re going to invent a new parameter and say that’s actually what they’re measuring. We’ll come back to that. [45:19] At each level, the upstream level is combined with effective input at the current level. What’s effective input at the current level? The way they measure it is just the amount of additional stuff you get at that level. But they think you can make effective inputs in three different ways, each corresponding to a different way to interact with AI. The first is a pure labor-augmenting way, and there they’re thinking about autocomplete. The way you write more lines of code with autocomplete is that it makes each hour a programmer spends writing code produce so much more code. Then they think, well, asynchronous agents aren’t like that. The way they think about agents is that the AI is going to write so much code, and then that’s a Leontief complement with a human who needs to go through and review all of that code before it turns into effective input. So they call that a different sort of complementarity between AI and humans. And then finally their last mode is pure substitution. Maybe if the asynchronous agent is good enough, you could just let it cook and write as many effective lines of code as you want without any input. [46:36] And Andrey, in our previous episode — or maybe two episodes ago now — we used a phrase I really enjoyed: ex cathedra. We get this ex cathedra pronouncement that autocomplete enters the production function this way and synchronous agents enter the production function that way. It’s not going to end up being super important, but it’s meant to get you thinking about what patterns you might expect to see as the boosted productivity of lines of code attenuates. So that’s the basic setup, those are the parameters we’re interested in, and then the question is what we can conclude from this model, and finally they try to plug in some parameters and simulate it. Andrey: [47:16] It is a useful model for thinking through the production process here, but I agree with you, the ex cathedra pronouncements are a bit hard to swallow in some dimensions. The fact that you can use an agent can affect all other parts of how you produce your code. Even the code that you’re writing, you might be writing differently. Or alternatively, just because you’re using an async agent doesn’t mean you’re not also using a sync agent at the same time and autocomplete at the same time. Seth: I don’t think they say you couldn’t. The part that’s a limitation, or maybe a little heroic, is the idea that they can do this mapping to a production function a priori. Andrey: They’ve set up the model so that the optimal production mechanism is a corner solution, meaning that you can’t both be writing code and having the AI agent write code, for example. Seth: Right. And then when they simulate it, they simplify it even further, and they really only simulate it for autocomplete. Given that they cannot actually bring all of these parameters to the data, and given that they have such a rich empirical setting already — if I were writing this model, I would not have gotten into the nitty-gritty of “here are three different ways that AI can plug into the production function” that we cannot distinguish between in the best possible data. Oh, and then this is the delicious part, Andrey. The delicious part is that after setting up this highly complex production function, they then say: we’re not going to solve this production function for the optimal reallocation problem, because that’s too hard. Why would you write down a model that’s so complex that you can’t bring it to the data and can’t optimize it? I would drop some of these terms. But that’s a nitpick, because at a motivational level it does hit the different vibes that people have, even if it’s not an empirical question at this point. Andrey: I think the reason they did it is that they have seven outcome variables, and they needed a model that can rationalize those. Seth: They rationalize it with way fewer parameters than they write down. They only calibrate two of these parameters. Andrey: The model is general, but then they assume a bunch of the terms are identical — the layer substitutions are identical. Seth: And for example, there’s a parameter in the model which is the number of hours you need to spend reviewing each AI-written bit of code. If you could actually measure that, put it in the model. But if you can’t, I don’t know why this is in the model. There’s this temptation to overcomplicate models of automation, and I’ve run into this with commenters on stuff I’ve written down, where it’s “your model is just a CES production function, but what if something entered this way? What if it were complementary in this weird, different way?” The point of CES is to write something super general and super flexible down. To add more flexibility to the already super flexible— Andrey: Well, I don’t think CES is that flexible. I really don’t. Seth: You’re right to say CES is not super flexible. The way I’d put it is that in a lot of settings, the only things you observe are the things you can estimate with CES, so that’s why you write it down. Andrey: I think that’s what you’re trying to say, not that it’s flexible. Seth: [51:47] It’s a model that hits all of the empirically measurable things. All right, friends of the show, love you guys, but a couple more parameters in here than I would have written down. So what are the conclusions of this model? First of all, it’s intractable with reallocation, so we restrict attention to the partial equilibrium case where there’s no reallocation of efforts. What do they find? Two intuitive results. The first is that if you write infinity lines of code, you only get infinite output if there are gross substitutes at every link of the production hierarchy. That makes sense. So now we’ve got a story for why, even if you were really good at automating code, you wouldn’t have infinite output. Fair enough. The second theoretical result is that as long as there isn’t perfect substitution, you’re going to have attenuation of the boost as you move down the production hierarchy. If you have a big effect at the top of the waterfall, the effects attenuate as you go down the waterfall. Again, intuitive, but it’s good to see the model deliver the result we’re going to see in the data. [52:21] In the simulation, the goal is to estimate two parameters of the model. All the other parameters we’re going to forget about. The two we’re interested in are, first, the elasticity of substitution between production at a higher level and input at the current level, and second, what they call an elasticity of output between the higher level and the lower level. That’s not the way I would talk about it, because it’s not a primitive of the model — it’s going to depend on input levels. I would talk about it as a share term between production at the higher stage and the current level’s input. But given that, they estimate these two parameters. Call it what you will. The parameter that’s really the interesting one is the elasticity of substitution between input at the higher level and the human inputs. Why can we say it’s human inputs? They restrict attention to the autocomplete natural experiment. So based on their ex cathedra pronouncements, autocomplete works by boosting lines of code at the lines-of-code level, but it’s not going to have any effect on productivity down the cascade. Autocomplete isn’t going to help you produce commits or releases. Given that, they estimate the parameter such that you get that attenuating effect, and finally we get the headline result: a 0.25 elasticity of substitution between upstream production and lines of code. How does that make you feel, Andrey? Andrey: One way to think about it is that it’s just a translation of the fact that lines of code don’t result in that many new releases. There are a lot of things the model is obviously missing. For example, you can reallocate your effort, and that’s something the estimation is not capturing. But look — what do I view this as? The meat of this paper is the empirical exercise, and this is a back-of-the-envelope exercise for the elasticity of substitution parameter. Given that, I’m okay with it. I’m not going to go to the bank and say it’s 0.25. So that’s not the part I worry about. I’m more worried that all these things under the code are fungible with each other. What is even a commit? A commit-to-PR ratio is not a constant thing. There’s not even a standard way — different coding teams have different ways of splitting up this code production process. I don’t know whether to think of this as a hierarchy or not. Maybe that’s the deeper criticism. And then, as you point out, quality is really a missing element. Whether that code is comments, whether that code is unit tests that are maybe not that important — that might be something being produced with autosuggest. That’s very different from the other type of code being produced that was essential functionality. Seth: Very well put, Andrey. Are we ready to move to our posteriors? Aggregates, App Stores, and Posteriors [56:00] Andrey: No — one other thing to point out. They do this back-of-the-envelope exercise, and the final thing they do is think about aggregate outcomes. Another naive way to study the effect of AI tools on coding is just to look at the total number of GitHub pull requests over time. If we thought coding was making people much more efficient, we’d expect that to spike. What they find is a trend increase around the start of 2025, and their data doesn’t go long past 2026, so we don’t really know what happens after that. But that increase is substantially less than the per-programmer increase they find in the event studies. In particular, the increase in pull requests you can read as maybe a 50% increase. What might be driving that discrepancy goes back to the experienced versus inexperienced developers. The developers producing the most code — their measured productivity isn’t increasing as much as that of casual, occasional users. And they’re also the ones generating the most code, so the overall effect is not as big as the event study estimates suggest. The other thing to point out is that to the extent code is being used to generate apps, we do see increases in the number of apps being produced on, say, the iOS store — but we don’t seem to see an increase in downloads. So it’s not obvious, and it’s similar to our story with books, that a lot of these marginally produced apps are actually resulting in something people are using. To the extent that productivity is ultimately tied to value, we don’t seem to be seeing it yet. That points to a broader observation: if you look around, you see a lot more code, but it’s not clear it’s resulted in amazingly better software for us yet. Or, another way to put it — and this is shown in the OpenAI paper on how people use ChatGPT — there’s a lot of substitution from buying things to home production, or even just producing things at home that you weren’t going to do before. You might ask ChatGPT for advice, or for tech support, and before you might have hired someone to do that, or alternatively not solved the problem in the first place. That’s probably where most of the AI value is coming from right now. Maybe the most impactful use of AI agents is in the use of producing Claude and Codex. Seth: All super well-taken points, and the little empirical exercise on the App Store releases is a nice cherry on top. In the framework of their model, the explanation for why we get much less output of new products than lines of code is a complementarity with a human at one of those production process layers. But as you suggest, there are a lot of different things that bottleneck might be, or lots of reasons why economic value being created is not showing up in that final measure. So — one more reason to take with a grain of salt the idea that this gives us strong evidence of complementarity between AI coding agents and humans. Andrey: So do you want to move on to the posteriors? Seth: [1:00:07] For those of you playing along at home, now is your chance to think about how this conversation has changed your priors. This chance to contemplate your posteriors is sponsored by Revelio Labs. Revelio Labs is a leading provider of labor economics data and data services for companies, academics, and independent researchers. Andrey and I have been working in economics of AI, digitization, and automation for a long time, and we can confirm just how useful Revelio’s data is. Revelio’s team combines comprehensive micro-level data on employee professional profiles, job postings, and employee sentiment with standardizations, mappings, and enrichments available, all to make that data useful without making your modeling decisions for you. The data can be flexibly aggregated to company, market, or industry, and can be used to study questions ranging from career trajectories to occupational transformation to the returns to skills and the impact of AI on labor demand for tasks. Can’t imagine anyone who would be interested in that. And Revelio data is available on WRDS. So if you’re an academic with a good library, go see if you have access to their premier data already. And if you don’t, you can reach out to their excellent economics team and they’ll hook you up. Okay, so Andrey, remind me — what were our posteriors? Andrey: [1:01:32] What were our priors? Mine was 80% on the effect on developer lines of code. I’ve updated to probably 95% at that point. Does AI use boost the true economic value of software developers by less than 50% of the boost to lines of code? I’ve become more confident, so moving up to 95% there. Is AI coding a gross complement or substitute? I still think it’s a complement. I think this paper mildly supports that it is, at least in the relevant sense I was talking about. What about you? Seth: [1:02:10] To this paper’s credit — and I had some sharp words about the model — I think this paper actually does move my priors considerably. That’s the best thing I can say about a paper. On the first question, AI boosting lines of code written by greater than 100%, that moves me from 95% confidence to 99% confidence. It’s great to see these giant values. For AI boosting economic output by less than 50% of the boost to lines of code, I’m going up from 95% to 97.5%. Again, this confirms my priors. And then the most subtle question. This paper really only moves my view about complementarity between AI coding agents and humans at one very particular layer of the production process. I said there was maybe a 20% chance of complementarity between ordinary coders and AI coding agents. That’s unchanged. For human engineering departments, that’s going up from 33% to 40%. For software companies, that’s where I move the most. I’d move from a 60% chance of complementarity at the software company or open source project level to 85 or 90%. That’s a big move for me. Big-time move in the priors. And finally, for the economy as a whole, I’m unchanged. Andrey: Thanks for joining us for another episode of Justified Posteriors. Always remember to keep your posteriors justified, and to like, comment, and subscribe to our podcast. Thank you. Get full access to Justified Posteriors at empiricrafting.substack.com/subscribe