Software factories are a bad analogy

LLMs were not involved in any step of writing this.

Fully automated “software factories” are allegedly right on the horizon. Some companies claim their factories are “dark,” a phrase borrowed from manufacturing, where fully automated factories run with lights off, because robot arms don’t need to see. In software, this means there’s no humans in the “factory” because no one is reading or writing code. Instead, engineers are doing harness or “loop” engineering for Claude. In practice, this means hooking Claude up to some input (bugs, incidents, feature requests) and giving it access to all the tools to write, verify, and ship code to satisfy the input. It’s vibecoding, at scale.

Some people say it works! Some of those people are in the business of selling you tools to set up software factories! On the other side, there’s folks that say it doesn’t work and can never work. Some of those folks are also in the business of selling tools, just different tools with more humans in the loop. And of course, everyone yapping about this is either a software engineer with a desire to remain employed (e.g. me) or a manager/executive/shareholder with a desire to reduce labor costs by laying off expensive engineers.

I think the software “factory” analogy is flawed and refining it can help us think more clearly about the role of coding agents.

What do you mean “factory?”

Disclaimer: I don’t know anything about factories.

A “software factory” imagines that we can input a long line of specifications to the factory, maybe in the form of a bug report or feature request, and get working software as an output.

However, I think calling this a factory is deeply flawed. A factory outputs mass quantities of the same thing. Creating N of the same good is completely different from creating N different goods. Moreover, we can make a much better analogy. Specifications exist for physical goods too! A basic factory has 1 specification, takes raw material, and outputs finished goods.

Some factories may accept customized orders that build off of a base specification. Customizations may be surface level (laser engraving) or consist of combining other assemblies (RAM and storage in a laptop). Notably, customizations are clearly scoped, and engineers build the factory and base specification to manufacture these customizations efficiently.

Next, we have custom fabricators, which are not really factories at all. Custom fabricators really do receive unique specifications from clients.

In some contexts, like 3D printing, CNC, and laser cutting, this can be fully automated (“lights out”). So how come these fabrication methods can be fully automated? Again, I don’t really know what I’m talking about, but I’d guess that it’s partially because:

1. These involve a robot following instructions derived from a exact, detailed specification in the form of a CAD model, which has been translated deterministically into physical steps.
2. Design, integration, and testing are all left to the client.
3. Size and complexity limits are set based on the machine’s capabilities.

However, if you want something custom built and you don’t have a CAD file or it’s complex, you’ll need to contact a custom job shop and have some conversations with smart humans who will figure out how to build what you’re talking about. Maybe they will use a 3D print farm, maybe they won’t.

So… just call it a software-custom-fabricator?

OK, so did I write all this just to nitpick the term “software factory” and suggest we start using the convoluted term “software custom fabricator?”

No, but I think fleshing out the analogy properly helps us understand the challenges of agentic coding, and how the vision of “specs in, code out” may pan out.

Let’s notice a few thing about what kinds of fabrication are able to go “lights out” and compare to software, where zero marginal costs let us ignore the raw material inputs.

Factories

The factories that produced N of the exact same good can be run lights out, but the design of the factories themselves took human ingenuity and iteration. The software equivalent of this is copying your executable and sending it around, obviously trivial and uninteresting.

Factories with Customization

The factories that did some level of customization could be viewed most trivally as having settings in your software or distributing different versions to different platforms or operating systems.

A more interesting read would be software that can truly be customized by laypeople. For example, drag-and-drop flowchart programming used by Zapier let people make their own custom workflows. The blocks and Zapier app itself are maintained by professionals, but you can choose how to put them together without having to pay an integrator. In an AI-native world, you chat with a bot which builds the workflow for you.

A more software-development-centric analogy would be allowing non-engineers to make changes to software using a highly guided and constrained process. Pre-AI, DSLs would be built to enable configuration. Post-AI, “skills” can be used to guide non-engineers through specific types of changes. For example, you might let marketers configure the copy and basic styles of certain pages or create landing pages for campaigns. It resembles the factory in the sense that there remains a base specification (the rest of the code base) and changes are minor or cosmetic.

Another form of this might be highly scoped bug fixes. Flaky specs and minor UI issues can be fixed by an coding agent, and in some sense, these are cosmetic changes to a base spec.

Digital Direct Fabrication

Digital direct fabricators (3D printing etc.) receive an exact specification that is compiled to machine movements, allowing them to run autonomously. It would not work with a vague description of what you want. Historically, software was itself the exact specification that is compiled to machine instructions. With AI, we can accept specifications in English and generally translate that into software. But same as with the fabricators, this requires a very thorough specification, manual testing by the client, and size/complexity limits. This might be most analogous to greenfield vibecoding.

However, I think there are two major challenges here that constrain us to greenfield/small projects specifically.

First, producing a detailed spec is challenging, especially against an existing system. To accurately specify a change to a complex system you have to understand that complex system. Often, a specification sounds reasonable until you try to implement it and discover your flawed assumptions. This usually resulted in conversations between engineering, product, and design to come up with a new specification.

Second, unlike real manufacturing, software generally becomes more and more complex through iterations, with no obvious physical limitations. Eventually, ungroomed codebases reach a level of size and complexity where you can’t change anything without breaking something else and productivity grinds to a halt. Perhaps coding agents can become better at building maintainable, extensible code. Perhaps coding agents running in long term loops could pause to improve code quality instead of being RLHF’d into task completion at any cost.

Custom Fabrication Job Shop

Last, we have the job shop, which currently operates in an equivalent of just hiring engineers the old fashioned way. It shines in places where you don’t have an exact specification or you have high complexity, often for new feature development. You give the engineers a specification, which they interpret. They may find issues and come back to you with questions and compromises, and you work through it together. They advocate for investment in code quality and look after the long term health of the code. They may use automated tools and coding agents, but it doesn’t matter to the client.

Can LLM-powered coding agents do this? They don’t do a great job of it now. Can they never develop these capabilities? I don’t know.