Sculley et al., NIPS 2015 took Ward Cunningham’s technical debt metaphor and applied it to production machine learning. The claim: ML systems carry all the maintenance costs of ordinary software plus a second set of ML-specific costs that are worse because they hide at the system level, not in any line of code. The paper became a founding document of MLOps. Much of what it names still has no clean fix.
The ML code is the small box
The paper’s most-cited figure shows the actual model training and inference code as a small black box surrounded by much larger boxes: configuration, data collection, feature extraction, data verification, resource management, serving infrastructure, monitoring, process tools. The learning is a fraction of a real system. Everything else is plumbing, and the plumbing is where the debt lives. This is the same shape as the Theory of Constraints point that output is set by whatever isn’t the thing you’re optimizing, and the same shape as Writing Code vs. Shipping Code: AI Productivity Across Tool Generations, where gains in the visible activity get swallowed by the surrounding stages.
Entanglement: changing anything changes everything
ML mixes signal sources so that no input is independent. Add a feature, remove one, change a hyperparameter, or let the input distribution shift, and the learned weights all move. The paper calls this CACE: Changing Anything Changes Everything. A feature you improved in isolation can degrade the whole model. The paper offers two mitigations: isolate models and combine them in an ensemble so a change is contained, or monitor prediction differences to detect when a change has rippled. Neither removes the entanglement; they make it observable.
Feedback loops and undeclared consumers
Because model outputs are usually written somewhere accessible (a log, a table, a file), other systems start reading them without telling anyone. The paper calls these undeclared consumers, a form of visibility debt. They create hidden tight coupling: you can no longer change your model without breaking a system you didn’t know existed.
Worse are feedback loops, where a model influences its own future training data. A direct loop: a model that picks what to show shapes what gets clicked, and clicks become the next round of labels. Judging whether the model is actually good becomes intractable, because the model helped create the data it’s judged on. This is the mechanism behind Systems Thinking’s warning that feedback structure, not any single component, generates system behavior. It also looks like the seed of what later got called model collapse: outputs pollute the corpus that trains the next model.
Data dependencies cost more than code dependencies
Code dependencies get caught by compilers and linkers. Data dependencies have no such tooling, so they accumulate silently.
- Unstable dependencies. Your input is another team’s output (a lookup table, or another model) that updates over time. When it changes, your behavior changes and no test fires. The fix is versioned, frozen copies of the signal, at the cost of staleness.
- Underutilized dependencies. Features that add almost no accuracy but add a failure surface: legacy features subsumed by newer ones, features bundled in for convenience, epsilon features added to chase tiny gains. The paper recommends regular leave-one-feature-out evaluation to find and delete them.
Google reported building automated tooling to annotate and track data dependencies, because static analysis for data is roughly where code analysis was decades ago.
The pile of glue
The system-level anti-patterns are where the paper is bluntest.
- Glue code. Using a general-purpose ML package forces massive supporting code to get data in and out. The Google authors report from their own systems that glue code can comprise 95% or more of a mature ML codebase. It freezes you to one package and buries the ML in plumbing.
- Pipeline jungles. Data preparation grows organically as signals are bolted on (scrapes, joins, sampling steps) until the pipeline is a jungle no one can test or reason about. Often a symptom of research and engineering working in silos. The fix is usually a from-scratch redesign, not a patch.
- Dead experimental codepaths. Conditional branches for old experiments left in the code. They accumulate and interact. The paper cites Knight Capital’s loss of $440M in 45 minutes, triggered when obsolete experimental code was reactivated in production.
- Abstraction debt. ML has no widely agreed abstraction the way databases have the relational model. What is the “database” of machine learning? None has emerged, so everyone rebuilds the scaffolding.
- Configuration debt. In a mature system the configuration surface (which features, which data, which learning settings, which pre- and post-processing) can exceed the code in size. Every line is a chance to be wrong. The recommendation: treat config like code, versioned and reviewed and diffable.
Changes in the external world
An ML system is coupled to a world that shifts under it. Manually set decision thresholds go stale the moment the model retrains, so thresholds should be learned on held-out data, not hardcoded. Unit and integration tests are insufficient because the failure is often statistical, not logical. The paper argues for live monitoring of prediction bias (the distribution of predictions should track the distribution of observed labels) and action limits (alert or cap when the system starts taking an unusual number of real-world actions).
Measuring the debt
The paper closes not with a metric but with questions a team should be able to answer, and be alarmed if they can’t:
- How easily can an entirely new algorithmic approach be tested at full scale?
- What is the transitive closure of all data dependencies?
- How precisely can you measure the impact of a new change?
- Does improving one model or signal degrade others?
- How quickly can a new team member be brought up to speed?
The cultural point underneath: moving fast usually incurs debt, and paying it down (deleting a feature, tightening an API, killing a dead codepath) is real progress even though it moves no accuracy number. Teams that only reward accuracy gains will accumulate debt until the system ossifies.
What changed since 2015
The tooling caught up to much of the diagnosis. The same group followed with The ML Test Score (Breck et al., 2017), a 28-point rubric across data, model, infrastructure, and monitoring. The MLOps stack that grew up next answered several of the named debts head-on: feature stores directly attack unstable and underutilized data dependencies, data versioning is close to the frozen-copy fix the paper recommended, model registries and experiment tracking cut into configuration and reproducibility debt. That is a real response, not bookkeeping.
The split worth naming is which debts the tooling retires and which it cannot. Roughly half the paper’s list is ordinary software debt that any large system accumulates: glue code, dead codepaths, pipeline jungles, configuration sprawl. Tools and discipline handle those. The other half is statistical and system-level, and it is the genuinely ML-native part: entanglement, feedback loops, undeclared consumers. Those are properties of the problem, not gaps in a toolchain, and no registry retires them. Foundation models sharpen exactly this half. Prompts are untracked configuration, a shared model API has undeclared consumers by the thousand, and model output fed back into training data is a feedback loop at internet scale.
Try it
Make a feedback loop collapse a recommender (an afternoon, Python + numpy/sklearn). Simulate a recommender that trains only on items users clicked, where click probability depends on both true item quality and whether the item was shown. Loop: recommend top-scored items, sample clicks, retrain on those clicks, repeat. Watch the catalog of recommended items shrink to a handful regardless of true quality. You are reproducing the direct feedback loop from the paper: the model manufactures the data that confirms it, and true quality becomes unmeasurable from the logs alone. Then break the loop by injecting a small fraction of random exploration and watch recovery.
Find underutilized features by leave-one-out (1-2 hours, sklearn). Train a gradient-boosted model on any tabular dataset with 15+ features. Retrain dropping each feature in turn and record the change in held-out accuracy. Rank features by their marginal contribution. The near-zero and negative ones are the paper’s underutilized data dependencies: features you’re paying to maintain for no accuracy, and a source of hidden fragility.
See also
- Systems Thinking: feedback structure generates behavior; the loops here are the ML instance
- Theory of Constraints: why the ML code being small means the ML code isn’t the bottleneck
- Writing Code vs. Shipping Code: AI Productivity Across Tool Generations: gains in the visible step swallowed by the surrounding stages
- CACE Principle and Correction Cascades: two of the paper’s named traps
Sources
- Hidden Technical Debt in Machine Learning Systems. Sculley, Holt, Golovin, Davydov, Phillips, Ebner, Chaudhary, Young, Crespo, Dennison. NIPS 2015.
- The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Breck et al., 2017. The actionable follow-up.
- Nitpicking Machine Learning Technical Debt. A later re-reading of the paper against the MLOps tools that grew up around it.