“ML infrastructure” names two different things. One is the machinery that serves and trains models: GPUs, batching, parallelism. That layer is mapped in ML Infrastructure Stack Below the API. This note covers the other one: the platform layer above the GPU. The pipelines, stores, registries, gateways, and monitors that turn a trained model into a system a company can run without it quietly breaking. Most of a “platform team’s” work lives here, not in CUDA.
The stack is bifurcating
For a decade the production-ML stack had a stable shape: a feature store, a training pipeline, a model registry, a serving endpoint, and a drift monitor. The primary artifacts were training data and model weights. The primary risks were data skew and fairness. The LLM wave did not extend this stack. It grew a parallel one.
The replacement is close to one-for-one, which is why the shift is easy to miss:
| Layer | Classic MLOps | LLM-era stack |
|---|---|---|
| Context store | Feature store (Feast, Tecton) | Vector store (Pinecone, Qdrant, pgvector) |
| Primary artifact | Training data, weights | Prompts, evals, embeddings |
| Registry | MLflow, W&B | Prompt/version registries (LangSmith, Humanloop) |
| Orchestration | Airflow, Kubeflow | LangGraph, Temporal, agent runtimes |
| Evaluation | Held-out metrics (AUC, F1) | LLM-as-judge, Ragas, DeepEval |
| Safety | Fairness checks (Fairlearn) | Guardrails (Llama Guard, Lakera) |
Two structural differences sit under the table. Cost moved from training to inference: a classic model is expensive to train once and cheap to call, while an LLM feature is cheap to adopt and expensive on every call, which is why the gateway layer (rate limits, caching, fallback routing) is new and load-bearing. And evaluation went from a number to a judgment: you cannot compute an F1 score on “was this answer good,” so the eval harness itself became infrastructure. See Context Engineering for what the retrieval half of this stack is doing, and AI Inference Unit Economics for why the cost shape changed.
The classic stack did not die. Fraud scoring, recommendations, and forecasting still run on feature stores, and Feature Store tooling matured in 2025 (Tecton folded into Databricks; MLflow 3 shipped mid-2025 with GenAI features). The point is that “ML infrastructure engineer” is starting to split into two roles that share a name and little else, a framing the hiring market has an interest in naming, so weigh it as a direction rather than a settled fact.
GPUs are a scheduling problem, not a chip problem
The expensive resource in ML infrastructure is GPU-hours, and the number that governs the bill is utilization, not FLOPS. A cluster full of H100s that sits idle half the time costs the same as one running flat out. So a large share of platform work is keeping expensive silicon busy.
The public evidence for how hard this is comes from Meta’s Llama 3 405B run. Over 54 days on 16,384 H100s the cluster hit 466 interruptions, 419 of them unexpected. That is roughly one failure every three hours, most of them hardware: GPUs dropping, NVLink faults, HBM3 memory errors. The team still kept about 90% effective training time (goodput) through checkpointing and fast restart. The lesson generalizes downward: failure rate scales with device count, so at large cluster sizes failure is the steady state, and the infrastructure exists to make failure cheap rather than to prevent it.
The scheduling side is the same story for inference and shared research clusters. A stock Kubernetes scheduler treats a GPU as binary, allocated or not, with no notion of how much of the card a job uses. That leaves large clusters fragmented and half-idle. The 2025 response is a layer of GPU-aware schedulers (Run:ai, Kueue) plus Kubernetes Dynamic Resource Allocation reaching GA, which lets a scheduler partition and bin-pack fractional GPUs instead of handing out whole cards. Utilization is the metric this layer is paid to move.
Why a platform layer exists at all
The recurring question is why any of this is separate infrastructure instead of code in the application. The answer is the one Hidden Technical Debt in Machine Learning Systems named in 2015: the model is a small box in a large diagram, and the boxes around it (data collection, feature extraction, serving, monitoring, configuration) are where the cost and the failures live. A feature computed one way in training and another way in serving produces skew that no unit test catches. A prompt changed without versioning breaks a workflow no one can reproduce. Centralizing these concerns makes them governable.
This is a Team Topologies platform-team pattern applied to ML: the platform exists so application teams consume a paved road (register a model, get an endpoint, see its metrics) instead of each team rebuilding pipelines and monitors. The failure mode is the same as any platform: build too much and it becomes a bottleneck, build too little and every team reinvents the plumbing badly. None of it is day-one infrastructure: a small team calling one API needs none of these layers, and each one earns its place only when scale, team count, or a reliability requirement forces it. The current wrinkle is that the LLM stack is roughly where MLOps was in 2018: strong deployment tooling, weak governance. The Inference Gateway and eval layers are consolidating now, and which org owns them is still being settled.
Try it
Run a feature store and make training/serving skew visible (2-3 hours, Python + Feast). Install Feast, define a couple of features over a small CSV, and materialize them. Then request the same feature from the offline store (for training) and the online store (for serving) and diff them. Introduce a deliberate mismatch, compute the feature over a different time window offline than online, and watch the two values diverge. That divergence is the exact bug feature stores exist to prevent, and seeing it once explains why the layer earns its place the moment you have real training and serving paths.
Put a gateway in front of two model backends (1-2 hours, LiteLLM or an OpenAI-compatible proxy). Route the same request through a proxy that fronts two backends (say a local model via Ollama and a hosted API). Add a fallback rule and a response cache, then fire repeated and concurrent requests. Watch the cache collapse duplicate calls and the fallback fire when one backend errors. That is the inference-gateway layer the cost shift makes worth building at volume, running on your laptop.
See also
- ML Infrastructure Stack Below the API — the serving and training internals this note sits on top of.
- Hidden Technical Debt in Machine Learning Systems — the paper that named the glue.
- AI Inference Unit Economics — why inference cost drives the new stack.
- Context Engineering — the retrieval half of the LLM-era stack.
- Team Topologies — the platform-team pattern this layer instantiates.
Sources
- Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training — Tom’s Hardware, on the 466 interruptions and ~90% goodput figures from Meta’s Llama 3 report.
- MLOps vs LLMOps Engineer: The Split Nobody Has Named Yet — the layer-by-layer stack replacement table (a jobs-market source, incentivized to name new categories; weigh the split thesis accordingly).
- MLOps in 2026 — The Definitive Guide — feature store maturity, MLflow 3, orchestration tooling.
- AI Workload Scheduling / GPU utilization — vendor overview of the scheduling and utilization problem (treat specific loss figures with caution).