ML Infrastructure Stack Below the API

Most engineers who work with AI live above a single line: client.messages.create(...). That line hides an enormous amount of systems engineering. This note maps the layer below the API call, and ends with a concrete weekend project to turn reading into intuition.

The reason to care: “I use LLMs in my product” and “I build ML infrastructure” are different jobs with different markets. The first is the application layer. The second is the machinery that makes the first possible: serving models under load, and training them in the first place. Knowing the difference, and which one you actually want to do, is worth a weekend of hands-on work.

The inference layer (serving)

This is what turns weights on disk into tokens over the wire, fast and cheap.

• Serving engines. vLLM, TensorRT-LLM, SGLang. They own the hard part: packing many concurrent requests through the GPU efficiently.
• Continuous batching. Requests do not arrive in neat batches. The engine merges in-flight requests dynamically so the GPU is never idle waiting for a slow request to finish. This is the single biggest throughput lever.
• KV-cache. Attention re-reads every prior token. Caching those key/value tensors avoids recomputation, but the cache eats VRAM and caps how many requests fit at once. Managing it (paging, eviction) is core to serving.
• Quantization. Storing weights in fewer bits (INT8, INT4, FP8) shrinks memory and speeds inference, trading a little quality for a lot of headroom.
• The tension that defines the layer. Throughput (tokens/sec across all users) versus latency (time-to-first-token and inter-token latency for one user). You cannot maximize both. Every serving decision is a point on that curve. See AI Inference Unit Economics and Test-Time Compute.

The training layer

This is what produces the weights the inference layer serves.

• Why one GPU is never enough. A frontier model does not fit in one GPU’s memory, and would take lifetimes to train there anyway. Training is a distributed-systems problem first, a math problem second.
• Parallelism, three flavors. Data parallelism (same model, different data shards, gradients synced). Tensor parallelism (a single layer split across GPUs). Pipeline parallelism (different layers on different GPUs). Real runs combine all three.
• FSDP / ZeRO. Fully Sharded Data Parallel and the ZeRO family shard the model, gradients, and optimizer state across GPUs so no single device holds the whole thing. This is how large models fit at all.
• The daily reality. Runs fail. GPUs drop. Gradients spike. Utilization sags and someone has to find the stall. Checkpointing, restarts, and keeping thousands of GPUs busy is the actual job, not the model architecture.
• Fine-tuning as the accessible on-ramp. LoRA and QLoRA train small adapter matrices instead of the full model, so you can adapt a model on a single consumer GPU. This is where a non-specialist can get real hands on keyboard.

Why the layers feel invisible

The whole point of a good API is to hide this. That is a feature for product work and a blind spot for a career. If your goal is to lead or build at this layer, the abstraction you rely on daily is exactly the knowledge you are missing. The fix is not more reading. It is touching the machinery once. Related: Hidden Technical Debt in Machine Learning Systems.

The weekend project

Three weekends, roughly zero dollars (a rented GPU hour or a free Colab tier). The goal is not a product. It is to feel the two layers and find out whether the day-to-day is something you want more of.

1. Fine-tune. Take a small open model (a 1B to 8B class model). LoRA-tune it on a tiny dataset. Watch a training loop actually run: loss curve, a run that fails, a hyperparameter that matters.
2. Serve. Stand the result up behind vLLM. Send it real concurrent traffic, not one request at a time.
3. Benchmark and improve. Measure tokens/sec and p99 latency under load. Then change one thing (quantize the weights, change batch settings, resize the KV-cache) and measure again. Make one number better on purpose.

What the project actually tells you: whether OOM errors, throughput graphs, and config-tuning read as satisfying or as toil. That answer is worth more than any job description, and it costs a weekend instead of a career pivot.

Where to go deeper

• vLLM paper and docs (continuous batching, PagedAttention) for the serving side.
• FSDP tutorials and the ZeRO papers for the training side.
• Related notes: Mixture of Experts, Test-Time Compute, AI Inference Unit Economics, Hidden Technical Debt in Machine Learning Systems.