[!todo] Seed note. A starting point, not a finished note yet.
An inference gateway (or LLM gateway) sits between an application and one or more model backends and handles the operational concerns that the cost shape of LLMs made unavoidable: request routing across models and providers, response caching, rate limiting, retries, and fallback when a backend errors or slows. It exists because LLM features are cheap to adopt and expensive on every call, so the marginal request needs governance the classic serve-a-model-endpoint pattern never required. LiteLLM and OpenAI-compatible proxies are common implementations; cloud model routers play the same role. It is one of the consolidating layers of the LLM-era stack. Seeded from ML Infrastructure.