Inference Request Lifecycle
When an online inference request feels slow or unstable, the root cause is often not obvious because the request passes through many layers before a token is emitted. A useful way to reason about the system is therefore not only by stack layers, but also by time: what happens first, what waits on what, and where latency or throughput is lost along the path.
This page follows one request from the application layer down to GPU execution and back to the user. The goal is to make performance debugging more concrete by connecting abstract layers to a real execution sequence.
Why Lifecycle Matters
A request-level view is useful because many symptoms that appear identical at the API boundary come from different stages underneath.
- A high TTFT may come from prompt assembly, queueing, prefill cost, or runtime admission delays.
- Low throughput may come from routing fragmentation, weak batching, KV pressure, or kernel inefficiency.
- Latency jitter may come from upper-layer fan-out, runtime preemption, or communication imbalance.
Looking only at the final response time hides these distinctions. A lifecycle view exposes where the request spends time and which layer owns the bottleneck.
Start with the Request Path
The easiest way to read the lifecycle is to begin with the layered flow rather than with a detailed time diagram. A request starts in the application layer, descends into the serving system, reaches hardware execution, and then climbs back up the stack as tokens stream out. That path already explains a large part of inference behavior: who shapes the request, who decides when it runs, who owns memory state, and where the actual compute happens.
The flowchart is useful because it separates three kinds of responsibility that are easy to mix together in practice.
1. Application / Orchestration Layer Shapes the Workload
The lifecycle begins before the serving engine sees anything. A chat system, coding assistant, search product, or workflow service decides that the next action requires a model call. In an agentic system, this stage may also include tool planning, retrieval, memory lookup, and prompt templating.
This layer determines the workload that will enter the system. Long prompts, aggressive fan-out, duplicated retrieval context, or poorly structured tool traces all increase the burden on the lower layers. When a request is already inflated here, no scheduler or kernel optimization can fully undo that cost later.
2. The Serving System Turns an API Call into Managed Execution
Most of the operational complexity lives in the middle layer. The request is no longer just a payload; it becomes a scheduled object with routing decisions, KV state, execution steps, and streaming outputs.
Router / Gateway
This is also where a component such as LiteLLM typically fits when it is used as a unified API layer or model proxy.
At the outer edge of the path, an infrastructure load balancer such as ELB may sit even earlier and distribute traffic across multiple LiteLLM or API gateway instances.
The router decides which model endpoint, replica group, or engine instance will receive the request. This affects not only infrastructure utilization but also cache locality and batch formation. Two requests with the same prefix may or may not benefit from prefix caching depending on whether they are routed to the same backend.
In practice, LiteLLM usually sits above the serving engine and below the application or agent layer:
- It presents one OpenAI-compatible or unified API to the application.
- It translates or normalizes requests across multiple model providers or backends.
- It can enforce routing policy, fallback, rate limits, logging, or cost/account controls.
What it usually does not do is execute the model step itself. Once LiteLLM forwards the request to vLLM, SGLang, OpenAI, Anthropic, or another backend, the lower serving/runtime layers take over request admission, batching, KV management, and GPU execution.
So in this lifecycle, LiteLLM belongs to the Router / Gateway / API Proxy position, not to the Serving Engine, Scheduler, or Model Executor positions.
An ELB-like component belongs even farther outward. It is usually responsible for generic network-level or HTTP-level traffic distribution, not for model-aware routing semantics. In other words:
- ELB decides which gateway instance receives the connection or request.
- LiteLLM decides which model backend or provider should serve the request.
Serving Engine
Frameworks such as vLLM or SGLang receive the request and convert it into an internal engine object. At this point the external API boundary ends and the serving framework's own runtime lifecycle begins. The engine owns request admission, lifecycle tracking, and output streaming.
Scheduler and KV Manager
Once inside the engine, the scheduler decides when the request actually runs. It may enter prefill immediately, wait behind earlier work, or be interleaved with other requests through continuous batching. This is often where latency behavior becomes non-intuitive: a request may already be inside the engine but still not be making progress.
The scheduler does not act alone. It relies on the KV manager to allocate, attach, reuse, and reclaim KV state. During prefill, new KV blocks are materialized. During decode, existing blocks are extended or reused. Under memory pressure, the scheduler-KV-manager boundary becomes one of the most important control points in the system because it determines whether the engine can safely admit more work or must become more conservative.
Model Executor
The model executor turns runtime decisions into actual model steps. It launches prefill and decode work with the batch shape selected by the scheduler and the KV state prepared by the KV manager. This is the bridge between serving policy and model computation: batch formation, scheduling, and memory management all become visible here as execution shape.
3. The Hardware Layer Performs the Actual Computation
At the bottom of the stack, kernels run on the hardware: GEMMs, attention kernels, communication collectives, fused operations, sampling kernels, and memory movement. This is where operator efficiency, interconnect bandwidth, and kernel launch overhead finally show up as realized performance.
If utilization is low here, the root cause may still lie above. Too little work per decode step, excessive synchronization from the chosen parallelism strategy, or a request stream that never forms efficient batches can all surface as poor GPU utilization even when the kernels themselves are implemented well.
4. Tokens Return Through the Same Stack
After each decode step, outputs move back upward: from model executor to engine, from engine to router, from router to application, and finally to the user. For streaming products, this return path is part of the user experience rather than an implementation detail. The user does not experience one monolithic completion; the user experiences a sequence of partially delayed tokens.
Then Read It as a Time Sequence
The flowchart answers one question: which layers and components does a request pass through? The time sequence answers a different question: in what order do these components interact during one request?
Reading both diagrams together makes the control flow easier to interpret. The flowchart gives the stable structure of the system. The sequence diagram shows one concrete traversal of that structure. When a latency or throughput problem appears, the practical debugging question is usually: which node in the flowchart is responsible, and at what point in the sequence does it start to dominate?
Where LiteLLM Sits
LiteLLM is best understood as a unified model access and proxy layer. It gives the upper application a consistent API while hiding differences among downstream providers or serving backends.
In the request path, its most natural location is:
Application / Agent -> LiteLLM -> serving backend (vLLM, SGLang, OpenAI API, Anthropic API, etc.)
That means LiteLLM is part of the north-south request entry path into inference, not the inner execution loop of inference itself.
- If LiteLLM adds retries, fallback, or provider selection, it changes request routing behavior and tail latency.
- If LiteLLM adds observability, auth, budgets, or quotas, it acts as control-plane middleware around inference.
- If the backend itself is slow because of queueing, KV pressure, or GPU underutilization, that is below LiteLLM in the stack.
This distinction matters during debugging. A bad experience seen at the API boundary might come from LiteLLM-side routing or policy logic, or from the downstream serving engine and hardware path after the request has already been forwarded.
ELB and LiteLLM
ELB and LiteLLM are related, but they usually operate at different levels of the request path.
The common deployment pattern is:
Client / Application -> ELB -> LiteLLM -> serving backend
or, in a provider-mixed setup:
Client / Application -> ELB -> LiteLLM -> vLLM / SGLang / OpenAI / Anthropic / other providers
Their responsibilities are different:
- ELB is typically an infrastructure load balancer. It spreads incoming traffic across multiple LiteLLM or gateway instances and helps with availability, horizontal scaling, and entry-point traffic management.
- LiteLLM is a model-aware gateway. It understands model names, provider differences, retries, fallback, auth policy, rate limits, logging, and cost-aware routing.
This separation is useful because it keeps network-level balancing distinct from model-level decision making. If one LiteLLM instance is overloaded, ELB can steer traffic elsewhere. If one model provider is degraded or too expensive, LiteLLM can choose a different backend without changing the application's upstream integration.
Where Bottlenecks Usually Show Up
The same request-lifecycle view is useful for debugging.
- High TTFT often comes from orchestration overhead, queueing, prefill cost, or delayed admission.
- Low TPS often points to weak batch formation, KV pressure, communication overhead, or underfilled kernels.
- Preemption usually indicates pressure in the scheduler-KV-manager interaction.
- Poor prefix-cache benefit often originates in upper-layer routing rather than in the cache mechanism itself.
- Low GPU utilization may reflect too-small decode steps, communication-heavy topology, or a request stream that never forms efficient batches.
Relationship to the Rest of AI Infra
This lifecycle view complements the stack overview rather than replacing it.
- Overview explains where each component sits in the system.
- Metrics explains how the request lifecycle is measured.
- KV Cache explains the main runtime state object carried through prefill and decode.
- Serving Runtime explains how the scheduler and runtime policies shape request progress.
- Parallelism explains how execution is distributed once the request reaches model computation.