What Is Time to First Token?
Time to first token (TTFT) is the delay between sending a request to a language model and receiving the first token of its response. It is one of the two latency numbers that matter most for LLM serving, alongside inter-token latency (how fast tokens stream once generation starts).
TTFT is dominated by the prefill phase: before a model can emit anything, it has to process the entire input prompt and build the key-value cache for it. A long prompt, a large model, or a busy server all push TTFT up. Once prefill is done, generation proceeds token by token and inter-token latency takes over.
Why It Matters
TTFT is what a user actually feels as responsiveness. In a chat interface, a voice agent, or a coding assistant, the wait before anything appears is the wait that gets noticed. Two systems can have identical total throughput while one feels snappy and the other feels sluggish, purely because of TTFT.
It also drives design choices. Applications with long system prompts or heavy few-shot context pay for that context on every request in the form of higher TTFT, which is why techniques that avoid recomputing shared context are valuable.
Reducing TTFT
The main levers act on the prefill cost and on queueing:
- Prefix caching: reuse the KV cache for shared prompt prefixes so repeated context is not recomputed.
- Right-sizing the model and GPU: a model that fits on a single well-matched GPU prefills faster than one split across devices.
- Continuous batching: keep the server from stalling behind long-running requests.
- Autoscaling: add capacity before queue depth turns into latency.
Saturn Cloud exposes these as part of its inference platform. Its endpoints track TTFT alongside throughput and GPU utilization, and run on GPU infrastructure sized to the workload, so teams can tune for responsiveness rather than guess at it.
