What Is Time to First Token?
Time to first token (TTFT) is the delay between sending a request to a language model and receiving the first token of its response. It is one of the two latency numbers that matter most for LLM serving, alongside inter-token latency (how fast tokens stream once generation starts).
TTFT is dominated by the prefill phase: before a model can emit anything, it has to process the entire input prompt and build the key-value cache for it. A long prompt, a large model, or a busy server all push TTFT up. Once prefill is done, generation proceeds token by token and inter-token latency takes over.
Why It Matters
TTFT is what a user actually feels as responsiveness. In a chat interface, a voice agent, or a coding assistant, the wait before anything appears is the wait that gets noticed. Two systems can have identical total throughput while one feels snappy and the other feels sluggish, purely because of TTFT.
It also drives design choices. Applications with long system prompts or heavy few-shot context pay for that context on every request in the form of higher TTFT, which is why techniques that avoid recomputing shared context are valuable.
Measuring It Honestly
TTFT is normally held to a percentile target rather than an average, for example P95 under one second. Averages hide the queued requests, and the tail is what users notice.
It is also distinct from throughput. An endpoint can post high tokens per second and still feel slow if TTFT is poor under load, because throughput is measured once generation is running and TTFT is measured before it starts. The two numbers move independently, and an endpoint that publishes only one of them is publishing half the picture.
Because it is measured on production traffic rather than synthetic runs, TTFT is a truer read of the operating point a fleet actually sustains for real users than a benchmark taken on an idle cluster.
Reducing TTFT
The main levers act on the prefill cost and on queueing:
- Prefix caching: reuse the KV cache for shared prompt prefixes so repeated context is not recomputed.
- Right-sizing the model and GPU: a model that fits on a single well-matched GPU prefills faster than one split across devices.
- Continuous batching: keep the server from stalling behind long-running requests.
- Autoscaling: add capacity before queue depth turns into latency.
Saturn Cloud exposes these as part of its inference platform. Its endpoints track TTFT alongside throughput and GPU utilization, and run on GPU infrastructure sized to the workload, so teams can tune for responsiveness rather than guess at it.
