Inference, Latency and Throughput: The Mechanics Behind AI Response Speed

What this means

When a user submits a prompt, the AI system performs a forward pass through the model's neural network: the input tokens are processed layer by layer through the transformer architecture, and output tokens are generated one at a time in an autoregressive loop. The model does not generate the entire response at once — it calculates each token sequentially, which is why streaming responses appear word by word rather than all at once.

Latency is typically measured as:

• Time to First Token (TTFT): the interval between submitting a request and receiving the first output token. This determines perceived responsiveness — the moment a user sees the response start.
• Time to Last Token (TTLT): total time from request submission to completion. Relevant for tasks where the full response is needed before action can be taken.

Throughput is measured in tokens per second (across the system as a whole) or requests per second (concurrent requests handled). It reflects the system's capacity under load, not the speed of any individual request.

These two dimensions are related but distinct. A system optimised for low individual latency may batch fewer requests together; a system optimised for throughput may accept higher individual latency to process more requests simultaneously.

Why it matters for business

Latency thresholds vary by use case, and design decisions must match those thresholds. A conversational AI assistant that takes eight seconds to return the first word of a response will feel broken to most users — interactive applications typically require TTFT below two seconds. A batch document processing workflow that runs overnight has entirely different tolerance for latency.

For internal productivity tools, high latency degrades adoption. If the AI response is slower than reaching for the keyboard, users revert to manual processes. For customer-facing applications, latency is directly correlated with customer experience and, in high-traffic scenarios, with infrastructure cost.

Throughput matters when the system must serve multiple users or process large volumes simultaneously. An AI system with excellent individual latency but low throughput will degrade under concurrent load — a common failure mode when enterprise pilots scale to full teams without re-architecture.

How it works technically

Several factors govern inference latency:

Model size: Larger models (measured in parameters) require more computation per token. A 70-billion-parameter model will produce tokens more slowly on equivalent hardware than a 7-billion-parameter model. Frontier API models (GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro) are large but hosted on specialised, well-optimised infrastructure; smaller open-weight models self-hosted on enterprise GPUs may be faster for specific tasks.

Input context length: Processing the input (the "prefill" stage) scales with the number of input tokens. Longer prompts and larger context windows increase time to first token.

Output length: Each output token requires a separate forward pass. A response of 500 tokens takes roughly five times as long to generate as a response of 100 tokens, all else being equal.

Infrastructure factors: GPU memory bandwidth, quantisation (reducing model precision from 32-bit to 8-bit or 4-bit to fit larger models on less memory), batching (grouping multiple requests to share computation), and network distance between the application and the model host all affect observed latency.

KV cache: During generation, the model stores intermediate key-value pairs from the attention mechanism to avoid recomputing them for previous tokens. Efficient KV cache management reduces per-token generation time significantly.

Common techniques for improving throughput include continuous batching (dynamically grouping requests as they arrive rather than in fixed batches), speculative decoding (using a small fast model to propose tokens that the large model then verifies), and caching frequently used prompt prefixes.

Practical implementation considerations

Latency and throughput requirements should be specified before model selection, not after. A model that performs well on quality benchmarks but cannot meet response time requirements for the deployment context is not fit for purpose regardless of its capability.

Practical configuration decisions for enterprise deployments include:

• Streaming vs batch: For interactive applications, streaming (sending tokens as they are generated) dramatically improves perceived responsiveness even when TTLT is unchanged. Enable streaming wherever users are waiting for responses in real time.
• Output length limits: Setting maximum output token limits reduces both latency and cost for tasks where concise responses are appropriate. Many enterprise applications do not need 2,000-word responses.
• Caching: Deterministic or near-deterministic queries (e.g. frequently asked questions about a product) can be cached at the application layer, returning stored responses without invoking the model at all.
• Model tiering: Not every task requires a frontier model. Routing simple classification or extraction tasks to smaller, faster models while reserving large models for complex reasoning substantially improves system-level throughput and reduces cost.

When planning infrastructure for enterprise scale, organisations working with Edison AI's AI implementation team typically discover that latency and throughput requirements — specified early, tested under realistic load — drive architecture decisions more consequentially than model quality benchmarks alone.

Common mistakes

• Benchmarking latency on a quiet test environment and expecting the same in production. Under concurrent load, latency increases materially. Load test with realistic concurrency levels before go-live.
• Not specifying TTFT vs TTLT requirements separately. Interactive and batch use cases have fundamentally different latency needs. Conflating them leads to over-engineering for one and under-engineering for the other.
• Ignoring output token counts in cost and latency modelling. Organisations often budget for inference based on input tokens alone. Output tokens also have cost and latency implications and should be modelled from the start.
• Scaling a pilot system to production without re-architecture. A system built for 10 concurrent users may not serve 500 without changes to batching, caching and infrastructure. Plan for scale from the architectural design phase.
• Choosing the largest available model by default. Larger models are not always better for every task, and they are always slower and more expensive. Match model size to task complexity.

What leaders should do next

For each AI application in your portfolio, document the latency requirements (TTFT for interactive, TTLT for batch), expected concurrent users and daily volume. Run load tests against these requirements before production launch. If existing deployments have not been tested under realistic load conditions, prioritise that testing now — latency failures under load are one of the most common causes of enterprise AI adoption setbacks.

Edison AI runs practical AI training that turns this understanding into day-to-day team capability.