The dashboard says average latency is fine. Users say the product feels slow. Both are true, because averages hide the slowest requests, and those are the ones people remember. Tail latency (the 95th, 99th or slower percentile of response times) is where model endpoints tend to hurt, and it has a handful of well-known causes that you can work through in order.

Look at the Distribution, Not the Mean

Start by plotting a latency histogram or percentile chart over a window long enough to include a few spikes. Look for the shape:

• A smooth long tail usually means queuing or contention: requests wait their turn.
• Distinct clusters (a fast hump and a separate slow hump) usually mean two different code paths, such as cache hit versus miss, or warm versus cold.
• Periodic spikes at regular intervals suggest something scheduled: garbage collection, a health check, a batch job on the same machine, or autoscaling events.
• Spikes correlated with input size point at the payload itself.

Which shape you have narrows the investigation a lot before you open a profiler.

Separate Network, Queue and Compute Time

A single “latency” number hides where the time went. Break each request into stages and record them separately:

1. Time in the network and load balancer before the request reaches your server.
2. Time waiting in the server’s queue.
3. Time doing preprocessing.
4. Time in model execution.
5. Time in postprocessing and serialization of the response.

If your serving framework doesn’t expose these, add timestamps yourself. Without this breakdown, people optimize the model when the delay was in a queue. That’s a common and expensive misdiagnosis.

Cause One: Queuing From Too Little Headroom

An endpoint running near capacity builds queues from random bursts alone, even when average load looks manageable. Latency climbs sharply as utilization approaches the limit. If queue time dominates the slow requests, the fix is capacity or load shaping, not model optimization. Add replicas, lower the per-replica concurrency, or reject excess load early rather than letting it pile up.

Cause Two: Batching Settings That Trade Latency for Throughput

Many serving stacks group requests into batches to use the GPU efficiently. A batching window that waits for more requests before running adds delay to every request in the batch, and a long wait under low traffic hurts the tail. Review the maximum batch size and the maximum wait time. Tune them against your latency objective, not just against throughput.

Cause Three: Cold Paths

Some requests take a slower route. The first request after a deploy may load weights or compile kernels. A scaled-up replica may not be warm yet. A cache miss may trigger an expensive lookup. If your slow cluster aligns with deploys or scaling events, add warm-up requests before a replica receives live traffic, and make the readiness check reflect real readiness.

Cause Four: Variable Input Cost

Not all inputs cost the same. For language models, longer prompts and longer generated outputs take more time. For vision models, image size may matter. If the tail correlates with input size, you have two options: bound the input (truncate, cap, or reject) or route large requests to a separate pool so they don’t hold up small ones.

Cause Five: Noisy Neighbors and Resource Contention

CPU-side work can stall a GPU-bound service. Tokenization, image decoding or JSON handling that competes with the model server for cores can create stalls. On shared infrastructure, other tenants can also affect you. Check CPU saturation, memory pressure and garbage collection pauses alongside GPU metrics.

A Hypothetical Example

A team serves a model behind an API, with typical responses in a comfortable range but a small share of requests far slower. The histogram shows two clusters. Splitting timings by stage shows the slow cluster spends most of its time in preprocessing, and further inspection ties it to unusually large uploaded files.

They add an input size limit, move heavy preprocessing to a separate worker pool, and surface a clear error for oversized inputs. The tail shrinks. A second, smaller spike remains at regular intervals, which they trace to a nightly job sharing the same node. Moving the job off that node removes it. At no point did they need to touch the model.

A Triage Table

Observation	Probable cause	What to try
Slow requests mostly wait in queue	Insufficient headroom	Add replicas; cap concurrency; shed load
Latency rises when traffic is low	Batching wait too long	Lower max wait; tune batch size
Spikes right after deploy or scale-out	Cold start / unwarmed replica	Warm-up requests; stricter readiness checks
Slow requests have large inputs	Variable input cost	Limit input size; separate pool
Regular periodic spikes	Scheduled contention	Find the co-located job or GC cycle
Slow only for some clients	Network path	Check regional routing and payload size

Fix the Objective Too

Decide what you are actually promising. A target such as “99 percent of requests under a stated time” gives the team something to measure against. Set it based on what the product needs, then alert on that percentile, not on the average. Alerting on averages is precisely how tail problems stay invisible.

Frequently Asked Questions

Which percentile should I track? Track at least the median and one high percentile such as p95 or p99. Choose the high one based on traffic volume, since rare percentiles are noisy with few requests.

Can a faster GPU fix tail latency? Only if compute time is the dominant stage. Check the stage breakdown first.

Does autoscaling fix this? It helps with sustained load but can introduce cold-start spikes if new replicas receive traffic before they are warm.

Is client-side retry a good idea? Carefully. Retries can amplify load during an incident, so use backoff and limits.

How long should I collect data before concluding? Long enough to capture at least a few cycles of your normal traffic pattern, including peaks and any scheduled jobs.

Next Step

Add stage-level timing (network, queue, preprocessing, model, postprocessing) to one endpoint this week, then look at which stage owns your slowest one percent of requests.