Autoscaling a web service is a solved problem for most teams. Autoscaling a model endpoint is trickier, because the thing you’re scaling is slow to start and expensive to leave idle. A new replica might need to pull a large container image, download weights, load them into GPU memory and warm up kernels before it can serve one useful request. Scaling policies that ignore this produce the classic pattern: traffic spikes, new capacity arrives too late, and users see timeouts.

Know What Happens During a Cold Start

When a replica starts from nothing, the time goes into several stages. Measure each on your own deployment instead of assuming:

1. Scheduling: waiting for a node with a free GPU, which may itself require provisioning a new machine.
2. Image pull: fetching the container image, which can be large if it bundles frameworks and libraries.
3. Weight download: fetching model files from storage.
4. Loading: moving weights into GPU memory and initializing the runtime.
5. Warm-up: first requests that trigger compilation or cache building and run slower.

The sum is your cold-start time, and it is the number that decides which scaling strategies are viable. Record it from a real scale-out event, with timestamps for each stage.

Shrinking the Cold Start

You usually can’t eliminate it, but you can cut it. Practical levers:

• Slim the image. Keep only what serving needs; leave training tools out.
• Cache close to the node. Pre-pull images onto nodes, and keep weights on a fast local or nearby volume rather than fetching from far away each time.
• Use efficient weight formats that load quickly, and check whether your serving runtime supports faster loading paths. Look at its current documentation.
• Run warm-up before readiness. Send representative requests during startup so the replica only joins the pool when it is truly ready.

Each step typically saves a modest amount; together they can matter a great deal.

Choosing the Scaling Signal

What triggers a scale-out matters as much as how fast it happens. Common choices:

• CPU utilization is often a poor signal for GPU-bound inference, because the GPU can be saturated while CPU looks idle.
• GPU utilization is closer, though a busy GPU doesn’t always mean requests are suffering.
• Request concurrency or queue depth tends to track user-visible pain more directly. If requests are waiting, you need more capacity.
• Request rate is simple and predictable but ignores how heavy each request is.

For many teams, concurrency or queue length per replica is the most honest signal. Pair it with a target derived from load testing: find how many concurrent requests one replica handles while still meeting your latency objective, then scale to keep below that.

Scale-to-Zero: When It’s Fine and When It Isn’t

Scale-to-zero means running no replicas when there’s no traffic. It saves money, particularly with expensive GPUs, but the first request after an idle period pays the full cold-start cost.

It suits internal tools, batch-like or asynchronous workloads, demos, and low-traffic endpoints where a slow first response is acceptable. It does not suit user-facing, latency-sensitive paths, or anything where a timeout upstream would turn a slow start into a failure.

A middle path: keep a minimum of one warm replica during working hours and allow zero overnight, using a schedule. Another is to accept the cold start but return a quick “warming up” response or queue the job and notify when done.

Scale Out Fast, Scale In Slowly

Because cold starts are expensive, aggressive scale-in is costly in a different way. If you drop replicas the moment load dips, the next burst forces you to pay the startup cost again. Use a cool-down period before removing capacity, and keep a small buffer above current demand. Scaling out should be eager; scaling in should be patient.

Also set an upper bound. Without a maximum, a runaway client or a retry storm can scale you into a very large bill. Set a cap, alert when you hit it, and decide deliberately whether to raise it.

A Hypothetical Example

A team hosts a classification model for an internal tool used mostly in business hours. Their first setup scales on CPU utilization with a minimum of zero. Each morning, the first users hit a long delay, and some requests time out.

They measure the cold start and find most of the time goes to pulling a heavy image and downloading weights. They slim the image, keep weights on a nearby volume, and add warm-up before readiness. They also switch the scaling signal to concurrency per replica and set a schedule keeping one replica warm from morning to evening, with zero allowed overnight. Morning timeouts disappear, and they still save compute overnight.

A Decision Table

Situation	Reasonable setup
User-facing, latency-sensitive	Minimum of one or more warm replicas; scale on concurrency
Internal tool with predictable hours	Scheduled minimum; zero outside hours
Asynchronous / batch-style inference	Scale-to-zero with a queue
Spiky traffic with long cold start	Higher warm buffer; pre-pulled images; patient scale-in
Cost-sensitive, tolerant of delay	Scale-to-zero; return a warming response

Test Before You Trust

Run a load test that mimics a real spike: ramp traffic quickly and watch how long it takes for new replicas to become ready, what latency users see in the meantime, and whether any requests fail. Also test the quiet case, where an endpoint at zero receives its first request. Both tell you things the configuration page can’t.

Frequently Asked Questions

Is scale-to-zero worth it for GPU endpoints? Often yes for low-traffic endpoints, given how costly idle GPUs are, as long as the first-request delay is acceptable.

Why doesn’t CPU-based autoscaling work well? Inference is often limited by the GPU, so CPU may stay low while requests queue up.

How many warm replicas do I need? Enough to absorb the burst that arrives during one cold-start period. Estimate from your traffic pattern and measured startup time.

Do managed platforms remove cold starts? They may hide or shorten them, but the underlying work still happens. Check each service’s current documentation for specifics.

Should I scale on latency directly? It can work, but latency reacts after users already feel the problem. Queue depth or concurrency is usually earlier.

Next Step

Trigger one deliberate scale-out on a staging endpoint and time each cold-start stage. That breakdown tells you which lever to pull first.