Once a model works in a notebook, the next question is deceptively simple: where does it run? The main options fall into three groups, each with a distinct personality. Choosing among them depends far more on your traffic and your team than on any single feature.

The Three Hosting Models

Serverless hosting. You deploy a model and the platform allocates compute when requests arrive, often scaling down to zero when idle. You pay in proportion to use and manage little infrastructure.

Dedicated managed endpoints. You reserve specific compute for your model through a managed service. Capacity is always warm, performance is more predictable, and you configure instance types and scaling rules.

Self-managed serving. You run an inference server on your own clusters, typically with containers and an orchestrator. You gain maximum control and take on the operational responsibility that comes with it.

None of these is inherently superior. They sit on a trade-off between convenience, predictability, and control.

Traffic Shape Is the Deciding Factor

Think about how requests actually arrive.

• Sporadic or unpredictable traffic, with long quiet periods, suits serverless, since you avoid paying for idle capacity.
• Steady, high-volume traffic tends to suit dedicated or self-managed hosting, where keeping capacity well utilized is more economical than per-request allocation.
• Spiky traffic with strict latency needs is the hardest case. You may need warm capacity as a floor with autoscaling above it.

Plot a week of expected requests, even as a rough estimate, before choosing. Decisions made from a hunch about “scale” often turn out wrong in both directions.

Cold Starts and Latency

Serverless platforms may need to load your model into memory when a request arrives after an idle period. This delay, commonly called a cold start, can be substantial for large models because weights must be fetched and loaded. For internal tools or batch-like uses it may not matter. For user-facing interactions it can break the experience.

Mitigations exist, such as keeping a minimum number of warm instances, using smaller or optimized models, or caching weights close to compute, but each one reduces the cost advantage that made serverless attractive. Check the provider’s current documentation for how it handles scaling to zero, concurrency, and model loading, because behavior and limits differ and change.

Control, Customization, and Constraints

Managed offerings often restrict what you can customize. Model size limits, supported frameworks, request timeouts, payload sizes, and available accelerator types can all constrain you. If you need a custom runtime, specialized batching, or unusual hardware, you may outgrow serverless quickly. Conversely, if you only need to serve a standard model behind an API, the constraints may never bite.

Self-managed hosting gives you the ability to tune batching, caching, and parallelism, and to run exactly the software you want, but you must operate scaling, upgrades, monitoring, and incident response.

Comparison at a Glance

Dimension	Serverless	Dedicated managed endpoint	Self-managed serving
Best traffic fit	Sporadic, unpredictable	Steady, predictable	Steady or large, specialized
Cold starts	Possible after idle periods	Generally avoided	Avoided if capacity stays warm
Operational load	Lowest	Moderate	Highest
Customization	Most limited	Moderate	Greatest
Cost behavior	Tracks usage	Tracks reserved capacity	Tracks capacity plus staff effort
Failure modes	Platform limits, throttling	Capacity sizing errors	Anything you misconfigure

Do Not Ignore the Staffing Cost

Self-managed hosting can look economical on a spreadsheet that counts only compute. It rarely stays that way once you add the engineering time for deployments, autoscaling, security patching, observability, and on-call. If nobody on the team wants to own a serving platform, a managed option may be cheaper in total even when its unit cost is higher. Be honest about who gets paged.

A Hypothetical Example

A small team builds an internal assistant that summarizes support tickets. Usage is concentrated in working hours, with almost none overnight, and a few seconds of extra latency on the first request of the morning is acceptable. They start with serverless hosting, which costs little when quiet and requires almost no operations.

Six months later, the feature is embedded in a customer-facing workflow with steady daytime traffic and a tight latency expectation. The cold-start behavior now hurts, and keeping warm instances erodes the savings. They migrate to a dedicated managed endpoint with autoscaling and a minimum capacity floor. Because their model was already packaged as a standard container behind a clear API, the move takes days rather than weeks. Their original choice was right for its time, and the architecture allowed them to change.

Design for Migration Between Tiers

You are unlikely to stay on one hosting model forever. Keep the model artifact and inference code in a standard container, expose a stable API contract in front of it, and avoid scattering provider-specific calls through your application. Then moving from serverless to dedicated, or from managed to self-managed, becomes a deployment change rather than a rewrite.

Questions to Ask Before You Commit

• What is the worst acceptable latency for the first request after idle?
• How large is the model, and how long does it take to load?
• What limits does the platform impose on payload size, timeouts, and concurrency?
• What happens during traffic surges: queueing, throttling, or errors?
• Who handles incidents, and what are they equipped to debug?
• Where does the data being sent to the model go, and does that meet your compliance obligations? Requirements vary by jurisdiction, so consult qualified advisors when regulated data is involved.

Frequently Asked Questions

Is serverless always cheaper? Not always. It tends to be economical for sporadic traffic, but steady high-volume workloads can be cheaper on well-utilized dedicated capacity. Model your actual request pattern.

Can large models run on serverless platforms? It depends on the provider’s current limits for model size, memory, and accelerators. Check the vendor’s documentation, and consider cold-start behavior if the model is large.

How do we reduce cold starts? Options include minimum warm instances, smaller or quantized models, faster weight loading, and scheduled warm-up. Each carries a cost or complexity trade-off.

When should we self-host? When you need customization or scale that managed options cannot provide, and when you have the people to operate it reliably.

Can we combine approaches? Yes. Some teams use dedicated endpoints for production traffic and serverless for experiments, internal tools, or low-volume models.

Next Step

Sketch your expected request pattern for a typical week, note your latency tolerance for the first request after idle, and match both against the table above. If the answer is unclear, start with the lowest-operations option and keep your model packaged so you can move up later.