To deploy a machine learning model, put it behind an API your app can call. Package the model with its dependencies, pick the hardware it needs, deploy it to an endpoint, set how it scales when traffic rises or falls, connect your app, then watch latency and errors once real users arrive.
From notebook to endpoint: what deployment means
A model that only runs in your notebook can't be called by an app. Deployment is the work of turning it into a service with a stable URL: something your product sends an input to and gets a prediction back from, at any hour, under whatever traffic turns up. The model itself doesn't change. Everything around it does.
There are two broad ways to get there, and the choice shapes the rest of the job.
Serve it yourself or hand it to a managed platform
- Serve it yourself. Use an open-source framework to wrap the model as an API and build a container you run on Docker or Kubernetes. You keep control of where it runs, and you take on running it.
- Use a managed platform. Hand the model to a service that runs the containers, the GPUs and the scaling for you, and call it through an endpoint. You give up some control and skip the server work.
Neither is better in general. If your team already runs containers, serving it yourself is a short step. If nobody wants to look after servers, a managed platform is the quicker route to a working endpoint. Either way, the steps below apply; what changes is who does the infrastructure part of each one.
Six steps to a model your app can call
- Package the model. Wrap it in a small inference service and list its dependencies, so one command loads the weights and answers a request. BentoML turns an inference script into a REST API server using Python type hints.
- Choose the hardware. Pick the GPU or CPU the model needs, based on its size and the response time you want. Hardware is the main lever on speed and on how much capacity you reserve. On Modal, the GPU choice is written in Python alongside the function itself.
- Deploy to a dedicated endpoint. Put the service behind a stable URL instead of a notebook session. The endpoint is what your product calls in production, so treat its address as something that shouldn't change.
- Set the scaling rules. Decide the minimum number of warm instances, the maximum, and whether it may drop to zero when idle. Those rules decide whether the first request after a quiet spell is fast or slow.
- Connect your app. Call the endpoint with an API token and add your own retry and queue handling, because production traffic arrives in bursts and some endpoints don't queue requests for you.
- Watch it and keep a way back. Track latency, throughput and errors, and keep a route to roll back to the previous version. Problems show up under real traffic, not in a test notebook. Replicate's deployments include canary releases and instant rollbacks for this reason.
Five platforms and frameworks for serving models
One of these is a framework you host yourself; the other four are managed services. Each one covers a different mix of the steps above.
BentoML
An open-source serving framework that turns a model inference script into a REST API server, then builds a reproducible Docker image you can run on Docker or Kubernetes. Adaptive batching and multi-model composition are built in.
Modal
A serverless GPU platform for inference, batch jobs and training. Container environments and GPU choice are written as Python code rather than config files, so the image, the hardware and the function all live in one place.
Baseten
Managed model inference: call hosted models through an OpenAI-compatible API, or deploy your own open-source or custom model on dedicated infrastructure, with regional deployment options. It manages the model containers, GPU capacity, scaling and observability.
Replicate
Its deployments give a model private, dedicated API endpoints with a choice of hardware, autoscaling and request logs, plus real-time latency, throughput and error metrics, which covers the watching step as well as the hosting.
Hugging Face Inference Endpoints
A managed service for deploying a model to production without running the servers yourself. It offers optional automatic scale-to-zero when an endpoint gets no requests, which brings us to the trade-off in the next section.
For a fuller comparison, see my guides to machine learning model deployment platforms and LLM hosting platforms, and if a managed service on this page isn't the right shape for you, there's a list of alternatives to Baseten as well.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
The cold start that scale-to-zero brings with it
Scale-to-zero looks like an easy saving: when nobody's calling the model, nothing runs. It changes what your users see, though. Hugging Face's own documentation for its endpoints says that when an endpoint has scaled to zero, the next request meets a cold start. While the new replica starts, the HTTP server returns a 502 Bad Gateway, and there's currently no queueing system for incoming requests, so the docs advise building your own client-side queue and error handling. The cold start time depends on the size of your model.
That's one vendor's documentation, but the trade-off is general: an endpoint that can sleep has to wake up, and the first person to arrive waits for it. Decide which side of that you're on before you switch it on:
- If a slow or failed first response is acceptable, such as an internal tool or a nightly batch, scale-to-zero is a reasonable choice.
- If users expect an instant answer, keep at least one instance warm with a minimum instance setting.
- Whatever you choose, build retries and a queue into your app, so a 502 during a wake-up is retried rather than shown to a user.
My verdict on a first deployment
Get one endpoint working end to end before you tune anything. Package the model, pick sensible hardware, deploy it, and connect a test app with retries in place from the start. Then decide on scaling with the cold start in mind, not just the idle savings, and make sure you can roll back before real users arrive. A plain, monitored endpoint that recovers from errors is worth more than a clever one nobody's watching.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What does it mean to deploy a machine learning model?
It means hosting the trained model behind an API or endpoint so other software can send it inputs and get predictions back, rather than running it by hand in a notebook.
Do I need Kubernetes to deploy a model?
No. Managed platforms such as Modal host the model for you. If you'd rather run it yourself, BentoML builds Docker images that run on Docker or Kubernetes.
What's scale-to-zero?
It shuts down an idle endpoint so it uses no instances. Hugging Face's docs warn that the next request then meets a cold start and a 502 response while a replica starts.
How do I keep a model warm?
Set a minimum number of instances above zero. Replicate's deployment docs describe always-on instances and a minimum instance setting for this purpose.
Can I deploy a model without a GPU?
A CPU-only service is possible for smaller models. Check each platform's hardware list against the model you have before you choose.
