Model deployment strategies decide how much risk a new model carries on its first day in production. The right choice depends on what a wrong prediction costs, how fast you can detect it, and how much traffic and tooling you have.
This article covers the two decisions you make for every model: how predictions are served (batch or online), and how a new version is released (shadow, canary, blue/green, or A/B). They combine, and most teams use more than one. For what can go wrong after release, see AI model deployment challenges in production.
First decision: batch or online serving
Before any rollout pattern, choose how predictions get to the product.
| Batch | Online (real time) | Embedded or edge | |
|---|---|---|---|
| How it works | A scheduled job scores many records and writes results to a table or store | A service answers each request with a prediction | The model runs in the app or on the device |
| Good when | Nobody waits on the answer: nightly scores, recommendations lists, risk rankings | The input only exists at request time, or the answer must reflect the latest state | Connectivity is poor, latency is tight, or data must stay on the device |
| Freshness | As old as the last run | Current | Current, but updates depend on app releases |
| Operational load | Low: a job, a table, a failure alert | High: availability, latency, scaling, on-call | Distribution and version management across devices |
| Rollback | Rerun with the previous model | Redeploy or shift traffic | Slow; depends on update cycles |
If batch is acceptable, prefer it. It removes latency requirements, makes validation easier (you can inspect the whole output before publishing it), and rollback is a rerun. Many teams build online services for problems that a nightly job would solve. A hybrid is common: batch-compute the expensive parts, and do a light online step at request time.
Second decision: how to release a new version
Recreate or rolling replacement
The baseline: stop the old version, start the new one, or replace instances gradually. This is the default behavior of a Kubernetes Deployment. A rolling update protects availability, but not quality. Every user gets the new model as soon as its instance is ready, so it suits low-risk changes and models whose outputs are easy to verify first.
Shadow deployment
The new model receives a copy of real production requests and produces predictions that are logged but never shown to users. The old model still answers.
- Use it for: validating behavior, latency and resource use on real inputs with zero user impact, especially for a first release or a major architecture change.
- Limits: you can compare predictions between models, but you cannot measure effects on user behavior, because users never saw the new output. If outcomes depend on the prediction (a recommendation that changes what people click), shadow data will not tell you how the model performs. It also doubles inference cost while it runs, and any side effects in the serving path must be switched off.
Canary release
A small share of live traffic goes to the new version while the rest stays on the old one. If monitored metrics stay healthy, the share grows in steps until the new version takes everything. If they do not, traffic returns to the old version.
Tooling exists to automate this. Argo Rollouts supports canary and blue-green strategies for Kubernetes and can integrate with metric providers to promote or roll back automatically, and KServe supports canary traffic splitting between model revisions (check its docs for the current field names).
- Use it for: most online model releases. It limits the blast radius and gives you real user outcomes at low exposure.
- Watch for: a canary needs enough traffic to detect problems in a reasonable time, and it needs metrics you trust. For a service with few requests, the canary stage may take days to reach any confidence. Also decide whether routing is sticky per user; flipping a person between models mid-session can confuse both users and your analysis.
Blue/green deployment
Two full environments run side by side. Blue serves production; green runs the new version and can be tested with production-like traffic or synthetic checks. When ready, you switch all traffic at once, and switching back is the rollback. The Argo Rollouts documentation describes the same idea: both versions are deployed, but only the old one receives production traffic until you cut over.
- Use it for: changes where you want an instant, clean rollback, or where running two versions against the same live traffic would be a problem, such as a change coupled to a schema or an API version.
- Limits: it needs capacity for two environments, and the cutover exposes everyone at once. It validates that the new environment works, not that the model is better.
A/B testing
An A/B test splits users into groups, randomly and persistently, and compares a business or quality metric between models over a planned period. It answers the question that the other patterns do not: is the new model better for the product?
- Use it for: deciding between candidates when the objective is a behavioral outcome such as conversion, retention or resolution rate, and offline metrics do not predict it well.
- Limits: it needs enough users to detect the effect size you care about, a fixed analysis plan, and the discipline not to stop the test when a number looks good. It is an experiment, not a safety mechanism. Run a canary first to make sure the model is not broken, then test for benefit.
Multi-armed bandits
Where a fixed split wastes traffic on a clearly worse arm, an adaptive allocation shifts traffic toward the better-performing model as evidence accumulates. It is more complex to operate and to analyze, so most teams reach for it only after a standard A/B setup is working.
Choosing a pattern
| Situation | Reasonable choice |
|---|---|
| Nightly scoring job | Run the new model on the full data, diff the output against the old one, publish when checks pass |
| First online release of a new model | Shadow, then canary |
| Routine retrain of an existing online model | Automated validation gate, then canary with automatic rollback |
| Need to know if the model improves a business metric | A/B test after a canary confirms stability |
| Change coupled to an API or schema version | Blue/green |
| Low traffic, low stakes | Rolling update with a smoke test and a fast manual rollback |
What every strategy needs underneath
The patterns above are routing. They only work if some things are already in place.
- Versioned artifacts. You must be able to say "route 5 percent to version 14". A tracking server and a model registry provide the identifiers and lineage.
- Metrics per model version. Latency, errors, prediction distribution and, where possible, outcome metrics, all labeled with the version that produced them.
- Rollback criteria written down. Decide before the release what thresholds trigger a rollback. Deciding during an incident is how bad models stay live for hours.
- A fallback. What happens if the model service is down or returns an invalid result: a previous version, a rule, a default.
Delayed labels change the plan. If you only learn whether a prediction was right weeks later, a canary on accuracy cannot work. Use proxy signals (input and prediction distributions, downstream behavior) for the stage gates, and evaluate true quality afterward.
For related reading, what is MLOps puts release management in the wider lifecycle, and for LLM applications the same ideas apply to prompts and retrieval settings; see LLM monitoring and observability. We design and build release pipelines like these as part of our MLOps services, and you can request a quote to discuss yours.