Most AI model deployment challenges in production are not about the model. They are about everything the model depends on: the data arriving at it, the code that prepares that data, the infrastructure that serves it, and the people responsible when it misbehaves.
This is a field guide to the failures we see teams run into after the first release, ordered roughly by when they tend to appear, with a mitigation for each. It covers classical ML models and, where it differs, LLM applications. The foundation is described in what is MLOps; how to release safely is in model deployment strategies.
1. The model works in the notebook and not in the service
The first challenge is the handoff. A model trained in a notebook depends on whatever state the notebook had: imported libraries at particular versions, columns renamed in an earlier cell, a scaler fit on a slice of data that nobody saved.
Mitigation. Make training a script with explicit inputs and outputs. Save the preprocessing and the model together as one artifact, for example a scikit-learn Pipeline or an equivalent in your framework, so the service cannot apply the model without the exact transformations used in training. Pin dependency versions and build the serving image from the same lockfile used for training.
2. Training-serving skew
Even with a shared artifact, features computed in production can differ from the ones computed in training. A common cause is a feature that is easy to compute over a historical table and awkward in a live request, so someone reimplements it in a different language. Another is time: the training set accidentally includes information that would not have been available at prediction time.
Mitigation. Compute features with one implementation, shared between training and serving, or generate both from one definition. Log the exact feature vector the service used for each prediction, and compare its distribution against the training data. Build training sets with point-in-time correctness: each row may use only data that existed at its timestamp.
3. Data quality breaks upstream
The model service rarely owns its inputs. A source team renames a field, changes a unit from dollars to cents, starts sending nulls, or a batch job runs late and the service reads yesterday's table. The model does not fail. It returns a confident prediction on bad input.
Mitigation. Validate inputs at the service boundary: types, ranges, allowed categories, null rates. Decide in advance what the service does on invalid input (reject, fall back to a default, or route to a rule) and make that visible in metrics. Agree on a data contract with the producing team so a schema change is announced before it ships, not discovered through a drop in conversions.
4. Drift and silent degradation
Production data changes. Customer behavior moves, a new product line appears, a sensor is replaced. Inputs can drift (the distribution of features changes) and so can the relationship between inputs and outcomes (concept drift). Neither raises an exception.
The hard part is that ground truth often arrives late. A churn label may take months. You cannot wait for accuracy to fall before reacting.
Mitigation. Monitor in layers. Track input distributions and prediction distributions immediately, since they need no labels. Track business proxies that arrive sooner. Compute true quality metrics when labels do arrive, and chart them by model version. Libraries such as Evidently (Apache 2.0) provide drift and data quality checks you can run in a scheduled job. Treat drift alerts as prompts to investigate, not as automatic retrain triggers; a shift in inputs does not always mean the model is wrong.
5. Latency, throughput and cost
A model that scores in 20 milliseconds on a laptop can take far longer in a service with serialization, network hops, feature lookups and a cold container. Throughput problems show up when traffic is bursty, and cost problems show up when GPUs sit idle between requests.
Mitigation. Set a latency budget for the whole request and measure each stage against it. Load-test with realistic payloads before launch. Choose the serving pattern to match the need: if nobody is waiting on the answer, batch scoring on a schedule is cheaper and simpler than a low-latency API. Where you do need online serving, look at request batching, model quantization, and autoscaling that accounts for model load time. Measure, since the right fix depends on where the time goes.
6. Packaging, dependencies and environments
Native libraries, CUDA versions, large model files and mismatched Python versions cause a large share of "works on my machine" incidents. Images grow to many gigabytes and deploys become slow.
Mitigation. Build containers in CI from pinned dependencies, keep model weights out of the image where practical (load them from an artifact store by version at startup), and test the built image, not just the source tree. Use the same base image for training and serving where you can.
7. Rollout, rollback and versioning
Replacing a model is a release, and it needs the same safeguards as any release: the ability to see what changed, to limit exposure, and to go back quickly. Teams that overwrite a file in a bucket have none of those.
Mitigation. Register every model version with its lineage (data, code, parameters). Deploy by version identifier, and keep the previous version warm enough to restore in minutes. Use staged rollouts so a bad model reaches a fraction of traffic first. The patterns, and when to use each, are the subject of model deployment strategies.
8. Testing a thing that has no single right answer
Unit tests do not tell you whether a model got worse. A new version can improve the average and break an important segment.
Mitigation. Keep a versioned evaluation set, including hard cases and examples from past incidents. Gate releases on metrics across segments, not just overall, and compare against the current production model on the same data. Add behavioral tests: invariances (changing a name should not change a credit decision) and minimum functionality on known examples.
9. Security, privacy and compliance
Prediction logs contain user data. Training sets may include personal data that someone later asks to delete. Model endpoints are attack surfaces, and for LLMs, prompts and retrieved context add new exposure.
Mitigation. Decide what you log and for how long, and classify it. Make sure deletion requests can reach training data, feature stores and logs. Put authentication and rate limits in front of every endpoint. For LLM applications, read our guides on personal data in prompts and traces and privacy in LLM applications.
10. Ownership
The least technical challenge is often the one that causes the longest outages. Who is paged when the model degrades: the data scientist who trained it, the platform team that runs the cluster, or the product team that owns the feature? When the answer is unclear, nobody acts.
Mitigation. Name an owner per model, write a short runbook (how to check health, how to roll back, who to contact upstream), and review it after each incident. Operational ownership should be part of the definition of done for a model, not an afterthought.
A pre-launch checklist
- Preprocessing and model ship as one versioned artifact.
- Input validation exists, with a defined fallback behavior.
- Feature vectors and predictions are logged, within your retention policy.
- Latency and error metrics are dashboarded with alerts.
- At least one input or prediction drift signal is monitored.
- The previous model version can be restored without a rebuild.
- A named owner and a runbook exist.
If a model serves a decision that affects people, such as credit, hiring or access to a service, add legal and fairness review to this list. That is outside the scope of an engineering checklist.
For the operational side of these problems, see our MLOps services. If you have a model that is stuck between a prototype and production, request a quote and describe where it is stuck.