Model Risk Management for Machine Learning
Governing AI decisions in production is a discipline of controls and evidence, not good intentions.
Blake Aber · Predicate Ventures
The gap between a model and a controlled model
Most teams can build a model that scores well on a holdout set. Far fewer can prove, months later, that the model running in production is the one they approved, trained on the data they documented, and producing decisions they can defend.
That gap is where model risk lives. In a regulated enterprise, the model itself is the smaller problem. The larger one is the system of controls around it: who approved it, what it was tested against, how it is monitored, and what evidence exists to show all of that happened.
Model risk management for machine learning is the practice of making those controls real. It is not a document you write once. It is a set of mechanisms that produce evidence continuously, whether or not anyone is watching.
What regulators actually ask for
Supervisory guidance on model risk has existed for years. The principles predate the current wave of machine learning, and they translate cleanly.
Three questions sit at the center. Is the model conceptually sound? Is it being used for the purpose it was validated for? And is there independent review of both?
For machine learning, each question gets harder.
Conceptual soundness means you can explain why the model works, not just that it does. A gradient-boosted model with two thousand features and no documented reasoning for any of them fails this test even if its accuracy is excellent.
Use consistency means the production context matches the validation context. A model validated on last year's applicant population and deployed against a shifted population is being used outside its tested envelope, and that is a finding.
Independent review means someone who did not build the model can challenge it. That person needs access to the training data, the feature logic, the test results, and the deployment configuration. If those artifacts are scattered across notebooks and chat threads, independent review cannot happen.
The controls that matter in production
Controls fall into a few categories. Each one should generate an artifact a reviewer can inspect without asking the model owner for help.
Approval and change control
Every model version that reaches production should have a recorded approval tied to a specific artifact: a model hash, a training dataset version, a feature set, and a set of test results. When any of those change, the model is a new version and needs its own approval.
The common failure is silent change. Someone retrains on fresh data and pushes the update without re-approval because "it's the same model." It is not. Change control exists to make that impossible, or at least visible.
Lineage and reproducibility
You should be able to reconstruct any production decision. That means versioning the model, the input features at decision time, the data the model was trained on, and the code that produced the score.
Reproducibility is the control that makes every other control credible. If you cannot rebuild a decision, you cannot investigate a complaint, respond to a regulator, or diagnose a drift event.
Monitoring with defined thresholds
Monitoring is only a control if it has thresholds and consequences. Logging prediction distributions is observability. A rule that says "if population stability exceeds this value, the model is flagged for review" is a control.
Monitor inputs for drift, outputs for distribution shifts, and performance for degradation against outcomes as they arrive. Define what each threshold breach triggers before it happens, not after.
Access and segregation of duties
The person who can modify a model in production should not be the only person who can approve that change. Segregation of duties is standard in financial controls and applies directly here. It is also one of the first things an auditor checks.
Evidence is the product
The hardest shift for engineering teams is accepting that the evidence is a deliverable, equal in weight to the model.
A model that works but cannot be explained, reproduced, or audited is a liability in a regulated context. A slightly less accurate model with a clean evidence trail is often the better choice, because it can be defended.
Evidence should be a byproduct of the pipeline, not a manual effort before an audit. When a model is trained, the pipeline should emit the dataset version, feature definitions, and test metrics. When it is deployed, the deployment system should record what was deployed and who approved it. When it scores a decision, the serving layer should log the inputs and the model version.
If an auditor asks for the full history of a decision and the answer takes a week of manual archaeology, the controls have failed even if nothing went wrong.
Validation is a standing function, not a gate
Teams often treat validation as a one-time checkpoint before launch. In a mature program, validation is continuous.
Initial validation covers conceptual soundness, outcome analysis, and testing against alternatives. Ongoing validation covers whether the model still performs as approved. A model that passed validation two years ago and has not been reassessed is an open risk regardless of its past performance.
Set a review cadence tied to materiality. A model driving high-value decisions earns more frequent scrutiny than one with limited reach. The cadence should be written down and enforced, not left to whoever remembers.
Where this breaks in practice
Three patterns cause most failures.
The first is tooling sprawl. When models are trained in one system, deployed through another, and monitored in a third, lineage breaks at every seam. Consolidating the evidence trail matters more than any single tool choice.
The second is treating governance as paperwork. A control that exists only as a policy document, with no enforcement in the pipeline, is not a control. It is a hope.
The third is the shadow model. Someone builds a scoring routine outside the governed path and it starts influencing decisions. Discovery usually comes during an incident. Inventory discipline, knowing every model that touches a decision, is the only defense.
The practitioner's standard
The test is simple. Pick any decision your system made in production six months ago. Can you name the model version, the features it used, the data it was trained on, who approved it, and what it was tested against, without asking the person who built it?
When the answer is yes, every time, you have model risk management. Everything before that is a plan to build one.