# AI Model Monitoring After the Model Goes Live
> A playbook for insurers on AI model monitoring, validation, drift detection, and retesting records that satisfy NAIC Model Bulletin and Exhibit C expectations.
- Source: https://insureaiwire.com/ai-model-monitoring-insurance/
- Publication: InsureAI Wire
- Author: Simon Li
- Updated: 2026-07-31
---
NAIC Model Bulletin §3.4 and Exhibit C of the [NAIC AI Systems Evaluation Tool](/naic-ai-evaluation-tool/) both look past a model's launch date and ask what the insurer did to keep it fit for use [^1][^2]. That is the part most carriers cannot document. A validation memo from the day the model went live is easy to produce; the records for everything after that day usually are not. This playbook is for the teams that need to build those records.
The goal is not to collect every possible metric. It is to create a documented monitoring system that an examiner can trace: what was tested before deployment, what is watched in production, when a change triggers a retest, and who decided what to do next. The form of the records matters less than the fact that they exist, are dated, and connect to the model's risk tier. That tier comes from the governance program these records sit inside, mapped in our [AI governance in insurance](/ai-governance-in-insurance/) guide.
## What the regulator wants to see
The NAIC Model Bulletin's risk management section says the AIS Program's risk management and internal controls should address validating, testing, and retesting as necessary to assess the generalization of AI System outputs upon implementation. Validation, the bulletin adds, can take the form of comparing model performance on unseen data available at development time to the performance observed post-implementation, measuring performance against expert review, or other methods [^1].
[Exhibit C](/glossary/exhibit-c/), the high-risk model questionnaire, is more specific. For each model it asks for the model name and version, implementation date, risk classification, AI type, and limitations. Then it asks the insurer to discuss testing of model outputs, including [model drift](/glossary/model-drift/), accuracy, unfair trade practices, [unfair discrimination](/glossary/unfair-discrimination/), and performance degradation, and to explain how the model was validated before deployment and how performance is monitored on an ongoing basis. It also asks for the last date of model testing [^2].
The message is that validation continues throughout the system lifecycle. The examiner is looking for evidence that the model was checked, then watched, then checked again. Exhibit C is also not the only exhibit a regulator may reach for. Governance sits in Exhibit B, worked through in our [AI governance framework implementation guide](/ai-governance-framework-implementation/), and the data behind each model in Exhibit D, in the [NAIC AI Exhibit D checklist](/naic-ai-exhibit-d-checklist/). Each of them branches off the [Exhibit A](/glossary/exhibit-a/) counts rather than stacking on the exhibit before it.
## The three parts of a monitoring record
A regulator-ready monitoring record can be organized into three parts. Each part should be dated, named, and tied to a specific model version.
**1. Pre-deployment validation package.** This is the evidence that the model was fit to launch. It should include the model's intended goals and objectives, the data used for development and validation, the validation methodology, the metrics achieved, and the limitations identified. If the model uses third-party data, the package should include due diligence on that data and how it was tested for bias and suitability. The bulletin calls this out in §3.7 and §4: the insurer should be able to produce a narrative description of the model's intended goals and how it was validated [^1].
**2. Production monitoring log.** This is the running record of what the model did after it went live. It should include the metrics being tracked, the frequency of review, the thresholds that trigger escalation, and the actual values observed. The log does not need to be daily for every model, but it must be proportionate to the model's risk tier. A high-risk underwriting model should be watched more closely than a low-risk internal queue-ranking model.
**3. Retesting trigger trail.** This is the record of what happened when a threshold was breached, a vendor updated a feature, or the business environment changed. It should show the trigger, the retesting performed, the results, and the decision. If the model was retrained, the retesting trail should include the new validation package. If the model was retired, the trail should show why.
FIG. 1: THE THREE PARTS OF A MONITORING RECORD
## Define model drift before you claim to detect it
Model drift is one of the most misused terms in AI governance. The NAIC Model Bulletin defines it as "the decay of a model's performance over time arising from underlying changes such as the definitions, distributions, and/or statistical properties between the data used to train the model and the data on which it is deployed" [^1]. That definition is broad enough to cover several distinct problems.
In practice, insurers should document at least three types:
- **Concept drift**: the relationship between inputs and outputs has changed. A model trained during a period of stable loss costs may no longer predict well after a catastrophe or a regulatory change.
- **Data drift**: the distribution of input features has changed. This can happen when a new data source is introduced, a vendor changes its scoring methodology, or the population being underwritten shifts.
- **Performance degradation**: the model's predictive accuracy has declined, even if the input distributions look similar. This is usually measured by tracking the same metrics used in pre-deployment validation against live outcomes.
Each type needs a documented detection rule. The rule should specify the metric, the threshold, the comparison window, and the frequency of review. A rule that says "we monitor for drift" is not sufficient. A rule that says "we compare the distribution of feature X in the last 90 days of production data to the training distribution and flag a shift greater than two standard deviations" is.
## Choose metrics that match the risk
The right metrics depend on what the model does and how much harm it could cause. A model that sets prices needs different monitoring than a model that triages internal work queues. The NAIC Model Bulletin makes this proportionality explicit: controls should reflect the degree and nature of risk posed to consumers, including the degree of potential harm and the extent of human involvement in the final decision [^1].
For high-risk models in underwriting, pricing, claims, or health utilization, the monitoring log should include:
- Performance metrics: accuracy, precision, recall, false positive rate, or other metrics used in validation.
- Fairness metrics: what InsureAI Wire groups as disparate-impact measures, protected-class correlation tests, or proxy-variable screening. The insurance instruments use terms such as bias analysis, unfair discrimination, and disproportionate adverse effect; New York's wording is covered in our [NYDFS Circular Letter No. 7 analysis](/ny-dfs-circular-letter-7/).
- Stability metrics: population stability index, characteristic analysis, or distribution drift tests for key features.
- Outcome metrics: the rate of overrides, the rate of consumer complaints, and the rate of appeals or reversals.
- Business metrics: the model's effect on loss ratios, claim costs, or other material financial outcomes.
For lower-risk models, the log can be lighter, but it should still exist. The examiner's question is not whether every metric was perfect; it is whether the insurer had a defined and executed monitoring plan.
## Set thresholds that drive action
A monitoring metric without a threshold is a dashboard decoration. A threshold without an action is a source of panic. The playbook should define what happens when each threshold is crossed.
The most common threshold failures are thresholds that are too sensitive, producing noise that teams learn to ignore, or thresholds that are too vague, leaving the decision to whoever reads the alert. A useful threshold has four components:
- **Metric**: what is being measured.
- **Baseline**: the value at validation or the expected value under normal conditions.
- **Tolerance**: the range that is considered acceptable, with a rationale for why that range was chosen.
- **Action**: the step to take when the tolerance is exceeded, including who decides and within what timeframe.
For example, a pricing model might use this rule: "If the population stability index for the top ten features exceeds 0.15 in any 30-day window, the model-validation team will review the shift. Within 10 business days, it will recommend retraining, reweighting, or continued monitoring." The threshold, the owner, and the timeframe are all documented.
## Document retesting as a decision, not an event
Retesting should be triggered by something specific: a drift alert, a vendor update, a new data source, a material change in the model's environment, or a scheduled review. Each retest should produce a record that answers the same questions as the original validation.
- What triggered the retest?
- What data was used?
- What methodology was applied?
- What were the results?
- What was the recommendation?
- Who approved the decision?
If the retest led to a model change, the change should be versioned and the new version should have its own validation package. If the retest led to no change, the record should explain why the existing model remains fit for use. The worst answer is silence. The second-worst answer is a change made without a documented reason.
## Watch third-party models the same way
The NAIC Model Bulletin puts AI systems developed by a third party inside the same program the insurer writes for its own [^1]. In monitoring terms, that means vendor models need the same records as internal models, even if the insurer does not own the training data.
Vendor models take that same record and add four lines to it:
- The vendor's monitoring commitments and how the insurer verifies them.
- The contract terms that give the insurer audit rights or access to audit reports, negotiated per our [AI vendor risk assessment checklist](/ai-vendor-risk-assessment/).
- The process for receiving and acting on vendor notices about model updates, drift, or performance issues.
- The insurer's own independent checks, such as backtesting on the insurer's population or comparing vendor outputs to outcomes.
A vendor that will not share monitoring data or cooperate with regulatory inquiries leaves the insurer holding the gap. At the exam, the missing cooperation is the insurer's finding to answer for, not the vendor's [^1].
## Keep the records exam-ready from the start
The examiner does not want to see a perfect model. The examiner wants to see a model that was managed with discipline. The records that support this are usually simpler than teams expect.
For each model, maintain a single file or index that contains:
- The model version and implementation date.
- The risk classification and the rationale for it.
- The validation package from before deployment.
- The monitoring plan, including metrics, thresholds, and frequency.
- The production monitoring log, with dates and values.
- The retesting records, including triggers, results, and decisions.
- Any vendor contracts or audit reports that relate to the model.
This index should be updated when the model changes and reviewed on a regular cadence. Records assembled only after an exam notice arrives carry their assembly date with them; records maintained as part of the model's normal life cycle are evidence of governance in themselves.
## The one-page monitoring plan, filled in
Everything above compresses into a single page per model. Here is the skeleton with example values for a hypothetical homeowners pricing model; the example is illustrative, not a real carrier's plan.
The download has the plan fields with the example column pre-filled and a second tab with the review checklist from the end of this playbook. Prefer plain text? Open the **Ask AI** menu and choose **Copy for AI**.
**Table [row-headers]:** Example monitoring-plan fields, rules, evidence, and escalation owners
| Field | Example entry |
|---|---|
| Model and version | HO pricing model v3.1 |
| Risk tier and rationale | High: sets consumer premiums directly |
| Owner | Chief actuary (model), model-validation lead (monitoring) |
| Review cadence | Monthly metrics review; quarterly validation summary |
| Performance rule | Lift on holdout vs. live outcomes; alert if decay exceeds 10% of validation baseline over 60 days |
| Stability rule | PSI on top ten features, 30-day window; review at 0.15, mandatory retest recommendation at 0.25 |
| Fairness rule | Quarterly proxy screen on rating variables; any new correlation above the documented tolerance goes to compliance same week |
| Outcome rule | Override rate and complaint rate per 1,000 quotes; investigate any 2x move quarter over quarter |
| Escalation path | Validation team reviews within 10 business days; retrain/reweight/monitor decision signed by chief actuary |
| Last test and next review | 2026-06-30 quarterly summary; next due 2026-09-30 |
The specific thresholds are yours to set and defend. What the examiner reads off this page is that each metric has a number, each number has an action, and each action has an owner with a deadline. That is the difference between monitoring and watching.
## A checklist for the next review
Run this checklist before an examiner asks for the records; anything you cannot answer becomes a work item with an owner.
- Is there a documented monitoring plan for every high-risk model?
- Does each plan name the metrics, thresholds, frequency, and owner?
- Are the thresholds tied to specific actions?
- Is there a production log that shows the metrics were actually tracked?
- Are drift alerts documented, even when no action was taken?
- Is the last testing date recorded and within the period defined by the monitoring plan?
- Is there a retesting record for every material change or alert?
- Do third-party models have vendor monitoring records and audit rights?
- Can the entire record for a model be produced in one place?
Where the answers are thin, start with the models that touch pricing, claims, or utilization decisions. Exhibit C names no models itself, because the risk criteria are the company's to set, but those are the ones where a missing log is hardest to explain.
## Monitoring as a regulatory expectation
Monitoring is a regulatory expectation, and both documents behind it are careful about how far they take that. The bulletin states that its goal "is not to prescribe specific practices or to prescribe specific documentation requirements," and that an insurer may show compliance through alternative means [^1]. The evaluation tool bills itself as optional supplemental exhibits for state regulators [^2]. The care goes only so far, though. The same passage in the bulletin says a Department still expects an insurer to produce the documents when it asks, and the bulletin's examination list names documentation related to validation, testing, and auditing, including evaluation of model drift [^1]. Exhibit C reduces all of that to a single line when it asks for the last date of model testing [^2]. A date is either in the log or it is not, and no narrative fills that gap after the fact.
Start with the high-risk models. Write the plan down before the drift alert arrives, and when it arrives, record the decision as well as the number. Third-party models stay inside the program too: the bulletin's expectations follow the system wherever it was built.
Every threshold above is an example. A PSI trigger at 0.15, a quarterly review cadence, a two-week retest window: none of those come from the bulletin or the tool, and defending the numbers you pick is your job rather than the NAIC's. A monitoring record proves you were watching something. Whether you were watching the right number is a separate argument, and that is the one that gets had in the exam room.
[^1]: NAIC Model Bulletin, "Use of Artificial Intelligence Systems by Insurers," adopted December 4, 2023: https://content.naic.org/sites/default/files/inline-files/2023-12-4%20Model%20Bulletin_Adopted_0.pdf
[^2]: NAIC, "AI Systems Evaluation Tool 4.0," 2026: https://content.naic.org/sites/default/files/inline-files/AI%20Systems%20Evaluation%20Tool%204.0%20%28Clean%29.pdf