When Is an Insurance AI Pilot Ready to Scale?

After several years of experimentation, most large insurers have moved past the question of whether AI has a role in insurance. The question now is whether a pilot is ready to scale. This is a harder question because it is less about the model and more about the organization around it. A technically impressive pilot can collapse in production if the workflow, the oversight, and the documentation are not ready.

The answer, according to industry leaders, is to stop talking about the model and start talking about the business outcome. James Thom, chief product officer at Vertafore, put it directly on a panel at Carrier Management’s InsurTech Summit in May 2026¹: if an insurer is still discussing AI in terms of technology, models, or concepts, it is a long way from scaling. If it is discussing outcomes, process changes, and impact, it is getting close ¹. That shift sounds simple, but it is the difference between a pilot that survives a demo and a system that survives a regulator.

Why technical success does not guarantee scale

A pilot is designed to prove that an AI system can solve a specific problem under controlled conditions. It usually has clean data, a narrow scope, and a team that knows the model intimately. Production is different. The data changes, the users change, and the model encounters edge cases that did not appear in training. Writing in the SOA Research Institute’s March 2026 AI Bulletin, Marty Arnold, chief underwriting officer at Amerisure, describes AI as moving rapidly from pilot programs into core insurance operations, with regulators and industry bodies noting its expanding role in underwriting, pricing, claims, and risk selection ². That is one practitioner’s read on where the industry is; the bulletin prints a standing disclaimer that its contributors speak only for themselves ².

The risk is not just model failure. A model that works in a pilot but fails quietly in production can make bad decisions at scale before anyone notices. A fraud score that flags too many legitimate claims, a prior authorization tool that delays care, or an underwriting model that rejects protected classes can each create regulatory and legal exposure. The harder question comes after the model reaches production: can the organization detect and respond when it runs wrong?

The five dimensions of readiness

Scaling an AI pilot should be a structured decision. The five dimensions below are our working framework and carry no NAIC or state-standard status. No regulator publishes this readiness test, and none of these dimensions is a citable gate. Two are anchored in published expectations, and we say which. The rest reflects our judgment about what tends to break.

1. Clear business outcome and success metrics

A pilot is ready to scale when it has a measurable outcome that the business cares about, not just a model accuracy metric. The success metric should be tied to a real workflow: reduced claims cycle time, fewer false positives, faster underwriting turnaround, or improved producer productivity. The metric should be compared against a baseline, and the comparison should be documented.

The team should also know the failure criteria. If the model does not hit the target, what happens? If the cost of false positives exceeds the savings, what is the rollback plan? Expansion without these guardrails turns a measured pilot into a gamble.

2. Integration into core workflow

AI that is bolted on top of an existing process usually fails. Users have to switch systems, re-enter data, or ignore the AI recommendation. The result is low adoption and low trust. Vertafore’s Thom emphasized that AI must be the core of the process, not an add-on ¹. That means the AI output should appear at the moment the user makes a decision, with a clear path to accept, reject, or escalate.

For example, an underwriting recommendation should appear inside the underwriting workbench, not in a separate dashboard. A claims triage score should be part of the claims adjuster’s existing screen. The less friction, the more likely the AI is to be used correctly.

3. Meaningful human oversight

The NAIC Model Bulletin puts accountability squarely on people: the AI Systems program should vest responsibility for it with senior management answerable to the board ³. Human involvement in the decision itself it handles differently. That is one of five factors an insurer weighs to decide how tight the controls around a given system need to be, alongside the nature of the decision, the potential harm, the explainability of the outcome, and how much of the system came from a third party ³. The bulletin sets no floor on how much of a decision a person has to make ³. Writing in the SOA’s AI Bulletin, actuary Dave Ingram pushes further, and treats “human in the loop” as the comfortable answer worth distrusting. The copilot metaphor “suggests that as long as an Actuary remains ‘in the loop,’ control is preserved.” That framing, he writes, “quietly misleads”: the model suggests, the human reacts, and judgment migrates downstream ².

Meaningful oversight means more than a person who clicks approve. It means the reviewer understands what the model is doing, has the authority to override it, and is held accountable for the outcome. The system should record when the model was overridden and why. That data becomes evidence that the oversight is real and not a rubber stamp. How regulators expect this to work program-wide is covered in the AI governance framework for insurance.

4. Regulatory and documentation readiness

A pilot that cannot be explained to a regulator is not ready to scale. The documentation should include the model’s purpose, the data used, the training and validation approach, the performance metrics, the known limitations, and the human oversight process. The questions a regulator is likeliest to arrive with are laid out exhibit by exhibit in the NAIC AI Systems Evaluation Tool: optional supplemental exhibits written for regulators, which twelve states are piloting from March to September 2026, each free to modify the questions for its own purposes ⁴⁵. Three of its habits still bear on a scale decision. Exhibit A counts by operational area, and one column asks how many of your models touch a consumer directly ⁴. Exhibit B asks how governance runs, under a preface saying the questions create no new requirements ⁴. Exhibit C works from whichever systems the company itself has labeled high risk ⁴. The tool supplies no risk-tier scheme, so that label is a decision you take before any regulator sees it. Settle the counts, the governance answer, and that list before the scale meeting.

The documentation should also map to the insurer’s existing governance program. For a structured approach to inventorying AI systems before a regulator asks, see the AI inventory by line of business framework. Rather than building a separate fraud-AI or underwriting-AI binder, fold each AI system into a single, defensible program.

5. Infrastructure and vendor management

Scaling requires a production environment that can handle volume, latency, and drift. It also requires a plan for updating the model as data changes. Many insurers license AI models from third-party vendors, which means the carrier must understand how the vendor tests, updates, and monitors the model. The vendor may own the code, but the carrier owns the regulatory responsibility.

William Steenbergen, chief technology officer at Federato, noted that insurers must define whether AI is allowed to make decisions or is only a tool for human review ¹. That decision has infrastructure implications. If the AI is allowed to make decisions, the system needs stronger audit trails, fallback procedures, and real-time monitoring. If it is a tool, the interface must support efficient review and override. The vendor oversight questions are covered in more detail in the AI vendor risk assessment for insurers checklist.

Common traps that kill pilots at scale

Several patterns repeat when AI pilots fail to scale.

The science project. The pilot was built by a small team using bespoke data pipelines. There is no plan to operationalize the data flow, retrain the model, or support users. The model dies when the team moves on.

The metric mismatch. The pilot optimized for accuracy, but the business cares about cost, speed, or customer satisfaction. Put illustrative numbers on it: a model that is 95% accurate can still be unusable if it creates 1,000 false positives a day ⁶. Both figures are ours, chosen to make the arithmetic visible, and the pair that matters is your own error rate against the headcount that has to work the queue.

The oversight illusion. The workflow includes a human approval step, but the human has no time, no training, and no incentive to challenge the model. The model effectively decides without accountability.

The regulatory surprise. The pilot was built in isolation from compliance. When the regulator asks for documentation, the team has model cards but no governance records, no consumer impact analysis, and no testing for unfair discrimination.

When to stop or scale back

Scaling should also have a reverse gear. Not every pilot deserves to move forward. A pilot should be paused or scaled back if the live error rate exceeds the threshold, if the business metric is not improving, if the cost of oversight exceeds the savings, or if users are bypassing the system. The decision should be documented just as carefully as the decision to scale. A carrier that only documents successes will have a hard time explaining a failure to a board or regulator.

The same discipline applies to systems that are already in production. A model that was ready to scale six months ago may not be ready today if the data distribution has shifted, if the vendor has changed the model, or if new laws have altered the consumer impact. Scaling works less like a one-time gate and more like a continuous readiness posture.

Who owns the scale decision

The final readiness question is organizational. Someone has to sign off that the pilot is ready for production, and that person must have both the authority and the accountability. In practice, the decision should not sit with the data science team alone, because the risks are not technical. It should be a joint sign-off among the business owner, the compliance or legal function, and the risk or actuarial team.

The business owner owns the outcome metric. Compliance owns the regulatory and consumer-impact documentation. Risk or actuarial owns the model soundness and testing. IT or security owns the infrastructure and vendor diligence. If any of these four cannot sign off, the pilot is not ready. That structure is what turns the readiness checklist from a paper exercise into a real gate.

A practical readiness checklist

Use the following checklist to evaluate a pilot before scaling. Like the five dimensions, it is ours rather than any regulator’s, and the thresholds inside it are yours to set and defend.

• Do you have a documented business outcome, baseline, and success threshold?
• Is the AI output embedded in the user’s primary workflow, with minimal friction?
• Is there a trained human reviewer who can override the AI and is accountable for the outcome?
• Have you documented the model’s purpose, data, limitations, and testing results?
• Have you tested for proxy discrimination or unfair outcomes across protected classes and geographies?
• Do you have a monitoring plan for drift, degradation, and unexpected behavior?
• Is there a rollback or fallback procedure if the model fails?
• Does the vendor contract include audit rights, change notification, and performance SLAs?
• Have you mapped the system to your AI governance program and inventory?
• Is there a consumer notice and recourse plan if the AI affects a decision?

The checklist does not replace judgment; it makes the judgment visible and defensible.

Why readiness is organizational

Insurance AI pilots do not fail because the technology is bad. They fail because the organization around them is not ready. Scaling is a decision about business outcomes, workflow integration, human oversight, regulatory documentation, and infrastructure. Where AI is already deployed across the industry is mapped in the AI use cases in insurance by business line hub, and the governance program a scaled system has to enter is the subject of the framework piece linked above. A pilot that scales without them does not stay a data science problem for long: the first adverse decision it produces at volume becomes a complaint file, and the complaint file becomes the exhibit somebody reads back to you.

What none of this can tell you is where your own line sits. The readiness bar is not published anywhere, and it moves with the line of business, the state, and how much of the decision the model is making. Before expanding authority, complete one real transaction with the decision evidence pack. If another qualified person cannot reconstruct the source facts, model version, human action, notice, and outcome, the pilot is not ready to become a larger evidence problem.

Footnotes

1. Insurance Journal, “How Insurers Know When It’s Time to Scale AI,” June 24, 2026, reporting a panel discussion at Carrier Management’s May InsurTech Summit (the article names the month, not the year; the year here follows from the June 24, 2026 publication date): https://www.insurancejournal.com/news/national/2026/06/24/874999.htm ↩ ↩² ↩³ ↩⁴

2. SOA Research Institute, Actuarial Intelligence Bulletin, March 2026. The pilot-to-production passage closes Marty Arnold’s contribution, p.12 (“Marty Arnold, FCAS, CERA is Chief Underwriting Officer at Amerisure Insurance”); the Ingram quotations are from “The Pilot and the Plane: Actuary-in-the-Model,” p.29. The cover carries the publication’s standing caveat: “The opinions expressed and conclusions reached by the authors are their own and do not represent any official position or opinion of the Society of Actuaries Research Institute”: https://www.soa.org/globalassets/assets/files/resources/research-report/2026/2026-03-ait170-ai-bulletin.pdf ↩ ↩² ↩³

3. NAIC, “Use of Artificial Intelligence Systems by Insurers,” Model Bulletin adopted December 4, 2023 (§1.3 on vesting responsibility with senior management accountable to the board; the Section 3 preamble on the five factors, of which human involvement in the final decision-making process is the third): https://content.naic.org/sites/default/files/inline-files/2023-12-4%20Model%20Bulletin_Adopted_0.pdf ↩ ↩² ↩³

4. NAIC, “AI Systems Evaluation Tool 4.0,” titled “Artificial Intelligence Systems Evaluation: Optional Supplemental Exhibits for State Regulators” (Exhibit A, “Quantify Regulated Entity’s Use of AI Systems,” four count columns by operational area; Exhibit B preface: the questions “are not intended to be interpreted as creating new requirements for AI Systems Governance Risk Assessments”; Exhibit C relies on “the company’s assessment of which AI System(s) are ‘high risk’”): https://content.naic.org/sites/default/files/inline-files/AI%20Systems%20Evaluation%20Tool%204.0%20%28Clean%29.pdf ↩ ↩² ↩³ ↩⁴

5. NAIC, “AI Systems Evaluation Tool Pilot: Pilot Project Background” (twelve participating states: California, Colorado, Connecticut, Florida, Iowa, Louisiana, Maryland, Pennsylvania, Rhode Island, Vermont, Virginia, Wisconsin; “States will use the Tool from March 2026 to September 2026”; “each jurisdiction has the authority to modify the Tool in the pilot to meet its needs”): https://content.naic.org/sites/default/files/call_materials/Pilot%20Project%20Summary.pdf ↩

6. InsureAI Wire illustrative scenario. The 95% and 1,000 figures explain why aggregate accuracy can conceal an operationally unmanageable error queue; they are not measurements from an insurer or external study. ↩