How CMS audits its AI prior authorization contractors

About 120 case files per company for each audited service each quarter, from services CMS picks without telling the contractor in advance, feed the accuracy score of the companies deciding prior authorization requests in CMS’s WISeR Model. The design is set out in CMS’s WISeR Data Reporting Guide of March 16, 2026, which names one other quality measure, the timeliness of determinations, and was made public in September through a Freedom of Information Act lawsuit.

This week’s story: the audit CMS wrote for its prior authorization contractors

WISeR covers Original Medicare only, not Medicare Advantage. It has run AI-assisted prior authorization in six states since January 2026. Page numbers below are those of the 443-page Combined Records file the Electronic Frontier Foundation posted from that lawsuit, EFF v. CMS. The March guide has both measures scored “pay-for-reporting” in Performance Period 1, the first quarter of 2026 (pp.98, 115). It says CMS “anticipates” scoring on results later in the first performance year (p.98).

CMS checks sampled determinations against Original Medicare coverage criteria, grouped by service, review method and outcome (p.99). The March guide has CMS ask for 30 files per group at the start and review the other 22 if any of the first eight fails (p.100). Per the guide, this “generally aligns with” the “8 and 30” procedure of the National Committee for Quality Assurance (p.100).

Each group is rated separately, so approvals are graded as well as non-affirmations, meaning requests found not to meet those criteria (p.100). Every non-affirmation must be reviewed by a licensed WISeR clinician or clinical reviewer before it is issued (p.80; Participant Guide §1.7.2, p.57). The February 2026 Participant Guide also says contractors “should use enhanced technology and WISeR Clinicians and Clinical Reviewers” to process each qualified request (§1.6, p.56).

Our reading, not a link either body draws: written comment closed on the NAIC supplement asking how insurers use and govern AI, while CMS’s records show contractors graded on their decisions. For scoring and payment details, see our September 25 report.

What to do this week

• How often does your own quality check pull decisions that an AI-assisted review helped make, and does the sample take in approvals as well as denials? CMS’s WISeR design, set out above, is one written reference to hold yours against, down to how many files it reads before widening the sample. It applies only to WISeR contractors; for a health plan, a utilization management vendor or a TPA it is a worked comparison, and our entry on utilization management covers where these reviews sit.

• A utilization management vendor or TPA that issues determinations for you should be able to hand over the full file on any one you pick, without being told in advance which it will be; your agreement is where to confirm that. WISeR contractors are “required to maintain documentation” of their determinations and provide it to CMS on request, and the March guide adds that “Failure to do so will negatively impact the WISeR-2 score” (pp.99 to 100). For a vendor whose AI supports adverse decisions, our guide to assessing AI vendor risk calls for human review of a sample of its decisions.

• Two numbers belong side by side for the last quarter: the accuracy your sampling reported for AI-assisted determinations, and the share of AI-assisted denials later reversed on appeal or resubmission. A high accuracy figure beside a high reversal rate is a reason to look again at how the sample is drawn.

On the Docket

• October 8, 2026: The NAIC’s big data and AI working group holds a one-hour public Webex at 10:00 AM CT to “continue public discussions on the AI Risk Evaluation Supplement,” its first session since written comments on version 5.0 came due, with no agenda or materials posted as of September 29. Insurers that filed a written comment and will listen for it, and those that missed the window and are following the draft from here. NAIC working group page

• October 9, 2026: Comments are due on the Arizona draft AI bulletin in the brief below, a date that rests on McDermott Will & Schulte’s September 28 alert rather than on any notice we could read. Insurers doing business in Arizona, a state our state tracker still files under “Nothing issued.” McDermott client alert

• October 26, 2026: The Colorado Attorney General’s hearing on rules for the Automated Decision-Making Technology and Chatbot Safety Acts opens at 10:00 AM in Denver and by video, and written comments are due by 11:59 PM MT that day, or on the hearing’s last day if it continues. Insurers weighing whether to testify as well as write, since the notice has Department of Law staff present any proposed revisions at the hearing before testimony begins. Notice of hearing

This week in brief

Governance

The NAIC took written comments on its AI Risk Evaluation Supplement, version 5.0, until September 29, and the July 22 panel minutes, posted September 22, close the discussion of generalized linear models (GLMs) with no motion or vote: Nevada regulator Gennady Stolyarov said GLMs “do not fit the definition of AI,” and the Casualty Actuarial Society’s Henry Liu replied that they “can also be made more unpredictable depending on the data used for training.” Our September 23 story quotes each panelist. July 22 minutes

Colorado’s Attorney General has reworded its rulemaking page on when a revised draft of the rules implementing SB 26-189 and the Chatbot Safety Act will be circulated, from “no later than September 23, 2026” to “at least five days prior to the public hearing,” which opens October 26; no revised draft was posted there as of September 29. The August 11 hearing notice set the September 23 date under the condition “If there are changes made to the proposed rules prior to the hearing.” Rulemaking page

Arizona’s insurance department, according to a September 28 client alert from the law firm McDermott Will & Schulte, issued a notice earlier in September inviting comment by October 9 on a draft AI bulletin, “most notably proposing to require a human to approve any consequential decision made by AI”; the Department of Insurance and Financial Institutions had posted no such draft on its bulletins page as of September 29. McDermott client alert

From the Guides

From framework diagram to file. Our new guide maps a NIST AI RMF or ISO/IEC 42001 framework onto an insurer’s AIS Program, the model bulletin’s Section 4 request list and Exhibits A to D of the NAIC’s draft evaluation tool, down to a worked example for one claims model: What an AI Governance Framework Actually Looks Like for an Insurer. On our reading, it covers the insurer-side record that the NAIC supplement in this week’s story asks about.

If you run quality sampling on AI-assisted prior authorization decisions, tell us in a reply to the email this issue arrived in how many files a round starts with and whether it adds files when one fails.

– The Editor, InsureAI Wire