How to Test ECDIS and AIS for Unfair Discrimination Under NY DFS Circular Letter No. 7

You already know what Circular Letter No. 7 says. This is the part after that. An ECDIS or an AIS is in front of you, and the letter expects it tested for unfair or unlawful discrimination ¹. Someone has to actually run the test, write down what came out, and have the file ready for the day the Department asks for it ¹. That someone is usually a compliance or model-risk function working with the actuaries who built the pricing, and the useful question for them is narrower than “what does New York require.” It is: which test, in what order, owned by whom, leaving what behind on paper.

The circular calls its own expectations “expectations,” not a new statute ¹. That framing matters for how you defend the work, because a DFS inquiry does not ask whether you obeyed a rule the letter invented. It asks whether you can demonstrate that the tools you use in New York are not based on a protected class and are not unfairly or unlawfully discriminatory, using the state and federal anti-discrimination law the circular points to rather than any definition the circular supplies itself ¹. The letter does not define “unfair or unlawful discrimination,” and it does not define “legitimate business necessity.” It defers the first to cited law and leaves the second to your judgment and record. So treat the testing as an evidence-production exercise with no fixed pass mark. The thing you are producing is a defensible answer to a question a regulator has not asked yet.

Separate the two tests before you run either

The most common way this goes wrong on the first pass is treating the circular as describing one test. It describes two, in different subsections, with different scope and different trigger language, and collapsing them produces a file that answers neither cleanly.

Read the scope column first. The proxy assessment in Section II.A never mentions AIS; the circular writes it entirely against ECDIS ¹. The three-step assessment in Section II.B applies to ECDIS or AIS ¹. “Comprehensive assessment” is the circular’s own name for that three-step exercise. It says nothing about how thorough your work should be, so keep the name attached to the thing it names when your file cross-references the letter.

The proxy assessment: ECDIS only

Section II.A sits under “Data Actuarial Validity” and is short. It opens with the one hard clause in the passage: insurers “must be able to demonstrate that the ECDIS employed for underwriting and pricing are not prohibited by the Insurance Law or regulations” ¹. Everything after that is “should.”

What to run. For each external data source feeding underwriting or pricing, evaluate the extent to which it is correlated with, in the circular’s own gloss, a proxy for, status in a protected class in a way that may result in unfair or unlawful discrimination. The circular says this correlation may be determined using data available to you or reasonably inferred using accepted statistical methodologies ¹. It also limits the exercise to protected classes for which data are available or can be reasonably imputed, so you are not expected to manufacture protected-class data you do not have ¹. If correlations turn up, the circular’s instruction is to consider whether a legitimate business necessity requires the use of that ECDIS. It does not define that phrase, and it does not tell you a correlation threshold that flips the answer. Both are yours to set and defend.

Who owns it. The correlation analysis is actuarial or data-science work. The judgment about business necessity, and the record of it, belongs with compliance or legal, because “required by a legitimate business necessity” is a legal-standard question dressed as a statistical one, and the person who signs it is answering to the cited law, not to a p-value.

What to retain. The circular puts no documentation clause inside Section II.A itself. The documentation obligation for the proxy assessment lives in Section II.C, which explicitly covers “paragraphs 12 and 15,” and paragraph 12 is the proxy assessment ¹. So retain: the data sources evaluated, the method used to determine or infer correlation with each protected class considered, the correlations found, and, where a correlated source stays in use, the written business-necessity rationale. A proxy assessment with no retained rationale for the sources you kept is the version that reads badly later.

The three-step assessment and its loop

Section II.B is where the disproportionate-adverse-effect testing lives, and it is the exercise most people mean when they say “the CL7 test.” The industry shorthand for the underlying idea is disparate impact, and that cross-regime concept is worth understanding, but note that “disparate impact” appears zero times in the body of this circular. DFS’s operative terms are disproportionate adverse effect, prima facie showing, and less discriminatory alternative, and your file should use the words the Department uses when it is describing this specific exercise ¹.

The gate into the sequence is the same establish-first clause the proxy assessment has. An insurer should not use ECDIS or AIS in underwriting or pricing unless it has first established, through the three-step assessment, that the resulting guidelines are not unfairly or unlawfully discriminatory ¹. Then three steps.

Step 1. Assess whether the use produces disproportionate adverse effects. Test whether use of the ECDIS or AIS has a disproportionate adverse effect, in underwriting or pricing, on similarly situated insureds or on insureds of a protected class, for any protected class whose membership you can determine from available data or reasonably infer using accepted statistical methodologies ¹. This is quantitative work, owned by actuarial or model-risk with the metrics discussed below. The branch here is the important one: if there is no prima facie showing of a disproportionate adverse effect, the insurer may conclude its evaluation after Step 1 ¹. Many models will legitimately stop here. Retain the metrics run, the classes tested, the results, and the reasoned conclusion that no prima facie effect was shown, because “we concluded after Step 1” is only defensible if the Step 1 record exists to point at.

Step 2. Assess whether there is a legitimate, lawful, and fair explanation for the differential effect. If Step 1 does show a disproportionate adverse effect, Step 2 asks whether there is a legitimate, lawful, and fair explanation or rationale for the differential effect on similarly situated insureds ¹. This is a legal and actuarial judgment, owned jointly, and it is the step where compliance and legal earn their seat. The branch: if there is no such explanation, the circular’s instruction is not to stop using the tool in a terminal sense but to modify the use and re-evaluate the modified use beginning at Step 1 ¹. Retain the articulated rationale, or, where there was none, the record of the modification and the fresh Step 1 run on the modified model.

Step 3. Search for a less discriminatory alternative. If there is a legitimate explanation, Step 3 is “conducting and appropriately documenting” a search and analysis for an alternative variable or methodology that is less discriminatory and would still reasonably meet the insurer’s legitimate business needs ¹. Note that Step 3 is the one step whose text builds documentation into the instruction itself: “appropriately documenting” is in the requirement, so a Step 3 with no written search is incomplete on its face. Two branches leave Step 3. If a less discriminatory alternative exists, modify the model and return to Step 1 ¹. If none exists, the circular sends you to ongoing model risk management under Section III and, critically, tells you to repeat Step 3 at least annually ¹. There is no endpoint that reads “may not be used in New York.” The endpoint is a standing obligation to keep looking.

That loop is the design of the test, so build your evidence to show the loop turning rather than a single pass. The file that survives an inquiry is the one that can show, for a model that stayed in use through Step 3 with no alternative found, a dated annual re-run of the Step 3 search and the model-risk monitoring that ran in between.

Documentation and the six example metrics

Section II.C is where testing becomes a defensible record. It expects insurers to document the processes and reasoning behind their testing methodologies and analysis for unfair or unlawful discrimination, commensurate with their use of ECDIS and AIS, and to be prepared to make that documentation available to the Department on request ¹. The commensurate-with-use phrasing means a model driving a large book of New York pricing carries a heavier documentation expectation than a low-stakes pilot; scale your file to the exposure.

The reach of Section II.C is easy to under-read. Its quantitative-assessment passage points at “paragraphs 12 and 15.” Paragraph 12 is the Section II.A proxy assessment; paragraph 15 is Section II.B Step 1 ¹. So the documentation and quantitative expectations cover the proxy assessment and Step 1 both, which is the textual reason the proxy assessment carries a documentation obligation even though Section II.A itself contains none.

For the quantitative side, the circular names six example metrics rather than mandating a single one: Adverse Impact Ratio, Denials Odds Ratios, Marginal Effects, Standardized Mean Differences, Z-tests and T-tests, and Drivers of Disparity ¹. The circular offers these as an illustration of what “multiple statistical metrics” can mean. Nothing in the text makes all six a required battery, so pick the ones that fit the decision the model makes. The circular also draws a boundary worth keeping in the file: there is “no expectation that insurers collect additional data … to perform exemplary analysis” ¹. If your team is proposing to acquire new protected-class data specifically to run these tests, the circular does not ask for that, and the decision to do it is a privacy and governance question in its own right.

If you also respond to NAIC information requests, the same test evidence tends to satisfy overlapping questions there. The NAIC AI Systems Evaluation Tool’s high-risk model exhibit asks how a model was validated and how it is tested for unfair discrimination, and its data exhibit asks which data elements a model uses and whether each is internal or third-party ². Those are optional supplemental exhibits, not New York law, but the retention you build for Section II.C is largely the retention those exhibits would draw on, so structure it once. Our NAIC Exhibit D data documentation playbook works the data-element side of that in detail.

The record below is the minimum a single run should leave behind, whichever exercise it was. The example entries are illustrative, not a real company’s data.

The worksheet mirrors this table and adds a run-by-run tracker, the six example metrics as a pick list, the three timing triggers with date columns, and the two-exercise comparison from the top of this piece.

Cadence: when the testing has to run

The circular sets the timing plainly. Unfair or unlawful discrimination testing and analysis should be administered prior to putting AIS into production and on a regular cadence thereafter, and whenever material updates or changes are made to either the ECDIS or AIS ¹. Three triggers, then, and the file needs a date against each: a pre-production test before go-live, a recurring test on a cadence you set, and an event-driven test tied to material change.

The circular does not fix the recurring interval, and the annual figure that attaches to Step 3 is specifically the “repeat Step 3 at least annually” instruction for the no-alternative path ¹. Reading it as a blanket annual cadence for all testing over-reads the letter. So the regular cadence is a number you choose and write into your AIS policies with a reason, the same way you would set any revalidation interval. What the circular does pin down is the change trigger, and that is the one most likely to be missed operationally, because a vendor pushing a model update or a swap in an external data feed can be a “material change” that nobody routed to model-risk. The control that catches it is a change-management hook: model or data changes cannot reach production without a check on whether the discrimination testing needs to re-run.

Governance and vendors: who answers for it

Section III turns the testing into part of a governance program someone owns. The governance expectations in Section III.B put policy approval at the top of the house: the board, another governing body or its committees, or senior management through delegated authority should review and approve the ECDIS and AIS-related policies and procedures at least annually ¹. Those policies should include clearly defined roles and responsibilities and monitoring-and-reporting requirements to senior management ¹. The practical read is that the testing owners named above should appear, by role, in an approved policy, and that the annual policy review is itself a dated artifact a regulator can ask to see.

Documentation is a program obligation too. The circular expects full documentation for the use of all AIS, including all ECDIS relied on for them, whether developed internally or supplied by third parties, consistent with 11 NYCRR 243 ¹. And it draws a hard line on one record type: insurers “must” be prepared to respond to consumer complaints and inquiries about AIS and ECDIS use, and “must” maintain complaint records under 11 NYCRR 243 ¹. That “must,” against the “should” that governs almost everything else in the letter, is worth flagging to whoever owns complaint handling.

Vendors are where responsibility gets tested. The circular is explicit on this: a third party’s claim of non-discrimination, or a proprietary third-party process, is no basis on its own for determining compliance, and the responsibility to comply remains with the insurer at all times ¹. That means a vendor’s assurance is not a substitute for your own Section II.B testing on a vendor-supplied model. Section III.D then tells you what to build into the contract, where appropriate and available ¹. The first term provides audit rights, or entitles you to receive audit reports by qualified auditing entities. The second requires the vendor to cooperate with regulatory inquiries and investigations related to your use of the product ¹. The circular notes this clause is drawn from the NAIC’s Model Bulletin ¹. The timing problem is the familiar one: these terms have to be in the contract before an inquiry lands, and a contract signed before AI entered the relationship usually does not have them. Our AI vendor risk assessment checklist covers the clause language, and the broad underwriting guide sets where third-party models sit in the underwriting stack.

Disclosure and the notice clock

The testing file connects to the transparency expectations in Section IV, because a decision your tested model produces may trigger a notice, and the notice content is prescribed. Where an insurer uses ECDIS or AIS, the notice should disclose whether it uses AIS in underwriting or pricing and whether it uses data about the person from external vendors ¹. It should also state that the person has the right to request information about the specific data that resulted in the decision, and give contact information for the request ¹. The circular also forecloses a common defense: the proprietary nature of a vendor’s algorithmic processes does not justify a lack of specificity about an adverse action ¹. If your Section II.B file documents which variables drove a differential effect, that is the same information the disclosure obligation reaches, so keep the two consistent.

Two specific mechanics deserve their exact register. Section IV.F carves out a narrow path: an underwriting process using ECDIS or AIS determines that an applicant will not be approved under that process, and the applicant can only obtain insurance by submitting to a non ECDIS or AIS-based process ¹. On that path the applicant has the right to know why. The insurer should provide written notice “within 15-days of such a determination,” and the non ECDIS or AIS-based process continues during the notice period ¹. That is the only determination this 15-day clock attaches to; it is not a general adverse-action clock. Separately, Section IV.F provides that an applicant who will not be approved based on specific ECDIS data should be given a process to review that data for accuracy, and that review process “needs to be provided at the time the applicant is notified” ¹. The letter uses “needs to be” there, a register a notch harder than the “should” running through the rest, so do not soften it in your procedures.

Where teams get this wrong

Four mistakes recur, and each is a gap a DFS inquiry can find quickly.

Running one test and calling it two. A proxy assessment on the ECDIS is not the Section II.B assessment, and that assessment does not discharge the proxy assessment for the data source. A model built on external data needs both, and the file should show both, separately.

Concluding after Step 1 with nothing written. “No prima facie effect” is a legitimate stopping point, but only as a documented finding. A conclusion after Step 1 with no retained metrics or classes tested is indistinguishable, on inspection, from not having tested.

Reading the end of Step 3 as a ban. The circular’s no-alternative path is ongoing model risk management plus an annual re-run of the search, not a prohibition. Teams that expect a terminal “banned” state sometimes stop maintaining the file for a model they kept in use, and the missing annual Step 3 is exactly what the standing obligation asks for.

Treating a vendor’s assurance as the test. The responsibility stays with the insurer at all times, and a proprietary process is not a compliance answer. If a vendor-supplied model has no insurer-run Section II.B evidence behind it, that is a gap the contract’s audit rights exist to close, which is why those rights have to be negotiated before anyone is asking for the file.

Scope is the last thing to keep straight. The circular addresses underwriting and pricing, is not intended to reach other phases of the product lifecycle, and does not apply to Child Health Plus, the Essential Plan, or Medicaid managed care ¹. If your testing program spans claims, marketing, or fraud models as well, those are examples of your own broader governance rather than of what this circular reaches, and the file should not imply the letter demands testing where it does not.