Why a checkbox cannot read your policy

Here is a paragraph from a real customer's information security policy, lightly changed to protect them:

Access to production systems is limited to engineers who need it. Access is reviewed periodically.

Now here is the criterion it is supposed to satisfy: the entity authorizes, modifies or removes access to data, software, functions and other protected information assets based on roles, responsibilities or the system design and changes, giving consideration to the concepts of least privilege and segregation of duties.

Does the paragraph satisfy the criterion? A rules engine has two options: match on keywords, in which case yes, both mention access and review, or require an exact template, in which case no, and it will also say no to the perfectly good policy the company writes next week in different words. Neither answer is useful.

What the right answer looks like

The right answer is that the paragraph is weak, for three specific reasons. Periodically is not a cadence, so nothing can be sampled. Need it is not a role definition, so least privilege is asserted rather than defined. And nobody is named, so there is no owner to interview. That answer requires reading the sentence, knowing what a reviewer will ask for, and noticing what the sentence carefully does not say.

That is the work. It is not classification and it is not extraction. It is judgement about prose, against a standard, in a context. Language models are the first tool that does it at a price a nine person company can pay.

Which is also why they are dangerous here

A model that reads your policy can also invent your policy. In compliance that failure is worse than useless, because a customer will act on it. Tell someone their encryption control passes and cite a sentence they never wrote, and they will not discover the problem until an auditor asks them to produce it. So most of our engineering effort is not spent on generating text. It is spent on refusing to.

Three checks do most of the work, and all three are boring:

• Quote verification. Any quote offered as support is matched against the submitted material verbatim, after normalising whitespace and quote characters. If it is not found, the quote is dropped and the finding is labelled unsupported. Short fragments are rejected outright, because an eight character match proves nothing.
• Evidence gating. A control cannot be marked met unless the finding names evidence an auditor could actually take. If the model says met and names nothing, we downgrade it to weak, in code, and say why.
• The score is not the model's. Readiness is computed from the statuses with a fixed weighting. We never ask a model for a number, because a number a model made up is a number you cannot defend to anyone.

Default to missing

The other decision that mattered was where to put the benefit of the doubt. Early on our mapping step was generous: given a vague policy and a helpful ticket, it leaned towards met. Customers liked those results and they were wrong. Now the instruction is explicit. Silence means missing. We plan to means missing. We usually means weak. Absence of evidence is not evidence.

This makes first runs feel harsh. A founder pastes everything they have and sees a score of 34. That is the honest number, and the alternative is a score of 71 that falls apart in a readiness call with an auditor who has done this two hundred times.

Why we route across models

Different steps of a run want different models. Deciding what kind of document a paste is wants the cheapest thing that can read. Mapping and drafting want a balanced model. A contested judgement across a long policy set wants the strongest model available, and that is worth paying for on the handful of controls where it matters.

Keeping that routing table in configuration rather than in prompt strings has a second benefit that has nothing to do with quality. When a provider changes price, rate limits or terms, and they all have, our response is a configuration change and an evaluation run, not a rebuild. The table is published on our How the AI works page, generated from the same configuration the product runs on.