Entropa results

What we measured on our own system, with the failures. Updated 2026-11-14.

1. Standards retrieval guard (the part that decides what an AI may read)

A read-only service gives an AI exact clauses from sealed, versioned standards (NIST SP 800-53, EU AI Act, DORA RTS, HIPAA 164.312, and a licensed PCI DSS copy held locally) and must refuse when it is not sure. Each row is a separate sealed round. Later rounds use new sets, so rows are not a controlled comparison.

Round	Test	Result	Bar	Verdict
R1	Citation accuracy	47 / 60	0.95	Failed
R1	Abstains on out-of-corpus questions	19 / 20	0.95	Passed
R1	Injection obeyed	1 / 10	0	Failed
R2	Description-only questions	45 / 53	0.90	Failed
R2	Abstention (name list off)	24 / 30	0.95	Failed
R2	Injection obeyed	4 / 20	0	Failed
R3	Abstention	29 / 30	0.95	Passed
R3	Injection obeyed	1 / 24	0	Failed
R3	Description-only questions (report only)	7 / 39	none	Over-cautious: it refuses too often
H1.1	Abstention	284 / 300	0.95	Failed by one case
H1.1	Injection obeyed	28 / 304 (upper bound 0.13)	0	Failed
H1.1	Id-named clause lookups	150 / 150	1.0	Passed
H1.1	Superseded-version questions	50 / 50	1.0	Passed
H1.1	Description-only questions (report only)	90 / 100	none	Easier set; not comparable with R3

Lesson stated plainly: our smaller early sets understated the injection rate. With 304 cases the real rate showed (28 obeyed). Exact id lookups and stale-version handling hold; free-text injection resistance does not yet meet its bar.

2. Hardening of our own code

Fuzz and property tests (H1.3): seven real defects found in our own code and fixed, one serious (a clause relabelled from licensed to open could have returned its full text). After the fixes: 0 violations in 100,000 cases per property in release builds, and 0 crashes in 8 fuzz targets run 10 minutes each. A debug-build run later exposed a too-tight timing budget in one test, now being corrected.
Mutation testing (in progress): deliberately breaking our defences to see whether a test notices. First target (flag detectors): 41 of 72 mutants caught, a score of 0.57, below our forecast. Survivors are being closed with new tests.

3. Access-token library (AT1)

10,001 planted attacks across 15 classes (forged, widened, expired, replayed, wrong audience, spliced and others): 0 accepted. Every valid control token was accepted.
22 of 22 hand-made defects in the verifier were caught by tests, after three test gaps were closed. Our first "all green" claim was release-only; a debug build exposed an arithmetic overflow in one test, fixed and disclosed.
Measured: about 7.2 KB per token and about 0.3 ms to verify (single signed token design).

4. Control self-assessment (E1)

We assessed our own estate against 17 controls from DORA RTS, PCI DSS 4.0.1, NIST SP 800-53, the EU AI Act and HIPAA: 1 met, 11 partial, 5 gaps, with every quoted control text machine-checked. Wording is always "evidence mapped to", never "compliant". See the full self-assessment.

5. What we have not done

No independent review of the cryptography, the code or the defences.
No customer has used it yet; no real-world traffic has been tested.
The sealed sets were written by us. The builder did not see them, but they are not independent either.
The retrieval guard is lexical (word matching), which is why description-style questions and injection are weak.