A hands-on guide for AI developers who have been asked "how do you know it works?" about a model or an LLM feature, and cannot answer with a proof. You will build, in about 400 lines of Python, the evidence that replaces one: an evaluation harness with slice metrics and a pass/fail contract, property-based and metamorphic tests, calibration and conformal prediction, output contracts and a regression eval for an LLM step, drift checks, a CI gate, and exact formal verification of a small neural network. Every block below was run, and the output shown is real.
This technote accompanies our blog post Provable correctness fails in AI: what evidence to build instead, which covers the business and regulatory side. Facts are as of 16 November 2026.
Who this is for, what you'll build, prerequisites
Who this is for: ML and AI engineers, data scientists who ship models, and platform engineers who own the pipelines around them. You know Python and pytest; you do not need a background in formal methods.
What you'll build: a small project, evidence/, around a loan-repayment classifier trained on synthetic data with a known ground truth, plus an invoice-extraction step that stands in for an LLM. Each file produces one kind of evidence; gate.sh runs them all and fails the build if any fails.
Prerequisites: Python 3.10 or later (we used 3.12), about 300 MB of disk for the libraries, any laptop CPU. No GPU and no API key are needed. We tested with NumPy 2.5.5, SciPy 1.18.1, scikit-learn 1.9.1, Hypothesis 6.170.15, Pydantic 2.14.3, MAPIE 1.6.0 and pytest 9.2.0.
Step 0: set up
mkdir evidence && cd evidence
python3 -m venv .venv && . .venv/bin/activate
pip install numpy scipy scikit-learn hypothesis pydantic pytest mapie
export OMP_NUM_THREADS=1 # see "Pitfalls": per-row predictions get ~100x slower without it on a busy machine
Step 1: data with a known ground truth
Real data is better for real decisions, but for learning the techniques you want a dataset where you know the right answer. Here repayment depends on income (up), debt ratio (down), years employed (up) and region, with the smallest region slightly worse off. Every test below therefore has a correct outcome.
"""Synthetic loan data with a known ground truth, so every test has a right answer."""
import numpy as np
FEATURES = ["income_k", "debt_ratio", "years_employed", "region"]
REGIONS = ["north", "south", "east", "west"]
def make_loans(n: int, seed: int = 0, income_shift: float = 0.0):
rng = np.random.default_rng(seed)
income = rng.lognormal(mean=10.4 + income_shift, sigma=0.5, size=n) # GBP a year
debt_ratio = rng.beta(2, 5, size=n)
years = rng.integers(0, 30, size=n)
region = rng.choice(4, size=n, p=[0.4, 0.3, 0.2, 0.1])
logit = (1.5 * (np.log(income) - 10.4) - 4.0 * (debt_ratio - 0.3)
+ 0.05 * (years - 8) - 0.3 * (region == 3))
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int) # 1 = repaid
X = np.column_stack([income / 1000, debt_ratio, years, region]).astype(float)
return X, y
Step 2: train the model, with two rules built in
scikit-learn's HistGradientBoostingClassifier accepts monotonic_cst, which forces the score to move in one direction as a feature rises [1]. We train one model with the constraints and one without, so we can see what the tests catch.
"""Train the model under test. monotonic_cst encodes two business rules by construction."""
import joblib
from sklearn.ensemble import HistGradientBoostingClassifier
from data import make_loans
def train(constrained: bool = True, seed: int = 0):
X, y = make_loans(20_000, seed=seed)
clf = HistGradientBoostingClassifier(
max_iter=300, learning_rate=0.05, random_state=seed,
categorical_features=[3],
# +1: more income never lowers the score; -1: more debt never raises it
monotonic_cst=[1, -1, 0, 0] if constrained else None,
)
return clf.fit(X, y)
if __name__ == "__main__":
joblib.dump(train(constrained=True), "model.joblib")
joblib.dump(train(constrained=False), "model_unconstrained.joblib")
print("saved model.joblib and model_unconstrained.joblib")
Step 3: an evaluation harness with a contract
The contract is the closest thing you have to a specification: thresholds agreed with the business before training. The harness reports held-out ROC AUC (how well the score ranks repayers above defaulters), Brier score and expected calibration error (ECE, the average gap between predicted probability and observed frequency), then the same per region with bootstrap 95% confidence intervals. It gates on the lower end of each slice's interval, so a small slice cannot pass on luck, and it exits non-zero on failure, which is what makes it usable in CI.
"""Held-out and per-slice evaluation with bootstrap confidence intervals and a pass/fail contract."""
import json
import sys
import joblib
import numpy as np
from sklearn.metrics import roc_auc_score, brier_score_loss
from data import make_loans, REGIONS
CONTRACT = { # agreed with the business before training
"auc_min": 0.70, # overall held-out ROC AUC
"slice_auc_lower_ci_min": 0.62, # worst slice, lower end of the 95% CI
"ece_max": 0.03, # expected calibration error
"min_slice_n": 300, # below this a slice is reported, not gated
}
def ece(y, p, bins=10):
"""Expected calibration error: weighted gap between confidence and frequency."""
edges = np.linspace(0, 1, bins + 1)
idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
return sum(abs(p[idx == b].mean() - y[idx == b].mean()) * (idx == b).mean()
for b in range(bins) if (idx == b).any())
def bootstrap_auc(y, p, n_boot=1000, seed=0):
rng = np.random.default_rng(seed)
stats = []
for _ in range(n_boot):
i = rng.integers(0, len(y), len(y))
if y[i].min() != y[i].max():
stats.append(roc_auc_score(y[i], p[i]))
return np.percentile(stats, [2.5, 97.5])
def evaluate(model_path="model.joblib", seed=1):
model = joblib.load(model_path)
X, y = make_loans(10_000, seed=seed) # held-out: a seed never used in training
p = model.predict_proba(X)[:, 1]
report = {"n": len(y), "auc": roc_auc_score(y, p),
"brier": brier_score_loss(y, p), "ece": ece(y, p), "slices": {}}
for code, name in enumerate(REGIONS):
m = X[:, 3] == code
lo, hi = bootstrap_auc(y[m], p[m])
report["slices"][name] = {"n": int(m.sum()), "auc": roc_auc_score(y[m], p[m]),
"auc_ci": [lo, hi]}
failures = []
if report["auc"] < CONTRACT["auc_min"]:
failures.append(f"auc {report['auc']:.3f} < {CONTRACT['auc_min']}")
if report["ece"] > CONTRACT["ece_max"]:
failures.append(f"ece {report['ece']:.3f} > {CONTRACT['ece_max']}")
for name, s in report["slices"].items():
if s["n"] >= CONTRACT["min_slice_n"] and s["auc_ci"][0] < CONTRACT["slice_auc_lower_ci_min"]:
failures.append(f"slice {name}: AUC lower CI {s['auc_ci'][0]:.3f}")
report["failures"] = failures
return report
if __name__ == "__main__":
r = evaluate(*sys.argv[1:2])
print(f"held-out n={r['n']} AUC={r['auc']:.3f} Brier={r['brier']:.3f} ECE={r['ece']:.3f}")
for name, s in r["slices"].items():
print(f" {name:<6} n={s['n']:>5} AUC={s['auc']:.3f} 95% CI [{s['auc_ci'][0]:.3f}, {s['auc_ci'][1]:.3f}]")
json.dump(r, open("eval_report.json", "w"), indent=2, default=float)
print("PASS" if not r["failures"] else "FAIL: " + "; ".join(r["failures"]))
sys.exit(1 if r["failures"] else 0)
$ python eval_harness.py
held-out n=10000 AUC=0.747 Brier=0.200 ECE=0.014
north n= 3989 AUC=0.749 95% CI [0.735, 0.764]
south n= 2920 AUC=0.759 95% CI [0.740, 0.778]
east n= 2065 AUC=0.735 95% CI [0.714, 0.756]
west n= 1026 AUC=0.723 95% CI [0.694, 0.753]
PASS
$ python eval_harness.py model_unconstrained.joblib
held-out n=10000 AUC=0.747 Brier=0.200 ECE=0.012
...
PASS
Note the widths: the west slice, with 1,026 cases, has an interval almost twice as wide as north's. Both models pass. On these numbers alone you could not choose between them, which is underspecification in miniature: one pipeline, many models with the same held-out score [2].
Step 4: property-based and metamorphic tests
A property-based test states a rule and lets a library generate the inputs. Hypothesis generates hundreds of cases, and when one fails it shrinks it to the simplest failing example [3]. A metamorphic test is a property that relates two runs: you do not know the right score for an applicant, but you know that adding income must not lower it. This idea comes from software testing, where it was developed for programs without a reliable test oracle [4]; CheckList applied the same idea to NLP models as invariance and directional tests [5].
"""Property-based and metamorphic tests: rules that must hold for every input, not just the test set."""
import os
import joblib
import numpy as np
from hypothesis import given, settings, strategies as st
MODEL = joblib.load(os.environ.get("MODEL", "model.joblib"))
applicant = st.tuples(
st.floats(5, 500), # income_k
st.floats(0, 1), # debt_ratio
st.integers(0, 45), # years_employed
st.integers(0, 3), # region code
)
def proba(rows):
return MODEL.predict_proba(np.asarray(rows, dtype=float))[:, 1]
@settings(max_examples=300, deadline=None)
@given(applicant)
def test_output_is_a_probability(a):
p = proba([a])
assert p.shape == (1,) and 0.0 <= p[0] <= 1.0
@settings(max_examples=300, deadline=None)
@given(applicant, st.floats(0.1, 200))
def test_more_income_never_lowers_score(a, extra): # metamorphic relation
richer = (a[0] + extra, *a[1:])
assert proba([richer])[0] >= proba([a])[0] - 1e-9
@settings(max_examples=300, deadline=None)
@given(applicant, st.floats(0.01, 0.5))
def test_more_debt_never_raises_score(a, extra):
worse = (a[0], min(1.0, a[1] + extra), *a[2:])
assert proba([worse])[0] <= proba([a])[0] + 1e-9
@settings(max_examples=100, deadline=None)
@given(st.lists(applicant, min_size=2, max_size=20), st.randoms())
def test_batch_order_does_not_change_answers(rows, rnd): # invariance
perm = list(range(len(rows)))
rnd.shuffle(perm)
p = proba(rows)
assert np.allclose(proba([rows[i] for i in perm]), p[perm])
$ python -m pytest -q test_properties.py
.... [100%]
4 passed in 4.51s
$ MODEL=model_unconstrained.joblib python -m pytest -q test_properties.py
.FF. [100%]
E assert np.float64(0.8262130572178492) >= (np.float64(0.8346077610546677) - 1e-09)
E Failing test case: test_more_income_never_lowers_score(
E a=(33.0, 0.0, 7, 0),
E extra=1.0,
E )
...
FAILED test_properties.py::test_more_income_never_lowers_score - assert np.fl...
FAILED test_properties.py::test_more_debt_never_raises_score - assert np.floa...
2 failed, 2 passed in 6.91s
The unconstrained model, with the same AUC, scores an applicant earning £33k at 0.835 and the same applicant earning £34k at 0.826. Hypothesis found it and shrank it to round numbers. Your failing example may differ, as generation is randomised; Hypothesis stores failures in its local database and replays them first on the next run. The fix here was to build the rule into the model; where you cannot, the test still tells you and the gate blocks the release.
Step 5: calibration and conformal prediction
A score of 0.8 should mean that about 80% of such applicants repay. Modern neural networks are often over-confident [6]; our trees were already well calibrated (ECE 0.014). If yours are not, scikit-learn's CalibratedClassifierCV fits a correction on held-out data.
Conformal prediction goes further: it turns any model's scores into prediction sets that contain the true label with a chosen probability, here 90%, assuming only that new data is exchangeable with the calibration data (roughly, drawn from the same population) [7]. It takes three steps in NumPy.
"""Split conformal prediction in plain numpy: prediction sets with a coverage guarantee."""
import joblib
import numpy as np
from data import make_loans
model = joblib.load("model.joblib")
alpha = 0.10 # target: true label in the set 90% of the time
X_cal, y_cal = make_loans(2_000, seed=2) # calibration data, unseen in training
X_test, y_test = make_loans(10_000, seed=3)
# 1. Non-conformity score: 1 - probability the model gave to the true class
p_cal = model.predict_proba(X_cal)
scores = 1 - p_cal[np.arange(len(y_cal)), y_cal]
# 2. Finite-sample-corrected quantile of the calibration scores
n = len(scores)
qhat = np.quantile(scores, np.ceil((n + 1) * (1 - alpha)) / n, method="higher")
# 3. Prediction set: every class whose score is within qhat
p_test = model.predict_proba(X_test)
sets = (1 - p_test) <= qhat # boolean matrix, shape (n_test, 2)
covered = sets[np.arange(len(y_test)), y_test].mean()
size = sets.sum(axis=1)
print(f"qhat={qhat:.3f} empirical coverage={covered:.3f} (target {1 - alpha:.2f})")
print(f"singleton sets={np.mean(size == 1):.1%} both classes={np.mean(size == 2):.1%} empty={np.mean(size == 0):.1%}")
# The guarantee is an average over calibration draws, so check it over many draws
cov = []
for s in range(100, 150):
Xc, yc = make_loans(2_000, seed=s)
sc = 1 - model.predict_proba(Xc)[np.arange(len(yc)), yc]
q = np.quantile(sc, np.ceil((len(sc) + 1) * (1 - alpha)) / len(sc), method="higher")
cov.append(((1 - p_test) <= q)[np.arange(len(y_test)), y_test].mean())
print(f"over 50 calibration draws: mean coverage={np.mean(cov):.3f} min={np.min(cov):.3f} max={np.max(cov):.3f}")
$ python conformal.py
qhat=0.689 empirical coverage=0.897 (target 0.90)
singleton sets=49.7% both classes=50.3% empty=0.0%
over 50 calibration draws: mean coverage=0.899 min=0.885 max=0.913
Two lessons. First, the guarantee holds on average over calibration draws, not for every single one: individual draws ranged from 0.885 to 0.913, and the mean was 0.899 against a fixed test set that adds its own noise. Second, and more useful to the business, half the applicants get the set {repay, default}: the model cannot decide at 90% confidence. That is your human-review queue, and its size is a cost you can now state.
MAPIE wraps the same method, with more conformity scores and regression support. Version 1 changed the API; this is the 1.x form [8]:
"""The same thing with MAPIE 1.x (prefit model, LAC score)."""
import joblib
import numpy as np
from mapie.classification import SplitConformalClassifier
from data import make_loans
X_cal, y_cal = make_loans(2_000, seed=2)
X_test, y_test = make_loans(10_000, seed=3)
mapie = SplitConformalClassifier(estimator=joblib.load("model.joblib"),
confidence_level=0.90, prefit=True)
mapie.conformalize(X_cal, y_cal)
_, sets = mapie.predict_set(X_test) # sets: (n, n_classes, n_confidence_levels)
sets = sets[:, :, 0]
print(f"MAPIE coverage={sets[np.arange(len(y_test)), y_test].mean():.3f} "
f"mean set size={sets.sum(axis=1).mean():.2f}")
MAPIE coverage=0.897 mean set size=1.50
Same coverage, same mean set size (half singletons, half pairs): the NumPy version and the library agree.
Step 6: output contracts and regression evals for an LLM step
An LLM's output is text; your system needs data. Treat every response as untrusted input. Pydantic validates structure (types, allowed values, required fields) and lets you add deterministic business checks such as "net plus VAT equals total" [9].
"""An output contract for an LLM step: schema, deterministic business checks, and a regression eval."""
import json
from datetime import date
from typing import Literal
from pydantic import BaseModel, Field, ValidationError, field_validator, model_validator
class InvoiceExtraction(BaseModel):
supplier: str = Field(min_length=2, max_length=120)
invoice_date: date
currency: Literal["GBP", "EUR", "USD"]
net: float = Field(ge=0)
vat: float = Field(ge=0)
total: float = Field(gt=0)
@field_validator("invoice_date")
@classmethod
def not_in_future(cls, v: date) -> date:
if v > date.today():
raise ValueError("invoice date is in the future")
return v
@model_validator(mode="after")
def totals_add_up(self):
if abs(self.net + self.vat - self.total) > 0.01:
raise ValueError(f"net + vat = {self.net + self.vat:.2f} but total = {self.total:.2f}")
return self
def check(raw: str) -> tuple[bool, str]:
"""Never trust the model's text: parse, validate, and say exactly what failed."""
try:
InvoiceExtraction.model_validate_json(raw)
return True, "ok"
except ValidationError as e:
return False, "; ".join(f"{'.'.join(map(str, err['loc'])) or 'model'}: {err['msg']}"
for err in e.errors())
# Three outputs an LLM really might produce for the same invoice
outputs = {
"good": '{"supplier":"Acme Ltd","invoice_date":"2026-10-31","currency":"GBP","net":100.0,"vat":20.0,"total":120.0}',
"bad_maths": '{"supplier":"Acme Ltd","invoice_date":"2026-10-31","currency":"GBP","net":100.0,"vat":20.0,"total":102.0}',
"bad_shape": '{"supplier":"Acme Ltd","invoice_date":"31/10/2026","currency":"£","net":"100","vat":20.0}',
}
if __name__ == "__main__":
for name, raw in outputs.items():
ok, why = check(raw)
print(f"{name:<10} {'PASS' if ok else 'FAIL'} {why}")
$ python llm_contract.py
good PASS ok
bad_maths FAIL model: Value error, net + vat = 120.00 but total = 102.00
bad_shape FAIL invoice_date: Input should be a valid date or datetime, invalid character in year; currency: Input should be 'GBP', 'EUR' or 'USD'; total: Field required
Notice what was not flagged: "net":"100", a string, was accepted and converted, because Pydantic's default lax mode coerces numeric strings. If you want the model to produce exact types, set model_config = ConfigDict(strict=True). Then route failures: retry once with the error message in the prompt, and send repeated failures to a person.
Validation proves shape, not truth. For truth you need a golden set: real cases with answers checked by a person, scored field by field, compared with the release in production before any prompt or model change ships.
"""Regression eval for an LLM step: golden cases, field-level scoring, compared against the last release."""
import json
import sys
from llm_contract import InvoiceExtraction, check
GOLDEN = [ # expected fields per case, written and reviewed by a person
{"id": "inv-001", "expect": {"supplier": "Acme Ltd", "total": 120.0, "currency": "GBP"}},
{"id": "inv-002", "expect": {"supplier": "Brightwater plc", "total": 54.0, "currency": "GBP"}},
{"id": "inv-003", "expect": {"supplier": "Nordlicht GmbH", "total": 238.0, "currency": "EUR"}},
{"id": "inv-004", "expect": {"supplier": "Kestrel Ltd", "total": 960.0, "currency": "GBP"}},
]
def extract(case_id: str, outputs: dict) -> str:
"""Replace with your model call (temperature 0, pinned model version, logged prompt)."""
return outputs[case_id]
def run(outputs: dict) -> dict:
valid = fields_ok = fields_total = 0
for case in GOLDEN:
raw = extract(case["id"], outputs)
ok, _ = check(raw)
valid += ok
got = InvoiceExtraction.model_validate_json(raw).model_dump() if ok else {}
for k, v in case["expect"].items():
fields_total += 1
fields_ok += got.get(k) == v
return {"schema_valid": valid / len(GOLDEN), "field_accuracy": fields_ok / fields_total}
if __name__ == "__main__":
baseline = json.load(open("baseline.json")) # scores of the release in production
candidate = run(json.load(open(sys.argv[1]))) # recorded outputs of the new prompt/model
print("baseline ", {k: round(v, 3) for k, v in baseline.items()})
print("candidate", {k: round(v, 3) for k, v in candidate.items()})
bad = [k for k in candidate if candidate[k] < baseline[k] - 0.02]
print("PASS" if not bad else f"FAIL: regression in {bad}")
sys.exit(1 if bad else 0)
$ python regression_eval.py outputs_v2.json
baseline {'schema_valid': 1.0, 'field_accuracy': 1.0}
candidate {'schema_valid': 0.75, 'field_accuracy': 0.667}
FAIL: regression in ['schema_valid', 'field_accuracy']
Here outputs_v1.json and outputs_v2.json hold recorded responses for four invoices (in v2, one currency is wrong and one total does not add up), so the example runs without an API key. In your project, extract() calls the model with temperature 0 and a pinned model version, and you store the raw responses with the score so a failure can be inspected later. Four cases is a demonstration; a useful golden set has a few hundred, chosen to cover your slices and your past incidents.
Step 7: drift monitoring
Every guarantee above assumes production data looks like your evaluation data. Check that it still does. The population stability index (PSI) measures how far a distribution has moved across fixed bins; the two-sample Kolmogorov-Smirnov (KS) test asks whether two samples could come from the same distribution.
"""Drift checks on inputs and scores: population stability index and the two-sample KS test."""
import joblib
import numpy as np
from scipy.stats import ks_2samp
from data import make_loans, FEATURES
def psi(expected, actual, bins=10):
"""Population stability index over quantile bins of the reference data."""
edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
return float(np.sum((a - e) * np.log(a / e)))
model = joblib.load("model.joblib")
X_ref, _ = make_loans(10_000, seed=1) # what the model was validated on
for label, shift in [("same population", 0.0), ("incomes 20% higher", np.log(1.2))]:
X_live, _ = make_loans(5_000, seed=7, income_shift=shift) # a week of production traffic
print(label)
for j, name in enumerate(FEATURES[:3]):
ks = ks_2samp(X_ref[:, j], X_live[:, j])
print(f" {name:<15} PSI={psi(X_ref[:, j], X_live[:, j]):.3f} KS p={ks.pvalue:.3g}")
s_ref, s_live = model.predict_proba(X_ref)[:, 1], model.predict_proba(X_live)[:, 1]
print(f" {'score':<15} PSI={psi(s_ref, s_live):.3f} mean {s_ref.mean():.3f} -> {s_live.mean():.3f}")
$ python drift.py
same population
income_k PSI=0.002 KS p=0.484
debt_ratio PSI=0.001 KS p=0.33
years_employed PSI=0.005 KS p=0.0319
score PSI=0.003 mean 0.572 -> 0.573
incomes 20% higher
income_k PSI=0.126 KS p=6.14e-59
debt_ratio PSI=0.001 KS p=0.33
years_employed PSI=0.005 KS p=0.0319
score PSI=0.066 mean 0.572 -> 0.627
The shifted income is caught by both measures, and the score distribution moves with it. But look at years_employed on the same population: KS gives p = 0.03, "significant" at the usual 5% level, with nothing wrong. Run enough tests every day and some will fire by chance; with large samples, tiny harmless shifts also become significant. Alert on effect sizes such as PSI (a common rule of thumb in credit scoring treats below 0.1 as stable and above 0.25 as a major shift; tune it on your own history) and, above all, on outcomes once labels arrive.
Step 8: formal verification of a small network
For a small ReLU network you can prove statements such as "for every applicant with income £50k to £200k, debt ratio at most 0.25 and any employment history, the network says likely to repay". Two methods, both in under 100 lines:
- Interval bound propagation pushes the input box through each layer. It is sound (if it says the property holds, it does) but incomplete (it can fail to prove a true property, because the bounds are loose).
- Mixed-integer linear programming encodes each ReLU exactly with one binary variable and asks a solver for the true minimum of the output over the box. It is complete, but the problem is NP-complete in general [10], so it only scales to small networks. SciPy's
milpuses the HiGHS solver [11].
"""Formal verification of a small ReLU network: interval bounds (fast, sound, incomplete)
and an exact mixed-integer program (complete) with scipy.optimize.milp."""
import sys
import numpy as np
from scipy.optimize import milp, LinearConstraint, Bounds
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from data import make_loans
X, y = make_loans(20_000, seed=0)
X = X[:, :3] # income_k, debt_ratio, years_employed
scaler = StandardScaler().fit(X)
net = MLPClassifier(hidden_layer_sizes=(8, 8), activation="relu", max_iter=500,
random_state=0).fit(scaler.transform(X), y)
layers = list(zip(net.coefs_, net.intercepts_)) # last layer gives the logit
def to_scaled(lo, hi):
return (np.array(lo) - scaler.mean_) / scaler.scale_, (np.array(hi) - scaler.mean_) / scaler.scale_
def interval_bounds(lo, hi):
"""Push a box through the network; returns pre-activation bounds per layer."""
bounds = []
for k, (W, b) in enumerate(layers):
Wp, Wn = np.clip(W, 0, None), np.clip(W, None, 0)
l, u = lo @ Wp + hi @ Wn + b, hi @ Wp + lo @ Wn + b
bounds.append((l, u))
lo, hi = np.maximum(l, 0), np.maximum(u, 0)
return bounds
def min_logit_milp(lo, hi):
"""Exact minimum of the logit over the box. Each unstable ReLU a = max(z, 0) becomes
linear constraints plus one binary d (d=1: active), using the interval bounds l <= z <= u."""
pre = interval_bounds(lo, hi)
n_in = len(lo)
sizes = [W.shape[1] for W, _ in layers]
# variable layout: x | (z, a, d) per hidden layer | logit
idx, off = {}, n_in
for k, n in enumerate(sizes[:-1]):
idx[k] = (off, off + n, off + 2 * n); off += 3 * n
out = off; nv = off + 1
rows, lb, ub = [], [], []
def add(coef, l, u):
r = np.zeros(nv)
for i, c in coef: r[i] += c
rows.append(r); lb.append(l); ub.append(u)
prev = list(range(n_in))
for k, (W, b) in enumerate(layers):
if k == len(layers) - 1: # logit = W a + b
add([(prev[i], W[i, 0]) for i in range(len(prev))] + [(out, -1)], -b[0], -b[0])
break
z0, a0, d0 = idx[k]
l, u = pre[k]
for j in range(W.shape[1]):
add([(prev[i], W[i, j]) for i in range(len(prev))] + [(z0 + j, -1)], -b[j], -b[j])
add([(a0 + j, 1), (z0 + j, -1)], 0, np.inf) # a >= z
add([(a0 + j, 1), (d0 + j, -max(u[j], 0))], -np.inf, 0) # a <= u d
add([(a0 + j, 1), (z0 + j, -1), (d0 + j, -min(l[j], 0))], -np.inf, -min(l[j], 0)) # a <= z - l(1-d)
prev = list(range(a0, a0 + W.shape[1]))
vlo, vhi, integ = np.full(nv, -np.inf), np.full(nv, np.inf), np.zeros(nv)
vlo[:n_in], vhi[:n_in] = lo, hi
for k in idx:
z0, a0, d0 = idx[k]; n = sizes[k]
vlo[a0:a0 + n] = 0; vlo[d0:d0 + n] = 0; vhi[d0:d0 + n] = 1; integ[d0:d0 + n] = 1
c = np.zeros(nv); c[out] = 1 # minimise the logit
res = milp(c, constraints=LinearConstraint(np.array(rows), lb, ub),
integrality=integ, bounds=Bounds(vlo, vhi))
assert res.success, res.message # never report a bound from a failed solve
return res.fun, res.x[:n_in] * scaler.scale_ + scaler.mean_
# Property: every applicant in the box must be scored "likely to repay" (logit > 0).
# required=True boxes are policy; the last one is exploratory, to find where the policy stops holding.
PROPERTIES = [
("income 80-200k, debt <= 0.15, 5-30 yrs", [80, 0.0, 5], [200, 0.15, 30], True),
("income 50-200k, debt <= 0.25, 0-30 yrs", [50, 0.0, 0], [200, 0.25, 30], True),
("income 40-200k, debt <= 0.30, 0-30 yrs", [40, 0.0, 0], [200, 0.30, 30], False),
]
failed = False
for name, box_lo, box_hi, required in PROPERTIES:
lo, hi = to_scaled(box_lo, box_hi)
ibp_lower = interval_bounds(lo, hi)[-1][0][0]
exact, worst = min_logit_milp(lo, hi)
verdict = "VERIFIED" if exact > 0 else "COUNTEREXAMPLE"
if ibp_lower > 0:
verdict += " (intervals alone were enough)"
print(f"{name}\n interval lower bound={ibp_lower:+.3f} exact min logit={exact:+.3f} {verdict}")
if exact <= 0:
print(f" worst case: income {worst[0]:.1f}k, debt {worst[1]:.3f}, {worst[2]:.1f} yrs")
failed |= required
# cross-check the verifier: no random sample may beat the exact minimum
S = np.random.default_rng(0).uniform(box_lo, box_hi, (100_000, 3))
h = scaler.transform(S)
for W, b in layers[:-1]:
h = np.maximum(h @ W + b, 0)
print(f" lowest logit in 100,000 random samples={(h @ layers[-1][0] + layers[-1][1]).min():+.3f}")
sys.exit(1 if failed else 0) # a required property failed
$ python verify_tiny_net.py
income 80-200k, debt <= 0.15, 5-30 yrs
interval lower bound=+0.254 exact min logit=+1.942 VERIFIED (intervals alone were enough)
lowest logit in 100,000 random samples=+1.960
income 50-200k, debt <= 0.25, 0-30 yrs
interval lower bound=-1.573 exact min logit=+0.589 VERIFIED
lowest logit in 100,000 random samples=+0.626
income 40-200k, debt <= 0.30, 0-30 yrs
interval lower bound=-2.774 exact min logit=-0.273 COUNTEREXAMPLE
worst case: income 40.0k, debt 0.300, 0.0 yrs
lowest logit in 100,000 random samples=-0.231
Read the three rows. The first property is proved by intervals alone. The second is true, but intervals could not show it; the exact solver could. The third is false, and the solver returns the worst applicant, a concrete counterexample to discuss with the business. Random sampling came close to the true minimum each time but never reached it, which is why sampling is a sanity check and not a proof.
The limits: this network has 16 hidden neurons. For larger networks use a dedicated verifier: α,β-CROWN, which has won VNN-COMP every year from 2021 to 2025 and reports scaling to convolutional networks with millions of parameters on local properties, or Marabou [12][13]. None of them proves anything about a large language model's answers. And a proof covers one model version and one input box: re-run it on every retrain, in the gate.
Step 9: the release gate in CI
Evidence that is not enforced decays. gate.sh runs everything above and stops at the first failure:
#!/usr/bin/env bash
# Release gate: every check must pass before a model or prompt change can ship.
set -euo pipefail
export OMP_NUM_THREADS=1 # stop OpenMP oversubscription in per-row tests
python model.py
python eval_harness.py model.joblib # held-out + slice contract
python -m pytest -q test_properties.py # properties and metamorphic relations
python conformal.py # coverage report (informational)
python regression_eval.py outputs_v1.json # LLM step: no regression against production
python verify_tiny_net.py # formal check: required boxes must verify
echo "ALL GATES PASSED"
A minimal GitHub Actions workflow that runs it on every pull request (shown for shape; we ran gate.sh locally, not in Actions):
name: model-gate
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with: { python-version: "3.12" }
- run: pip install numpy scipy scikit-learn hypothesis pydantic pytest mapie
- run: ./gate.sh
- uses: actions/upload-artifact@v7
if: always()
with: { name: evidence, path: eval_report.json }
Running it in production
- Pin everything. Pin library versions in a lock file and record the model artefact's hash, the data snapshot and the git commit in
eval_report.json. A report you cannot tie to a model version is not evidence. - Containers. Run the gate in the same image you deploy, so the tests see the same NumPy and scikit-learn builds as production. All of this runs on CPU; the property tests and verifier finished in seconds on a laptop.
- Threads. Set
OMP_NUM_THREADSdeliberately in both CI and serving. On our shared build machine, one single-row prediction took 150 ms with default threading and 1.3 ms with one thread. - Monitoring. Schedule
drift.pyagainst a daily sample of logged inputs and scores, export PSI per feature as metrics to your existing monitoring stack, and alert through the normal on-call route. Join outcomes to predictions when labels arrive and re-run the evaluation harness on them. - Go further. Google's ML Test Score lists 28 tests and monitoring checks for production readiness and is a good audit of what you have missed [17]; Sculley et al. explain why the data pipelines and glue code around a model need the same care as the model [16].
- Keep the record. Store every gate run's output with the release. When an auditor or a customer asks how you knew the model was fit for use, this is the answer.
Pitfalls and debugging
- Hypothesis tests are very slow. Thread oversubscription in OpenMP; set
OMP_NUM_THREADS=1. Also setdeadline=None, because model calls have variable latency. - A property test fails on a tiny difference. Floating-point noise; compare with a tolerance (we use 1e-9) and decide what difference matters to the business.
- Conformal coverage is below target. A single calibration draw can be; check the average over draws. If it is far below, your calibration data is not exchangeable with production, often because of a time trend: calibrate on recent data.
- The MILP is slow or fails. Tighten input bounds, because loose bounds make weak encodings, and check
res.success. Past a few hundred neurons, move to a dedicated verifier. - The LLM regression eval flaps. Set temperature to 0, pin the model version, and if outputs still vary, run each case several times and gate on the pass rate.
How to evaluate the evidence itself
Test your tests. Plant a known fault and confirm the gate catches it: the unconstrained model is exactly that for the property tests, outputs_v2.json is one for the LLM eval, and the shifted incomes are one for drift. Keep these planted faults in the repository and run them in CI as tests that are expected to fail, so a broken check cannot pass silently.
Alternatives
| Need | What we used | Alternatives | When to switch |
|---|---|---|---|
| Evaluation harness | Plain scikit-learn and NumPy | MLflow evaluation, your experiment tracker's eval features | Many models and teams; you need a UI and history |
| Property tests | Hypothesis | CheckList (NLP behavioural tests) | Text models where perturbations are linguistic |
| Uncertainty | NumPy split conformal, MAPIE | CalibratedClassifierCV for calibration only | You need calibrated scores, not sets |
| LLM output | Pydantic plus a golden set | Provider structured-output modes, LLM eval frameworks | Many prompts; you want dashboards and LLM-as-judge scoring |
| Drift | PSI and KS in SciPy | Evidently and similar monitoring libraries | Many features; you want reports and dashboards |
| Verification | Interval bounds and SciPy MILP | α,β-CROWN, Marabou | Networks beyond a few hundred neurons |
FAQ
Is this enough for the EU AI Act? It produces the kind of evidence Articles 9 and 15 ask for (testing against prior defined metrics and probabilistic thresholds, declared accuracy, robustness), but compliance also needs risk management, data governance, documentation and human oversight processes [14]. After the Digital Omnibus, stand-alone Annex III high-risk obligations apply from 2 December 2027 [15].
Do I need all of this for every model? No. Scale it to the risk: an internal search ranker might need steps 3, 6 and 7; a credit decision needs all of them, plus independent validation.
Can I verify an LLM formally? Not its answers. You can verify the deterministic code around it, constrain its output format, and verify small classifiers that sit next to it, such as a router or a guard model.
Why synthetic data? So every test has a known right answer. Swap in your data and keep the structure; the contract thresholds will change.
If you would like help putting evaluation harnesses, release gates and monitoring around your own models, see our MLOps setup service.
References
- scikit-learn user guide: histogram-based gradient boosting, monotonic constraints
- D'Amour et al., "Underspecification presents challenges for credibility in modern machine learning", arXiv 2011.03395 (JMLR, 2022)
- Hypothesis documentation
- Chen et al., "Metamorphic testing: a review of challenges and opportunities", ACM Computing Surveys, 2018
- Ribeiro et al., "Beyond accuracy: behavioral testing of NLP models with CheckList", ACL 2020
- Guo et al., "On calibration of modern neural networks", ICML 2017
- Angelopoulos and Bates, "A gentle introduction to conformal prediction and distribution-free uncertainty quantification", arXiv 2107.07511
- MAPIE documentation
- Pydantic documentation: validators
- Katz et al., "Reluplex: an efficient SMT solver for verifying deep neural networks", CAV 2017
- SciPy documentation: scipy.optimize.milp
- α,β-CROWN neural network verifier (GitHub)
- Marabou neural network verifier (GitHub)
- Regulation (EU) 2024/1689 (Artificial Intelligence Act)
- Digital Omnibus on AI, Regulation (EU) 2026/1744: AI Act Explorer
- Sculley et al., "Hidden technical debt in machine learning systems", NeurIPS 2015
- Breck et al., "The ML Test Score: a rubric for ML production readiness and technical debt reduction", IEEE Big Data 2017