La tienda ya está abierta

Construimos máquinas Workstation AI para sitios web, agentes de IA y entrenamiento de LLM — configúrelas y cómprelas en línea.

Obtener precios profesionales
Productos de IA
Labs de IAAgentes OpenAIAgentes ClaudeGrok BotCRM nativo de Workstation AIMarketingTodos los productos
Soluciones de IA
IA aplicadaAgentes de IAIA generativaAutomatización con IAFormación en IAPaquetes de IA para pymesLaboratorio de IA empresarialEstaciones de trabajo de IAHardware y software NVIDIAComputación acelerada NVIDIAIA privadaNube privada de GPUClústeres de GPUIA en el bordeIA por sector
Servicios de IA
Modernización de plataformasIngeniería digitalFundamentos de datos e IAOperaciones autónomasConsultoría de IAAutomatización DevOpsCiberseguridadDesarrollo de softwareCreación de agentesConfiguración MLOpsAlquiler de GPUNube privada de GPU
Sobre nosotros
EmpresaNuestra historiaSociosCasos de éxito
Centros de datos de IA
Centros de datos de IANube privada de GPUClústeres de GPUAlquiler de GPU
Análisis de IASerie JevLaboratorio de benchmarksSelección de modelos
Notas técnicas
Documentación
Linux OSUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
Plataformas de IA en la nubeAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI FoundryNubes y mercados de GPU
Centros de datos de IA privadosPara despachos de abogadosPara firmas de contabilidadPlataformas localesNube totalmente protegida
HerramientasRing PromoterJobshout
Noticias
Blog
ContáctenosIniciar sesión

Estaciones de trabajo de IA, software multiagente de IA, infraestructura de GPU y soluciones de agentes inteligentes para empresas modernas.

Contáctenos

Soluciones de IA

IA aplicadaAgentes de IAIA generativaAutomatización con IAFormación en IAPaquetes de IA para pymesLaboratorio de IA empresarialIA por sector

Productos

Todos los productosWorkstation AI native CRM y ERPMarketingAgentes OpenAIRing PromoterJobshout

Empresa

Sobre nosotrosEmpresaCentros de datos de IAPor Qué WorkstationSociosCasos de éxitoPreciosContactoTenthMatrix

Recursos

Análisis de IANotas técnicasDocumentaciónLinux OSPlataformas de IA en la nubeCentros de datos de IA privadosBlogBuscarMapa del sitio
Sede central en el Reino Unido
77-79 Marlowes, Hemel Hempstead HP1 1LFCómo llegar: tome la salida 20 de la M25, Outer LondonN.º de empresa: 11641870Lun - Vie: 9:00 - 18:00 GMT
+44 7515 356 146
Oficina de Bélgica
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683Lun - Vie: 9:00 - 18:00 CET
+32 492 45 67 46
Oficina de India
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
Oficina de Taiwán
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓Lun - Vie: 9:00 - 18:00, hora de Taiwán (GMT+8)
+886 926 892 701+32 492 45 67 46

© 2026 Workstation AI. Todos los derechos reservados.

PrivacidadEliminación de datosCookiesTérminos del servicioMapa del sitio web

NVIDIA, the NVIDIA logo, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX and HGX are trademarks and/or registered trademarks of NVIDIA Corporation in the US and other countries. AMD, the AMD Arrow logo, EPYC, Instinct, Ryzen and Threadripper are trademarks of Advanced Micro Devices, Inc. Dell and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Intel and Xeon are trademarks of Intel Corporation or its subsidiaries. AMAX, AceleMax and LiquidMax are trademarks of AMAX Engineering Corporation. All other product names, logos and brands are the property of their respective owners. We use them only to identify the products we describe, supply or support; their use does not mean their owners endorse us.

Inicio / Notas técnicas / Technology
AIMLOpsLLMPython

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Evidence instead of proofs, in about 400 lines of Python with real output: slice metrics with bootstrap intervals and a pass/fail contract, Hypothesis property and metamorphic tests, calibration and conformal prediction, Pydantic contracts and regression evals for LLM output, drift checks, a CI gate, and exact verification of a small neural network

16 de noviembre de 2026Technology27 min read

A hands-on guide for AI developers who have been asked "how do you know it works?" about a model or an LLM feature, and cannot answer with a proof. You will build, in about 400 lines of Python, the evidence that replaces one: an evaluation harness with slice metrics and a pass/fail contract, property-based and metamorphic tests, calibration and conformal prediction, output contracts and a regression eval for an LLM step, drift checks, a CI gate, and exact formal verification of a small neural network. Every block below was run, and the output shown is real.

A terminal running a release gate: evaluation PASS, four property tests passed, conformal coverage 0.897, LLM regression PASS, two verified properties and one counterexample, ending with ALL GATES PASSED.

Summary. You cannot prove a model correct, because there is no complete specification to prove it against. You can build evidence that it is reliable enough and keep producing it: agreed metrics with confidence intervals per slice, tests of rules that must hold for every input, uncertainty with a stated coverage, validated LLM output, drift alerts, and a gate that blocks releases when any of these fail. Where a small network makes a bounded, high-stakes decision, you can also verify properties exactly. The headline result from our run: two models with an identical held-out AUC of 0.747 behaved differently, and only a property test told them apart.

This technote accompanies our blog post Provable correctness fails in AI: what evidence to build instead, which covers the business and regulatory side. Facts are as of 16 November 2026.

Who this is for, what you'll build, prerequisites

Who this is for: ML and AI engineers, data scientists who ship models, and platform engineers who own the pipelines around them. You know Python and pytest; you do not need a background in formal methods.

What you'll build: a small project, evidence/, around a loan-repayment classifier trained on synthetic data with a known ground truth, plus an invoice-extraction step that stands in for an LLM. Each file produces one kind of evidence; gate.sh runs them all and fails the build if any fails.

Prerequisites: Python 3.10 or later (we used 3.12), about 300 MB of disk for the libraries, any laptop CPU. No GPU and no API key are needed. We tested with NumPy 2.5.5, SciPy 1.18.1, scikit-learn 1.9.1, Hypothesis 6.170.15, Pydantic 2.14.3, MAPIE 1.6.0 and pytest 9.2.0.

Seven layers of evidence from broad to narrow: held-out metrics, slice metrics, property and metamorphic tests, calibration and conformal prediction, LLM output contracts, drift monitoring, and formal verification of small networks, each with the file that produces it.
The evidence you will build, one file per layer. Lower layers cover more of the system with weaker guarantees; formal verification gives the strongest guarantee over the smallest scope.

Step 0: set up

mkdir evidence && cd evidence
python3 -m venv .venv && . .venv/bin/activate
pip install numpy scipy scikit-learn hypothesis pydantic pytest mapie
export OMP_NUM_THREADS=1   # see "Pitfalls": per-row predictions get ~100x slower without it on a busy machine

Step 1: data with a known ground truth

Real data is better for real decisions, but for learning the techniques you want a dataset where you know the right answer. Here repayment depends on income (up), debt ratio (down), years employed (up) and region, with the smallest region slightly worse off. Every test below therefore has a correct outcome.

"""Synthetic loan data with a known ground truth, so every test has a right answer."""
import numpy as np

FEATURES = ["income_k", "debt_ratio", "years_employed", "region"]
REGIONS = ["north", "south", "east", "west"]


def make_loans(n: int, seed: int = 0, income_shift: float = 0.0):
    rng = np.random.default_rng(seed)
    income = rng.lognormal(mean=10.4 + income_shift, sigma=0.5, size=n)  # GBP a year
    debt_ratio = rng.beta(2, 5, size=n)
    years = rng.integers(0, 30, size=n)
    region = rng.choice(4, size=n, p=[0.4, 0.3, 0.2, 0.1])
    logit = (1.5 * (np.log(income) - 10.4) - 4.0 * (debt_ratio - 0.3)
             + 0.05 * (years - 8) - 0.3 * (region == 3))
    y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)  # 1 = repaid
    X = np.column_stack([income / 1000, debt_ratio, years, region]).astype(float)
    return X, y

Step 2: train the model, with two rules built in

scikit-learn's HistGradientBoostingClassifier accepts monotonic_cst, which forces the score to move in one direction as a feature rises [1]. We train one model with the constraints and one without, so we can see what the tests catch.

"""Train the model under test. monotonic_cst encodes two business rules by construction."""
import joblib
from sklearn.ensemble import HistGradientBoostingClassifier
from data import make_loans


def train(constrained: bool = True, seed: int = 0):
    X, y = make_loans(20_000, seed=seed)
    clf = HistGradientBoostingClassifier(
        max_iter=300, learning_rate=0.05, random_state=seed,
        categorical_features=[3],
        # +1: more income never lowers the score; -1: more debt never raises it
        monotonic_cst=[1, -1, 0, 0] if constrained else None,
    )
    return clf.fit(X, y)


if __name__ == "__main__":
    joblib.dump(train(constrained=True), "model.joblib")
    joblib.dump(train(constrained=False), "model_unconstrained.joblib")
    print("saved model.joblib and model_unconstrained.joblib")

Step 3: an evaluation harness with a contract

The contract is the closest thing you have to a specification: thresholds agreed with the business before training. The harness reports held-out ROC AUC (how well the score ranks repayers above defaulters), Brier score and expected calibration error (ECE, the average gap between predicted probability and observed frequency), then the same per region with bootstrap 95% confidence intervals. It gates on the lower end of each slice's interval, so a small slice cannot pass on luck, and it exits non-zero on failure, which is what makes it usable in CI.

"""Held-out and per-slice evaluation with bootstrap confidence intervals and a pass/fail contract."""
import json
import sys
import joblib
import numpy as np
from sklearn.metrics import roc_auc_score, brier_score_loss
from data import make_loans, REGIONS

CONTRACT = {                       # agreed with the business before training
    "auc_min": 0.70,               # overall held-out ROC AUC
    "slice_auc_lower_ci_min": 0.62,  # worst slice, lower end of the 95% CI
    "ece_max": 0.03,               # expected calibration error
    "min_slice_n": 300,            # below this a slice is reported, not gated
}


def ece(y, p, bins=10):
    """Expected calibration error: weighted gap between confidence and frequency."""
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
    return sum(abs(p[idx == b].mean() - y[idx == b].mean()) * (idx == b).mean()
               for b in range(bins) if (idx == b).any())


def bootstrap_auc(y, p, n_boot=1000, seed=0):
    rng = np.random.default_rng(seed)
    stats = []
    for _ in range(n_boot):
        i = rng.integers(0, len(y), len(y))
        if y[i].min() != y[i].max():
            stats.append(roc_auc_score(y[i], p[i]))
    return np.percentile(stats, [2.5, 97.5])


def evaluate(model_path="model.joblib", seed=1):
    model = joblib.load(model_path)
    X, y = make_loans(10_000, seed=seed)  # held-out: a seed never used in training
    p = model.predict_proba(X)[:, 1]
    report = {"n": len(y), "auc": roc_auc_score(y, p),
              "brier": brier_score_loss(y, p), "ece": ece(y, p), "slices": {}}
    for code, name in enumerate(REGIONS):
        m = X[:, 3] == code
        lo, hi = bootstrap_auc(y[m], p[m])
        report["slices"][name] = {"n": int(m.sum()), "auc": roc_auc_score(y[m], p[m]),
                                  "auc_ci": [lo, hi]}
    failures = []
    if report["auc"] < CONTRACT["auc_min"]:
        failures.append(f"auc {report['auc']:.3f} < {CONTRACT['auc_min']}")
    if report["ece"] > CONTRACT["ece_max"]:
        failures.append(f"ece {report['ece']:.3f} > {CONTRACT['ece_max']}")
    for name, s in report["slices"].items():
        if s["n"] >= CONTRACT["min_slice_n"] and s["auc_ci"][0] < CONTRACT["slice_auc_lower_ci_min"]:
            failures.append(f"slice {name}: AUC lower CI {s['auc_ci'][0]:.3f}")
    report["failures"] = failures
    return report


if __name__ == "__main__":
    r = evaluate(*sys.argv[1:2])
    print(f"held-out n={r['n']}  AUC={r['auc']:.3f}  Brier={r['brier']:.3f}  ECE={r['ece']:.3f}")
    for name, s in r["slices"].items():
        print(f"  {name:<6} n={s['n']:>5}  AUC={s['auc']:.3f}  95% CI [{s['auc_ci'][0]:.3f}, {s['auc_ci'][1]:.3f}]")
    json.dump(r, open("eval_report.json", "w"), indent=2, default=float)
    print("PASS" if not r["failures"] else "FAIL: " + "; ".join(r["failures"]))
    sys.exit(1 if r["failures"] else 0)
$ python eval_harness.py
held-out n=10000  AUC=0.747  Brier=0.200  ECE=0.014
  north  n= 3989  AUC=0.749  95% CI [0.735, 0.764]
  south  n= 2920  AUC=0.759  95% CI [0.740, 0.778]
  east   n= 2065  AUC=0.735  95% CI [0.714, 0.756]
  west   n= 1026  AUC=0.723  95% CI [0.694, 0.753]
PASS
$ python eval_harness.py model_unconstrained.joblib
held-out n=10000  AUC=0.747  Brier=0.200  ECE=0.012
  ...
PASS

Note the widths: the west slice, with 1,026 cases, has an interval almost twice as wide as north's. Both models pass. On these numbers alone you could not choose between them, which is underspecification in miniature: one pipeline, many models with the same held-out score [2].

Step 4: property-based and metamorphic tests

A property-based test states a rule and lets a library generate the inputs. Hypothesis generates hundreds of cases, and when one fails it shrinks it to the simplest failing example [3]. A metamorphic test is a property that relates two runs: you do not know the right score for an applicant, but you know that adding income must not lower it. This idea comes from software testing, where it was developed for programs without a reliable test oracle [4]; CheckList applied the same idea to NLP models as invariance and directional tests [5].

"""Property-based and metamorphic tests: rules that must hold for every input, not just the test set."""
import os
import joblib
import numpy as np
from hypothesis import given, settings, strategies as st

MODEL = joblib.load(os.environ.get("MODEL", "model.joblib"))

applicant = st.tuples(
    st.floats(5, 500),                 # income_k
    st.floats(0, 1),                   # debt_ratio
    st.integers(0, 45),                # years_employed
    st.integers(0, 3),                 # region code
)


def proba(rows):
    return MODEL.predict_proba(np.asarray(rows, dtype=float))[:, 1]


@settings(max_examples=300, deadline=None)
@given(applicant)
def test_output_is_a_probability(a):
    p = proba([a])
    assert p.shape == (1,) and 0.0 <= p[0] <= 1.0


@settings(max_examples=300, deadline=None)
@given(applicant, st.floats(0.1, 200))
def test_more_income_never_lowers_score(a, extra):          # metamorphic relation
    richer = (a[0] + extra, *a[1:])
    assert proba([richer])[0] >= proba([a])[0] - 1e-9


@settings(max_examples=300, deadline=None)
@given(applicant, st.floats(0.01, 0.5))
def test_more_debt_never_raises_score(a, extra):
    worse = (a[0], min(1.0, a[1] + extra), *a[2:])
    assert proba([worse])[0] <= proba([a])[0] + 1e-9


@settings(max_examples=100, deadline=None)
@given(st.lists(applicant, min_size=2, max_size=20), st.randoms())
def test_batch_order_does_not_change_answers(rows, rnd):    # invariance
    perm = list(range(len(rows)))
    rnd.shuffle(perm)
    p = proba(rows)
    assert np.allclose(proba([rows[i] for i in perm]), p[perm])
$ python -m pytest -q test_properties.py
....                                                                     [100%]
4 passed in 4.51s

$ MODEL=model_unconstrained.joblib python -m pytest -q test_properties.py
.FF.                                                                     [100%]
E       assert np.float64(0.8262130572178492) >= (np.float64(0.8346077610546677) - 1e-09)
E       Failing test case: test_more_income_never_lowers_score(
E           a=(33.0, 0.0, 7, 0),
E           extra=1.0,
E       )
...
FAILED test_properties.py::test_more_income_never_lowers_score - assert np.fl...
FAILED test_properties.py::test_more_debt_never_raises_score - assert np.floa...
2 failed, 2 passed in 6.91s

The unconstrained model, with the same AUC, scores an applicant earning £33k at 0.835 and the same applicant earning £34k at 0.826. Hypothesis found it and shrank it to round numbers. Your failing example may differ, as generation is randomised; Hypothesis stores failures in its local database and replays them first on the next run. The fix here was to build the rule into the model; where you cannot, the test still tells you and the gate blocks the release.

Step 5: calibration and conformal prediction

A score of 0.8 should mean that about 80% of such applicants repay. Modern neural networks are often over-confident [6]; our trees were already well calibrated (ECE 0.014). If yours are not, scikit-learn's CalibratedClassifierCV fits a correction on held-out data.

Conformal prediction goes further: it turns any model's scores into prediction sets that contain the true label with a chosen probability, here 90%, assuming only that new data is exchangeable with the calibration data (roughly, drawn from the same population) [7]. It takes three steps in NumPy.

"""Split conformal prediction in plain numpy: prediction sets with a coverage guarantee."""
import joblib
import numpy as np
from data import make_loans

model = joblib.load("model.joblib")
alpha = 0.10                                   # target: true label in the set 90% of the time

X_cal, y_cal = make_loans(2_000, seed=2)       # calibration data, unseen in training
X_test, y_test = make_loans(10_000, seed=3)

# 1. Non-conformity score: 1 - probability the model gave to the true class
p_cal = model.predict_proba(X_cal)
scores = 1 - p_cal[np.arange(len(y_cal)), y_cal]

# 2. Finite-sample-corrected quantile of the calibration scores
n = len(scores)
qhat = np.quantile(scores, np.ceil((n + 1) * (1 - alpha)) / n, method="higher")

# 3. Prediction set: every class whose score is within qhat
p_test = model.predict_proba(X_test)
sets = (1 - p_test) <= qhat                    # boolean matrix, shape (n_test, 2)

covered = sets[np.arange(len(y_test)), y_test].mean()
size = sets.sum(axis=1)
print(f"qhat={qhat:.3f}  empirical coverage={covered:.3f} (target {1 - alpha:.2f})")
print(f"singleton sets={np.mean(size == 1):.1%}  both classes={np.mean(size == 2):.1%}  empty={np.mean(size == 0):.1%}")

# The guarantee is an average over calibration draws, so check it over many draws
cov = []
for s in range(100, 150):
    Xc, yc = make_loans(2_000, seed=s)
    sc = 1 - model.predict_proba(Xc)[np.arange(len(yc)), yc]
    q = np.quantile(sc, np.ceil((len(sc) + 1) * (1 - alpha)) / len(sc), method="higher")
    cov.append(((1 - p_test) <= q)[np.arange(len(y_test)), y_test].mean())
print(f"over 50 calibration draws: mean coverage={np.mean(cov):.3f}  min={np.min(cov):.3f}  max={np.max(cov):.3f}")
$ python conformal.py
qhat=0.689  empirical coverage=0.897 (target 0.90)
singleton sets=49.7%  both classes=50.3%  empty=0.0%
over 50 calibration draws: mean coverage=0.899  min=0.885  max=0.913

Two lessons. First, the guarantee holds on average over calibration draws, not for every single one: individual draws ranged from 0.885 to 0.913, and the mean was 0.899 against a fixed test set that adds its own noise. Second, and more useful to the business, half the applicants get the set {repay, default}: the model cannot decide at 90% confidence. That is your human-review queue, and its size is a cost you can now state.

Split conformal prediction: score the calibration set, take the corrected 90th percentile of non-conformity scores as qhat equals 0.689, then include every class whose score is within qhat. Confident cases get one label; uncertain cases get both and go to human review.
Split conformal prediction in three steps. Uncertain cases get both labels and go to a person; the share of such cases (50.3% here) is a number you can put in front of the business.

MAPIE wraps the same method, with more conformity scores and regression support. Version 1 changed the API; this is the 1.x form [8]:

"""The same thing with MAPIE 1.x (prefit model, LAC score)."""
import joblib
import numpy as np
from mapie.classification import SplitConformalClassifier
from data import make_loans

X_cal, y_cal = make_loans(2_000, seed=2)
X_test, y_test = make_loans(10_000, seed=3)
mapie = SplitConformalClassifier(estimator=joblib.load("model.joblib"),
                                 confidence_level=0.90, prefit=True)
mapie.conformalize(X_cal, y_cal)
_, sets = mapie.predict_set(X_test)            # sets: (n, n_classes, n_confidence_levels)
sets = sets[:, :, 0]
print(f"MAPIE coverage={sets[np.arange(len(y_test)), y_test].mean():.3f}  "
      f"mean set size={sets.sum(axis=1).mean():.2f}")
MAPIE coverage=0.897  mean set size=1.50

Same coverage, same mean set size (half singletons, half pairs): the NumPy version and the library agree.

Step 6: output contracts and regression evals for an LLM step

An LLM's output is text; your system needs data. Treat every response as untrusted input. Pydantic validates structure (types, allowed values, required fields) and lets you add deterministic business checks such as "net plus VAT equals total" [9].

"""An output contract for an LLM step: schema, deterministic business checks, and a regression eval."""
import json
from datetime import date
from typing import Literal
from pydantic import BaseModel, Field, ValidationError, field_validator, model_validator


class InvoiceExtraction(BaseModel):
    supplier: str = Field(min_length=2, max_length=120)
    invoice_date: date
    currency: Literal["GBP", "EUR", "USD"]
    net: float = Field(ge=0)
    vat: float = Field(ge=0)
    total: float = Field(gt=0)

    @field_validator("invoice_date")
    @classmethod
    def not_in_future(cls, v: date) -> date:
        if v > date.today():
            raise ValueError("invoice date is in the future")
        return v

    @model_validator(mode="after")
    def totals_add_up(self):
        if abs(self.net + self.vat - self.total) > 0.01:
            raise ValueError(f"net + vat = {self.net + self.vat:.2f} but total = {self.total:.2f}")
        return self


def check(raw: str) -> tuple[bool, str]:
    """Never trust the model's text: parse, validate, and say exactly what failed."""
    try:
        InvoiceExtraction.model_validate_json(raw)
        return True, "ok"
    except ValidationError as e:
        return False, "; ".join(f"{'.'.join(map(str, err['loc'])) or 'model'}: {err['msg']}"
                                for err in e.errors())


# Three outputs an LLM really might produce for the same invoice
outputs = {
    "good":      '{"supplier":"Acme Ltd","invoice_date":"2026-10-31","currency":"GBP","net":100.0,"vat":20.0,"total":120.0}',
    "bad_maths": '{"supplier":"Acme Ltd","invoice_date":"2026-10-31","currency":"GBP","net":100.0,"vat":20.0,"total":102.0}',
    "bad_shape": '{"supplier":"Acme Ltd","invoice_date":"31/10/2026","currency":"£","net":"100","vat":20.0}',
}
if __name__ == "__main__":
    for name, raw in outputs.items():
        ok, why = check(raw)
        print(f"{name:<10} {'PASS' if ok else 'FAIL'}  {why}")
$ python llm_contract.py
good       PASS  ok
bad_maths  FAIL  model: Value error, net + vat = 120.00 but total = 102.00
bad_shape  FAIL  invoice_date: Input should be a valid date or datetime, invalid character in year; currency: Input should be 'GBP', 'EUR' or 'USD'; total: Field required

Notice what was not flagged: "net":"100", a string, was accepted and converted, because Pydantic's default lax mode coerces numeric strings. If you want the model to produce exact types, set model_config = ConfigDict(strict=True). Then route failures: retry once with the error message in the prompt, and send repeated failures to a person.

Validation proves shape, not truth. For truth you need a golden set: real cases with answers checked by a person, scored field by field, compared with the release in production before any prompt or model change ships.

"""Regression eval for an LLM step: golden cases, field-level scoring, compared against the last release."""
import json
import sys
from llm_contract import InvoiceExtraction, check

GOLDEN = [  # expected fields per case, written and reviewed by a person
    {"id": "inv-001", "expect": {"supplier": "Acme Ltd", "total": 120.0, "currency": "GBP"}},
    {"id": "inv-002", "expect": {"supplier": "Brightwater plc", "total": 54.0, "currency": "GBP"}},
    {"id": "inv-003", "expect": {"supplier": "Nordlicht GmbH", "total": 238.0, "currency": "EUR"}},
    {"id": "inv-004", "expect": {"supplier": "Kestrel Ltd", "total": 960.0, "currency": "GBP"}},
]


def extract(case_id: str, outputs: dict) -> str:
    """Replace with your model call (temperature 0, pinned model version, logged prompt)."""
    return outputs[case_id]


def run(outputs: dict) -> dict:
    valid = fields_ok = fields_total = 0
    for case in GOLDEN:
        raw = extract(case["id"], outputs)
        ok, _ = check(raw)
        valid += ok
        got = InvoiceExtraction.model_validate_json(raw).model_dump() if ok else {}
        for k, v in case["expect"].items():
            fields_total += 1
            fields_ok += got.get(k) == v
    return {"schema_valid": valid / len(GOLDEN), "field_accuracy": fields_ok / fields_total}


if __name__ == "__main__":
    baseline = json.load(open("baseline.json"))          # scores of the release in production
    candidate = run(json.load(open(sys.argv[1])))       # recorded outputs of the new prompt/model
    print("baseline ", {k: round(v, 3) for k, v in baseline.items()})
    print("candidate", {k: round(v, 3) for k, v in candidate.items()})
    bad = [k for k in candidate if candidate[k] < baseline[k] - 0.02]
    print("PASS" if not bad else f"FAIL: regression in {bad}")
    sys.exit(1 if bad else 0)
$ python regression_eval.py outputs_v2.json
baseline  {'schema_valid': 1.0, 'field_accuracy': 1.0}
candidate {'schema_valid': 0.75, 'field_accuracy': 0.667}
FAIL: regression in ['schema_valid', 'field_accuracy']

Here outputs_v1.json and outputs_v2.json hold recorded responses for four invoices (in v2, one currency is wrong and one total does not add up), so the example runs without an API key. In your project, extract() calls the model with temperature 0 and a pinned model version, and you store the raw responses with the score so a failure can be inspected later. Four cases is a demonstration; a useful golden set has a few hundred, chosen to cover your slices and your past incidents.

Step 7: drift monitoring

Every guarantee above assumes production data looks like your evaluation data. Check that it still does. The population stability index (PSI) measures how far a distribution has moved across fixed bins; the two-sample Kolmogorov-Smirnov (KS) test asks whether two samples could come from the same distribution.

"""Drift checks on inputs and scores: population stability index and the two-sample KS test."""
import joblib
import numpy as np
from scipy.stats import ks_2samp
from data import make_loans, FEATURES


def psi(expected, actual, bins=10):
    """Population stability index over quantile bins of the reference data."""
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))


model = joblib.load("model.joblib")
X_ref, _ = make_loans(10_000, seed=1)                       # what the model was validated on
for label, shift in [("same population", 0.0), ("incomes 20% higher", np.log(1.2))]:
    X_live, _ = make_loans(5_000, seed=7, income_shift=shift)  # a week of production traffic
    print(label)
    for j, name in enumerate(FEATURES[:3]):
        ks = ks_2samp(X_ref[:, j], X_live[:, j])
        print(f"  {name:<15} PSI={psi(X_ref[:, j], X_live[:, j]):.3f}  KS p={ks.pvalue:.3g}")
    s_ref, s_live = model.predict_proba(X_ref)[:, 1], model.predict_proba(X_live)[:, 1]
    print(f"  {'score':<15} PSI={psi(s_ref, s_live):.3f}  mean {s_ref.mean():.3f} -> {s_live.mean():.3f}")
$ python drift.py
same population
  income_k        PSI=0.002  KS p=0.484
  debt_ratio      PSI=0.001  KS p=0.33
  years_employed  PSI=0.005  KS p=0.0319
  score           PSI=0.003  mean 0.572 -> 0.573
incomes 20% higher
  income_k        PSI=0.126  KS p=6.14e-59
  debt_ratio      PSI=0.001  KS p=0.33
  years_employed  PSI=0.005  KS p=0.0319
  score           PSI=0.066  mean 0.572 -> 0.627

The shifted income is caught by both measures, and the score distribution moves with it. But look at years_employed on the same population: KS gives p = 0.03, "significant" at the usual 5% level, with nothing wrong. Run enough tests every day and some will fire by chance; with large samples, tiny harmless shifts also become significant. Alert on effect sizes such as PSI (a common rule of thumb in credit scoring treats below 0.1 as stable and above 0.25 as a major shift; tune it on your own history) and, above all, on outcomes once labels arrive.

Step 8: formal verification of a small network

For a small ReLU network you can prove statements such as "for every applicant with income £50k to £200k, debt ratio at most 0.25 and any employment history, the network says likely to repay". Two methods, both in under 100 lines:

  • Interval bound propagation pushes the input box through each layer. It is sound (if it says the property holds, it does) but incomplete (it can fail to prove a true property, because the bounds are loose).
  • Mixed-integer linear programming encodes each ReLU exactly with one binary variable and asks a solver for the true minimum of the output over the box. It is complete, but the problem is NP-complete in general [10], so it only scales to small networks. SciPy's milp uses the HiGHS solver [11].
"""Formal verification of a small ReLU network: interval bounds (fast, sound, incomplete)
and an exact mixed-integer program (complete) with scipy.optimize.milp."""
import sys
import numpy as np
from scipy.optimize import milp, LinearConstraint, Bounds
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from data import make_loans

X, y = make_loans(20_000, seed=0)
X = X[:, :3]                                            # income_k, debt_ratio, years_employed
scaler = StandardScaler().fit(X)
net = MLPClassifier(hidden_layer_sizes=(8, 8), activation="relu", max_iter=500,
                    random_state=0).fit(scaler.transform(X), y)
layers = list(zip(net.coefs_, net.intercepts_))         # last layer gives the logit


def to_scaled(lo, hi):
    return (np.array(lo) - scaler.mean_) / scaler.scale_, (np.array(hi) - scaler.mean_) / scaler.scale_


def interval_bounds(lo, hi):
    """Push a box through the network; returns pre-activation bounds per layer."""
    bounds = []
    for k, (W, b) in enumerate(layers):
        Wp, Wn = np.clip(W, 0, None), np.clip(W, None, 0)
        l, u = lo @ Wp + hi @ Wn + b, hi @ Wp + lo @ Wn + b
        bounds.append((l, u))
        lo, hi = np.maximum(l, 0), np.maximum(u, 0)
    return bounds


def min_logit_milp(lo, hi):
    """Exact minimum of the logit over the box. Each unstable ReLU a = max(z, 0) becomes
    linear constraints plus one binary d (d=1: active), using the interval bounds l <= z <= u."""
    pre = interval_bounds(lo, hi)
    n_in = len(lo)
    sizes = [W.shape[1] for W, _ in layers]
    # variable layout: x | (z, a, d) per hidden layer | logit
    idx, off = {}, n_in
    for k, n in enumerate(sizes[:-1]):
        idx[k] = (off, off + n, off + 2 * n); off += 3 * n
    out = off; nv = off + 1
    rows, lb, ub = [], [], []
    def add(coef, l, u):
        r = np.zeros(nv)
        for i, c in coef: r[i] += c
        rows.append(r); lb.append(l); ub.append(u)
    prev = list(range(n_in))
    for k, (W, b) in enumerate(layers):
        if k == len(layers) - 1:                         # logit = W a + b
            add([(prev[i], W[i, 0]) for i in range(len(prev))] + [(out, -1)], -b[0], -b[0])
            break
        z0, a0, d0 = idx[k]
        l, u = pre[k]
        for j in range(W.shape[1]):
            add([(prev[i], W[i, j]) for i in range(len(prev))] + [(z0 + j, -1)], -b[j], -b[j])
            add([(a0 + j, 1), (z0 + j, -1)], 0, np.inf)                  # a >= z
            add([(a0 + j, 1), (d0 + j, -max(u[j], 0))], -np.inf, 0)       # a <= u d
            add([(a0 + j, 1), (z0 + j, -1), (d0 + j, -min(l[j], 0))], -np.inf, -min(l[j], 0))  # a <= z - l(1-d)
        prev = list(range(a0, a0 + W.shape[1]))
    vlo, vhi, integ = np.full(nv, -np.inf), np.full(nv, np.inf), np.zeros(nv)
    vlo[:n_in], vhi[:n_in] = lo, hi
    for k in idx:
        z0, a0, d0 = idx[k]; n = sizes[k]
        vlo[a0:a0 + n] = 0; vlo[d0:d0 + n] = 0; vhi[d0:d0 + n] = 1; integ[d0:d0 + n] = 1
    c = np.zeros(nv); c[out] = 1                         # minimise the logit
    res = milp(c, constraints=LinearConstraint(np.array(rows), lb, ub),
               integrality=integ, bounds=Bounds(vlo, vhi))
    assert res.success, res.message                      # never report a bound from a failed solve
    return res.fun, res.x[:n_in] * scaler.scale_ + scaler.mean_


# Property: every applicant in the box must be scored "likely to repay" (logit > 0).
# required=True boxes are policy; the last one is exploratory, to find where the policy stops holding.
PROPERTIES = [
    ("income 80-200k, debt <= 0.15, 5-30 yrs", [80, 0.0, 5], [200, 0.15, 30], True),
    ("income 50-200k, debt <= 0.25, 0-30 yrs", [50, 0.0, 0], [200, 0.25, 30], True),
    ("income 40-200k, debt <= 0.30, 0-30 yrs", [40, 0.0, 0], [200, 0.30, 30], False),
]
failed = False
for name, box_lo, box_hi, required in PROPERTIES:
    lo, hi = to_scaled(box_lo, box_hi)
    ibp_lower = interval_bounds(lo, hi)[-1][0][0]
    exact, worst = min_logit_milp(lo, hi)
    verdict = "VERIFIED" if exact > 0 else "COUNTEREXAMPLE"
    if ibp_lower > 0:
        verdict += " (intervals alone were enough)"
    print(f"{name}\n  interval lower bound={ibp_lower:+.3f}  exact min logit={exact:+.3f}  {verdict}")
    if exact <= 0:
        print(f"  worst case: income {worst[0]:.1f}k, debt {worst[1]:.3f}, {worst[2]:.1f} yrs")
        failed |= required
    # cross-check the verifier: no random sample may beat the exact minimum
    S = np.random.default_rng(0).uniform(box_lo, box_hi, (100_000, 3))
    h = scaler.transform(S)
    for W, b in layers[:-1]:
        h = np.maximum(h @ W + b, 0)
    print(f"  lowest logit in 100,000 random samples={(h @ layers[-1][0] + layers[-1][1]).min():+.3f}")
sys.exit(1 if failed else 0)                             # a required property failed
$ python verify_tiny_net.py
income 80-200k, debt <= 0.15, 5-30 yrs
  interval lower bound=+0.254  exact min logit=+1.942  VERIFIED (intervals alone were enough)
  lowest logit in 100,000 random samples=+1.960
income 50-200k, debt <= 0.25, 0-30 yrs
  interval lower bound=-1.573  exact min logit=+0.589  VERIFIED
  lowest logit in 100,000 random samples=+0.626
income 40-200k, debt <= 0.30, 0-30 yrs
  interval lower bound=-2.774  exact min logit=-0.273  COUNTEREXAMPLE
  worst case: income 40.0k, debt 0.300, 0.0 yrs
  lowest logit in 100,000 random samples=-0.231
Three number lines, one per input box, comparing the interval lower bound, the exact minimum from the solver and the lowest of 100,000 random samples against zero. Box one: all above zero. Box two: interval bound below zero but exact minimum above, so verified. Box three: exact minimum below zero, a counterexample.
Interval bounds are fast but loose; the exact solver settles the middle case. Random sampling never beats the exact minimum, which is a useful check that the encoding is right, but sampling alone could never prove the property.

Read the three rows. The first property is proved by intervals alone. The second is true, but intervals could not show it; the exact solver could. The third is false, and the solver returns the worst applicant, a concrete counterexample to discuss with the business. Random sampling came close to the true minimum each time but never reached it, which is why sampling is a sanity check and not a proof.

The limits: this network has 16 hidden neurons. For larger networks use a dedicated verifier: α,β-CROWN, which has won VNN-COMP every year from 2021 to 2025 and reports scaling to convolutional networks with millions of parameters on local properties, or Marabou [12][13]. None of them proves anything about a large language model's answers. And a proof covers one model version and one input box: re-run it on every retrain, in the gate.

Step 9: the release gate in CI

Evidence that is not enforced decays. gate.sh runs everything above and stops at the first failure:

#!/usr/bin/env bash
# Release gate: every check must pass before a model or prompt change can ship.
set -euo pipefail
export OMP_NUM_THREADS=1          # stop OpenMP oversubscription in per-row tests
python model.py
python eval_harness.py model.joblib          # held-out + slice contract
python -m pytest -q test_properties.py       # properties and metamorphic relations
python conformal.py                          # coverage report (informational)
python regression_eval.py outputs_v1.json    # LLM step: no regression against production
python verify_tiny_net.py                    # formal check: required boxes must verify
echo "ALL GATES PASSED"

A minimal GitHub Actions workflow that runs it on every pull request (shown for shape; we ran gate.sh locally, not in Actions):

name: model-gate
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install numpy scipy scikit-learn hypothesis pydantic pytest mapie
      - run: ./gate.sh
      - uses: actions/upload-artifact@v7
        if: always()
        with: { name: evidence, path: eval_report.json }
A release candidate passes through five gates in turn: evaluation contract, property tests, conformal coverage report, LLM regression eval and formal verification. A candidate that fails any gate is stopped and the report is kept.
The gate in CI: a candidate model or prompt moves through each check in order; any failure stops it, and the reports are kept as the release record.

Running it in production

  • Pin everything. Pin library versions in a lock file and record the model artefact's hash, the data snapshot and the git commit in eval_report.json. A report you cannot tie to a model version is not evidence.
  • Containers. Run the gate in the same image you deploy, so the tests see the same NumPy and scikit-learn builds as production. All of this runs on CPU; the property tests and verifier finished in seconds on a laptop.
  • Threads. Set OMP_NUM_THREADS deliberately in both CI and serving. On our shared build machine, one single-row prediction took 150 ms with default threading and 1.3 ms with one thread.
  • Monitoring. Schedule drift.py against a daily sample of logged inputs and scores, export PSI per feature as metrics to your existing monitoring stack, and alert through the normal on-call route. Join outcomes to predictions when labels arrive and re-run the evaluation harness on them.
  • Go further. Google's ML Test Score lists 28 tests and monitoring checks for production readiness and is a good audit of what you have missed [17]; Sculley et al. explain why the data pipelines and glue code around a model need the same care as the model [16].
  • Keep the record. Store every gate run's output with the release. When an auditor or a customer asks how you knew the model was fit for use, this is the answer.

Pitfalls and debugging

  • Hypothesis tests are very slow. Thread oversubscription in OpenMP; set OMP_NUM_THREADS=1. Also set deadline=None, because model calls have variable latency.
  • A property test fails on a tiny difference. Floating-point noise; compare with a tolerance (we use 1e-9) and decide what difference matters to the business.
  • Conformal coverage is below target. A single calibration draw can be; check the average over draws. If it is far below, your calibration data is not exchangeable with production, often because of a time trend: calibrate on recent data.
  • The MILP is slow or fails. Tighten input bounds, because loose bounds make weak encodings, and check res.success. Past a few hundred neurons, move to a dedicated verifier.
  • The LLM regression eval flaps. Set temperature to 0, pin the model version, and if outputs still vary, run each case several times and gate on the pass rate.

How to evaluate the evidence itself

Test your tests. Plant a known fault and confirm the gate catches it: the unconstrained model is exactly that for the property tests, outputs_v2.json is one for the LLM eval, and the shifted incomes are one for drift. Keep these planted faults in the repository and run them in CI as tests that are expected to fail, so a broken check cannot pass silently.

Alternatives

NeedWhat we usedAlternativesWhen to switch
Evaluation harnessPlain scikit-learn and NumPyMLflow evaluation, your experiment tracker's eval featuresMany models and teams; you need a UI and history
Property testsHypothesisCheckList (NLP behavioural tests)Text models where perturbations are linguistic
UncertaintyNumPy split conformal, MAPIECalibratedClassifierCV for calibration onlyYou need calibrated scores, not sets
LLM outputPydantic plus a golden setProvider structured-output modes, LLM eval frameworksMany prompts; you want dashboards and LLM-as-judge scoring
DriftPSI and KS in SciPyEvidently and similar monitoring librariesMany features; you want reports and dashboards
VerificationInterval bounds and SciPy MILPα,β-CROWN, MarabouNetworks beyond a few hundred neurons

FAQ

Is this enough for the EU AI Act? It produces the kind of evidence Articles 9 and 15 ask for (testing against prior defined metrics and probabilistic thresholds, declared accuracy, robustness), but compliance also needs risk management, data governance, documentation and human oversight processes [14]. After the Digital Omnibus, stand-alone Annex III high-risk obligations apply from 2 December 2027 [15].

Do I need all of this for every model? No. Scale it to the risk: an internal search ranker might need steps 3, 6 and 7; a credit decision needs all of them, plus independent validation.

Can I verify an LLM formally? Not its answers. You can verify the deterministic code around it, constrain its output format, and verify small classifiers that sit next to it, such as a router or a guard model.

Why synthetic data? So every test has a known right answer. Swap in your data and keep the structure; the contract thresholds will change.

If you would like help putting evaluation harnesses, release gates and monitoring around your own models, see our MLOps setup service.

References

  1. scikit-learn user guide: histogram-based gradient boosting, monotonic constraints
  2. D'Amour et al., "Underspecification presents challenges for credibility in modern machine learning", arXiv 2011.03395 (JMLR, 2022)
  3. Hypothesis documentation
  4. Chen et al., "Metamorphic testing: a review of challenges and opportunities", ACM Computing Surveys, 2018
  5. Ribeiro et al., "Beyond accuracy: behavioral testing of NLP models with CheckList", ACL 2020
  6. Guo et al., "On calibration of modern neural networks", ICML 2017
  7. Angelopoulos and Bates, "A gentle introduction to conformal prediction and distribution-free uncertainty quantification", arXiv 2107.07511
  8. MAPIE documentation
  9. Pydantic documentation: validators
  10. Katz et al., "Reluplex: an efficient SMT solver for verifying deep neural networks", CAV 2017
  11. SciPy documentation: scipy.optimize.milp
  12. α,β-CROWN neural network verifier (GitHub)
  13. Marabou neural network verifier (GitHub)
  14. Regulation (EU) 2024/1689 (Artificial Intelligence Act)
  15. Digital Omnibus on AI, Regulation (EU) 2026/1744: AI Act Explorer
  16. Sculley et al., "Hidden technical debt in machine learning systems", NeurIPS 2015
  17. Breck et al., "The ML Test Score: a rubric for ML production readiness and technical debt reduction", IEEE Big Data 2017
Compartir este artículo

Más en Technology

TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

Install torchcurves, fit and size a curve, choose B-spline or Legendre and the input map, build a monotone calibration curve and an additive model on tabular data, compare with SciPy, scikit-learn, a KAN and an MLP, export coefficients to JSON, test and run it in production, with real outputs

Leer más
Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Install cactus-compute, make tool calls, gate on confidence with a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, and ship an offline container, with every example run and a comparison with llama.cpp, ONNX Runtime, ExecuTorch, MLC LLM and TensorRT Edge-LLM

Leer más
AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

Close the AI guidance gap in code: a versioned policy file, an OpenAI-compatible FastAPI gateway, Presidio redaction, per-team tool and MCP allow-lists, red-team tests, an audit log, Prometheus metrics and a usage report

Leer más