Key takeaways
- Select by workload and constraints, not by launch blogs.
- Use the matrix as a starting hypothesis.
- Measure on your data; keep historical eval versions.
Why this matters
Model selection fails when teams chase leaderboards. Workloads differ. This manual teaches a decision framework. It is not a permanent ranking of brands.
Decision framework
- Does the step need to generate novel text/code/images?
- If no: consider decision models / classifiers / rankers.
- If yes: does it need deep multi-step reasoning?
- If yes: reasoning model. If no: fast/small LLM may suffice.
- Add constraints: latency SLO, privacy, tool use, modality, offline/on-prem.
- Measure on your golden set before committing.
Selection matrix
| Workload | Candidate model type | Key metric |
|---|---|---|
| Classification | Decision model / classifier | Accuracy + latency |
| Routing | Decision model | Correct routing + escalation quality |
| Tool safety | Decision model | False negatives |
| Simple extraction | Small/fast LLM or decision model | Accuracy/cost |
| Coding | Coding / agentic model | Task success |
| Deep reasoning | Reasoning model | Correctness |
| Creative writing | Generative LLM | Quality rubrics |
| RAG generation | LLM | Answer quality |
| RAG evaluation | Judge / decision model | Calibration |
| Real-time voice | Fast multimodal | End-to-end latency |
| Long-running agent | Agentic frontier model + harness | Task completion |
| High-volume automation | Fast decision model | Cost × accuracy |
Evaluation dimensions
- Intelligence & reasoning depth
- Latency & tail latency
- Cost & cost per success
- Context window needs
- Structured output reliability
- Tool use / MCP fitness
- Coding ability
- Multimodal needs
- Reliability & uptime
- Calibration
- Privacy / data residency
- Deployment model
- Rate limits
- Observability hooks
- Safety tooling
- Task-specific accuracy
Production recommendations
Keep a written model policy. Version it. Revisit when providers change defaults. Prefer explicit model ids over moving aliases in production paths that encode thresholds.
Conclusion
The right model is the cheapest model that meets a measured quality bar under your latency and risk constraints — sometimes that is Jev-like, sometimes it is a frontier reasoner, often it is both in one system.
Worked example 1: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 2: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 3: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 4: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 5: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 6: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 7: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 8: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 9: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 10: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 11: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Worked example 12: define the user journey, list each decision or generation step, assign a candidate model type from the matrix, write the metric, and only then pick a vendor. Skipping the matrix step is how organisations accidentally spend premium reasoning tokens on boolean gates. Document the owner of each step so model policy changes have a human inbox. Re-run a fixed eval harness monthly; store results beside the policy document. If a step is both generative and decisive, split it. If a step is high volume, invest in calibration. If a step is irreversible, invest in false-negative analysis and human escalation UX — not only in model IQ.
Related on Workstation
- Benchmark Lab — Interactive charts
- Model selection manual
- AI Agents
- AI Automation
- Jobs
Sources
- TypeSafe AI — Introducing System One Models & Jev — Primary vendor source (15 Sep 2026).
- LangChain — Building a Harness with Jev — Harness integration (17 Sep 2026).
- TypeSafe documentation index — Concepts, primitives, API, cookbooks.
- LangChain — Can Jev Be a Better Agent Evaluator? — 20 Sep 2026 LangSmith experiment.
- OpenRouter — Jev vs LLM when to use each — Published triage cost/latency/accuracy.
- AyAutomate — Independent Jev benchmark — 791 decisions; challenges headline multipliers vs small models.
- JevBench RESULTS v1.2 — Public community harness.
- OpenAI — Introducing the Agents API — Landscape: agent platform primitives.