Compare decision models

Jev vs Laya vs Kev vs d1

Start with the inputs and deployment route your app needs. Jev is hosted; Laya and Kev have public weights, while d1 spans hosted services and open siblings. A family name does not guarantee the same inputs, licence or accuracy across its checkpoints.

Decision model comparisons by task

Use a comparison that measures the decision your app needs. The tests below compare different systems on different tasks; their scores cannot form a single decision models leaderboard. Keep each result attached to its model version, supplied evidence and error budget.

Jev, Laya, Kev and d1 access comparison · Separate benchmark boards and methods · Cost per decision and billing units

Decision model API examples

Start from the example for your endpoint and model. A shared question type does not make TypeSafe, OpenAI and Parallel requests interchangeable. The examples below have different jobs and different evidence of execution.

Local runtime choices · Evaluate on your own task

Decision model API limits

A context window, a validated task length and a question limit describe different constraints. Check the exact model, version and endpoint before sending a large batch. Missing limits here mean unrecorded, not unlimited.

Where decision model integrations run

A model endpoint, an MCP tool and a hosted cloud service connect at different layers. The projects below record working routes. Their results do not establish support for a different provider or client.

Evaluate on your own task

Use this checklist before choosing a model or an automation threshold. The cited results explain the checks; they are not performance targets for your workload.

Decision models versus classifiers · Repeated-call and batching evidence

1. Collect labelled examples of the decisions your app will make, including errors it must avoid. Keep threshold-fitting examples separate from the final test. Record dataset reuse rights and keep final questions out of prompt tuning. SILT registered its plan before testing 799 unpublished questions; its calibration conclusions applied to that question set and rule.

   Evidence: SILT · Do Decision Models Know When They're Wrong? · 11 Oct 2026 · Source ↗

2. Give each model the same state, options and external context. If one service searches the web, evaluate that as a separate system condition. Parallel's PWS-Decisions gave its own API live search while Jev and other competitors had neither web access nor supplied external context. VENDOR-REPORTED: its ranking measures those different system conditions.

   Evidence: Parallel · PWS-Decisions v0.1.0-rc1 · blog, repository results and dataset terms · 01 Nov 2026 · Source ↗

3. Repeat each question with reordered options and track which individual answers change, alongside aggregate accuracy. In the 1,700-question option-order study, Jev's accuracy moved by under two points while the set of questions it ever got wrong grew by 27% to 29%. Compare against identical-order repetition to separate order effects from repeat noise.

   Evidence: Song, Zheng, Ma, Zhou · Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? · arXiv 2610.35826 · 26 Oct 2026 · Source ↗

4. Compare predicted probabilities with observed outcomes on your labels. Report calibration separately for the question types you use. Do not treat the training goal as a calibration result: SILT found only one of seven models passed its chosen calibration rule; Jev's result was inconclusive.

   Evidence: SILT · Do Decision Models Know When They're Wrong? · 11 Oct 2026 · Source ↗

5. Define the cost of false accepts and false rejects, then fit a threshold on calibration data. On untouched test examples, report both the share automated and its error rate. With Vals AI's 1% claim-verification error budget, Jev automated 95% at 1.6% error, over budget. Mercury Decide automated 94% at 0.8% error.

   Evidence: Vals AI · Jev evaluation 6 Oct 2026, Mercury Decide evaluation 8 Oct 2026 · Source ↗

6. Measure accuracy and threshold errors on labelled examples in each language you will serve. XDecideBench found that the English-best model, chosen in hindsight, lost significantly in 56 of 232 suite-language cells. A threshold fitted for 5% errors in English admitted 7.8% to 12.4% on German and French XNLI.

   Evidence: Marek Šuppa, Jaroslav Kopčan · XDecideBench · arXiv 2611.01370 · 01 Nov 2026 · Source ↗

7. Pin model versions and retain the full request, prompt, option order, batch composition and response. Run identical requests repeatedly, then rebatch the same inputs. In pgdocket's dated Jev 1.13 observations on 2,000 complaints, 4.8% changed top answers across three identical runs; rebatching changed about 16.5%. Those figures do not establish every model's determinism.

   Evidence: ElyasEbrahim · pgdocket Phase 0 · 17 to 19 Oct 2026 · Source ↗

8. Keep billed usage and end-to-end timing for each tested configuration, including batching and fallback calls. Report uncertainty alongside accuracy and automation coverage. Vals AI reports paired intervals, held-out automation errors, and latency at different batch sizes. pgdocket's single-row configuration changed accuracy, token use and elapsed time relative to batching, so a batch-size change needs a quality check too.

   Evidence: Vals AI · Jev evaluation 6 Oct 2026, Mercury Decide evaluation 8 Oct 2026 · Source ↗ · Evidence: ElyasEbrahim · pgdocket Phase 0 · 17 to 19 Oct 2026 · Source ↗