Hybrid Execution Feature Matrix

This matrix compares the thirty-one hybrid-execution notes profiled in this section, feature by feature: two vendor API features that constrain decoding, six members that validate, mask, or repair their way to typed output (Instructor, the Atomic Agents framework built on it, Outlines, BAML, the typed language whose LLM calls are functions with the same error-correcting parser class, Guardrails, the validator-hub framework whose Guards compose installable risk validators around the call, and Marvin, Prefect’s AI-functions toolkit whose four functions cast, classify, extract, and generate typed values), eighteen decision-model implementations of the no-generation contract (TypeSafe’s closed Jev, the seven open answers the community shipped in Jev’s launch week, CUA-S1, Jevlike, Kev, Laya, NanoJev, Nimble, and SemIf, the second-generation lines Jeff and Jeeves, Nokia’s training-free AnyJev, and the board-era additions Decider, H2O-Lightning-4B, Von, Winnow, AWS’s Strands Decider 2B, the architectural dissent CLM, and the full-pipeline multimodal entrant Valen), the local runtime that packages most of them (Ollaya), and four benchmarks that measure the rest of them (JevBench, the Decision Index, tool-eval-bench, the tool-calling auditor, and System1-Arena, the academic arena that hands decision models and generative LLMs identical requests across 92k questions and 138 closed-loop environments).

These thirty are less competitors than mechanisms on one spectrum, and the decision that matters is where the schema guarantee lives, in sampling, in post-hoc checks, or in the architecture itself: I would take decoding-time enforcement everywhere it exists, treat most single-provider Instructor deployments written after 2025 as incidental complexity (and Atomic Agents as the way to keep the pattern when a whole pipeline needs it, with BAML as the language-shaped bet that the whole pattern gets absorbed into the toolchain, Guardrails as the hub-shaped bet that the checking itself becomes installable packages, and Marvin as the smallest-API bet that the loop needs no wrapping at all), and no longer take the architectural class purely on faith, because the launch week put seven independently inspectable open brackets next to Jev’s closed claim, the weeks since produced two outside scoreboards for all of them plus a second generation of fork lines, a reasoning entrant, a training-free readout, a deterministic tool-calling auditor, an academic arena that scores both halves of the contract on identical requests, and a board era that has since promoted a company-built open leader (H2O-Lightning-4B), a multi-size workhorse family (Decider), the community family with the strongest third-party record (Winnow), a CPU-scale encoder (Von), AWS’s eval-first entry (Strands Decider 2B, the best-published training discipline over the weakest board row), and one architectural dissent (CLM) the boards score near zero, while Ollaya packages the open side into one installable runtime.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell below traces to a source cited there or in the references.

The matrix

#

Reading the matrix

#

The guarantee-mechanism row is the distinction that carries the most weight, and every other row is downstream of it. OpenAI, Anthropic, and Outlines enforce the schema while the tokens are being sampled, so an invalid token is never drawn in the first place. Instructor inspects the finished output and re-asks when Pydantic rejects it. Jev, the seventeen open implementations, and the runtime that packages them sit at the end of the spectrum the other eight approach: with no text generation there is nothing to constrain and nothing to validate, which is why their rows guarantee retries and type errors away the same way the decoding-time options do. The difference among the eighteen is verifiability, scope, and evidence, not mechanism: Jev’s numbers began as vendor claims behind a closed API, and the JevBench board is the first outside reading of them (Jev led the v1.2 board at 74.4 and the initial sealed v1.4 revision at 63.3, which dropped SemIf from 73.1 to 47.7, before v1.4.2’s eleven additions moved decider-4b v2, 64.1, half a point ahead and the v1.4.2.1 Plumb-4B, 65.8, and v1.4.2.2 Imajev-4B, 67.4, point releases took the top spot on consecutive days, from one runner whose methodology its own thread contests), CUA-S1 grew from a 2.8 MB MIT checkpoint anyone can re-run into a four-checkpoint family (its card now adds a 196-decision eval over actual forms and a Jev head-to-head, all still vendor-run), Laya is an Apache-2.0 multilingual family whose benchmark comparisons are likewise self-run, the launch-week wave adds a dormancy spectrum from Kev’s pre-registered locked-test harness down to Jevlike’s single day of commits and NanoJev’s unreplicated game benchmarks, the second generation arrived as Jeff’s AutoJev-fork fine-tunes, whose benchmark table matches Jev’s published overall while conceding every reasoning benchmark to it, and Jeeves’s reasoning-then-deciding 9B, which beats the same published numbers on its own tests at ten times the latency, while AnyJev takes the evidence lead with a peer-readable arXiv report, verified early-exit bounds, and a third-party benchmark (local-jev-bench) scoring it level with the trained engines on the public transfer sources. The board era then filled the middle of the table: Decider fields the widest checkpoint range and the deepest (and most humbling) scoreboard history, H2O-Lightning-4B is the company-built 4B that took the open composite lead, Winnow is the family three independent readers agree on, Von puts the contract on CPUs at 395M, and CLM, the one architectural dissent, sees its contrastive-encoder argument scored at the bottom of both boards. Ollaya packages the open side into one installable runtime whose core feature is parity-checking every model against its author’s code. Since October the scoreboard layer has doubled and the boards disagree: JevBench’s open-weights main board passed Jev’s unranked reference (71.5) on 2026-10-08, H2O-Lightning-4B v1.1 taking the open composite lead at 72.5 before the capability-era reshuffle, and the board at revision v1.7.47 (fetched 2026-10-24) now carries 197 ranked open-weights systems ordered by a Capability Score headline (Chon-1.0-12B 85.3 first yet only twenty-eighth by the composite, basal-omni-1.0-ultra 82.5 second, Quyet-1.0-Large 81.7 and Jeff-1.0-Large 81.5 next, the former composite leader H2O twelfth by capability and thirty-sixth by composite), while the split hosted-API board holds 44 ranked rows where kettei-fat leads the official composite and the vendor’s Jev 1.14.0 row now sits unranked as a reference at 70.7 in fifteenth position (fourth by composite on the 2026-10-15 reading, second by capability), and the Decision Index 0.4.1 board (regenerated 2026-10-22, 229 configurations after the same-day regeneration) rebuilt its Full score as entirely private (60 percent same-skills, 40 percent new-domains, the public index shown but uncounted), where Fastino’s shipped GLiDE V1.2 leads at 71.52 and forty systems sit above the vendor’s Jev 1.14 row (60.24, forty-first) with the 1.13 reference row forty-eighth at 59.22, and the same checkpoints score tens of points apart on the two scales (Kev 27B: forty-sixth on the Decision Index, near the bottom of JevBench); JevBench v1.7.47 also announces the coming v1.7.0 protocol (landing not before 2026-10-24), which adds a 10-second limit per decision and scores over-limit decisions as failed, so the slow rows will move again, so read any board standing as a property of a protocol rather than of a model. System1-Arena, the 2026-10-15 academic arrival, adds a fourth layer: identical requests to System-1 models and generative LLMs scored on probability quality across 92k questions plus closed-loop environments, where the largest System-1 models match fast closed LLMs off the loop at a quarter of the latency and the small ones rarely survive the loop. In plain words: one approach makes the mistake impossible to emit, the other catches the mistake after you have paid for it, and the last two never draw a token in the first place.

A decoding-time guarantee changes the failure mode rather than removing it. OpenAI staff described a confused model looping in technically valid output until max_tokens, and you pay for every token of it. Anthropic refusals come back as HTTP 200 with non-schema text you must handle yourself. The retry loop leaves your code; error handling does not.

Given the mechanism, the library columns survive only in the niches the vendors cannot reach. Instructor’s remaining value is the cross-provider surface (15+ providers, Ollama and vLLM included) and business rules no grammar expresses. Atomic Agents keeps that surface but adds the piece Instructor leaves out: a component structure for pipelines where many typed calls need to compose, which is a maintainability argument rather than a guarantee upgrade. BAML is the same class expressed one layer down: the schema lives in the language and the repair parser in its runtime, so the guarantee travels with the codebase instead of with an import, paid for in pre-1.0 toolchain churn and a provider matrix its own pivot undocumented. Guardrails holds the hub niche: the checks themselves become installable packages (65 risk validators from PII to prompt injection to provenance), composed around any LiteLLM-reachable provider or served outside your process, with the same post-hoc failure class and a 2026 that asks who maintains the hub (hub install retired, hosted inference cut off, the company acquired by Harvey). Marvin is the floor of the family: the same post-hoc loop exposed as four plain functions over Pydantic types, which is the right size for scripts and the wrong size for pipelines, and a library that has twice rebuilt itself (2.0, the ControlFlow merge) and published no release since March 2026. Outlines’ is regex, CFG, and recursive coverage, plus one type-driven API across local engines and hosted APIs. The Instructor note itself argues the minimal correct design on a single vendor is native strict mode plus a thin Pydantic validation step, and I agree.

Strict tool calls are the sharpest vendor win, because a malformed call in the middle of an agent loop is the failure retries never fixed well. The Anthropic note calls this the highest-value use of the feature. Both vendors guarantee tool arguments at sampling time, Instructor can only validate and re-ask, nothing in the Outlines note grounds tool-call guarantees at all, and CUA-S1 scores form elements rather than tool calls.

Costs follow the mechanism as well. The decoding-time options bill injected prompts (286 to 675 tokens per request on Anthropic) or first-request compilation (OpenAI) but nothing per failure. Instructor pays a full model round trip per failed validation, which is the price of enforcing rules the grammar cannot express. Outlines compiles once per schema and then runs at microseconds of overhead, at the cost of running your own stack. CUA-S1 is free beyond your own hardware, but the general planning model above it still bills every request.

Choosing from the matrix

#

• Already committed to OpenAI or Anthropic and need schema-valid JSON or tool arguments: use the native feature first and add nothing.
• Constraints are semantic (business rules, field meanings) or the stack spans providers: Instructor, layered over native strict mode where it exists.
• Serving your own models, or needing regex, CFG, or recursive structures the vendor subsets reject: Outlines as the front end.
• Running a high-throughput serving stack: prefer the engine’s own backend (xgrammar in vLLM) over Outlines’ engine.
• Agent loops where one malformed tool call wrecks a run: a vendor with strict tool use beats any library.
• Zero tolerance for retry latency: any decoding-time option, budgeting for Anthropic’s injected tokens and OpenAI’s compile latency.
• Many small decisions in latency- or cost-sensitive code paths (routing, scoring, guardrails) and no need for generated text: Jev is the hosted column priced and timed for that job, pending third-party confirmation of its claims.
• Want the strongest measured open checkpoint today and can serve vLLM on data-centre hardware: H2O-Lightning-4B, reading its open-composite lead as one protocol’s verdict.
• Want one contract across laptop-to-data-centre budgets: Decider, reading its numbers per checkpoint size and per protocol, with its sealed-board collapse attached.
• Want the community family with the strongest third-party record, with text and image states, on your own GPU: Winnow, budgeting a 16 GB-class card for the 12B and accepting the private training data.
• Want a vendor-team open checkpoint with the training run disclosed seed by seed and can wait on the score: Strands Decider 2B, reading its published calibration contract as the thing to test before its 107th-place board row.
• Want typed decisions on a CPU, English-only, mid-accuracy, and accept an archived project: Von, the 395M encoder with the dual-SDK and Docker packaging, archived 2026-10-17 with weights and SDKs preserved.
• Want accuracy back without abandoning the contract: Jeeves, accepting a 3.3 s median thinking budget, self-run numbers, and an unboarded checkpoint.
• Want typed decisions from a model you already serve, without training one: AnyJev, paying rotation prefills where the stopping rule asks for them and validating calibration on your own label splits first.
• Building a multi-component Python pipeline where every call needs a typed contract: Atomic Agents over Instructor, reading it as conventions and glue, not a stronger guarantee.
• Want the least ceremony a typed LLM call can have (extract, cast, classify, generate, no wrapping): Marvin, accepting the post-hoc guarantee, the twice-rebuilt history, and a release train quiet since March 2026.
• Need risk validators (PII, prompt injection, toxicity, provenance) around LLM calls on multiple providers more than schema guarantees: Guardrails, budgeting for post-hoc validation and reading the post-acquisition maintenance signals first.
• Starting agent-native code where the typed calls should live in the language rather than an import, and pre-1.0 single-vendor churn is acceptable: BAML, budgeting for the pivot’s documentation gaps on providers.
• Want the whole open wave runnable locally behind one TypeSafe-compatible API: Ollaya, reading its self-run accuracy tables as directional and its parity checks against the authors’ code as the feature worth trusting.
• Want to experience the contract locally before trusting anyone’s numbers: SemIf, the only column whose browser demo runs client-side with no backend, reading its explicit not-calibrated warning first.
• Shopping the wave by evidence rather than stars: Nimble hosts the only human-labeled head-to-head with Jev (76.0 versus 74.8 macro), NanoJev ships the most complete but unreplicated pipeline, and Jevlike is the dormant one-day starter whose value is historical.
• Curious whether a different architecture beats the LoRA-plus-pointer pattern: CLM is the dissent, and both boards’ near-zero readings are the current answer.
• Want the systems measured rather than described: JevBench, the one column that scores every other one on frozen items with per-task artifacts, read with its thread’s methodology objections and one-runner caveats attached.
• Want breadth instead of depth (42 chance-corrected benchmarks over 110,201 requests, five skill areas, private generalization tests): the Decision Index, reading its 80-percent-private Full score as one maintainer’s measurement and its public suite as the re-runnable part.
• Want the same questions put to decision models and generative LLMs, with closed-loop environments on top: System1-Arena, the two-day-old academic arena, reading its missing independent reproduction first.
• About to route agent tool calls through your own serving stack and want an audit rather than a leaderboard: tool-eval-bench, seeded multi-turn scenarios with full traces, reading its author-estimated difficulty tiers as exactly that.

Changes

#

• 2026-08-24 - Created with four columns and category rows including the guarantee-mechanism comparison.
• 2026-09-18 - Extended from four to five columns with Jev, the first member that is a model rather than a mechanism around one, and updated the intro, reading, and choosing sections for the third guarantee class.
• 2026-09-20 - Extended from five to six columns with CUA-S1, the open-weights counterpart to Jev’s no-generation contract, and updated the intro, thesis, guarantee-spectrum reading, and choosing sections.
• 2026-09-21 - Extended from six to seven columns with Laya, the open-weights multilingual decision-model family, inserted alphabetically between Jev and OpenAI Structured Outputs, and updated the intro, thesis, spectrum reading, choosing bullets, and the CUA-S1 evidence framing after its model-card expansion.
• 2026-09-21 - Extended from seven to twelve columns with the launch-week open wave (Jevlike, Kev, NanoJev, Nimble, SemIf, all inserted alphabetically), rewrote the intro and thesis for the seven-open-brackets framing, extended the spectrum reading with the dormancy-and-evidence spread, and added four choosing bullets for the wave.
• 2026-09-22 - Extended from twelve to thirteen columns with JevBench, the category’s first independent benchmark, inserted alphabetically between Jev and Jevlike, updated the kev (MLX shipped, thread cleared the bar), nimble (temperature fitted 2026-09-22), and semif (calibration PRs) cells, and added a choosing bullet for the scoreboard column.
• 2026-09-24 - Removed the verification preamble line on owner request.
• 2026-09-25 - JevBench’s sealed-decision v1.4 re-scoring updated the spectrum reading (Jev first on both boards, SemIf 73.1 to 47.7), and the JevBench, Kev, and SemIf maintenance cells moved to the 2026-09-24 re-scoring and later pushes.
• 2026-09-26 - JevBench’s v1.4.2 additions (eleven systems, a v1.4.3 roster pending) updated the spectrum reading, where decider-4b v2 (64.1) now sits half a point ahead of Jev (63.3), and the JevBench maintenance cell moved to the 2026-09-24 additions.
• 2026-09-27 - JevBench’s v1.4.2.1 point release (Plumb-4B 65.8, the new #1) updated the spectrum reading again, and the JevBench maintenance cell moved to the 2026-09-27 point release.
• 2026-09-29 - Extended from thirteen to fourteen columns with Jeff, the AutoJev-fork fine-tune family that clears the wave’s evidence bar (471-point launch thread), inserted alphabetically between Instructor and Jev, and updated the intro, thesis, spectrum reading, and choosing sections; the JevBench maintenance cell moved to the v1.4.2.2 point release (Imajev-4B 67.4, the new #1), the Kev cell moved to twelve days old, and the Laya API-surface cell moved to PyPI 0.3.21.
• 2026-10-02 - Moved the Kev cells for the Kev 1.0 family release and the new 27B flagship (kind now names the full-weights 27B, the API-surface cell records kev-1.0, maintenance moves to sixteen days old and three releases, cost adds the data-centre GPU) and the Laya API-surface cell to PyPI 0.3.23; the JevBench text board itself is unchanged since v1.4.2.2.
• 2026-10-02 - Moved the JevBench maintenance cell to the frozen v1.5 method publication of 2026-09-29 (equal-axis, equal-type headline amendment), with the v1.4.2.2 board still current and no v1.5 results published.
• 2026-10-03 - Moved the Laya API-surface cell to PyPI 0.3.24 and pointed the SemIf reference at the repository’s canonical SemIf-OpenJev name (the /SemIf URL redirects); every other cell is unchanged.
• 2026-10-04 - Moved the Laya API-surface cell to PyPI 0.3.26 (0.3.25 and 0.3.26 shipped October 3) and the Kev maintenance cell to seventeen days old; every other cell is unchanged.
• 2026-10-06 - Extended from fourteen to sixteen columns with Jeeves, the reasoning-then-deciding 9B from PostHog, and Ollaya, the local runtime that packages the open families behind the TypeSafe API, both inserted alphabetically, and updated the intro, thesis, spectrum reading, and choosing sections.
• 2026-10-06 - Moved the CUA-S1 kind, local-models, and maintenance cells to the four-checkpoint family, the Jeff maintenance and API-surface cells to the v1.3 adapter-first base plus Jeff-Code, the Kev maintenance cell to twenty days old, and the Laya API-surface cell to PyPI 0.3.28.
• 2026-10-06 - Corrected the maintenance-row ages (Jeff eight days and Ollaya thirteen, not three weeks; Kev nineteen, not twenty) and refreshed the stale day counts for Jeeves, Laya, NanoJev, Nimble, and SemIf.
• 2026-10-07 - Extended from sixteen to seventeen columns with the Decision Index, the second independent scoreboard (HF Space board plus an MIT reproduction kit), inserted alphabetically between CUA-S1 and Instructor, and updated the intro, thesis, spectrum reading, and choosing sections.
• 2026-10-07 - Moved the JevBench maintenance and provider-breadth cells to the v1.6.1 boards (open-weights main board, split hosted-API leaderboard, Jev the unranked reference), the Nimble maintenance cell to the v3 checkpoint (CC BY-NC), refreshed the day counts (Kev twenty, Jeeves eight, Jeff nine, Laya nineteen, NanoJev twenty, Nimble nineteen, Ollaya fourteen), and canonicalized the Anthropic and OpenAI docs references after both vendors moved their docs domains (docs.claude.com to platform.claude.com, platform.openai.com to developers.openai.com).
• 2026-10-07 - Extended from seventeen to nineteen columns with AnyJev (Nokia Applied Research’s training-free readout with a technical report and third-party benchmark coverage, reversing the note’s 2026-09-25 rejection) and Atomic Agents (the Instructor-built component framework), inserted in sorted position and traced to the new notes; the intro, thesis, spectrum reading, library reading, and choosing sections updated.
• 2026-10-08 - Extended from nineteen to twenty columns with tool-eval-bench, the MIT tool-calling auditor for self-hosted serving stacks with a /v1/systemone decision track, inserted in sorted position and traced to the new note; moved the JevBench maintenance and provider-breadth cells to board revision v1.7.18 (123 ranked open-weights systems, H2O-Lightning-4B the new open leader above Jev’s reference), the Laya API-surface cell to PyPI 0.4.0, and the Ollaya provider-breadth cell to 19 families; the intro, thesis, spectrum reading, and choosing sections updated.
• 2026-10-09 - Moved the JevBench maintenance and provider-breadth cells to board revision v1.7.28 (128 ranked open-weights systems, decisio v0.8.0 the new second open entry at 71.7; the hosted-API board re-ranked around Inception’s Mercury Decide at 72.4 and holds 7 ranked rows), the Laya API-surface cell to PyPI 0.4.1, corrected the Decision Index maintenance cell to the 2026-10-07 board regeneration its note records, and refreshed the day counts (Jeeves ten, Jeff eleven, Kev twenty-two, Laya twenty-one, NanoJev twenty-two, Nimble twenty-one, Ollaya sixteen, tool-eval-bench pushed 2026-10-09); every other cell is unchanged.
• 2026-10-10 - Moved the JevBench maintenance and provider-breadth cells to board revision v1.7.31 (128 ranked open-weights systems unchanged; Microsoft-Decision-1 joined the API board at 69.1, 8 ranked rows), the Decision Index maintenance and provider-breadth cells to the 0.3.1 regeneration of 2026-10-10 (117 configurations), and the tool-eval-bench maintenance and API-surface cells to v2.8.0 (2026-10-10, Strata and TabbyAPI joining the backend list); refreshed the day counts (AnyJev nineteen, Jeeves eleven, Jeff twelve, Kev twenty-three, Laya twenty-two, NanoJev twenty-three, Nimble twenty-two, Ollaya seventeen, SemIf 24, Atomic Agents pushed 2026-10-10); every other cell is unchanged.
• 2026-10-11 - Extended from twenty-five to twenty-six columns with BAML (Boundary’s typed agent language whose LLM calls are functions, carrying the error-correcting parser from its structured-output origin), inserted alphabetically between AnyJev and Atomic Agents, every row gaining a cell traced to the new note; the intro, thesis, library reading, and choosing sections updated, and its references added.
• 2026-10-11 - Extended from twenty to twenty-five columns with CLM (the contrastive-encoder dissent the boards score near zero), Decider (Mapika’s 2B-to-35B family and the sealed-board collapse record), H2O-Lightning-4B (the company-built open-composite leader), Von (the 395M CPU-scale encoder), and Winnow (the community family with the strongest third-party record), all inserted in sorted position and traced to the new notes; moved the JevBench maintenance and provider-breadth cells to board revision v1.7.44 (148 ranked open-weights systems, the hosted-API board grown to 27 ranked rows), the Laya API-surface cell to PyPI 0.4.2, and refreshed the day counts (AnyJev twenty, CLM three weeks, Decider 25, H2O-Lightning-4B nine, Jeeves twelve, Jeff thirteen, Kev twenty-four, Laya twenty-three, NanoJev twenty-four, Nimble twenty-three, Ollaya eighteen, SemIf 25, Von 23, Winnow three weeks); the intro, thesis, spectrum reading, and choosing sections updated.
• 2026-10-12 - Extended from twenty-six to twenty-seven columns with Strands Decider 2B (AWS Strands team’s open-weights 2B decision model, the wave’s best-published training discipline over its weakest board row), inserted alphabetically between SemIf and tool-eval-bench, every row gaining a cell traced to the new note; moved the JevBench maintenance and provider-breadth cells to board revision v1.7.46 (152 ranked open-weights systems ordered by the new Capability Score, the API board at 27 rows with Jev second by capability), the Decision Index maintenance and provider-breadth cells to the 2026-10-12 regeneration (144 configurations), and the Laya API-surface cell to PyPI 0.4.4, and refreshed the day counts (AnyJev twenty-one, CLM nineteen, Decider 26, H2O-Lightning-4B ten, Jeeves thirteen, Jeff fourteen, Kev twenty-five, Laya twenty-four, NanoJev twenty-five, Nimble twenty-four, Ollaya nineteen, SemIf 26, Von 24, Winnow twenty-two); the intro, thesis, and spectrum reading updated.
• 2026-10-13 - Extended from twenty-seven to twenty-eight columns with Valen (the open-weights multimodal Jev-family decision model), inserted in sorted position after tool-eval-bench, all eleven cells traced to the new note, intro counts bumped.
• 2026-10-13 - Extended from twenty-eight to twenty-nine columns with Guardrails (the Apache-2.0 validator-hub framework composing Input and Output Guards from installable risk validators, the validate-and-retry family’s hub kind), inserted alphabetically between Decision Index and H2O-Lightning-4B, every row gaining a cell traced to the new note; repaired the intro’s enumeration, which still summed to twenty-seven after the morning’s Valen extension (the decision-model count now eighteen with Valen named among the board-era additions, and the open-implementation and spectrum counts in the reading brought in line); the intro, thesis, library reading, and choosing sections updated, and its references added.
• 2026-10-13 - Moved the JevBench maintenance and provider-breadth cells to 155 ranked open-weights systems and 28 API rows (still board revision v1.7.46), the Decision Index maintenance and provider-breadth cells to the 0.3.2 board (149 configurations), the Laya API-surface cell to PyPI 0.4.6, the Ollaya provider-breadth cell to 21 families and its maintenance cell to v0.13.0 (pushed 2026-10-13), and refreshed the pushed dates and day counts (AnyJev, Decider, Kev, Strands Decider 2B, Valen’s star count); every other cell is unchanged.
• 2026-10-15 - Extended from twenty-nine to thirty columns with Marvin (Prefect’s Apache-2.0 AI-functions toolkit, the validate-and-retry family’s smallest API: extract, cast, classify, and generate over Pydantic types, with the ControlFlow-derived task-agent-thread layer), inserted alphabetically between Laya and NanoJev, every row gaining a cell traced to the new note; moved the JevBench maintenance and provider-breadth cells to the 2026-10-14 refresh of v1.7.46 (162 ranked open-weights, 30 API rows), the Decision Index maintenance and provider-breadth cells to the 2026-10-14 regeneration (166 configurations), the BAML maintenance cell to baml-language 0.22.2 (2026-10-13) pushed 2026-10-15, the Laya API-surface cell to PyPI 0.4.7, and the Guardrails maintenance cell to pushed 2026-10-15 (7,524 stars); the intro, thesis, library reading, board reading, and choosing sections updated, and its references added.
• 2026-10-17 - Extended from thirty to thirty-one columns with System1-Arena (the academic benchmark handing decision models and generative LLMs identical requests across 92k questions and 138 closed-loop environments), inserted in sorted position between Strands Decider 2B and tool-eval-bench, every row gaining a cell traced to the new note; repaired every row’s cell alignment, which had been rotated one column out of place since the 2026-10-15 Marvin extension prepended Marvin’s cells before the first column instead of inserting them at its sorted position, so the first twenty columns of every row showed the neighboring member’s cell (all rows re-aligned and spot-verified); moved the JevBench maintenance and provider-breadth cells to board revision v1.7.46 refreshed 2026-10-16 (181 ranked open-weights systems, 39 API rows, H2O-Lightning-4B ninth by capability), the Decision Index maintenance and provider-breadth cells to the 2026-10-16 regeneration (174 configurations, new leader StartLux-Decision v1.1 27B at 69.12, Jev 1.14 thirteenth and the 1.13 reference eighteenth), the Laya maintenance and API-surface cells to 0.4.9 (2026-10-17), the Atomic Agents maintenance cell to v2.12.0 (2026-10-15), the tool-eval-bench maintenance cell to the 2026-10-14 serve UI, the NanoJev maintenance cell to the System1-Arena README leads, refreshed the pushed dates (BAML 10-17, Kev 10-16, Jeeves 10-16, Ollaya 10-17), moved the Von maintenance cell to its 2026-10-17 archival, and caveated the Von choosing bullet; the intro, thesis, spectrum reading, and choosing sections updated, and its references refreshed to the fetched board states.
• 2026-10-18 - Repaired the separator row, whose System1-Arena cell carried a stray “? not verified” left from the previous run’s insertion instead of a dash; refreshed the maintenance cells for the rotation and pushed-since members (CLM twenty-five days, Kev thirty-one, Decider 32, Strands Decider 2B nineteen with pushed 2026-10-16 and the five-checkpoint v1 line named); every other cell unchanged.
• 2026-10-20 - Moved the maintenance-row cells for the push-signal members and the rotation slice: Atomic Agents to the three 2026-10-19 releases (v2.13.1 current, pushed 2026-10-20), BAML to the nightly train through the 20261018 build (pushed 2026-10-20), Ollaya to v0.14.0 and 23 families (provider-breadth cell moved), Strands Decider 2B to pushed 2026-10-19 with PyPI 0.2.0 (API-surface cell moved), Jeff to the Jeff-Code repository split with its beyond-schema cell re-based to the 254-option Qwen ceiling and the 8,192-token input cap, Nimble to the v4 and v4-max hosted API with its beyond-schema cell re-based to the per-revision temperature story, H2O-Lightning-4B to v1.3.1 on main with two-way reading (API-surface cell moved), tool-eval-bench’s pushed date to 2026-10-20, Von to the archived notice pointing at LiquidAI/d1-omni-600M, and Winnow’s card date to 2026-10-13; moved the JevBench maintenance and provider-breadth cells to board revision v1.7.46 updated 2026-10-18 (184 ranked open-weights, 39 API rows), refreshed the stale day counts in the same row, updated the board paragraph (basal-omni-1.0-ultra first by capability, Jeff-1.0-Large third, H2O tenth, the API composite dropping Jev to seventh), and refreshed the board, local-jev-bench, Ollaya site, Decision Index, and BAML npm reference lines to the 2026-10-20 fetches; the intro, thesis, and choosing sections unchanged.
• 2026-10-22 - Moved the Decision Index maintenance, provider-breadth, and guarantee-mechanism cells to the edition-0.4 board (label 0.4.1, regenerated 2026-10-22, 226 configurations, the Full score entirely private with the public index shown but uncounted) and updated the board paragraph around it (GLiDE V1.2 first at 71.52, Jev 1.14 forty-first at 60.24, the 1.13 reference forty-eighth, Kev 27B forty-sixth); moved the maintenance cells for the push-signal members (AnyJev pushed 2026-10-21, Atomic Agents to v2.15.0 and the GenAI-span tracing rework, BAML to the 20261020 nightly and pushed 2026-10-22, Decider pushed 2026-10-20, Guardrails pushed 2026-10-21, Kev pushed 2026-10-21, Laya to 0.4.13 (2026-10-21) with its API-surface cell moved and a community Rust runtime named, Ollaya to v0.16.0 and 24 families (provider-breadth cell moved) pushed 2026-10-22, Strands Decider 2B pushed 2026-10-21, Valen pushed 2026-10-22) and refreshed the stale day counts in the same row; the intro, thesis, and choosing sections unchanged.
• 2026-10-24 - Moved the JevBench maintenance and provider-breadth cells to board revision v1.7.47 (main board updated 2026-10-23, API board 2026-10-22: 197 ranked open-weights systems, 44 API rows, Chon-1.0-12B first by capability at 85.3, H2O-Lightning-4B twelfth by capability, both Jev rows unranked references with the API row at 70.7 in fifteenth position) and updated the board paragraph around it, including the announced v1.7.0 protocol (a 10-second per-decision limit landing not before 2026-10-24); moved the Decision Index maintenance and provider-breadth cells to 229 configurations (the same-day regeneration, named rows unchanged); moved the maintenance cells for the push-signal members (BAML to the 20261022 nightly and pushed 2026-10-24, Guardrails pushed 2026-10-24, Kev pushed 2026-10-23, Laya to 0.4.14 (2026-10-23), Nimble’s day count to thirty-six, Ollaya to v0.17.0 and 26 families (provider-breadth cell moved) pushed 2026-10-24, Strands Decider 2B pushed 2026-10-23, Valen pushed 2026-10-24) and refreshed the stale day counts in the same row; refreshed the board, Decision Index, and Ollaya reference lines to the 2026-10-24 fetches; the intro, thesis, and choosing sections unchanged.

See also

#

• Harness Feature Matrix - the same treatment for terminal coding agents
• Model Selection for Coding Tasks - the model side of the decision, since the mechanism choice follows the model choice
• MCP - the tool ecosystem whose calls get schema guarantees
• LangChain - the framework layer that wraps these features alongside its own output parsers

References

#

• https://developers.openai.com/api/docs/guides/structured-outputs - strict mode, API surfaces, JSON-mode comparison for the OpenAI column (canonical after the platform.openai.com move; verified 200 on 2026-10-07)
• https://platform.claude.com/docs/en/build-with-claude/structured-outputs - schema subset, complexity limits, injected-token costs for the Anthropic column (canonical after the docs.claude.com move; verified 200 on 2026-10-07, supported-model list re-verified unchanged 2026-10-12)
• https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use - sampling-time guarantees behind the strict-tool-calls row (canonical after the docs.claude.com move; verified 200 on 2026-10-07)
• https://github.com/567-labs/instructor - license, reask retries, provider list for the Instructor column (14,005 stars as of 2026-10-12)
• https://dottxt-ai.github.io/outlines/latest/ - output types, integrations, pluggable backends for the Outlines column
• https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/data/v03.json - the Decision Index board data grounding its column and the new members’ rows (CLM 6.49, Decider 31B 66.03, Von 11.65, Winnow 46.57/32.72; label 0.4.1, generated 2026-10-22, fetched 2026-10-22 and re-fetched 2026-10-24 at 229 configurations)
• https://typesafe.ai/blog/introducing-system-one-models-and-jev - primitives, pricing, and claims behind the Jev column
• https://docs.typesafe.ai/ - the Choice, Score, and Noul surfaces behind the Jev column
• https://news.ycombinator.com/item?id=49717558 - the launch thread and its skepticism behind the Jev column’s early-access status
• https://github.com/trycua/cua - the host repository (29,429 stars, MIT) and CUA-S1 component behind the new column (GitHub API, as of 2026-10-12)
• https://huggingface.co/cua-ai/cua-s1-forms - the MIT weights grounding the CUA-S1 open-source and local-model cells
• https://huggingface.co/cua-ai/cua-s1-forms/raw/main/cua-s1-forms.json - the checkpoint metrics grounding the CUA-S1 calibrated-probabilities and synthetic-only cells
• https://news.ycombinator.com/item?id=49767564 - the 2026-09-19 launch thread grounding the CUA-S1 research-artifact status
• https://github.com/firelex/jeff - the Jeff column: MIT code, Apache-2.0 weights, the benchmark table and speed claims, the AutoJev lineage (GitHub API and README, as of 2026-10-12)
• https://raw.githubusercontent.com/firelex/jeff/main/README.md - the v1.3 adapter-first base, the fifteen adapters, the Jeff-Code paired-run table (re-based to 47 percent faster per task, 32 percent less time), and the per-adapter licence split behind the Jeff cells (re-fetched 2026-10-20)
• https://github.com/PostHog/jeeves - the Jeeves column: MIT code, Apache-2.0 weights, the benchmark tables and latency trade (GitHub API and README, as of 2026-10-12)
• https://news.ycombinator.com/item?id=49891290 - the 242-point Jeeves launch thread grounding its evidence framing
• https://ollaya.dev/ - the Ollaya column: the TypeSafe-compatible API surface, family list, and the Ollama 0.40.6 comparison behind its cells (26 families re-verified 2026-10-24)
• https://github.com/ollaya-dev/ollaya - the Ollaya repository: Apache-2.0, created 2026-09-23, 1,298 stars (GitHub API, as of 2026-10-12)
• https://news.ycombinator.com/item?id=49848269 - the 618-point Ollaya launch thread
• https://news.ycombinator.com/item?id=49883844 - the 576-point Jeff launch thread grounding the Jeff column’s evidence framing
• https://docs.vllm.ai/en/latest/features/structured_outputs.html - vLLM backend names behind the maintenance-status row
• https://github.com/NandhaKishorM/laya - the Laya column: Apache-2.0 license, checkpoint table, Router, PyPI package (GitHub API and PyPI, as of 2026-10-12)
• https://huggingface.co/convaiinnovations/laya - the Laya model card: the self-critical limits section, post-temperature ECE, and where-Jev-leads tables behind the Laya cells
• https://news.ycombinator.com/item?id=49765348 - the Laya launch thread criticisms behind the context-limit and calibration cells
• https://github.com/vinnylarouge/jevlike - the Jevlike column: MIT, one day of commits, the option-attention starter (GitHub API, as of 2026-10-12)
• https://news.ycombinator.com/item?id=49731282 - the 166-point Jevlike thread behind its dormant-since-launch status cell
• https://github.com/jaredpalmer/kev - the Kev column: Apache-2.0, the SDK-compatible server, releases and author standing (GitHub API, as of 2026-10-12)
• https://raw.githubusercontent.com/jaredpalmer/kev/main/PLAN.md - the pre-registered eval discipline behind Kev’s beyond-schema and cost cells
• https://github.com/TianyuCodings/NanoJev - the NanoJev column: MIT code, weights and dataset on Hugging Face (GitHub API, as of 2026-10-12)
• https://github.com/bespokelabsai/nimble - the Nimble column: stars and pushed date, with the missing license behind its partial open-source cell (GitHub API, as of 2026-10-12)
• https://raw.githubusercontent.com/bespokelabsai/nimble/main/docs/PUBLIC_BENCHMARKS.md - the human-labeled Jev head-to-head behind the Nimble evidence framing
• https://github.com/TheoLeeCJ/SemIf-OpenJev - the SemIf column: MIT, the rename from OpenJev, and the frozen-model method (GitHub API, as of 2026-10-12; the /SemIf URL redirects here)
• https://news.ycombinator.com/item?id=49752041 - the OpenJev thread behind SemIf’s community-footprint status (720 points as of 2026-09-22)
• https://github.com/fstandhartinger/jevbench - the JevBench column: MIT harness, 48-row board, frozen artifacts (GitHub API and README, 257 stars as of 2026-10-12)
• https://raw.githubusercontent.com/fstandhartinger/jevbench/main/RESULTS-v1.2.md - the per-axis scores grounding the JevBench column and the spectrum-reading update
• https://news.ycombinator.com/item?id=49800574 - the 92-point JevBench thread and its methodology objections behind the contested scoreboard framing
• https://simonwillison.net/2024/Aug/6/openai-structured-outputs/ - independent record of failure modes and of Instructor’s influence
• https://github.com/nokia-applied-research/AnyJev - the AnyJev column: repository, readout design, and Tacit line (1,202 stars as of 2026-10-12)
• https://arxiv.org/abs/2610.00831 - the AnyJev column: rotation-averaging results, prior correction, and the verified early-exit bounds
• https://raw.githubusercontent.com/tak-bro/local-jev-bench/HEAD/README.md - the third-party Apple Silicon benchmark behind the AnyJev, CLM, Decider, Von, and Winnow evidence cells (re-fetched 2026-10-20 after its full 2026-10-14 to 16 re-measurement)
• https://github.com/Eigenwise/atomic-agents - the Atomic Agents column: repository, component model, and Instructor foundation (6,284 stars as of 2026-10-12)
• https://eigenwise.github.io/atomic-agents/ - the Atomic Agents column: schema-bounded components and provider integrations
• https://github.com/Contrastive-LM/CLM - the CLM column: the contrastive recipe and vLLM-plus-head architecture (2,988 stars as of 2026-10-12)
• https://huggingface.co/Contrastive-LM/CLM-v0.1-8B - the CLM column: the Apache-2.0 checkpoint (4,367 downloads, 765 likes, as of 2026-10-12)
• https://benchmarkheaven.com/api/jevbench/v1.6.1 - the JevBench aggregate JSON behind the board cells and the CLM, Decider, H2O-Lightning-4B, and Winnow rows (fetched 2026-10-15 and re-fetched 2026-10-20; the served snapshot remains a pre-v1.7.46 refresh)
• https://benchmarkheaven.com/jev-models - the live open-weights main board: 197 ranked systems ordered by the Capability Score (Chon-1.0-12B first at 85.3, basal-omni-1.0-ultra second, Jeff-1.0-Large next at 81.5), H2O-Lightning-4B v1.1 twelfth by capability and thirty-sixth by composite, both Jev rows the unranked references (board v1.7.47 updated 2026-10-23, fetched 2026-10-24)
• https://benchmarkheaven.com/jev-models/api - the live hosted-API board: 44 ranked rows, kettei-fat first on the official composite and the vendor’s Jev 1.14.0 row an unranked reference at 70.7 in fifteenth position (board v1.7.47, API page updated 2026-10-22, fetched 2026-10-24)
• https://github.com/fstandhartinger/jevbench/issues/181 - the add-request behind the H2O-Lightning-4B column’s entry route (fetched 2026-10-11)
• https://github.com/Mapika/decider - the Decider column: repository, family design, and surfaces (1,128 stars as of 2026-10-12)
• https://huggingface.co/Mapika/decider-2b - the Decider column: the 2B checkpoint (331,532 downloads, 105 likes, as of 2026-10-12)
• https://huggingface.co/h2oai/h2o-lightning-4b - the H2O-Lightning-4B column: the model card, vLLM-plus-shim serving path, and the vendor’s board claim (fetched 2026-10-11)
• https://github.com/wfzyx/von - the Von column: repository, 395M encoder design, SDK and Docker surfaces (884 stars as of 2026-10-12)
• https://pypi.org/project/von-sdk/ - the Von column: the Python SDK, 1.3.7 uploaded 2026-10-02 (fetched 2026-10-12)
• https://huggingface.co/EldanRing/Winnow-12B - the Winnow column: the model card, GGUF targets, and the self-run panel (fetched 2026-10-11)
• https://github.com/EldanRing/winnow-inference - the Winnow column: the MIT server and its surfaces (26 stars as of 2026-10-12)
• https://github.com/BoundaryML/baml - the BAML column: repository, Apache-2.0, stars, and activity (GitHub API, as of 2026-10-12)
• https://boundaryml.com/explore.md - the BAML column: the language tour, runtime types, LLM calls as functions, the error-correcting parser, testsets and quorum runners, and the Agent Tries BAML arena
• https://api.npmjs.org/downloads/point/last-month/@boundaryml/baml - 1,019,496 monthly downloads behind the BAML maintenance framing (month ending 2026-10-18, re-fetched 2026-10-20)