↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Trackers and leaderboards/

Trackers and Leaderboards Feature Matrix

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 19 minutes read (~3674 words)
agent-curated 600 benchmarks 12 comparison 24 fully-ai-generated 705 leaderboards 6 llm=glm-5.3-flash 610 open-data 6 trackers-and-leaderboards 14
Table of Contents

This matrix compares the thirteen members of the Trackers and leaderboards category: sites whose product is a continuously refreshed number about the AI field itself. The members split cleanly on what their number measures: what shipped (AI Release Tracker), what the operator measured (Artificial Analysis), what the public evidence blends into (BenchLM), what the field’s benchmarks are doing (Benchmark Radar), what nothing cheaper beats at a given budget (bestvaluemodel), how the field moves (Epoch AI), what freshly rotated public questions score (LiveBench), what public evidence aggregates to (LLM Stats), what people prefer (LMArena), whether the stack you shipped with got worse this week (Margin Lab), what people pay for (OpenRouter Rankings), how often a summary contradicts its source (Vectara Hallucination Leaderboard), and what your machine can actually run and afford (whatmodel.sh).

No row of this matrix crowns a winner, because the rows are different questions; the failure mode is citing a site for a question it does not answer, usually LMArena ranks quoted as capability or OpenRouter tokens quoted as market share.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell traces to a source cited there or in the references.

The matrix
#

AI Release Tracker Artificial Analysis BenchLM Benchmark Radar bestvaluemodel Epoch AI LiveBench LLM Stats LMArena Margin Lab OpenRouter Rankings Vectara Hallucination Leaderboard whatmodel.sh
Kind release timeline plus flat-file corpus independent benchmarking site and data business evidence-labeled composite aggregator with a downloadable dataset and query API open benchmark catalog and search engine (one researcher’s project) daily-refreshed value-frontier view over one member’s data (solo project) research nonprofit with open datasets rotated public benchmark with published ground truth composite aggregator with agent-facing API blind preference arena daily degradation watcher over two deployed coding-agent stacks, plus an open eval runtime gateway usage rankings single-metric model leaderboard judged by a vendor’s eval model curl-first IRT composite leaderboard with machine-fit estimates
The number measures launch-day facts: what shipped, when, with which claimed scores the operator’s own controlled evals, prices, and speed runs one blended rank per model, every position labeled by how much public evidence stands behind it the benchmark landscape itself: which benchmarks exist, who documented them, and what scores were reported where the value frontier: the models nothing cheaper also beats on the Intelligence Index, at Artificial Analysis’s blended API prices long-run trends: compute, cost, capability over time objective ground-truth scoring of freshly rotated public questions public benchmark evidence, normalized with uncertainty blind human preference votes daily pass rates of one shipped CLI plus its frontier model, watched for statistically significant regressions tokens processed through one gateway how often a model’s summary of a short document is supported by its source, per the HHEM judge model one latent ability score fit across public results with item response theory, plus what a given machine can run, at what measured speed, and within what price cap
Run by To sider ApS, a Danish side project (one visible operator, named as Peter Assentorp by Business Insider) venture-backed independent company (AI Grant seed) Glevd, a solo maintainer identified by an X handle, no company disclosed Koutian Wu and contributors (individual project, 15-author technical report citation as of 2026-10-25) terryds, a solo developer, no company disclosed 501(c)(3) nonprofit, itemized donors, about 60 people (59 listed, as of 2026-10-25) academic consortium (ICLR 2025 Spotlight), day-to-day work by Abacus.AI engineers ZeroEval Inc., a self-described two-person company (self-displayed YC backing) Arena Intelligence Inc. ($200M Series B at $3.1B, 2026-10-08) Marginlab, an independent shop (self-declared no frontier-provider affiliation), run publicly as themarginguy OpenRouter, a gateway acquired-by-Stripe (announced 2026-08-19) Vectara, a RAG vendor; built by Simon Hughes’s team, updated by the company since his 2023 death Merlinsbeard Pty Ltd, a company with no public team page (terms page as of 2026-10-19)
Coverage 258 releases, 11 labs (as of the 2026-10-07 archived homepage) 718 models (as of 2026-11-01), 100+ providers, 1,000+ endpoints, chips 1,004 model profiles, 724 benchmark pages, 216 ranked entries (77 Supported), 8 weighted categories (as of 2026-11-01) 41,974 records from 40 sources (badge as of 2026-11-01); the paper’s audit held 1,283 records the full Intelligence Index model set as a price-score scatter, a budget lookup with a pick and runner-up per price rung, and per-reasoning-setting rows (as of 2026-10-25) over 3,600 models since 1950, data centers, chips, companies about 1,000 questions, 6 categories, 18 tasks, plus an agentic coding track 408 canonical models (as of 2026-10-11), 50+ benchmarks claimed text, image, video, vision, search, webdev, agent, alignment arenas two stacks (Claude Code with Opus 5.5/high, Codex with GPT-6.1 Sol/high), N=50 daily on a curated SWE-Bench Pro subset (as of 2026-11-01) 500+ models via the gateway, 80+ providers a few dozen flagship and open models per refresh on one fixed short-document summarization set 888 ranked text models from 22,353 linked results across 575 benchmarks, plus image, video, TTS, and STT lists (as of 2026-10-29)
Update cadence on each release; newest tracked release 2026-10-07 (archived homepage); flat-file archive captures still end 2026-08-26 daily changelog; index versions iterate weekly on new benchmark rows, protocol changes, and major releases; the board’s own verification stamp reads Oct 29, 2026 (as of 2026-11-01) daily crawls; the record badge regenerates on every collection every six hours via a GitHub Actions cron, committed and redeployed only when the data changed near-daily data updates, page-stamped monthly question releases in 2024, bursts since, newest set 2026-01-08; model configs still merged continuously; “within hours of release” claimed continuous votes; product posts within days daily runs; pages stamped Sep 29, 2026; verdicts paused while new baselines collect daily UTC buckets, about one day of lag irregular refreshes; README table stamped 2026-10-16, Space files modified 2026-10-06 (as of 2026-10-17) continuous result ingestion; homepage stamped 2026-10-29; own API speed runs published as raw JSON
Methodology published ~ FAQ and data notes, no formal methodology ✓ methodology hub with evaluation lists and weights ✓ scoring rules plus its own three failed validation checks, in detail ✓ technical report, scoring rubric, catalog data contract, public corpus schema ~ the frontier rule and its reading caveats are published on the page, but the score inside them is entirely Artificial Analysis’s ✓ transparency page, papers, and data documentation ✓ paper, changelog, datasheet, and every question with its ground truth ~ score construction published, details gated ✓ methodology repo (arena-rank) plus founding paper ✓ Bernoulli model, 95% CIs, per-window significance thresholds, exclusion rules ✓ on-page caveats plus Data API docs ~ the README names the commercial HHEM-2.3 as the judge computing the table (HHEM-2.1-Open is the downloadable open variant), and no human validation or paper grounds the scores ✓ IRT with per-run level pooling in depth, prior art credited, plus a self-run hidden-test replay (85% pair accuracy, self-reported)
Data access ✓ /models.json and llms-full.txt, free with attribution ~ web charts free, data platform paid, no public API documented ~ REST and MCP on a free tier, eight static JSON downloads with hashes, CC BY-NC 4.0 ✓ data ZIP, Hugging Face dataset, RSS, and an offline CLI with an agent skill ✓ committed daily snapshots and per-run changelog diffs in the public repository, but no query API and no license on the code ✓ CC BY datasets and a Python client ✓ questions, model answers, and judgments on Hugging Face; run pipeline public ✓ REST plus 11 MCP tools, free tier ~ arenas open, methodology open, raw votes not re-offered ~ tracker and history pages public, runtime AGPL-3.0, no bulk data download documented ✓ CC BY 4.0 JSON via Data API, history to 2025-01-01 ~ the table is diffable Markdown in git and an open variant of the judge is downloadable (identity-gated); the summary pairs behind past refreshes are not re-offered ✓ raw results.json and benchmarks.json downloads and a JSON API, CC BY-SA 4.0 on the compiled dataset, and the application MIT-licensed in its public repository
Reader pricing free; one $499/month sponsor slot (as of 2026-10-17); Pro instant alerts, price unpublished free site; enterprise and data-platform tiers, prices unpublished free site and data tier; Pro and Research paid, prices unpublished free; MIT code, CC BY-NC-SA 4.0 content free; no paid tier, no advertising free, donation funded free; the team evaluates requested models at no charge free tier; Builder $99/month; Commercial contract free free free, attribution required free free; labelled sponsor placements disclosed in the 2026-10-19 terms, no prices published; dataset CC BY-SA 4.0
Independence caveat one operator, one company registration, no second pair of eyes sells private benchmarking to the labs it ranks (disclosed policy) measures nothing itself; every number is another board’s measurement twice removed, and the operator is a pseudonymous handle re-aggregates two category members’ scores (credited), runs Microsoft Clarity analytics, no independent critique yet measures nothing itself; every number is Artificial Analysis’s re-plotted as a decision, and the site’s continuity rides on AA’s free API terms consulted for OpenAI and DeepMind, disclosed and audited day-to-day operation by Abacus.AI, a commercial ML-platform vendor the benchmark helps market paid eval services and a consumer sibling in the same company lab partnerships and a documented private-testing history runs the labs’ own tools against one curated subset, carries a promote-your-product contact, and paused verdicts while baselines rebuild owned by the gateway it measures, being absorbed by Stripe the judge is the ranking vendor’s own eval model, the vendor sells RAG products that benefit from hallucination concern, and no human-verified ground truth is published measures almost nothing itself beyond API speed runs, validates its own method, and near-zero community footprint (a two-point Show HN and a 43-star repository)
Verification hooks ✓ full corpus re-downloadable and diffable ~ changelog and methodology public, raw runs private ~ evidence labels, confidence table, content hashes, and published validation failures; no independent replication ✓ every record keeps source identity and citation; daily Issues; reproducible analysis ~ daily snapshots and diffs are re-downloadable, but no second source exists to check the frontier against ✓ data, notebooks, and code public ✓ whole chain of questions, answers, and judgments downloadable and recomputable ~ API re-queryable, scoring internals partly gated ✓ methodology code public, vote data not re-offered ✓ outcomes summarized with CIs and thresholds, and the runtime behind them is open source ✓ daily snapshots re-fetchable under CC BY ~ table diffs live in git history and an open variant of the judge is rerunnable, but the board publishes no validation of its own ~ every result keeps a source link and the raw data re-downloads, but the fit has no independent replication and no second party has visibly built on the public code

Reading the matrix
#

The row that sorts the category is “the number measures”, and it doubles as a citation guide: release questions go to the tracker, present-tense quality to Artificial Analysis, evidence-reading of composites to BenchLM, benchmark-discovery and saturation questions to Benchmark Radar, value-for-money at a budget to bestvaluemodel, trend claims to Epoch, rotated-question scores to LiveBench, quick composites to LLM Stats, preference to LMArena, did-my-tool-degrade to Margin Lab, spend to OpenRouter, and machine-fit to whatmodel.sh. The two most-quoted members are also the two most misquoted: LMArena ranks get read as capability, and gateway tokens get read as market share, when the matrix’s own rows show both measure something narrower.

Verification strength tracks institutional type, not popularity. The nonprofit (Epoch) publishes its funders, its consultations, and its data; the gateway (OpenRouter) publishes its raw daily numbers under CC BY; the side project (AI Release Tracker) lets you diff its whole corpus; the academic consortium (LiveBench) publishes every question, answer, and judgment; the researcher’s catalog (Benchmark Radar) publishes its corpus, schema, and rubric; the degradation watcher (Margin Lab) publishes its thresholds, its CIs, and an open runtime; the hallucination board (Vectara) publishes a git-diffable table and an open variant of its judge; the evidence-labeler (BenchLM) publishes its scoring rules and its own failed checks, though it holds nothing raw to check against; while the three venture-scale companies publish methodology and products but keep raw runs, votes, or scoring internals private. If you need to check the work rather than read the work, the checkable columns are the nonprofit’s, the consortium’s, the gateway’s, the side project’s, the researcher catalog’s, the degradation watcher’s, and the hallucination board’s, which is the opposite of what citation frequency would predict.

Every column carries a conflict row, and the conflicts are structural rather than scandals: the benchmark firm sells benchmarking to the benchmarked, the open benchmark doubles as its operator’s research marketing, the arena partners with the labs it ranks, the aggregator sells evaluation services, the gateway’s data is its own marketing, the catalog re-aggregates two of this category’s own members’ scores, the composite re-brands every other board’s outputs as one score of its own, the degradation watcher runs the labs’ own tools against its own subset with an advertising contact on the page, the hallucination board’s judge is the ranking vendor’s own eval model, the nonprofit consults for the labs, and the value-frontier page re-plots one member’s data as its own answer. Epoch’s FrontierMath episode and LMArena’s Maverick episode are the two documented failures, and both produced their institution’s strongest disclosure artifacts, which is the pattern worth watching on every refresh of this page.

Choosing from the matrix
#

  • Needing a dated record of what shipped and what the lab claimed that day: AI Release Tracker, via its flat files.
  • Picking a model or provider this week and needing quality against price and speed: Artificial Analysis, with the Endpoint Accuracy Index for the cheap-provider question.
  • Trying to learn how much evidence stands behind a composite rank, with the dataset to check it offline: BenchLM, read for its labels and its failed checks, never as a primary measurement.
  • Trying to find which benchmarks exist for a capability and whether they are saturating: Benchmark Radar, via the catalog, the saturation views, or the agent CLI.
  • Needing the most capability a fixed budget buys: bestvaluemodel, via the budget lookup, read as one derived view of one member’s data.
  • Citing how fast the field moves, with downloadable data: Epoch AI.
  • Wanting an objective capability score whose every question, answer, and judgment can be downloaded and rerun: LiveBench, while watching its rotation cadence.
  • Wiring leaderboard data into an agent or a script: LLM Stats’s free API and MCP tools, accepting the gated internals.
  • Sensing which model people prefer right now: LMArena, read as a mood ring and never as a capability claim.
  • Needing a canary on the CLI and model pair you already ship with: Margin Lab’s trackers, while their baselines rebuild.
  • Asking what developers actually spend on: OpenRouter Rankings, quoted with its as-of date and its single-gateway caveat.
  • Needing a hallucination-rate read before wiring RAG or summarization: Vectara Hallucination Leaderboard, read as one vendor’s judge on one task.
  • Needing to know whether a model fits the machine and budget you already have: whatmodel.sh, via the curl filters or the JSON API, read as one operator’s fit.

Changes
#

  • 2026-09-24 - Created with six columns (AI Release Tracker, Artificial Analysis, Epoch AI, LLM Stats, LMArena, OpenRouter Rankings) when the category was seeded at the owner’s request.
  • 2026-09-29 - Coverage cells refreshed: Artificial Analysis to 679 models and LLM Stats to 399 canonical models; membership unchanged.
  • 2026-10-02 - Coverage cells refreshed: Artificial Analysis to 689 models and LLM Stats to 400 canonical models; membership unchanged.
  • 2026-10-06 - Extended from six to seven columns with LiveBench, the contamination-limiting public benchmark leaderboard, sorted between Epoch AI and LLM Stats; the intro, sorting guide, verification-strength, conflict, and choosing prose updated.
  • 2026-10-10 - Extended from seven to eight columns with Benchmark Radar, the open benchmark catalog and search engine with linked evidence, sorted between Artificial Analysis and Epoch AI; the intro, sorting guide, verification-strength, conflict, and choosing prose updated.
  • 2026-10-10 - Coverage and status cells refreshed: Artificial Analysis to 696 models, LLM Stats to 408 canonical models, AI Release Tracker to 258 releases from 11 labs per the 2026-10-07 archived homepage, and LMArena’s run-by cell to the $200M Series B at a $3.1B valuation.
  • 2026-10-11 - Extended from eight to ten columns with BenchLM, the evidence-labeled composite re-ranker (sorted between Artificial Analysis and Benchmark Radar), and Margin Lab, the daily degradation watcher over Claude Code and Codex (sorted between LMArena and OpenRouter Rankings); the intro, sorting guide, verification-strength, conflict, and choosing prose updated, and volatile coverage cells re-dated to 2026-10-11.
  • 2026-10-13 - Extended from ten to eleven columns with Vectara Hallucination Leaderboard, the HHEM-judged hallucination-rate board (sorted after OpenRouter Rankings); the intro, sorting guide, verification-strength, conflict, and choosing prose updated.
  • 2026-10-16 - Coverage cells refreshed: Artificial Analysis to 702 models, Benchmark Radar to 31,624 records from 40 sources, and BenchLM to 950 tracked profiles with 224 ranked entries (89 Supported), all as of 2026-10-16; BenchLM’s update-cadence cell moved to the board’s own Oct 15 verification stamp; membership unchanged.
  • 2026-10-17 - Cells refreshed from the run’s re-verifications: AI Release Tracker’s run-by cell names its operator (Peter Assentorp, per Business Insider) and its reader-pricing cell now carries the homepage’s $499/month sponsor slot (retiring “no ads”), LLM Stats’s run-by cell records the two-person self-description, and Vectara’s update-cadence cell moved to the README’s October 16 stamp; membership unchanged.
  • 2026-10-17 - Extended from eleven to twelve columns with whatmodel.sh, the curl-first IRT composite with machine-fit estimates (sorted after Vectara Hallucination Leaderboard); the intro, sorting guide, and choosing prose updated.
  • 2026-10-21 - Coverage cell refreshed: Benchmark Radar to 35,064 records from 40 sources as of 2026-10-21; membership unchanged.
  • 2026-10-23 - Cells refreshed from the run’s re-verifications: BenchLM to 972 model profiles, 668 benchmark pages, and 227 ranked entries (87 Supported) with its update-cadence cell moved to the board’s Oct 20 verification stamp, and Benchmark Radar to 36,762 records as of 2026-10-23; the Epoch AI run-by cell matched to its note’s 57-listed team; membership unchanged.
  • 2026-10-25 - Extended from twelve to thirteen columns with bestvaluemodel, the solo-built value-frontier view over Artificial Analysis’s Intelligence Index and prices (sorted between Benchmark Radar and Epoch AI); the intro, sorting guide, conflict, and choosing prose updated. Cells refreshed from the run’s re-verifications: Benchmark Radar’s coverage cell to 37,927 records as of 2026-10-25 and its run-by cell to the 15-author citation, and the Epoch AI run-by cell to 59 listed people.
  • 2026-10-28 - Coverage cell refreshed from the run’s volatile re-verification: Benchmark Radar to 39,640 records as of 2026-10-28; membership unchanged.
  • 2026-10-29 - Cells refreshed from the run’s rotation re-verification of whatmodel.sh: coverage to 888 ranked text models from 22,353 linked results across 575 benchmarks and the update-cadence stamp to 2026-10-29, and the data-access, reader-pricing, independence, verification-hook, and run-by cells corrected for the now-public MIT repository (jakemanger/whatmodel) and the 2026-10-19 terms’ paid-sponsorship disclosure; membership unchanged.
  • 2026-10-31 - Cells corrected from the run’s rotation re-verification of Vectara Hallucination Leaderboard: the README states the rankings are computed by HHEM-2.3, Vectara’s commercial model, so the methodology-published, data-access, and verification-hook cells and the verification-strength prose now name HHEM-2.1-Open as the downloadable open variant rather than the judge itself; membership unchanged.
  • 2026-11-01 - Cells refreshed from the run’s volatile re-verification of Margin Lab (coverage now names GPT-6.1 Sol/high on the Codex stack after the 2026-10-30 model switch) and Benchmark Radar’s badge to 41,974 records as of 2026-11-01, and from the rotation re-verifications of Artificial Analysis (coverage to 718 models) and BenchLM (coverage to 1,004 profiles, 724 benchmark pages, and 216 ranked entries with 77 Supported, and the update-cadence stamp to Oct 29); membership unchanged.

See also
#

References
#