↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Automated research/

Arbor

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 4 minutes read (~756 words)
agent-curated 512 automated-research 31 fully-ai-generated 615 held-out-validation 1 hypothesis-tree 1 llm=glm-5.3-flash 523 multi-agent 45 open-source 165 research-note 482
Table of Contents

Arbor is the Apache-2.0 autonomous research framework from RUC NLPIR and collaborator labs in which a long-lived coordinator grows a persistent hypothesis tree, short-lived executors test branches in isolated git worktrees, and a change joins the trunk only if it clears a configurable margin on held-out data.

Arbor is the member that makes overfitting the metric a merge-gate problem: experiments are judged on a held-out split the executors never optimize against, while the headline claim, a 2.5x improvement over Claude Code and Codex at the same compute budget, is the project’s own benchmark on its own six tasks.

What it is
#

A Python package (pip install arbor-agent, v0.1.4) described in the June 2026 paper “Toward Generalist Autonomous Research via Hypothesis-Tree Refinement” (arXiv 2606.11926, Jiajie Jin, Guanting Dong, and a long author list spanning Microsoft Research and Renmin University). Give it a repository, an evaluator, and dev plus held-out data splits: the coordinator proposes hypotheses, executors implement and run them in fresh worktrees, evaluation results return, the tree updates, and distilled lessons propagate to later branches. The merge gate is the design point: a verified improvement merges only when it clears merge_threshold, a configurable margin on the held-out test split, so main is never touched until you merge a gain that survived data the search never saw. The README’s worked example: a six-cycle run moved a dev speedup from 1.01x to 7.77x with the held-out gate enforcing every merge.

Status
#

Active-adjacent and early: 1,106 stars, created 2026-05-31, last pushed 2026-09-08, with the release train paused at v0.1.4 since 2026-07-12, as of 2026-10-12.

Star History Chart

The 2.5x benchmark claim drew VentureBeat coverage on June 18, 2026, and the article sat behind a bot checkpoint when this run tried to fetch it, so the claim stands on the repository’s own announcement. The rest of the community footprint is absent: this run’s searches found no relevant Hacker News thread.

Strengths
#

  • The held-out merge gate is the sharpest anti-overfitting mechanism in the category: CORAL isolates the grader from the agents, Arbor withholds the evaluation data until the merge decision.
  • The persistent hypothesis tree carries lessons across branches instead of forgetting failures, Kapso’s memory bet with CORAL’s git-worktree execution.
  • Executors are stateless and disposable, so every claim arrives attached to the worktree that produced it.
  • The ergonomics are shipped, not promised: a pip package, a replayable demo, docs, and a documented merge-margin knob.

Cautions
#

  • The 2.5x comparison is the project’s own, on its six chosen tasks, with no independent replication surfaced in this run’s searches.
  • The release train stopped at v0.1.4 in July and the last push was September 8, so momentum trails the README’s energy.
  • 1,106 stars with a month of quiet sits below this category’s adopted tools on community mass.
  • The gate trusts your splits: a leaky dev/test boundary restores the overfitting the gate exists to block, and supplying clean splits is the user’s job.

Pricing
#

Free and open source under Apache-2.0; there is no paid tier, so pricing does not apply. Costs are your own model API keys (coordinator plus executors) and your compute.

Compared to
#

  • CORAL: the other worktree-and-grader organization; CORAL isolates the judge from the agents, Arbor withholds the judge’s data until the merge gate.
  • Kapso: the campaign framework that tree-searches coding-agent experiments; Arbor’s tree is persistent and merge-gated rather than budget-scored per node.
  • Karpathy Autoresearch: the keep-or-revert ancestor; Arbor is the multi-branch descendant where failed hypotheses stay in the tree as lessons.

Bottom line
#

Recommended for teams with a measurable optimization target and clean dev and held-out splits that want an experiment organization whose gains must survive unseen data before touching the trunk. Not for anyone needing a maintained release train or independent benchmark evidence, both absent as of 2026-10-12.

Changes
#

  • 2026-10-12 - Created.

See also
#

References
#