Google Harness2: What It Is, How to Run It and What a Real Run Cost

A step by step guide to Google Research's Harness2: run the free demo, install a benchmark, price a run, read a real $22 business task, block rubric leaks and plug in your own tasks.

Jahanzaib Ahmed
Updated 36 min read

Explore with AI

Harness² title card with a frozen glass cube inside copper rings of rotating file tiles and a looping arrow, with the tagline Setup, cost and risks

Google Research released Harness2 (written Harness² in the paper, and often searched as "harness squared") on October 8, 2026. It is an Apache 2.0 framework that rewrites the prompts, skills, scripts and plugins around a frozen model while it works on a task, then saves what it learned for the next one. The repo has passed 21,000 stars as of November 1, and its paper reports gains of up to 11.5 points on WorkBuddy-Bench.

This guide covers a first run, what a run costs, what the release leaves unprotected and how to use it on your own tasks. I read the setup docs, the core modules, the paper and the open issues, and I ran the credential free demo, the help output, the setup dry run and the install check myself at commit 1cfe254e on October 18. I did not run a full benchmark, because that spends real money, but I did run one model backed business task on October 22 and report what it did and what it cost in "A real run on a business task" below. Where a result comes from my own run, I say so. One command the README tells you to run fails today (Step 2).

Updated November 1, 2026. The repository has not changed since commit 1cfe254e on October 14, so every command below still matches it. I checked the tracker again, and every pull request and issue cited in this guide is still open.

Harness2

Apache 2.0Version 0.1.0

  • Runs on Linux is documented (Python 3.12 or 3.13 and uv). The demo and one JobBench task ran natively on my Mac. Windows is not documented or tested here.
  • Local demo Credential free: harness2-demo --output runs/demo --k 2
  • Best result +11.5 points (GLM-5 under OpenCode, WorkBuddy-Bench), from the paper
  • Cost No spend cap in the code. My one real task cost $22.04 as run, about $15.7 clean, at list price.
  • Repo status 21,670 stars, 147 open items and no merged pull request on Nov 1

Checked Nov 1, 2026 against commit 1cfe254 and the paper.

The nine steps at a glance

Roadmap of the nine Harness2 steps from the free demo to your own tasks, with steps 4 and 5 marked as spending model credits
The path this guide follows. Price the run (step 7) before you start the refinement (step 4), because only the demo is free.
StepWhat you doMain command or fileCost
1. Run the demoProve the install works with scripted editorsharness2-demoFree, no credentials
2. Models and credentialsPoint it at Vertex AI.env and gcloud auth application-default loginFree
3. Install a benchmarkSet up LAB, JobBench or WorkBuddy-Bench and dry run itharness2-setup and harness2-checkFree, but it downloads code and data
4. One small refinementRun one task at k of 1harness2 --bench lab --k 1Model credits
5. Judge and reportScore the chosen output and read the resultharness2-judge and harness2-reportJudge credits
6. Lock down the networkBlock the hosts that hold the answersAn egress firewall or DNS policyFree
7. Price the runWork out calls and dollars before you startCost estimatorFree
8. Read the adopted harnessOpen the files the improver wroteruns/ and library/Free
9. Your own tasksWrite an adapterBenchmarkAdapterFree to write, credits to run

Requirements and costs before you start

Harness2 is a research release: documented for Linux, Vertex AI by default and heavy on third party benchmark code. The package itself has no runtime dependencies. The weight is in the benchmarks it drives.

NeedFor whatSource
Linux and Python 3.12Everything. Python 3.13 is also declared supported for the packageSetup guide
uvCreating the environment and installing the packageREADME
Node.js 20 or newer, plus git, curl, unzip and ca-certificatesSetup uses npm to install a pinned Bun runtime and the Codex CLI. The setup guide lists the packages in an apt lineSetup guide
Google Cloud CLIApplication Default Credentials for Vertex AI modelsSetup guide
Pandoc and PopplerThe LAB legal benchmark onlySetup guide
Docker EngineWorkBuddy-Bench only, and you must be able to run docker infoSetup guide

Which platforms work

PlatformStatusBasis
LinuxDocumented. The README says to use Linux, Python 3.12 and uvREADME
macOSNot claimed by the authors, but the demo, the install check and one JobBench task ran natively on my Apple silicon Mac (Python 3.12.12 from uv). I did not install LAB or WorkBuddy-Bench thereMy runs
Windows 11Not documented, and I did not try it. I would start in a Linux environment such as WSL2 or a Linux virtual machine, which is my assumption and not something the authors sayNot tested
DockerNeeded for WorkBuddy-Bench only. docker info must workSetup guide

Each setup run downloads a pinned Bun runtime, clones two upstream repositories at fixed commits (OpenCode and the benchmark you chose), applies patches, builds a separate Python environment for the benchmark, and downloads task data. JobBench and WorkBuddy-Bench fetch theirs from Hugging Face, while LAB tasks come with its pinned checkout. I have no install time to report.

Runs cost money. Every model backed command spends credits on executor runs, improver sessions and judge calls, and the docs recommend starting with a small task selection. Step 7 shows the arithmetic: the repository documents exact counts of runs and sessions, though not prices.

Google Harness2 in plain terms

A harness is everything around a model that lets it act as an agent: the system prompt, tool guidance, memory, reusable skills, helper scripts, subagents and plugins. Most agent work holds the model fixed and tunes that scaffolding by hand. Harness2 automates the tuning per task, while the agent works on something it has not seen, and adds a persistent library on top. Its authors describe two connected loops.

The first loop works inside one task. The pipeline runs your current harness once to see how it behaves. Then it proposes two edited harnesses: an invariant one aimed at general weaknesses such as planning, tool use, context management or delegation, and a domain one aimed at the knowledge, procedures and constraints of the current field. It runs both edited harnesses on the same task and compares what happened against the reference run. Finally a composer session reads the diffs and the evidence and builds a successor harness, which it may do by merging edits, revising conflicts, adopting one candidate or keeping the original. The authors name the three phases Propose, Probe and Compose, and the key design choice is that no step sees a score or a rubric. Decisions have to be justified by observed behavior.

Diagram of Harness2's loop: a reference run, then propose, probe and compose steps, an adopted harness, and a library whose lessons feed the next task
One task through Harness2. The library on the right is the second loop: lessons from this task become evidence for the next one.

The second loop works across tasks. After each task, a reflection step reads the whole improvement record, keeps the adopted harness and writes down lessons that the execution contrasts support. Both go into a per task library, grouped by domain. A later task in the same domain can read earlier harnesses and lessons before it proposes anything. The paper calls the thing that does all this the improvement harness, to keep it apart from the execution harness that does the actual work.

Interactive explainer

How Harness2 refines a harness

Press Play to walk through one task, or switch tabs for the budget and the eight files.

Step 1 of 5

Run it once

The agent runs the task with the base harness. This reference execution is what every candidate is compared against.

Without a reference run there is nothing to compare the candidates against.

Reads: the base harness files

Source: the Harness2 paper (arXiv 2610.14224) and the repository README.

Two knobs control cost and quality. The budget k sets how much exploration each task gets. The mode picks how that exploration is spent. Parallel mode probes k invariant and k domain candidates and then writes k independent compositions from the whole pool. Sequential mode runs k Propose, Probe, Compose steps in a chain and carries the refined harness and the accumulated evidence from one step to the next. At k of 1 the two modes are the same single step.

Diagram comparing parallel recursion, where candidates are composed together, with sequential recursion, where each step refines the previous harness
Parallel recursion spreads out and merges. Sequential recursion goes deeper one step at a time.

The release supports two executors, OpenCode by default and Codex, and three benchmarks: JobBench for professional work, LAB for legal work and WorkBuddy-Bench for office, web and code tasks. It also contains the pieces you need to plug in something else, which Step 9 covers.

Step 1: Run the credential free demo

Start here, because it proves the plumbing without spending anything. The README gives these commands from the repository root. I ran them in a fresh clone at commit 1cfe254e:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

The install pulled eight packages in a few seconds: Harness2 itself plus openpyxl, python-pptx, lxml, pillow, xlsxwriter, et-xmlfile and typing-extensions. The demo then finished in about 1.6 seconds and printed this:

  [accounting] 1/2 ok           ledger-1
  [accounting] 2/2 ok           ledger-2
Local demo: 2 tasks completed; 0 failed. .../runs/demo/demo.json

No model was called. The demo runs the real recursion and library code against a local adapter with scripted editors, and it executes real Python scripts as the agent. The task is a ledger reconciliation. The base harness ships scripts/summarize.py with a filter that keeps only positive amounts, so it drops credits. For the first ledger, the base run printed Total: 105.25. Every edited candidate printed Total: 85.15, which is the correct signed sum of 100.25, minus 20.10, plus 5.00. For the second ledger the base total was 35.50 and the candidates gave 20.25. The scripted editor removed the filter, the scripted selector picked an output whose total matched the source ledger, and reflection wrote down what happened.

The reflection step wrote an EXPERIENCE.md for each adopted harness. This is the one for the second task in parallel mode, trimmed and with the file's Markdown bold removed:

- adopted: `agg2`  (run `demo-oc-agg2/ledger-2/agg2`)
## Why this was chosen
The signed total matches the source ledger.
## Transferable lesson
- cue: When a ledger includes credits and debits
- lesson: Include both signs when summing in scripts/summarize.py.
- confidence: high for this observed arithmetic contrast

The result file demo.json also records "execution_count": 28 and "editor_calls": 32 for that run, and the first task shows library_entries_before of 0 in both modes while the second shows 1. That is the library working: the second task could read what the first one left. Those two counts are worth a closer look, and Step 7 does the arithmetic with more runs. You can restrict modes with --mode parallel or --mode sequential. The default is both.

Step 2: Set up models and credentials

The provided configuration targets Google Vertex AI. The README tells you to run cp .env.example .env. I ran it and got cp: .env.example: No such file or directory, because the file is missing from the repository at commit 1cfe254e. An open pull request reports the same failure. Until it lands, create .env yourself from the settings in the table below, with placeholders swapped for model IDs your project can use:

GOOGLE_CLOUD_PROJECT=your-project-id
HARNESS2_SOLVER_MODEL=google-vertex/<a Gemini model ID>
HARNESS2_IMPROVER_MODEL=<provider/model, in the same form as the solver>
HARNESS2_JUDGE_MODEL=<a bare Gemini model ID>

Then authenticate with Application Default Credentials:

gcloud auth application-default login

Shell variables override the file. These are the settings that matter (the IDs above are placeholders because I will not guess which models your account can reach):

SettingMeaning
GOOGLE_CLOUD_PROJECTYour Vertex AI project. GOOGLE_CLOUD_LOCATION defaults to global
HARNESS2_SOLVER_MODELThe executor model, written in OpenCode's provider/model form
HARNESS2_IMPROVER_MODELThe model that edits and composes harnesses. Defaults to the executor
HARNESS2_JUDGE_MODELA bare judge model ID. The adapter adds the Vertex prefix
HARNESS2_RUNS_DIRWhere results go. Defaults to runs/

Three choices deserve a decision rather than a default. First, the improver. The paper ran its editor and selector on Claude Opus 4.8 at high reasoning effort, and its analysis section also tests the same model in both roles. With five models from GLM-5 to Claude Opus 5, every one improved on every benchmark at k of 2, which suggests a stronger improver is not required, though I would still not assume your cheapest model behaves the same. Second, the judge. Changing the judge model changes the measurement, so pick one and keep it for the whole experiment. Third, authentication. If you cannot open a browser on the machine, the docs list --no-browser login, a service account key through GOOGLE_APPLICATION_CREDENTIALS, and service account impersonation. Even with a key file, keep gcloud on your PATH, because the install check calls gcloud auth application-default print-access-token.

For a Codex executor you also need a Responses API endpoint. The release includes harness2-bridge, a local LiteLLM server that presents Gemini on Vertex AI that way, installed with uv pip install -e '.[render,bridge]'. The bridge prints the Codex settings to add to your .env and stores its port and generated key under dependencies/codex/. For WorkBuddy, which runs Codex inside Docker containers, the bridge must listen on 0.0.0.0, which exposes it on every network interface of the host. I come back to that in Step 6.

Step 3: Install one benchmark and dry run it

Install only the benchmark you plan to use, then check the install without calling a model:

harness2-setup lab
harness2-check --bench lab

Use jb for JobBench or wb for WorkBuddy-Bench. WorkBuddy also needs --subset office, web or code on every command after setup, and harness2-setup wb without a subset installs all three. Add --substrate cx to any command to use Codex instead of OpenCode. Setup clones the upstream repositories at pinned commits and refuses to overwrite a checkout at a different revision. The pin table in dependencies/PINNED.md lists them:

RepositoryUsed for
harveyai/harvey-labsLAB, the legal benchmark
Job-Bench/job-bench-evalJobBench evaluation code
Tencent/workbuddy-benchWorkBuddy-Bench
opencodeThe default executor

The release also carries four patch files. They fix a moving Git dependency in OpenCode, add judge changes for JobBench and LAB, and register the executor adapters for WorkBuddy while refreshing short lived credentials. Each benchmark gets its own virtual environment inside its upstream checkout, for example dependencies/harvey-labs/.venv. Harness2 picks those interpreters itself, so you keep the main environment active and never activate the others.

Run harness2-setup lab --dry-run first, especially if your machine sits behind a restrictive firewall. I ran it, and it prints the plan without changing anything. For LAB the plan is:

  • Initialise an OpenCode checkout, add github.com/anomalyco/opencode as its origin, fetch the pinned commit and apply a patch.
  • Install Bun 1.3.11 through npm and run bun install --frozen-lockfile --ignore-scripts.
  • Do the same clone, fetch and patch for harveyai/harvey-labs and copy the runner overlays in.
  • Run uv sync --locked --python 3.12 and add google-auth.

There is no data download step for LAB, which matches the docs saying its tasks ship in the pinned checkout. Setup talks to github.com, registry.npmjs.org, huggingface.co and its CDN, a Docker registry for WorkBuddy, and astral.sh when it needs to fetch Python 3.12. Those hosts matter again in Step 6, because the list of hosts setup needs overlaps with the list you must block during evaluation.

The check command is worth understanding. On my fresh clone, before any setup, harness2-check --bench lab ended with Setup incomplete: 9 issue(s), and the list included the missing solver model setting, no LAB tasks, a missing OpenCode checkout, failed patches and a missing pandoc. It reads like a to do list, and one line still points you to the absent .env.example. It verifies installed dependencies, credentials and configuration. It does not test whether your account can reach a given model. That test happens the first time a model is called, so a clean check does not guarantee a clean run.

A real refinement run repeats the same checks, including the judge configuration, before it starts any rollout. The code comment gives the reason: the namespace records the judge model and the judge command refuses to grade under a different one, so finding a missing judge after the sweep would mean paying for the sweep twice.

Two benchmark notes before you run anything. LAB needs Pandoc and Poppler. WorkBuddy-Bench has a custom Tencent license with a territorial limitation, and the docs tell you to read it before downloading or using the benchmark, since the derived runner files and patches keep that license. For WorkBuddy, setup refuses to run unless your solver model is a google-vertex/gemini ID and your judge a bare gemini ID with the project set. The fetch script it patches also requires a checksum file and will not unpack an archive it cannot verify.

Step 4: Run one small refinement

The README's example runs a single legal task:

harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1

Before you spend anything, add --dry-run. It prints the exact stream of domain and task pairs, the namespaces and the available domains, creates no model client and writes nothing. These are the flags you will use most:

FlagWhat it does
--kThe task level recursion budget
--mode (or --modes)How that budget is spent. Takes parallel or sequential, or a comma list of both, and defaults to parallel
--substrate cxUse Codex instead of OpenCode
--only-domainsRestrict the run to named domains
--max-tasksOne cap per domain, with no global cap
--subsetWorkBuddy only: office, web or code
--workers NRun N domains at once. Tasks inside a domain stay in order
--seed NShuffle the task stream reproducibly. It does not control model sampling
--split-fileUse a train and test partition made by scripts/split_tasks.py

Watch the --max-tasks wording. It caps tasks per domain, so on a benchmark with many domains, a cap of 1 still means one task in every domain you did not exclude. Always pair it with --only-domains for a first run. If you forget the WorkBuddy subset, you get a clear message: error: --bench wb needs --subset from ('office', 'web', 'code'), got None. Running harness2 --bench lab --dry-run before setup stops with a similar one-line error telling you to run harness2-setup lab.

The order of tasks matters more than it seems. Tasks inside a domain run in sequence and each one can read the library entries from earlier ones, so the stream order is part of the experiment. With no seed, the stream uses a fixed domain order and sorted tasks. With a seed, domains and the tasks inside each domain are shuffled with a generator seeded from the seed and the domain name. That detail has a practical consequence: dropping one domain from a sweep leaves the order of every other domain unchanged, so your partial runs line up with later full ones.

Results go into a namespace built from the benchmark, executor, mode, budget and an optional seed and tag, such as lab-oc-agg1 for a parallel run at k of 1. On disk the parallel mode appears as agg and the sequential mode as seq. Repeating the same command resumes the run. The code marks a rollout complete only when it saw it return, and it fingerprints the harness that produced it, so a rollout from a different configuration is never reused. When you change models or task selection, use a fresh HARNESS2_RUNS_DIR. The saved configuration is meant to block incompatible resumes.

Step 5: Judge the run and read the report

Refinement produces harnesses and deliverables. Judging is separate, and it runs afterwards:

harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

The judge and report commands must use the same benchmark, executor, subset, k, mode and seed as the refinement run. The method never imports the judge module, as its docstring says. The judge only scores the one output that selection chose, after it chose, and a judge failure returns None instead of zero. A missing score is excluded from the denominator and reported as unscored. The repository's rule is that a broken evaluation should not look like a bad agent.

The headline figure depends on the benchmark. For LAB it is Partial, the fraction of criteria passed, alongside All-pass, where every criterion passed. For JobBench it is the weighted rubric score. For WorkBuddy-Bench it comes from the verifier's own score file. If you want to know whether the refinement helped at all, run harness2-judge --bases, which also scores the reference executions of the base harness and writes base_judged.json next to judged.json. The report command covers only the refined harness, so the baseline comparison is a separate file you read yourself. A failed judge writes its output to judge.log in the run directory. A Vertex 429 RESOURCE_EXHAUSTED there means quota, so wait and rerun the judge.

Step 6: Lock down the network before you trust a number

The docs treat this step as mandatory, and it is easy to skip. All three benchmarks are public. JobBench publishes each task's RUBRICS.json on Hugging Face. The release does not restrict executor network access, and Codex runs with its sandbox disabled. In a validation run of this release, a Codex rollout on JobBench downloaded its own task's rubric without being told to. A number produced that way measures how well the agent looks things up.

The docs list what to block for the executor after setup is finished:

  • huggingface.co, hf.co and *.hf.co
  • github.com and raw.githubusercontent.com, for the benchmark repositories
Diagram of the Harness2 network lockdown: the setup hosts are allowed during installation, then huggingface.co, hf.co, github.com and raw.githubusercontent.com are blocked for the executor at the network layer
Block the hosts after installation, at the network layer, and keep a dated copy of the rules beside your runs.

Setup itself needs those hosts, so apply the restriction after installation. Because Codex has no sandbox here, do it at the network layer, with an egress firewall or a DNS policy on the executor host, and do not rely on a tool setting. The docs also ask you to report the restrictions you used next to your results. I would add one habit of my own: keep a note of the date and a copy of the firewall rules beside the runs/ folder, because a number without its conditions is hard to compare six months later.

Two more safety points come from the setup guide:

  • The bridge. The Codex bridge for WorkBuddy binds to all interfaces, so restrict who can reach it.
  • The machine. The executor is an agent with a shell, so run the whole thing on a machine or container that holds nothing you care about: no personal credentials, no company tokens, no mounted home directory.

Pull request 37 describes subprocesses inheriting the runner's full environment, so credentials in your shell may reach processes the run starts. Give the process access to nothing it does not need. A guardrail file the agent can edit is advice to the agent, so treat sandboxing as a job for the machine, not the prompt. The OpenAI wiki egress case shows what an open route out allows. The same habit applies to the tools an agent can reach: every MCP server a coding agent has loaded is another route out, so I audit them first with the steps in how to remove an MCP server from Claude Code before I leave anything running unattended.

Step 7: Price the run before you start it

Diagram of what one Harness2 task costs in calls per mode: 4k executor rollouts, 3k improver sessions, selection and reflection outside the ledger, and 7k rollouts when both modes share the base runs
The 4k and 3k counts come from the repository's ledger. Selection and reflection are extra calls, and judge calls and retries are not counted.

The repository documents the cost structure exactly. For every task, each mode schedules 4k executor rollouts and 3k improver sessions, where the k base executions are counted inside the 4k. The breakdown is k base runs, 2k candidate runs (k invariant and k domain) and k final runs. The 3k sessions are k invariant proposals, k domain proposals and k compositions. A ledger in the code asserts this identity per task and raises an error if the counts or labels do not match, which catches a stage that silently overwrote another. Retries are counted separately, and selection and reflection sessions are recorded outside the ledger.

WhatAmountBasis
The credential free demo$0No models are called
One task, one mode, k of 14 rollouts and 3 improver sessions, plus 1 reflection callThe repository's ledger and my demo counts
My one real task (HR, Gemini 3.8 Flash)$22.04 as run, about $15.7 cleanList prices, OpenCode totals, one task
Full LAB stream, k of 14,572 rollouts and 3,429 sessions1,143 tasks, my arithmetic, before judging and retries
Full LAB stream, k of 522,860 rollouts and 17,145 sessionsSame arithmetic

The same counts by budget, per task:

Budget kRollouts per taskImprover sessions per task
143
286
3129
52015

I checked these formulas against the demo, which logs how many executor runs and improver calls it makes. With two tasks and one mode, the counts were exactly what 4k and 3k predict:

Demo run, 2 tasksExecutor runsEditor callsReading
k of 1, one mode882 x 4 runs. 2 x (3 sessions + 1 reflection). No selection at k of 1
k of 2, one mode16162 x 8 runs. 2 x (6 sessions + 1 selection + 1 reflection)
k of 3, one mode24222 x 12 runs. 2 x (9 + 1 + 1)
k of 5, one mode40342 x 20 runs. 2 x (15 + 1 + 1)
k of 1, both modes1416The base runs are shared: 16 minus 2
k of 2, both modes283232 minus 4 shared base runs

The table also settles two questions the docs leave open. Selection and reflection are real extra calls on top of the 3k sessions, and running both modes costs a little less than double in executor runs because the base executions are shared. Editor calls simply add up. In a real run each of those calls is a model session, so the dollar figure is yours to work out. These are call counts from scripted editors, not prices.

Scale that by tasks. The docs say the full LAB stream has 1,143 tasks. My arithmetic: one mode at k of 1 is 4,572 rollouts and 3,429 improver sessions, and at k of 5 it is 22,860 rollouts and 17,145 improver sessions, before judging and before any retries. Multiply by what one rollout and one improver session cost on your models and you have a floor.

It is a floor because above k of 1 an agentic selector session compares anonymized deliverables and picks one, and reflection runs after adoption. Neither is in the ledger. At k of 1 selection is skipped.

In the paper's setup each rollout is capped at 200 model turns and 3,600 seconds, except WorkBuddy-Bench, which uses its container timeout at twice the benchmark default, so a single long rollout can be expensive. My Harness2 cost estimator applies these counts to your own task count, k and prices.

By default the code runs two rollouts at a time per phase and allows two improver sessions at once, and WorkBuddy runs one job at a time with two trials. The --workers flag adds concurrency across domains. More concurrency shortens the wall clock time but does not change the bill.

The repository has no spend cap yet. An open proposal asks for a --max-usd limit, so until that lands the only guard you have is your own task selection and a budget alert on your cloud project. Set that alert before the first run. The ledger in the code records realized dollars for improver sessions but logs executor rollouts without a cost, so take executor spend from your cloud billing. A follow up pull request, #127, would record executor cost for OpenCode, but it is unmerged as of November 1, so on the main branch executor spend is still logged without a cost.

In the demo, each task's summary.json carries a ledger block with R, expected_R, P, expected_P and the dollar fields, all zero there because the editors are scripted. Run one real task at k of 1 first and compare that block with your billing.

The paper adds the quality side of the trade. Parallel recursion at k of 5 on Gemini 3.5 Flash under OpenCode raised the gain over the base harness from 4.5 to 7.5 points on JobBench and from 5.1 to 8.4 on WorkBuddy-Bench compared with k of 1. Both recursive forms at k of 2 beat every baseline the authors plotted at k of 5 on two WorkBuddy domains. The authors say they tested a bounded range of budgets and did not look for the best one. My read: start at k of 1, look at what the adopted harness changed, and only then decide whether a larger budget is worth four times the rollouts.

Warning: set a budget alert on your cloud project before the first run. The code has no spend cap, and the open proposal for a --max-usd limit has not been merged.

Note: my one real run, a single HR task on Gemini 3.8 Flash at k of 1, added up to $22.04 at list prices. About $6 of that was a mistake of mine (two copies of the harness running at once), so a clean single task came to about $15.7. Most of it is the agent re-reading its own long context. The breakdown is in "A real run on a business task" below.

Step 8: Read the adopted harness and the library

Output lands under runs/lab/, runs/jb/ or runs/wb/<subset>/. The demo shows the layout, and benchmark runs may arrange the executor folders differently. Each run namespace, such as demo-oc-agg2, has one folder per task, and what is inside depends on the mode:

  • Parallel namespace. Each task folder holds a summary.json that covers both modes with one block per mode, a context/ folder per improver stage, the harnesses/ tried, a selection/ record and a reflection/ record.
  • Sequential namespace. It keeps its own context/, selection and reflection folders but no separate summary.
  • Executions. Executor runs live in a separate executions/ tree. In parallel mode for k of 2, each task got base1 and base2, inv1 and inv2 for the invariant candidates, dom1 and dom2 for the domain candidates, and agg1 and agg2 for the compositions. Sequential mode used c1 and c2 for its chained steps and reused the shared base runs.

The adopted harness is the thing to read, because it shows what the improver decided mattered.

Two records are worth opening early. selection/agg2/selection.json (the folder is named after the final) shows the selector seeing anonymized candidates: in my run it mapped A to agg2 and B to agg1, chose A, recorded "chosen_is_base": false and gave its reason. And summary.json lists every stage with its dimension (such as INV, DOM or AGG), its attempts, any fallback and its notes, plus a similarity block. In my run mean_line_jaccard was 1.0 across the two finals, which is what you get when scripted editors make the same edit twice.

A harness here is a tree of eight components. The paper's description of the editable surface under OpenCode lists them, and the code's placeholder files label the last four as c5 to c8, which leaves c1 to c4 for the four always on files:

PathWhat it isWhen it costs context
systemprompt.mdRole and procedureAlways
guardrails.mdStanding rulesAlways
LongTermMEMORY.mdDurable domain factsAlways
tool_descriptions/tool_guidance.mdHow to use the toolsAlways
sub_agents/<name>/AGENT.mdA delegated subagent in its own contextWhen called
skills/<name>/SKILL.mdA reusable workflow loaded on demandWhen invoked
scripts/<name>.pyA deterministic helper run through the shellWhen run
plugin/<name>.tsA plugin that acts inside the tool loopRuns on tool calls

Codex lays the surface out differently. The always on text lives in AGENTS.md, skills sit under .agents, and the plugin lever becomes a hooks file under .codex. The stock harness starts nearly empty, and each on demand folder holds only a placeholder that names the lever, which is how the editor learns the component exists.

The code also guards the edit surface. Under OpenCode, always on text is capped at 24,000 characters, with the code noting that every token there competes with the agent's own output budget. Codex has a smaller cap.

A skill needs a name that matches its folder and a description, and it must be named in the always on text so the model knows to load it. A script must compile, and a helper that a skill calls must ship with the skill. An invalid proposal gets one retry with the validator's message, and if it fails again the stage keeps the harness it started with and records that fallback. An adopted harness can therefore end up identical to the base one, either because validation failed twice or because the composer chose to preserve the current harness.

Look next at the library. In the demo the library sat at library/k2/agg/accounting/ and library/k2/seq/accounting/: one folder per mode and domain, with an INDEX.md listing earlier tasks, and for each task the full adopted harness tree plus an EXPERIENCE.md explaining why that harness was picked and what weakness it addressed. The code keeps the whole tree, not a prose summary, so executable pieces survive intact. Libraries are scoped per run and mode, so each sweep starts every domain empty and a library does not carry over between experiments by default. The paper studies transfer separately, and I did not dig into that part.

One more thing to read: the evidence the improver saw. It receives its inputs as files in a working directory: the task, full trajectories with no prompt budget truncation, deliverables, the current harness, the library and a mission file. Guards refuse to hand over a directory that contains a grader verdict, a rubric or a score. The paper reports a case where a baseline run scored 43.1 because it never reconciled data across source files and never reopened outputs to verify them. The invariant edit added a guardrail forcing an input inventory and a verification script. The domain edit added a skill for reconciliation. Read your own adopted harnesses with that lens: edits that fix process failures tend to be more portable than edits that restate domain facts.

Step 9: Plug in your own tasks with an adapter

The shipped command line only knows the three benchmarks. The release does include the seam you need for your own. The core method is written against a protocol called BenchmarkAdapter, and the docstring says that if you implement it, the whole method works. An open proposal on the repository asks for a --bench local mode that runs on a folder of tasks, so this may get easier, but today you write an adapter.

The adapter has to supply these pieces:

  • tasks(split), a stable ordered list of task ids, because library state must be reproducible.
  • domain_of(task_id), the grouping unit such as practice area, occupation or customer type. Tasks in one domain share a library.
  • task_prompt(task_id), the task as the agent saw it, with criteria and expected output stripped.
  • task_files(task_id), optional source material.
  • execute(task_id, harness_dir, run_id), which runs one rollout and returns its directory, idempotently by run id. A harness directory of None means the unmodified base harness.
  • delivered(run_dir), trajectory(run_dir), deliverable_text(run_dir) and optionally deliverable_files(run_dir), so the improver can read what happened and what was produced.
  • base_harness(), a directory with your starting component tree, and validate_tree(), which raises an error whose message is fed straight back to the improver as retry feedback.
  • score(run_dir, task_id), which only the judge calls, and which returns None when judging fails.

A static description, BenchSpec, tells the method your install mode, the proposer and selector model channels, glob patterns for files that must never reach the improver, and whether subagents are allowed. The code rejects measurement words in the BenchPrompts text, namely benchmark, evaluation, grader, rubric and score, which is a small piece of leak prevention I found clever. The shipped demo adapter is the best working reference, and it is short enough to read in one sitting. Here is a skeleton, an untested sketch based on the protocol and that demo adapter, not a working integration:

class MyAdapter:
    spec = BenchSpec(
        name="support-replies",
        runs_root=root / "runs",
        install_mode="workspace_config",
        prompts=BenchPrompts(work="answer customer questions from the help center"),
        channels={"proposer": channel, "selector": channel},
        leak_globs=("expected_reply.txt",),
    )
    substrate = OpencodeSubstrate()

    def tasks(self, split="all"): ...
    def domain_of(self, task_id): ...
    def task_prompt(self, task_id): ...
    def execute(self, task_id, harness_dir, run_id, *, timeout=None): ...
    def delivered(self, run_dir): ...
    def trajectory(self, run_dir, *, obs_trunc): ...
    def deliverable_text(self, run_dir): ...
    def base_harness(self): ...
    def validate_tree(self, tree): ...
    def score(self, run_dir, task_id): ...

Before you write it, ask one question the docs only imply: can your task be run several times safely? The Probe phase executes candidate harnesses on the same task, and k above 1 multiplies that. For analysis, document drafting or code that runs in a throwaway directory, that is fine. For a task that sends email, edits a live database, spends money or posts anywhere, repeated runs are repeated side effects, and with network access unrestricted and the Codex sandbox off, the blast radius is yours to contain. That is my judgment, not the authors' statement, and it is the main reason I would start with offline tasks.

A real run on a business task

Everything above comes from the repository and the paper, so I also ran it for real. On October 22, 2026 I ran one JobBench task on my Mac, with Gemini 3.8 Flash as the executor, the improver and the judge, at k of 1 in parallel mode. It is one task and one run each. It shows what a run looks like and what it costs. It does not show that Harness2 works in general.

58 to 64Score out of 64, base to adopted harness, one run each
$22.04Cost as run, at list price
about $15.7Estimated cost of a clean single task
41.5M vs 0.4MCached input tokens against output tokens

The scenario

The task comes from the Human Resources Specialists folder. A biotech company has worrying staff turnover and the VP of HR has an executive review coming. The agent gets three files: past employee attrition data, a list of job applicants, and a sheet of requirements for each open position. It has to produce a Word report and an Excel workbook. The report must find the three most important problems, compare what departed employees were paid against the current pay bands, show which roles lose people too fast and how overtime relates to leaving, and screen every applicant against the position rules.

It is harder than it sounds. Salaries in the attrition file are monthly while the pay bands are annual, and nothing says to multiply by 12. Education arrives as a numeric code that only means something after you join it to a lookup sheet. Licensed roles need both an acceptable license type and an Active status. Overtime has to be compared role by role, or the effect hides inside the average.

What happened

HarnessScoreRubric items passedChange
Base (almost empty)58 of 647 of 8Start
Adopted by Harness264 of 648 of 8+6

Source: my run of JobBench human_resources_specialists/task1, rubric weighted, judged by gemini-3.8-flash, one run each.

Provenance: one missed rubric item, weight 6 of 64, is the whole difference, and the base was already at 90.6 percent. I scored both outputs by calling the JobBench scoring code directly, because I stopped the run before its reflection step to stay inside my spend limit. The judge is the same model family as the executor, which can favor its own style. I did not read the rubric.

The base harness is almost empty: every file is a placeholder. The harness Harness2 composed for this task has a role prompt, fourteen lines of domain memory, five guardrails, a tool guidance file, a 13.9 KB skill with a screening procedure and a 7.2 KB script that checks the deliverables before the agent finishes.

Table of the harness files before and after Harness2 for one HR task, with sizes in bytes, and the score going from 58 of 64 to 64 of 64
What Harness2 wrote for the HR task. Sizes are from the real files. The score is one run each, so it is an anecdote.

One effect I could check in the files. The improver noticed from the base run that LibreOffice was not installed, so Excel formulas written by a script carry no stored value and any reader other than Excel sees blank cells. It wrote that into the tool guidance and into the check script. The base workbook has 256 formula cells and all 256 have no stored value. The refined workbook has 81 formulas and none are missing a value. I recounted both files myself. I cannot say this earned the extra points, because I did not read the rubric.

Two cards: the base workbook has 256 formula cells with 256 missing a stored value, the refined workbook has 81 formula cells with none missing
Counted from the two Excel files the run produced.

What it cost

The eleven Gemini 3.8 Flash sessions added up to $22.04 at list prices, using OpenCode's own cost totals. About $6 of that was my mistake: I started a second copy of the harness on the same run directory while the first was still alive. A clean single run would have been about $15.7. The judge added an estimated $0.8 to $2.0, because it does not log tokens.

Bar chart of the cost of one Harness2 task by stage, as run and as a clean single run, totalling 22.04 dollars as run and about 15.7 dollars clean
Cost by stage at list prices. The clean column is an estimate from the same sessions.

Most of that money is the agent re-reading its own long context. Across the eleven sessions there were 41.5 million cached input tokens against 0.4 million output tokens. These are list rates. Google lists Gemini 3.8 Flash at $0.75 input, $3.75 output and $0.075 cached input per million tokens through December 31, 2026, doubling to $1.50, $7.50 and $0.15 on January 1, 2027, so read my figures as an upper bound while the introductory price runs. At about $15.7 per task, four tasks did not fit under my $25 limit, so I stopped after one.

What tripped me up on a Mac

  • macOS has no GNU timeout command and the JobBench runner calls it. The first launch died at once, before any model call, and a small stand-in script fixed it. The problem is now tracked upstream in issue #77 and pull request #128, both still open.
  • The docs say Linux, but JobBench ran natively on macOS with no Docker.
  • The executor can reach Hugging Face, where JobBench's rubrics are public. I could not change firewall rules, so I ran a local proxy that refuses huggingface.co, github.com and similar hosts. It saw no blocked attempts, but it only stops programs that honor proxy settings, so it is weaker than a network block. An open pull request, #118, notes that if the executor fetches a task's own rubric, that text lands in the trajectory the improver reads, so the leak can end up inside the adopted harness. Block the hosts before the run, not after.
  • When two OpenCode processes started together they hit a database error and the harness retried on its own.
  • The call counts did not match 4 rollouts and 3 improver sessions. I saw 7 and 4, because of my duplicate process. I also stopped before the reflection step, so there is no lessons file from this run.

What I take from it

The pipeline works end to end on a Mac and produces files you can read and diff. Budget before you run it, because one task at k of 1 cost real money. And treat one task as an anecdote: the base was already near the top, so a single rubric item moves the score by about nine points.

Reading the paper's numbers

The paper, submitted on October 9, tests two models, GLM-5 and Gemini 3.5 Flash, under two executors, OpenCode and Codex, on the three benchmarks. The headline claims are one step gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB's Partial pass rate. Those maxima come from different configurations, so do not read them as one setup's results. The authors' project page shows the Gemini 3.5 Flash numbers under OpenCode for the three WorkBuddy domains:

DomainBase harnessMeta-HarnessHarness2 at k of 5Gain over base
Code73.475.580.8+7.4
Web69.975.178.3+8.4
Office74.078.683.4+9.4

Source: the Harness2 project page, Gemini 3.5 Flash under OpenCode, WorkBuddy-Bench domains.

Provenance: the first four columns are copied from the project page. The gain column is my subtraction (Harness2 minus base), not a figure the authors print.

A weaker executor with a better harness can pass a stronger one with its original harness. After one step, GLM-5 reached 75.7 under OpenCode on WorkBuddy-Bench, above the base Gemini score of 72.4. Other methods were less steady. The authors report that ReasoningBank fell as much as 6.6 points below the base harness on coding tasks, and Harness Scaling underperformed the base in seven of twelve benchmark and configuration pairs. On LAB Partial, Meta-Harness led for Gemini 3.5 Flash under Codex, so the picture is not a clean sweep.

The ablations show what carries the weight. Removing the library lowered the WorkBuddy-Bench and JobBench scores by 3.1 and 1.8 points. Removing the invariant and domain scoping cost 2.9 and 3.7. Removing composition cost 1.6 and 1.9. Every component contributed, and scoping mattered most on JobBench. All of that was measured at k of 1 with Gemini 3.5 Flash under OpenCode.

Limits of these numbers: they are the authors' measurements, averaged over three runs with shuffled task order. The repository's pin file warns that historical paper runs may have used different tasks, model deployments and extra local tooling than the public workflow. The authors' own limitations section says the signals depend on execution variability and shared model biases, that they do not guarantee correctness, and that an adopted change may not transfer to another task. Cost is also real: every extra probe and composition is more inference. I treat the paper as strong evidence that the method works on these benchmarks and as no evidence yet about your tasks.

Troubleshooting

SymptomLikely causeFix
cp: .env.example: No such file or directoryThe file is missing from the repositoryCreate .env by hand, as in Step 2
harness2-check lists many failures on a fresh cloneNothing is installed yetRun harness2-setup <bench>, then check again
--bench wb needs --subsetWorkBuddy needs a subset on every commandAdd --subset office, web or code
Check reports stale Codex configurationYou changed the Codex endpoint or model after setupRerun harness2-setup <bench> --substrate cx
Judge writes nothing and judge.log shows 429Vertex quotaWait and rerun harness2-judge
WorkBuddy setup refuses to runModel IDs are not in the required form, or the project is unsetUse a google-vertex/gemini solver ID, a bare gemini judge ID and set GOOGLE_CLOUD_PROJECT
Judge or report finds nothing, or refuses to gradeArguments differ from the refinement runMatch benchmark, executor, subset, k, mode and seed exactly
Resume does not pick up a different setupThe saved configuration blocks incompatible resumesUse a new HARNESS2_RUNS_DIR
split_tasks.py is missing from a wheel installThe partitioner lives in the source treeWork from a source checkout for splits
Docker errors on WorkBuddyYour user cannot run DockerConfirm docker info works first
Unscored tasks in the reportA judge or verifier result was missingRerun the judge. Unscored is not counted as zero

The repository was ten days old when I first read it, and its open pull requests show real rough edges. As of November 1 the tracker held 147 open items, 107 pull requests and 40 issues, and no pull request had been merged. On October 23, nobody with a maintainer role had commented on any thread. Several touch resume behavior, such as re-running tasks when prompt text changes or after an improver outage, and one reports tracebacks for some invalid arguments to the judge and report commands. Expect early friction, pin the commit you used, and write it down with your results. Use the google-research URL linked above: lookalike forks already exist.

When Harness2 is the wrong tool

It fits best where you have a repeatable task type, a stream of similar tasks, a way to re-run safely and a measurement you trust. It is a poor fit in five cases.

Your situationWhy it is a poor fitWhat to do instead
Tasks with side effects you cannot repeatThe Probe phase runs candidates on the same task, and k above 1 multiplies it (Step 9)Start with offline tasks in a throwaway directory
You need a hard security boundaryNothing in the release provides oneRun it on an isolated machine and block the hosts (Step 6)
One task type and one deploymentHand tuning may cost less than a recursive search and be easier to explainTune against a real eval harness with human review
You cannot afford the multiplierAt k of 1 each task costs four rollouts and three improver sessions per mode, compared with one rollout for the plain agentPrice it first with the cost estimator
You need one fast yes or no call per message, such as sorting an inboxA decision model answers that directlyI tested one in What Is Jev AI? and Jev vs Laya on 16 test messages

Where it does fit, I would use it as a research instrument before production machinery. Run it on a held out set of your own tasks, read the adopted harnesses and keep the edits that make sense to a human. Process fixes, like inventory and verification steps, tend to port well, which is what agentic engineering work often comes down to.

If you would rather have someone build and measure this kind of agent setup for your business, I do that: see how I build AI agents.

Frequently asked questions

What is Google Harness2?

Harness2 is an open source research framework from Google Cloud AI Research. It refines the prompts, skills, scripts and plugins around a frozen model for each task, using a Propose, Probe, Compose step, then keeps adopted harnesses and lessons in a library to guide later tasks. Its paper is arXiv 2610.14224.

Is Harness2 an official Google product?

No. The README says it is not an officially supported Google product and is not eligible for Google's open source vulnerability rewards program. It is released under Apache 2.0, and third party code keeps its own licenses.

Can I run Harness2 without paying for models?

Only the local demo. It uses scripted editors and real Python executions and needs no credentials. It finished in about 1.6 seconds on my Mac. Every benchmark run calls executor, improver and judge models and costs money, so start with one task at k of 1.

Does Harness2 change the model's weights?

No. The README says model weights stay fixed, and the paper calls the executors frozen. Only the surrounding harness changes: the system prompt, guardrails, memory, tool guidance, skills, subagents, scripts and plugins. That makes the result easy to inspect, because the edits are readable files.

Is it safe to run on my work laptop?

I would not. The release leaves executor network access unrestricted and disables the Codex sandbox. Use an isolated machine with no credentials and block the benchmark hosts for the executor before you trust any score.

Does Harness2 work on Windows?

The authors document Linux only, and I did not try Windows. The demo, the install check and one JobBench task ran on my Mac. On Windows 11 I would start with WSL2 or a Linux virtual machine, which is my assumption and not something the repository says. WorkBuddy-Bench also needs Docker.

How much does a Harness2 run cost?

The code has no spend cap. One task at k of 1 in one mode makes 4 executor rollouts and 3 improver sessions, plus one reflection call. My one real HR task on Gemini 3.8 Flash cost $22.04 at list price as run and about $15.7 for a clean run. Price yours first with the Harness2 cost estimator.

What does Harness² or "harness squared" mean?

Harness2, Harness² and Harness^2 all name the same project, google-research/harness2. The authors describe two connected loops: one that refines the harness inside a single task, and one that carries lessons across tasks.

Can Harness2 use Codex or other models?

It supports two executors, OpenCode by default and Codex, and three benchmarks. Models are called through Vertex AI. For Codex you need a Responses API endpoint, and the release includes harness2-bridge, a local LiteLLM server that presents Gemini on Vertex AI that way.

Feed to Claude or ChatGPT

Last updated

November 1, 2026

Jahanzaib Ahmed

Jahanzaib Ahmed

AI Systems Engineer & Founder

AI Systems Engineer with 126 production systems shipped. I run AgenticMode AI (AI agents, RAG systems, voice AI) and ECOM PANDA (ecommerce agency). I build AI that works in the real world for businesses across home services, healthcare, ecommerce, SaaS, and real estate.

Want this built for your business? See AI Agent Development, or browse the free tools and guides.