🤯 pi-reason-harness

Recursive self-improving reasoning harness for pi

20-layer meta-system that discovers, adapts, evolves, transfers, and validates strategies autonomously.

---

Builds task-specific reasoning strategies on top of any LLM by running iterative solve-verify-feedback loops with multi-expert ensembling, voting, and a 20-layer meta-system that discovers, adapts, evolves, transfers, and validates strategies autonomously.

JS-exclusive — LLM calls go through pi's native LLM infrastructure (@earendil-works/pi-ai). Code sandbox uses Node's vm module. Zero Python dependency.

How It Works

The core insight (from first-principles analysis of SOTA reasoning systems): LLMs are knowledge stores that require intelligent probing strategies to extract reliable answers. The harness layer (open-source) iteratively generates, verifies, and refines. The meta-system layer (proprietary, rebuilt here) discovers and evolves the strategies themselves.

The 20-Layer Meta-System

Layer	Name	What it does
0	Problem Critic	Inspects problems, proposes targeted deltas to proven templates (not writing from scratch)
1	Strategy Library	Persistent store of proven strategies with ROI + quality metrics
2	Meta-Rule Engine	Extracts cross-strategy principles that compound over time
3	Model Router	Thompson sampling for intelligent model selection per category
4	Budget Bandit	Early stopping, budget reallocation, re-exploration when stuck
5	Auto-Trigger	Self-improvement runs automatically (on success rate drops, new categories, periodic)
6	Recursive Harness Generation	Generates entire solve approach configurations (the "solver of solvers")
7	Ensemble Diversification	Each expert uses a fundamentally different approach strategy
8	Sub-problem Decomposition	Break hard problems into sub-problems, solve independently, combine
9	Budget Optimization	Marginal ROI estimation, reallocate iterations to high-ROI experts
10	Cross-Domain Transfer	Transfer proven strategies across analogous categories automatically
11	Confidence-Weighted Voting	Weight votes by self-assessed quality, not just output match
12	Progressive Difficulty	Train on easiest examples first, build up to harder ones
13	Auto-Transfer	Automatically transfer strategies when new categories are encountered
14	Per-Problem Prompt Synthesis	Generate + validate specialized prompts for novel problem types
15	Meta-Meta Level	Harness-of-harnesses — generate new approach types from performance data
16	Gradient-Based Budget Optimization	Trajectory-based improvement estimation with finite-difference gradients
17	Recursive Meta-Meta Nesting	Meta-harnesses feed back into solve; recursive evolution of underperformers
18	Multi-Model Decomposition	Route sub-questions to different models in parallel based on strengths
19	Per-Iteration Prompt Adaptation	Evolve the solver prompt mid-solve based on failure patterns
20	ARC-AGI Benchmark Integration	Validate against real ARC-AGI-2 challenges with scoring

Layer 6: Recursive Harness Generation — the "solver of solvers"

The biggest gap with Poetiq: their open-source code shows ONE harness configuration with fixed prompts. Their blog results prove they generate MULTIPLE different configurations per problem type.

A HarnessSpec defines a complete solve approach:

• Approach type: code-sandbox, decomposition, chain-of-questions, analogy, counter-factual, exhaustive-search, code-direct
• Solver/feedback prompts: Full templates with $$problem$$ placeholders
• Config overrides: Temperature, iterations, reasoning level
• Decomposition config: Max sub-problems, depth, combine strategy
• Validation data: Score on held-out data, production stats

The system generates multiple specs per problem, validates them, and evolves them over time.

Layer 7: Ensemble Diversification

Instead of N experts with the same prompt (just different seeds/models), each expert uses a fundamentally different approach:

Expert	Approach	When to use
1	code-sandbox	Grid/array problems — generate code, execute, verify
2	decomposition	Complex problems — break into sub-problems
3	analogy	Hard problems — solve simpler version first
4	chain-of-questions	Knowledge tasks — hierarchical probing
5	counter-factual	Stubborn problems — generate wrong solutions, invert
6	exhaustive-search	Small search spaces — enumerate, filter

Each approach has its own specialized prompt template.

Layer 8: Sub-problem Decomposition

For hard problems that resist direct solving, the decomposer breaks them into independent sub-problems:

1. LLM analyzes the problem and proposes 2-4 sub-problems
2. Each sub-problem is solved independently
3. Sub-solutions are combined (sequentially, in parallel, or hierarchically)

Example: "Rotate 90° clockwise" → Sub-problem 1: "Transpose the grid" → Sub-problem 2: "Reverse each row"

Layer 9: Budget Optimization via Marginal ROI

Not just "stop when stuck" but "spend where ROI is highest":

• Estimate marginal ROI per expert based on recent improvement rate
• Reallocate remaining iterations to experts with highest expected improvement
• Phase-based execution: run all experts for half-iterations, then reallocate

Layer 10-13: Cross-Domain Transfer + Auto-Transfer

When a strategy works in one domain, the system automatically transfers it to analogous domains:

• Category similarity map: grid-transformation ↔ pattern-completion ↔ spatial-reasoning
• Transfer adapts domain-specific parts while keeping universal insights
• Auto-triggered when a new category is encountered with no existing strategies
• Creates both a strategy entry and a harness spec for the new category

Layer 11: Confidence-Weighted Voting

Voting is weighted by self-assessed quality:

• Solutions that pass in fewer iterations count more (efficiency bonus)
• Solutions with high soft scores count more (partial accuracy bonus)
• Failed solutions grouped by output similarity, ranked by total confidence

Layer 12: Progressive Difficulty

Training examples are ordered from easiest to hardest:

• Difficulty proxy: grid size + unique value count + input/output asymmetry
• The solver sees simpler patterns first, building up to complex ones
• Mirrors Poetiq's per-iteration shuffle but with intelligence

Layer 14: Per-Problem Prompt Synthesis

For truly novel problems where no proven strategy exists, the system synthesizes specialized prompts:

• Computes a problem fingerprint based on structural features (grid size, unique values, operation type)
• If a validated synthesized prompt matches the fingerprint, uses it instead of the generic template
• If no match, generates a new specialized prompt via LLM and validates it on training data
• Prompts with validation score > 0.5 are persisted for future use
• Fingerprint-based matching allows cross-problem generalization

Layer 15: Meta-Meta Level — Harness-of-Harnesses

The biggest architectural gap with Poetiq: their system doesn't just generate strategies, it generates new types of harness approaches. Our meta-meta level:

• Analyzes performance data across all harness specs and meta-harnesses
• Uses LLM to propose NEW approach types that combine strengths of successful ones
• Each meta-harness has a name, description, solver prompt, config overrides, and rationale
• Meta-harnesses can evolve (generation counter, parent lineage) like strategies
• Example: "Decomposed-Sandbox-Synthesis" — combines decomposition's cognitive offloading with code-sandbox's deterministic verification

Layer 16: Gradient-Based Budget Optimization

Replaces simple proportional reallocation with finite-difference gradient estimation:

• Estimates dScore/dIteration (improvement rate) using a 5-point window
• Estimates d²Score/dIteration² (acceleration/deceleration)
• Predicts expected next score: current + gradient + 0.5 × acceleration
• Allocates iterations proportional to (expected improvement × confidence)
• Detects when an expert should switch approaches: stuck (gradient ≈ 0) + decelerating (acceleration < 0)

Layer 17: Recursive Meta-Meta Nesting

Meta-harnesses don't just get generated — they feed back into solve:

• selectMetaHarnessExpertConfig() assigns the best meta-harness to one expert in the ensemble
• Meta-harness performance is tracked (useCount, avgScore, successCount)
• Underperforming meta-harnesses (useCount ≥ 2, avgScore < 0.5, generation < 3) are recursively evolved via recursiveMetaEvolve()
• This creates a true recursive loop: solve → generate meta-harness → use in solve → evolve if underperforming → repeat

Layer 18: Multi-Model Decomposition

When multiple models are available, the system routes sub-questions to the best-suited model:

• decomposeAndRoute(): LLM analyzes the problem and assigns sub-problems to models based on heuristic strengths
• Model strength heuristics: Anthropic (complex reasoning, code), OpenAI (math, creative), Google (multimodal), Groq (fast), Wafer (reasoning), DeepSeek (code, math)
• Dependency tracking: sub-problems can depend on previous results
• solveRoutedDecomposition(): solves each sub-problem with its assigned model, combines results
• Triggered automatically in solve when models.length > 1 and useMeta=true

Layer 19: Per-Iteration Prompt Adaptation

The solver prompt adapts mid-solve based on failure patterns:

• After 3 consecutive failed iterations (score < 0.5), adaptPromptMidSolve() is called
• LLM analyzes the failure trajectory and suggests a prompt modification
• Three adaptation types: pre-insert (add before problem), anti-pattern (warn after problem), section-replace (replace a named section)
• applyIterationAdaptation() modifies the prompt for subsequent iterations
• The adaptation persists within the expert's solve loop

Layer 20: ARC-AGI Benchmark Integration

The system can validate against real ARC-AGI-2 challenges:

• loadArcChallenges(): loads challenges from ARC-AGI JSON files
• runArcBenchmark(): runs the harness on a batch of challenges with budget limits
• Re-verifies test outputs against ground truth (when available)
• Computes: solved, partial solved, avg best score, total cost, total time
• CLI: pi-reason-harness arc-benchmark --data-path ... --max-challenges 5
• Benchmark results with wafer/GLM-5.1: 5/7 unique challenges solved (71%), 1 near-miss (0.97), cost ~$0.04/challenge

Layers 0-5: Core Meta-System

These were implemented in the previous iteration and remain the foundation:

• Layer 0: Critique, Don't Create — The critic receives proven templates and proposes targeted deltas (insertions, anti-patterns, examples). This is code review, not writing from zero.
• Layer 2: Meta-Rules Compound — When a child strategy outperforms its parent, generalizable principles are extracted and applied to other categories.
• Layer 3: Thompson Sampling — Beta(α,β) sampling with Laplace smoothing picks the best model per category.
• Layer 4: Budget Bandit — Early stopping, re-exploration when all experts fail.
• Layer 5: Auto-Trigger — Runs automatically on success rate drops, new categories, and every 5th problem.

The Harness Layer

Below the meta-system, the harness implements the iterative solve loop with Poetiq-parity features:

1. Iterative solve-verify-feedback loops — Generate code, sandbox-execute, build detailed feedback
2. Multi-expert ensembling — Parallel experts with diverse approaches
3. Confidence-weighted voting — Group by output, rank by confidence
4. Poetiq-parity feedback — Element-by-element diff grids, shape mismatch detection
5. Poetiq-parity formatting — <Diagram> text with Fisher-Yates shuffle
6. Self-audit verification — LLM checks its own answers
7. Budget tracking — Per-problem cost/time limits

Task Types

Type	Strategy	Verification
code-reasoning	Generate JavaScript code → sandbox execute → verify against examples → feedback loop	Sandbox (default) or external
knowledge-extraction	Chain-of-questions probing → self-audit → confidence bucketing	Self-audit (recommended)
hybrid	Decide per-problem: code or direct answer → verify → feedback	Any method

Approach Types (for ensemble diversification)

Approach	Description	Best for
code-sandbox	Generate JS code, execute in sandbox, verify output	Grid/array transformations
code-direct	Generate code, extract answer without execution	Computation-heavy
decomposition	Break into sub-problems, solve each, combine	Multi-step problems
chain-of-questions	Hierarchical probing from broad to specific	Knowledge questions
analogy	Solve simpler version first, then scale up	Hard spatial problems
counter-factual	Generate wrong solutions, analyze failures, invert	Stubborn problems
exhaustive-search	Enumerate possibilities, filter by constraints	Small search spaces

Persistent Data

The meta-system persists across server restarts at ~/.pi-reason-harness/:

File	Contents
strategies.json	Strategy library with ROI, quality metrics, lineage
meta-rules.json	Cross-strategy principles with validation stats
model-routes.json	Per model×category routing stats
harness-specs.json	Complete harness specifications per category×approach
synthesized-prompts.json	Per-problem-type specialized prompts with validation
meta-harnesses.json	Generated approach types with evolution lineage

Quick Start

# Initialize a reasoning session
pi-reason-harness init --name "ARC solver" --type code-reasoning \
  --models '["anthropic/claude-sonnet-4-5","openai/gpt-4o"]' --num-experts 3

# Solve with the full 13-layer meta-system pipeline
pi-reason-harness solve --meta --problem "Transform the grid..." \
  --train-inputs '[[1,2],[3,4]]' \
  --train-outputs '[[4,3],[2,1]]' \
  --test-inputs '[[5,6]]'

# Analyze a problem without solving
pi-reason-harness meta-analyze --problem "Rotate a 2x2 grid 90 degrees clockwise"

# Decompose a hard problem into sub-problems
pi-reason-harness decompose --problem "Rotate a 3x3 grid 90 degrees clockwise. Input: [[1,2,3],[4,5,6],[7,8,9]]"

# Check harness specs
pi-reason-harness harness-specs

# Evolve the worst-performing spec
pi-reason-harness evolve-harness

# Check the strategy library
pi-reason-harness strategies

# Transfer a strategy from grid-transformation to pattern-completion
pi-reason-harness transfer --source-category grid-transformation --target-category pattern-completion

# Check meta-rules
pi-reason-harness meta-rules

# Check model routing stats
pi-reason-harness model-routes

Architecture

┌───────────────────────────────────────────────────────────┐
│  META-SYSTEM V3 (16 layers — the proprietary layer)       │
│                                                           │
│  Layer 0: Problem Critic (critique-don't-create)          │
│  Layer 1: Strategy Library (ROI + quality metrics)        │
│  Layer 2: Meta-Rule Engine (cross-strategy principles)    │
│  Layer 3: Model Router (Thompson sampling)                │
│  Layer 4: Budget Bandit (early stopping + re-explore)     │
│  Layer 5: Auto-Trigger (self-improving loop)              │
│  Layer 6: Recursive Harness Generation (solver-of-solvers)│
│  Layer 7: Ensemble Diversification (different approaches) │
│  Layer 8: Sub-problem Decomposition (break & combine)     │
│  Layer 9: Budget Optimization (marginal ROI realloc)      │
│  Layer 10: Cross-Domain Transfer (analogous categories)   │
│  Layer 11: Confidence-Weighted Voting (quality-ranked)    │
│  Layer 12: Progressive Difficulty (easiest-first)         │
│  Layer 13: Auto-Transfer (new category handling)          │
│  Layer 14: Per-Problem Prompt Synthesis (novel types)     │
│  Layer 15: Meta-Meta Level (harness-of-harnesses)         │
│  Layer 16: Gradient-Based Budget Optimization             │
│  Layer 17: Recursive Meta-Meta Nesting (harness↔solve)    │
│  Layer 18: Multi-Model Decomposition (model routing)      │
│  Layer 19: Per-Iteration Prompt Adaptation (mid-solve)    │
│  Layer 20: ARC-AGI Benchmark Integration (validation)     │
└───────────────────────┬───────────────────────────────────┘
                        │ generates (with deltas + rules + specs)
                        ▼
┌─────────────────────────────────────────────────────────┐
│  HARNESS (iterative solve-verify-feedback)              │
│                                                         │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐     │
│  │ Expert 1    │  │ Expert 2     │  │ Expert N    │     │
│  │ code-sandbox│  │ decomposition│  │ analogy     │     │
│  │ (pi-ai      │  │ (pi-ai       │  │ (pi-ai      │     │
│  │  LLM call   │  │  LLM call    │  │  LLM call   │     │
│  │  + sandbox  │  │  + sub-      │  │  + analogy  │     │
│  │  + verify   │  │  solve       │  │  + verify   │     │
│  │  + feedback │  │  + combine   │  │  + feedback │     │
│  └────┬────────┘  └─┬────────────┘  └──┬──────────┘     │
│       └─────────────┼──────────────────┘                │
│                     ▼                                   │
│     CONFIDENCE-WEIGHTED VOTING                          │
│     (group by output, rank by confidence)               │
│                     │                                   │
│                     ▼                                   │
│     LEARN + ADAPT + EVOLVE + TRANSFER                   │
│     (update strategies, extract rules, evolve specs,    │
│      transfer to new categories, auto-improve)          │
└─────────────────────────────────────────────────────────┘

CLI Reference

Command	Description
init	Initialize session with task config, models, verification
solve	Run iterative solve-verify-feedback loop
status	Show session state, budget, learned adaptations
results	Show iteration results
learn	Inspect strategy adaptations
reset-learn	Clear learned strategies
clear	Clear session
meta-analyze	Analyze a problem with the critic (no solving)
meta-improve	Manually trigger strategy evolution + rule extraction
strategies	List strategy library with ROI + quality metrics
meta-rules	List meta-rules with validation stats
model-routes	List model routing stats per model×category
harness-specs	List harness specifications with validation + production stats
evolve-harness	Evolve the worst-performing harness spec
transfer	Transfer strategy from one category to another
decompose	Decompose a problem into sub-problems
synth-prompts	List synthesized prompts with validation stats
meta-harnesses	List meta-harnesses (generated approach types)
generate-meta-harness	Generate a new approach type from performance data
arc-benchmark	Run ARC-AGI benchmark validation against real challenges
route-decompose	Decompose a problem across multiple models

init flags

--name, --type, --models, --num-experts, --verification, --verify-command, --max-cost, --max-time

solve flags

--problem, --train-inputs, --train-outputs, --test-inputs, --meta / -m

transfer flags

--source-category, --target-category

decompose flags

--problem

LLM Integration

The harness uses @earendil-works/pi-ai for all LLM calls. Models are specified in provider/model format (e.g., anthropic/claude-sonnet-4-5, openai/gpt-4o). API keys are resolved from the same environment variables pi uses:

• ANTHROPIC_API_KEY — Anthropic models
• OPENAI_API_KEY — OpenAI models
• GEMINI_API_KEY — Google models
• GROQ_API_KEY — Groq models
• WAFER_API_KEY — Wafer Pass models (GLM-5.1, Qwen3.5-397B-A17B)
• etc.

Custom Providers

The harness also supports custom providers (like Wafer Pass) that aren't in pi-ai's built-in model registry. Custom providers use direct OpenAI-compatible API calls. Currently supported:

Provider	Base URL	Models	Notes
wafer	https://pass.wafer.ai/v1	GLM-5.1, Qwen3.5-397B-A17B	Reasoning models with reasoning_content field

To add a new custom provider, add it to the CUSTOM_PROVIDERS map in server.ts.

No additional setup required — if pi can call the model, so can the harness.

Tests

npm test

104 tests covering: vm sandbox, formatProblem, arrayDiff, buildDetailedFeedback, PromptDelta application, budget bandit, Thompson sampling, meta-rule engine, prompt quality metrics, harness specs, ensemble diversification, budget optimization, cross-domain transfer, confidence-weighted voting, progressive difficulty, decomposition, problem fingerprinting, synthesized prompts, meta-harnesses, gradient estimation, approach switching, recursive meta-meta nesting, multi-model decomposition routing, per-iteration prompt adaptation, ARC-AGI benchmark.

License

MIT