@alexleekt/pi-shared

The glue that holds the monorepo together.

Shared types and utilities for Pi extensions in this monorepo.

Exports

Path	Description
@alexleekt/pi-shared/session	manageSessionSubscription() — per-session subscription lifecycle helper
@alexleekt/pi-shared/types	Shared Pi extension type definitions
@alexleekt/pi-shared/prompt-eval	Generalized prompt evaluation framework

@alexleekt/pi-shared/prompt-eval

A reusable framework for evaluating LLM prompts against test cases. Extracted from pi-heading's duplicated prompt-eval-*.ts scripts.

Core types

export interface EvalSuite<T extends TestCase = TestCase> {
  name: string;
  testCases: T[];
  promptBuilder: (testCase: T) => PromptMessage;   // { system, user, maxTokens }
  extractMode: "json" | "raw";
  scorers: Scorer<T>[];
  modelConfig?: ModelConfig;
}

export type Scorer<T extends TestCase = TestCase> = (params: {
  text: string;      // extracted result
  raw: string;       // raw LLM output
  testCase: T;
}) => ScoreResult;

Key functions

Function	Purpose
runSuite(suite, model)	Evaluate all test cases, score outputs, print progress
generateReport(results, suite, model)	Build markdown report string
callLLM(system, user, model, config?)	Call local proxy with retry
extractResult(raw)	Parse {"result": "..."} or fall back to raw
optimizeSuite(factory, promptPath, config)	Iterative prompt optimization loop

Built-in scorers

scorers.wordCount(max)
scorers.withinLimit(max)
scorers.noMetaCommentary()
scorers.noQuotes()
scorers.noMarkdown()
scorers.validJson()
scorers.presentContinuous()
scorers.pastTense()
scorers.noTrailingPeriod()
scorers.concise(threshold)
scorers.alignsWithExpected(field, threshold)
scorers.echoesGoal(goalField, threshold)
scorers.specificConcrete()
scorers.noVagueFiller()

Iterative optimization

import { optimizeSuite } from "@alexleekt/pi-shared/prompt-eval";

const result = await optimizeSuite(
  (promptText) => mySuiteFactory(promptText),   // factory builds EvalSuite from prompt text
  "/path/to/prompt.md",                         // file to mutate in-place
  {
    desirableOutcome: "Concise past-tense summaries under 12 words, no meta-commentary",
    targetPassRate: 85,
    maxIterations: 5,
    evalModel: "firepass",
    criticModel: "firepass",
  }
);

The optimization loop:

1. Load prompt from file
2. Run suite → check pass rate
3. If below target, collect top N failing cases
4. Call critic LLM with failures + desirable outcome description
5. Write revised prompt back to file
6. Repeat until target met or max iterations

License

MIT