pi-langfuse

English | 简体中文

Langfuse observability extension for Pi Coding Agent. It sends complete Pi runs to Langfuse so the prompt, agent workflow, LLM generations, tool calls, final response, usage, cost, and health scores appear in one trace.

What This Adds to Pi

• One Langfuse trace per user prompt, grouped by Pi session.
• Root agent, per-request generation, and per-tool tool observations.
• Final assistant output capture, tool error visibility, and trace-level scores.
• Privacy controls for inputs, outputs, tool I/O, system prompt, and cwd.
• Secret redaction and local path hashing before upload.
• Capability-gated REST fallback for self-hosted Langfuse setups that expose the legacy trace API when OTel spans arrive but traces do not materialize. Langfuse v4 events_only deployments use OTel without legacy fallback ingestion.

Prerequisites

• Node.js >= 22
• Pi Coding Agent installed and configured
• A Langfuse account (cloud or self-hosted)

Quick Start

1. Install the extension:

   pi install npm:pi-langfuse
   

2. Run Pi once. If no credentials are configured yet, Pi prompts for:

   • Langfuse public key, starting with pk-lf-...
   • Langfuse secret key, starting with sk-lf-...
   • Langfuse host, defaulting to https://cloud.langfuse.com

3. Run Pi normally:

   pi "Explain the architecture of Redis"
   

4. Open Langfuse and inspect the new trace.

Configuration

Langfuse API keys are available in Langfuse Cloud -> Settings -> API Keys.

Method 1: Interactive setup

Run any pi command with the extension loaded. On first run without configuration, Pi prompts in the CLI or TUI and saves the result to ~/.pi/agent/pi-langfuse/config.json.

To run setup again:

/langfuse-setup

To inspect the active configuration without exposing secrets:

/langfuse-status

The status command reports the config source, host, masked public key, capture policy, active-run state, config path, and last runtime error.

Method 2: Environment variables

Set these before starting Pi:

export LANGFUSE_PUBLIC_KEY="pk-lf-xxxx"
export LANGFUSE_SECRET_KEY="sk-lf-xxxx"
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"  # optional; LANGFUSE_HOST is also supported

Saved config takes precedence. Environment variables are only used when ~/.pi/agent/pi-langfuse/config.json is missing or incomplete.

For short-lived SDK hosts, set the bounded final score-delivery attempt during shutdown:

export PI_LANGFUSE_SCORE_SHUTDOWN_TIMEOUT=2  # seconds; defaults to 2 seconds

The extension attempts queued trace-level scores before other shutdown telemetry work. This value cannot extend the overall shutdown deadline.

Privacy controls can also be set through environment variables:

export LANGFUSE_PRIVACY_PRESET="full-debug"

Available presets:

Preset	Captures
metadata-only	Metadata only; omits inputs, outputs, tool I/O, system prompt, and cwd
prompts-only	Prompt/provider inputs plus metadata
conversations	Inputs and assistant outputs, but omits tool I/O, system prompt, and cwd
full-debug	Full trace detail; this is the default

Fine-grained flags override presets:

export LANGFUSE_CAPTURE_INPUTS=true
export LANGFUSE_CAPTURE_OUTPUTS=true
export LANGFUSE_CAPTURE_TOOL_IO=false
export LANGFUSE_CAPTURE_SYSTEM_PROMPT=false
export LANGFUSE_CAPTURE_CWD=false
export LANGFUSE_CAPTURE_SOURCE_METADATA=false
export LANGFUSE_CAPTURE_PATHS=false

Source metadata remains off in every preset unless LANGFUSE_CAPTURE_SOURCE_METADATA=true is set explicitly. The same holds for absolute paths and LANGFUSE_CAPTURE_PATHS.

All captured payloads are redacted before upload. The extension masks common API keys, bearer tokens, passwords, cookies, private keys, Langfuse keys, GitHub/npm/AWS-style tokens, and local absolute paths.

Absolute paths

By default, local absolute paths (/Users/..., /home/..., /tmp/..., C:\Users\...) are replaced everywhere with a stable [PATH_HASH:<12 hex chars>] digest, so usernames and repository names never reach Langfuse. This applies to inputs, outputs, tool I/O, tool error messages, and the cwd metadata field. Opt in to see real paths in traces:

export LANGFUSE_CAPTURE_PATHS=true

Or persist it in config.json:

{ "capture": { "LANGFUSE_CAPTURE_PATHS": "true" } }

Like LANGFUSE_CAPTURE_SOURCE_METADATA, this stays off in every privacy preset until it is set explicitly, and it only affects paths — secret masking (tokens, keys, cookies, passwords) is always on regardless. Note that LANGFUSE_CAPTURE_CWD=false is a different control: it drops the cwd metadata field entirely rather than changing how paths are rendered.

/langfuse-status reports the current setting under Capture: absolute paths.

Payload limits

Before upload, payloads are shaped: strings are truncated and deeply nested or very wide structures are trimmed. These caps keep traces small and protect the Langfuse ingestion pipeline. Override any of them (no rebuild needed):

export PI_LANGFUSE_MAX_STRING_LENGTH=12000       # per-string chars (system prompt, inputs)
export PI_LANGFUSE_MAX_TOOL_PAYLOAD_LENGTH=24000 # per tool input/output chars
export PI_LANGFUSE_MAX_DEPTH=6                    # max nesting depth
export PI_LANGFUSE_MAX_ARRAY_ITEMS=50            # max array elements kept
export PI_LANGFUSE_MAX_OBJECT_KEYS=80            # max object keys kept
export PI_LANGFUSE_MAX_PAYLOAD_NODES=2000        # max total nodes per payload

Set any limit to 0, off, none, or unlimited to disable that cap entirely (captures the full value). Unset or invalid values fall back to the defaults shown above. To capture a very large system prompt or big tool payloads in full, raise or disable the relevant limit (e.g. PI_LANGFUSE_MAX_STRING_LENGTH=off).

The REST fallback ingestion is chunked so each request body stays well below the Langfuse gateway's payload limit (~4.5MB). These knobs control the chunk budget and a hard ceiling for the whole fallback payload:

export PI_LANGFUSE_MAX_INGESTION_BATCH_BYTES=4194304  # per-request body budget, default 4MB

export PI_LANGFUSE_MAX_FALLBACK_TOTAL_BYTES=33554432  # whole-payload ceiling, default 32MB

When the accumulated fallback payload exceeds the 32MB ceiling, ingestion is skipped with a warning instead of attempting an unrecoverably large upload.

Reasoning tokens

Pi reports reasoning (thinking) tokens for Anthropic, OpenAI Codex, OpenRouter, opencode-go and Qwen. Providers count them inside output, and by default the extension reports output whole, so the reasoning share is not visible in Langfuse. Opt in to report it as its own usage bucket:

export PI_LANGFUSE_SPLIT_REASONING_TOKENS=true

Or persist it in config.json:

{ "capture": { "PI_LANGFUSE_SPLIT_REASONING_TOKENS": "true" } }

With the split on, a generation that used 37 output tokens of which 10 were reasoning is reported as output: 27 plus output_reasoning_tokens: 10. Both keys contain output, so the Output row in Langfuse still shows 37; only the breakdown beneath it gains the reasoning share. Reasoning is clamped to the reported output, so the buckets always add up to the total.

Before enabling — check your model prices. Langfuse matches prices to usage by exact key, and model definitions you created in your project take precedence over Langfuse's maintained defaults. A custom model priced on input and output only would cost output_reasoning_tokens at zero, so reasoning-heavy generations would look cheaper than they are. Add a price for output_reasoning_tokens to every custom reasoning model in Settings → Models first, then turn the split on. Langfuse's built-in prices for reasoning models already include it. Providers that report their own cost are unaffected: Langfuse uses the reported cost as-is and does not recompute it from usage.

The split is off by default, so upgrading changes nothing until you enable it. Unset PI_LANGFUSE_SPLIT_REASONING_TOKENS (or set it to false) to go back; traces already ingested keep their buckets.

Method 3: Persistent config.json

Create or update ~/.pi/agent/pi-langfuse/config.json:

{
  "publicKey": "pk-lf-xxxx",
  "secretKey": "sk-lf-xxxx",
  "host": "https://cloud.langfuse.com",
  "privacyPreset": "conversations"
}

Fine-grained capture flags can also be persisted:

{
  "publicKey": "pk-lf-xxxx",
  "secretKey": "sk-lf-xxxx",
  "host": "https://cloud.langfuse.com",
  "capture": {
    "LANGFUSE_PRIVACY_PRESET": "metadata-only",
    "LANGFUSE_CAPTURE_INPUTS": "true"
  }
}

Security: Keep ~/.pi/agent/pi-langfuse/config.json private. Never commit API keys to version control. When the extension writes this file itself, it creates the config directory with 0700 permissions and the file with 0600 permissions where the host filesystem supports POSIX modes.

Verify the Extension

Check that Pi has loaded the package:

pi list

pi-langfuse should appear in the installed package list.

To verify the Langfuse host and API keys from inside Pi, run:

/langfuse-test

This command makes a timeout-bounded authenticated request to Langfuse and, if it succeeds, sends a small test trace.

What Appears in Langfuse

• Each Pi session gets its own Langfuse session ID.
• Each user prompt within that session becomes a separate trace.
• The trace contains the final assistant output shown in Pi.
• Tool runs appear as tool observations with arguments, results, and error state.
• LLM requests appear as generation observations, including usage and cost when the provider exposes them. Reasoning tokens are reported as their own usage bucket when PI_LANGFUSE_SPLIT_REASONING_TOKENS is enabled.
• Trace-level scores include tool counts, tool success rate, and whether the run had errors.

The package also includes a Langfuse CLI skill, so Langfuse data can be queried directly from Pi:

/pi-langfuse-langfuse <your-query>

Source Metadata

Repository source capture is independent of the privacy presets and disabled by default. Enable it only after deciding that commit identity is appropriate for the Langfuse project:

export LANGFUSE_CAPTURE_SOURCE_METADATA=true

For a Git worktree, the extension records only revision state:

{
  "source_type": "git-repo",
  "vcs.ref.head.revision": "0123456789abcdef...",
  "git_detached": "false",
  "git_dirty": "false",
  "metadata_source": "git-detection"
}

The revision is the full HEAD commit. Dirty state includes tracked changes and untracked files, but never their paths or contents. Detached state is reported without a branch or tag name. Git remotes, URLs, credentials, usernames, branches, absolute paths, and repository names are never inspected or uploaded by this collector.

When capture is off, the collector does not invoke Git and reports source_type: "disabled". Non-Git directories report non-git; a missing or unusable Git executable and incomplete Git state report unavailable.

Use native Langfuse and OpenTelemetry settings for deployment identity and explicit operator-owned overrides instead of repository files:

export LANGFUSE_RELEASE="1.2.3"
export LANGFUSE_TRACING_ENVIRONMENT="production"
export OTEL_SERVICE_NAME="pi-agent"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.2.3,vcs.repository.name=public-repo"

LANGFUSE_RELEASE and LANGFUSE_TRACING_ENVIRONMENT keep their Langfuse semantics. OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are loaded as process-scoped OpenTelemetry resource attributes; values such as vcs.repository.name apply to every session sharing the runtime, may identify private source, and require a runtime restart to change.

Compatibility and rollback

Earlier versions emitted git_commit, branch, remote, owner, repository, and repo-local .pi-langfuse.metadata.json values by default. New traces use vcs.ref.head.revision and stop reading that file. Update dashboards before enabling source capture; historical traces are unchanged.

Unset LANGFUSE_CAPTURE_SOURCE_METADATA to stop collection immediately. Pin pi-langfuse@1.5.12 only if the old schema is required during migration; doing so also restores its broader default source disclosure.

Troubleshooting

No traces appearing?

• Verify the API keys and run /langfuse-setup again if needed.
• Run /langfuse-status to confirm the loaded host, config source, privacy mode, and last runtime error.
• Confirm the Langfuse project is active and accepts writes.
• Confirm the keys have write permission.
• Look for 📊 Langfuse: log messages in Pi output.

Extension not loading?

pi list
pi install npm:pi-langfuse

"Missing config" on startup?

• Run /langfuse-setup.
• Or set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY before starting Pi.

Model or cost not showing?

• Some providers do not expose cost information.
• Inspect the raw observation data in Langfuse traces.
• The model field can come from provider events, finalized assistant messages, model_select, or ctx.model.

API key errors?

• Public keys start with pk-lf-.
• Secret keys start with sk-lf-.
• For self-hosted deployments, verify the host URL.

Development Docs

Development setup, source installation, runtime architecture, trace model, tracked fields, and validation steps are documented in DEVELOPMENT.md and DEVELOPMENT_CN.md.

License

MIT