pi-voice-loop

pi-voice-loop is a Pi extension package for operating a Pi-native voice loop. The current provider adapter is xAI, exposing TTS, STT, streaming TTS/STT, custom voices, voice discovery, realtime client secrets, and realtime Grok voice-agent sessions through one command:

/voice-loop

The corrected product direction is Pi-native bidirectional voice I/O:

push/toggle-to-talk -> provider STT -> review/edit transcript
-> pi.sendUserMessage(normal | steer | followUp)
-> Pi commentary/final output -> provider TTS/playback queue

Pi remains the coding agent. Provider adapters provide ears, speech, and later realtime bridge capability; xAI is the first concrete adapter.

Canonical docs

• VISION.md: product north star and user stories.
• README.md: operator-facing usage and current capability surface.
• ARCH.md: architecture, boundaries, process edges, and future seams.
• PLAN.md: remaining implementation sequence.
• CHANGELOG.md: release-facing change log.

What it is for now

• Generate speech from Pi-authored text or files.
• Transcribe local or explicitly approved remote audio.
• Use streaming TTS/STT paths when latency behavior matters.
• Run controlled realtime Grok voice-agent sessions with local audio output and status.
• Exercise provider voices, custom voices, client secrets, and xAI realtime capabilities from a repeatable local cockpit.
• Prove the next voice-loop contract through fake adapters before live mic, playback, or paid media validation. The fake loop avoids xAI/media calls but still submits real Pi user messages and can trigger the configured Pi model.

Where it is going

• Pi-native voice loop: explicit listen control, transcript review/edit, delivery as normal/steer/follow-up, and phase-aware speech output.
• macOS media: SoX-backed mic recording and explicit playback queue.
• Pi narrator: later optional output policy that reuses the same speech queue.
• xAI realtime bridge: later explicit voice-agent functions for Pi status, steer, follow-up, and summaries.

Current boundary

The batch voice_loop_tts and voice_loop_stt tools are TTS/STT helpers. Realtime voice-agent work is command-owned in the current package. The voice-loop foundation is also command-owned until its UI and shortcut behavior are proven.

Narrator/status speech is roadmap scope, not the current implementation target.

Install

From a local checkout:

cd /path/to/pi-voice-loop
pi install "$PWD"
pi list | grep pi-voice-loop

For one run without installing:

cd /path/to/pi-voice-loop
pi -e "$PWD"

A Pi user-package install points at the local checkout, so uncommitted local changes are live until you remove or repoint the package. Pi packages run with your local user permissions. Review package source before installing third-party packages.

Credentials

Shell environment has highest precedence:

export XAI_API_KEY="your_key_here"

The stable Pi user auth file is the fallback source:

~/.pi/agent/extensions/pi-voice-loop.env

Template and manual setup instructions ship in:

examples/pi-voice-loop.env.example

Safe manual setup:

mkdir -p ~/.pi/agent/extensions
cp examples/pi-voice-loop.env.example ~/.pi/agent/extensions/pi-voice-loop.env
chmod 600 ~/.pi/agent/extensions/pi-voice-loop.env
$EDITOR ~/.pi/agent/extensions/pi-voice-loop.env

Use /voice-loop auth in a UI-capable Pi session to edit the file, or /voice-loop auth path to print the destination and template. Validate without printing the key:

/voice-loop auth status

The auth command writes the file with 0600 permissions and never stores secrets in tracked JSON config. Shell environment values still override the file at runtime. Base URL remains non-secret JSON config, guarded by allowCustomBaseUrl.

Command surface

Only /voice-loop is registered. Subcommands are arguments.

Command	Purpose
/voice-loop status	Show config paths, auth presence/provenance, output root, voice-loop state, and realtime state.
/voice-loop auth	Edit the stable xAI auth file in UI mode.
/voice-loop auth status	Show auth provenance without printing secrets.
/voice-loop auth path	Show the auth-file path and template guidance.
/voice-loop auth reset --yes	Remove the stable auth file after approval.
/voice-loop config paths	Show global and project config paths.
/voice-loop config defaults	Print default config JSON.
/voice-loop config reset [project|global]	Delete package config after confirmation.
/voice-loop loop status	Show voice-loop config and speech queue state.
/voice-loop loop listen --fake-transcript "text" --accept-transcript	Run the fake transcript-to-Pi loop without xAI/media calls; it still submits a real Pi user message.
/voice-loop loop listen --fake-transcript "text" --delivery steer --accept-transcript	Deliver the reviewed fake transcript as a steering message when Pi is busy.
/voice-loop loop listen --record --seconds 5 --yes	Explicitly record microphone audio, transcribe it with xAI STT, review/edit the transcript, then request Pi processing.
/voice-loop loop listen --record --speak-output --yes	Also synthesize selected Pi output with xAI TTS and play it through the explicit playback adapter.
/voice-loop loop media status	Show SoX/afplay/ffmpeg tool availability without recording or playback.
/voice-loop loop media record --seconds 5 --yes	Explicitly record microphone audio with SoX rec into the output root.
/voice-loop loop media play audio.wav --yes	Explicitly play a saved audio artifact with afplay or --adapter sox.
/voice-loop loop mute	Mute the speech queue.
/voice-loop loop unmute	Unmute the speech queue.
/voice-loop loop pause	Pause queued playback after current speech.
/voice-loop loop resume	Resume queued playback.
/voice-loop loop skip	Skip the current queued speech item.
/voice-loop voices	List built-in voices from the current provider.
/voice-loop tts "text" --yes	Generate batch TTS audio into the package output root.
/voice-loop tts --file input.txt --yes	Generate batch TTS from a UTF-8 text file.
/voice-loop tts-stream "text" --yes	Generate streaming TTS audio over WebSocket.
/voice-loop stt audio.mp3 --yes	Transcribe a local audio file.
/voice-loop stt https://example.com/audio.mp3 --yes	Transcribe a remote URL only when config allows remote URLs.
/voice-loop stt-stream raw.pcm --yes	Stream raw audio frames to xAI STT.
/voice-loop realtime start --yes	Start one realtime voice-agent WebSocket session.
/voice-loop realtime send-text "hello"	Send text to the active realtime session and request a response.
/voice-loop realtime send-audio-file raw.pcm	Send raw audio chunks to the active realtime session.
/voice-loop realtime commit	Commit buffered input audio in manual-turn mode.
/voice-loop realtime clear	Clear buffered input audio.
/voice-loop realtime cancel	Cancel an in-progress response.
/voice-loop realtime stop	Close the realtime session.
/voice-loop realtime status	Show active realtime state.
/voice-loop custom-voices list	List custom voices owned by the team.
/voice-loop custom-voices get VOICE_ID	Show metadata for one custom voice.
/voice-loop custom-voices create --file ref.wav --name "Name" --language en --yes	Create an Enterprise-gated custom voice.
/voice-loop custom-voices update VOICE_ID --name "New name" --yes	Patch custom voice metadata.
/voice-loop custom-voices download VOICE_ID --yes	Download reference audio into the package output root.
/voice-loop custom-voices delete VOICE_ID --yes	Delete a custom voice.
/voice-loop client-secret create --seconds 300 --yes	Create a realtime client secret; --show-secret is UI-only so the token is not written into model-visible command history.

Paid one-shot operations, remote mutations, and realtime session start require --yes or an interactive confirmation. Commands sent inside an active realtime session use the session opened by /voice-loop realtime start --yes. Non-interactive paid calls require allowNonInteractivePaidCalls and --yes. Set enabled: false to deny active xAI, realtime, and voice-loop operations while preserving status/config/auth recovery commands.

Voice-loop delivery modes

The voice-loop bridge maps reviewed transcript text to Pi messages. Fake listen uses operator-provided text; recorded listen uses explicit SoX microphone capture followed by xAI STT, then the same review/edit and Pi handoff contract:

• normal: valid when Pi is idle; sends as a normal user message.
• steer: when Pi is busy, sends with deliverAs: "steer" to steer the active run.
• followUp: when Pi is busy, sends with deliverAs: "followUp" and runs after the current work finishes.

Default policy:

• idle Pi: normal
• busy Pi: configured voiceLoop.busyDelivery, default followUp
• busy Pi plus explicit normal: denied rather than silently converted

Commentary extraction

Default speech output is commentary only. Fake speech artifacts are the default no-network proof path. Live TTS/playback requires --speak-output on the voice-loop command, paid xAI approval, local playback approval, idle normal delivery, and playback-tool availability before Pi handoff. Output capture is queued only for an idle normal voice-loop delivery, after the transcript artifact is durably written. The queue arms only after Pi emits the matching extension input event and reaches before_agent_start preflight for that same prompt; stale unmatched capture requests expire instead of arming on arbitrary later work. Busy followUp and steer delivery submit the input but defer speech capture until Pi exposes a reliable turn-correlation signal. Installing the extension does not passively write speech artifacts for unrelated Pi turns. The extractor reads finalized Pi assistant text from message_end and parses TextContent.textSignature JSON:

{ "v": 1, "id": "text-block-id", "phase": "commentary" }

Default behavior:

• phase: "commentary" is spoken
• phase: "final_answer" is silent unless configured
• missing or malformed phase is silent unless configured
• user, tool, thinking, and tool-call content is not spoken

message_update streaming speech is deferred until phrase buffering and phase proof are implemented. message_end is the correctness baseline.

Capability matrix

xAI/Pi capability	Package support	Notes
Built-in voices	implemented	GET /v1/tts/voices.
Custom voice IDs in TTS/realtime	implemented	Any voice id can be configured.
Custom voice list/get/create/update/download/delete	implemented	Create is Enterprise-gated by xAI.
TTS REST	implemented	Saves raw audio bytes locally.
Voice-loop output TTS/playback	implemented, live validation pending	Requires --speak-output; uses selected Pi commentary/final phases only.
TTS WebSocket streaming	implemented	Saves streamed audio deltas locally.
STT REST local file	implemented	File max 500 MB.
STT REST remote URL	implemented but disabled by default	Requires config opt-in because xAI server fetches the URL.
STT WebSocket streaming	implemented	Raw PCM/µ-law/A-law file input only.
Realtime voice agent	implemented	Text and raw audio-file input; output audio saved locally.
Realtime client secrets	implemented	Creates ephemeral tokens for browser/mobile client experiments.
xAI realtime server-side tools	configurable but empty by default	Tool invocations may add xAI charges.
Auth file fallback	implemented	Env first, then ~/.pi/agent/extensions/pi-voice-loop.env.
Fake transcript-to-Pi loop	implemented	No xAI/media contract proof for review/edit and delivery; submits real Pi messages.
Phase-aware speech extraction	implemented	Commentary-only default from matched idle input plus finalized message_end; busy output capture is deferred.
Fake TTS/playback queue	implemented	No-network artifact and queue proof.
Mac microphone capture	implemented behind explicit commands	SoX rec through CoreAudio; no hidden recording.
Recorded voice input to xAI STT	implemented, live validation pending	/voice-loop loop listen --record --yes records, transcribes, reviews, and requests Pi processing.
Automatic playback	not implemented	Live playback is opt-in per voice-loop command with --speak-output; no hidden autoplay.
Pi narrator	planned	Optional later policy over the same speech queue.
Pi/local tool bridging into realtime	planned	Explicit bridge seam for Pi status, steer, follow-up, and summaries.

Configuration

Global config:

~/.pi/agent/extensions/pi-voice-loop.json

Project config:

<project-root>/.pi/extensions/pi-voice-loop.json

Defaults:

{
  "enabled": true,
  "apiKeyEnv": "XAI_API_KEY",
  "baseUrl": "https://api.x.ai/v1",
  "allowCustomBaseUrl": false,
  "confirmPaidCalls": true,
  "allowNonInteractivePaidCalls": false,
  "allowRemoteAudioUrls": false,
  "outputRoot": ".pi/extensions/pi-voice-loop",
  "modelTools": { "networkPolicy": "deny" },
  "voiceLoop": {
    "reviewTranscript": true,
    "busyDelivery": "followUp",
    "speechPhases": {
      "commentary": true,
      "finalAnswer": false,
      "unknown": false
    },
    "fakeMedia": true,
    "media": {
      "recordSeconds": 5,
      "recordSampleRate": 16000,
      "recordChannels": 1,
      "playbackAdapter": "afplay"
    }
  },
  "tts": {
    "voiceId": "eve",
    "language": "auto",
    "codec": "mp3",
    "sampleRate": 24000,
    "bitRate": 128000,
    "optimizeStreamingLatency": 0,
    "textNormalization": true
  },
  "stt": {
    "language": null,
    "format": true,
    "diarize": false,
    "multichannel": false,
    "channels": 1
  },
  "sttStreaming": {
    "sampleRate": 16000,
    "encoding": "pcm",
    "interimResults": true,
    "endpointing": 500,
    "chunkMs": 100
  },
  "realtime": {
    "model": "grok-voice-think-fast-1.0",
    "voice": "eve",
    "instructions": "You are a helpful voice assistant inside Pi.",
    "maxMinutes": 30,
    "turnDetection": "server_vad",
    "inputFormat": "audio/pcm",
    "inputSampleRate": 16000,
    "outputFormat": "audio/pcm",
    "outputSampleRate": 24000,
    "saveOutputAudio": true,
    "autoStartMic": false,
    "autoPlay": false,
    "tools": []
  }
}

Local macOS audio tools

Stage 2 uses macOS local tools only:

• SoX rec: primary microphone recorder through CoreAudio.
• SoX play: optional playback/conversion support.
• afplay: built-in playback fallback for saved WAV/MP3 artifacts.
• ffmpeg: fallback conversion, AVFoundation device listing, and probe support.

Check local prerequisites before recorded or playback commands:

command -v sox rec play afplay ffmpeg

afplay ships with macOS. Install SoX/ffmpeg through your workstation package manager when missing, then use /voice-loop loop media status for package-level readiness.

/voice-loop loop media status is read-only. Recording and playback are never started by install, status, or fake loop commands; they require explicit loop media record ... --yes, loop media play ... --yes, or loop listen --record ... --yes commands or an interactive confirmation. Recorded listen also needs paid xAI approval before STT and UI transcript review by default; --accept-transcript is only for fake transcripts because a recorded STT transcript cannot be pre-reviewed before it exists. Live output speech needs --speak-output plus paid TTS and local playback approval before Pi handoff. The fake adapter remains the no-xAI/no-media test path.

Model-visible tools

The extension registers these helper tools:

• voice_loop_status
• voice_loop_realtime_status
• voice_loop_voices
• voice_loop_tts
• voice_loop_stt

The default model-tool network policy is deny, so model-initiated calls do not call xAI until the operator changes config. Paid model-visible calls still need tool-level paid-call confirmation. Realtime session start and voice-loop control are intentionally slash-command operations in this version.

Output files

Generated files live under:

.pi/extensions/pi-voice-loop/
  audio/
  transcripts/
  realtime/
  custom-voices/

Command results return useful paths and metadata. Generated audio, transcripts, fake-media receipts, realtime output, and playback artifacts remain first-class product output.

Validation

No-network gate:

pnpm run gate
git diff --check
printf '{"type":"get_commands"}\n' \
  | pi --mode rpc --no-session --no-context-files --no-skills --no-extensions \
    -e "$PWD"

pnpm run gate runs package-local TypeScript source typechecking, the no-network runtime test suite, JSON validation, and an npm package dry-run.

Live xAI checks spend money and send text/audio to xAI. Mic capture and playback use local devices. They are not part of the no-network gate.