English | 简体中文

pi-voicekit

Community continuation of codexstar69/pi-listen (upstream, MIT — dormant since v7.2.2 in May 2026). Not affiliated with the original author. Old name: pi-listen. This fork removes every cloud backend (0.6.0): dictation, punctuation and speech all run on your machine, so upstream's two-backend setup no longer applies — see Privacy.

Voice in and voice out for Pi. Hold-to-talk dictation with 21 offline models and a live preview above the editor, plus speech that reads the agent's replies aloud. Recognition, punctuation and synthesis all run in process on sherpa-onnx: there is no key, no endpoint and no account to set up, and no audio is uploaded to a remote speech service.

Language scope: Chinese and English. Recognition and punctuation are validated for Chinese, English and mixed Chinese–English dictation only. Every other language a model happens to cover is out of scope and unvalidated — see Language scope.

0.6.0 — one path, the local one. Both cloud backends are gone from the code: no streaming transport, no cloud voices, nothing to authenticate. Upgrading from 0.5.x erases the retired fields from settings.json on the first start — see Privacy. Earlier releases are documented in the changelog.

See How It Works

Setup (2 minutes)

1. Install the extension

# In a regular terminal (not inside Pi)
pi install npm:pi-voicekit

2. Pick a model

Open the panel with /voice-settings, go to the Models tab and press ↵ on the model you want. It downloads once (43 MB for the smallest, 671 MB for the recommended Parakeet) and becomes the active model. The language row then offers exactly the languages that model supports.

/voice-setup opens the same panel — it is an alias, not a separate wizard.

Nothing here costs money or needs an account: the models are public downloads and the only budget they take is disk space.

3. Check audio capture

pi-voicekit auto-detects your audio tool. No manual install needed if you already have sox or ffmpeg.

When PULSE_SERVER is set (SSH audio tunnel or remote PulseAudio) the order becomes ffmpeg → sox → arecord — network Pulse sources need ffmpeg.

4. What first start looks like

On launch pi-voicekit reports what it found and what is missing:

• A local model. With none selected the status line reads MIC SETUP and voice stays off — a config that finished setup without a model has no working path, so the extension sends you to /voice-setup instead of guessing a default. With a model, it reads MIC LOCAL.
• An audio tool. /voice test reports which one was detected, whether the microphone actually produced audio, and whether the sherpa-onnx runtime loaded.

When both are in place, hold SPACE and dictate.

Settings Panel

Everyday configuration lives in one place: /voice-settings. Its five tabs are General, Models, Downloaded, Speak and Device. A few optional fields have no row of their own and are set in settings.json instead — the preview's three are listed under Streaming preview.

General — model, language, scope, voice

Set the local model, the dictation language, the scope (global or this project) and the voice switch. Every row is a keyboard shortcut away — ↵ changes, ← → switches tab.

Models — browse, search, install

Browse 22 models from Parakeet, Whisper, Moonshine, SenseVoice, GigaAM, Paraformer, and Qwen3. Each model shows accuracy and speed ratings (●●●●○/●●●●○), fitness badges, and download status. Fuzzy search to find models fast. Press Enter to activate and download. The preview entry is the one exception: Enter downloads it when it is missing, and it can never be activated.

One of the 22 sits in its own Preview (streaming) group: a Chinese–English streaming Paraformer (226 MB) that is download-only and can never become the final model. Install it and, while you record, the words appear in a six-line panel above the editor, scrolled to the newest line. The final transcript still comes from your offline model — the preview never writes the editor. With the preview model not installed the preview adds nothing: no panel, no download, no prompt, no notice. previewEnabled (a settings.json field, on by default) turns the panel off without deleting the model.

Screenshots were captured on a v7.0 build: the panel now has five tabs (Speak was added) and the catalogue has grown to 22 models.

Downloaded — manage installed models

See what's installed, total disk usage, and which model is active. Press Enter to activate, x to delete. The preview model appears here too, marked preview only; Enter does not activate it. Models from Handy are auto-detected and can be imported without re-downloading.

Speak — TTS models and voices

Twenty offline TTS models (Kitten, Kokoro, Piper) from ~13 MB, downloaded on selection; then pick a voice and set the speed. Replies are read aloud automatically once TTS is on — ttsAutoSpeak (on by default) is a settings.json field with no row in this tab. The panel has a Test row that synthesises a sample sentence through the same pipeline.

Device — hardware profile and dependencies

See your hardware profile (RAM, CPU, GPU), dependency status (sherpa-onnx runtime), available disk space, and total downloaded models. Model recommendations are based on this profile.

Punctuation — offline, marks-only

Not a tab: punctuation is an automatic step between the recogniser and the editor, on by default. When a transcript contains Chinese and is essentially unpunctuated (fewer than one mark per 20 characters), an in-process model inserts full stops, commas and question marks. It can only insert marks — every other byte of the text it is given is preserved — and any problem leaves the text with no mark added. English is not supported (the model measured F1 0.175 on English), and a transcript whose punctuation is already dense enough — one mark per 20 characters or more — is left alone; a sparser one is punctuated even if it carries a mark or two.

The model is a one-time ~285 MB download, fetched in the background from the moment a session starts — or, on a machine where that did not happen, the first time a dictation needs it. Either way a dictation that arrives while the download is still running gets no mark added. Once the download has completed, verified against its digest and the engine has been constructed — any of which can fail — later qualifying dictations are punctuated; until then they get no mark added, and /voice-punctuation status reports the state. The switch is the punctuationEnabled setting (a settings.json field, on by default — the panel has no row for it), and /voice-punctuation status shows whether the step ran, why it did not, and the state of the model.

Usage

Keybindings

How recording works

1. Hold SPACE — a countdown shows how long is left. Audio capture starts when the threshold is reached, so begin speaking then: anything said during the countdown is not recorded
2. Keep holding — audio streams into a memory buffer, and a live preview above the editor shows the words as the streaming model recognises them (only when the preview model is installed)
3. Release SPACE — recording continues for 1.5s (tail recording) to catch your last word, then finalizes
4. The active local model transcribes the buffer on a worker thread (2–10 seconds), the transcript is cleaned and punctuated, and the text is inserted at the cursor in chunks small enough for the editor to take whole, ready to send

Commands

v7.1 keyboard

While in the settings panel:

While an install widget or playback indicator is mounted (no overlay in front):

Language scope

Supported and validated: Chinese, English, and mixed Chinese–English dictation. Every recognition, punctuation and quality decision in this repository is made against those two languages.

Other languages are not supported: the models that cover them stay in the catalogue because they are general-purpose models, but nothing about them is validated here — no accuracy figures, no punctuation behaviour, no device recommendation. They are on the roadmap, not in the support matrix.

The language row offers whatever the active local model supports — a single-language model auto-sets the language and an English-only model hides the others, so pick the model first if the language you want is missing.

Local Models

21 offline models across 7 families, plus the streaming preview entry described above. The tables here cover the offline ones, the only models that can produce the final transcript. Sorted by quality — best models first. Only Chinese and English are validated; every other language listed below is out of scope — see Language scope.

Top picks

Fast and lightweight

Specialist

Plus 8 language-specialised Moonshine v2 variants — two for Japanese, one each for Korean, Arabic, Chinese, Ukrainian, Vietnamese and Spanish — and only the Chinese one is inside the supported scope.

How local models work

Hold SPACE → audio captured to memory buffer (plus the live preview, when installed)
                ↓
Release SPACE → buffer sent to sherpa-onnx inside a worker thread
                ↓
         ONNX inference on CPU (2–10 seconds)
                ↓
         Final transcript inserted at the cursor

Dictation is batch, not streaming: the final transcript appears when you finish speaking, a few seconds later — the live preview above the editor is a separate streaming model and never becomes the text that is inserted. Models download automatically on first use — resumable, verified after completion, and deduplicated (no double-downloads). The streaming preview model is the exception: it is fetched only when you press ↵ on its row in the Models tab, never on first use. The settings panel shows real-time download progress with speed and ETA.

Models from Handy (~/Library/Application Support/com.pais.handy/models/) are auto-detected and can be imported via symlink (zero disk duplication).

Performance

Measured on the maintainer's machine, local CPU, no network, over 70 published utterances (28 Chinese, 28 English, 14 mixed Chinese–English). RTF is processing time divided by audio duration — lower is better, and below 1.0 is faster than real time.

End to end, the punctuation step adds almost nothing when it runs: on 27–32 character inputs the model call measures p50 2.9 ms / p95 3.1 ms after a one-time 544 ms load, all on CPU (see docs/BENCHMARKS.md). whisper-turbo is the slowest of the three by an order of magnitude and the least accurate on this corpus — measured for comparison, not recommended for CPU-only use.

Protocol, all result tables and the honest limits live in docs/BENCHMARKS.md — GitHub only, because npm ships the extension and this README.

Features

Architecture

# core
extensions/voice.ts                         Main extension — state machine, recording, UI, command surface
extensions/voice/config.ts                  Config loading, saving, migration
extensions/voice/onboarding.ts              Model and language pickers, first-run flow
extensions/voice/audio-tool.ts              Capture tool detection (sox / ffmpeg / arecord)
extensions/voice/hold-to-talk.ts            Hold detection, Kitty and non-Kitty terminals
extensions/voice/release-controller.ts      Recording lifecycle, release handling

# speech-to-text
extensions/voice/local.ts                   Model catalog (22 models), in-process transcription
extensions/voice/sherpa-engine.ts           sherpa-onnx bindings — recognizer lifecycle, inference
extensions/voice/sherpa-loader.ts           Lazy native module loading
extensions/voice/model-download.ts          Download manager — resume, progress, verification, Handy import
extensions/voice/device.ts                  Device profiling — RAM, GPU, CPU, container detection
extensions/voice/transcript-cleanup.ts      Filler removal and spoken-numeral restoration
extensions/voice/transcript-join.ts         Segment stitching — CJK-aware, no space inside a sentence

# live preview and workers
extensions/voice/preview-entrypoint.ts      One preview session and one final pass per recording
extensions/voice/preview-session.ts         Preview session — events, text, degradation, display lifetime
extensions/voice/preview-stream.ts          Streaming segmentation — endpoint and 60 s segment cap
extensions/voice/preview-worker-client.ts   Preview worker client — claims, partials, flushes, retirement
extensions/voice/local-preview-worker.ts    Preview worker entry — streaming recogniser off the main thread
extensions/voice/final-pass-client.ts       Final-pass worker client — timeouts, reuse, invalidation
extensions/voice/final-pass-worker.ts       Final-pass worker entry — offline recogniser off the main thread
extensions/voice/transcript-composer.ts     Committed prefix plus the mutable partial tail
extensions/voice/preview-panel.ts           Live draft widget — six rows, tail visible, no header
extensions/voice/editor-sink.ts             Cursor insertion in chunks the editor takes whole

# offline punctuation
extensions/voice/punctuation.ts             Offline punctuation — marks-only splice, fail-open step
extensions/voice/punctuation-model.ts       Punctuation model — catalogue, digest verification, download

# text-to-speech
extensions/voice/speak.ts                   Speak entry point, auto-speak wiring
extensions/voice/tts-engine.ts              sherpa-onnx TTS synthesis
extensions/voice/tts-local-models.ts        Local TTS catalog — 20 models (Kitten, Kokoro, Piper)
extensions/voice/tts-playback.ts            Playback, buffering, player detection
extensions/voice/tts-text-filter.ts         Code-block stripping, sentence prep
extensions/voice/tts-onboarding.ts          TTS onboarding flow
extensions/voice/tts-onboarding-overlay.ts  TTS onboarding overlay
extensions/voice/tts-install-progress.ts    Model install progress widget
extensions/voice/tts-playback-indicator.ts  Speaking indicator widget

# settings and UI
extensions/voice/settings-panel.ts          Settings panel — overlay, 5 tabs
extensions/voice/ui-picker.ts               Generic list picker
extensions/voice/ui-help-overlay.ts         Keyboard and command reference
extensions/voice/ui-aura.ts                 Visual primitives (Liquid Braille, Aurora)
extensions/voice/ui-widget-base.ts          Widget registry and base class
extensions/voice/ui-render-ticker.ts        Shared render ticker
extensions/voice/ui-icons.ts                Glyph and icon set
extensions/voice/ui-width.ts                CJK-aware visual width helpers
extensions/voice/ui-locale-labels.ts        Native language and voice labels

# benchmarks (repository only — never published)
bench/manifest.jsonl                        Benchmark slices: dataset, split, license, selection rule
bench/fetch.py                              Deterministic fetcher — writes bench/data/ (gitignored)

# types
extensions/voice/sherpa-onnx-node.d.ts      Type declarations for the optional native module

Configuration

Settings stored in Pi's settings files under the voice key:

Both paths follow Pi: the global one sits in Pi's agent directory, so a build run with PI_CODING_AGENT_DIR set uses that directory instead of ~/.pi/agent; the project one uses Pi's config-directory name rather than a literal .pi. Downloaded models are separate and always live under ~/.pi/models.

{
	"voice": {
		"version": 6,
		"enabled": true,
		"language": "zh",
		"localModel": "parakeet-v3",
		"scope": "global",
		"onboarding": { "completed": true, "schemaVersion": 6 }
	}
}

A file carried over from 0.5.x may still hold the retired speech-backend fields; the first start erases them and says so — see Privacy.

Hold-to-talk delay defaults to 700 ms (/voice-hold-delay accepts 200–3000 ms).

Streaming preview

Three ordinary optional fields: an explicit project value wins, then the global one, then the default. All three are settings.json fields — the panel has no rows for them.

The recording budget and the preview's 60 s segment cap are separate limits: the cap ends a recognition segment early, it does not stop the recording. The preview model lives in ~/.pi/models/streaming-paraformer-bilingual-zh-en/ and appears in the Downloaded tab like any other download, marked preview only; pressing Enter does not activate it.

Punctuation

Punctuation is automatic and on by default. When a dictation finishes, the transcript is cleaned first — see the cleanup switches in the table below — then punctuated locally if the text contains Chinese and its punctuation density is below one mark per 20 characters; the step only inserts marks and changes nothing else. The model runs in process, and the one-time download of the model itself is the only traffic it causes. English is not supported in this version, and English text is skipped by the same rule. A transcript the rule declines gets no mark added, and neither does any transcript whose punctuation fails at any point.

Both engines stay resident for the session: the recogniser as before, and the punctuation engine built when a session starts, so no dictation waits for it (measured 0.2 ms for the call, engine usable within 100 ms, 50-544 ms to build). On a machine that has never fetched the model, that first session fetches it: 285 MB on disk, and roughly 300 MB of memory per engine on the reference machine — 292 MB for punctuation, 316 MB for the paraformer-zh recogniser. See Engine residency for the numbers.

All three switches resolve the same way. A machine-wide false is final: no project block can turn the step back on, even one that saved true before the global value was set to false. Below that, an explicit project value wins — a repository that saved false keeps the step off when the global value later turns on — then the global value, then the default. An omitted value in a project block inherits the global one rather than the default. The cleanup runs before punctuation and on its own switches, so turning punctuation off still leaves a cleaned transcript.

The first project-scope save writes the values the config resolved to — including inherited ones — into the repository's .pi/settings.json, so all three keys become concrete booleans there. That pins the project value; it does not exempt the repository from a later machine-wide false, which still applies everywhere.

/voice-punctuation status prints the punctuation switch state, whether the model is downloaded and digest-verified, and what the step decided on the last dictation of the session. The model lives in ~/.pi/models/punct-ct-transformer-zh-en/, appears in the Downloaded tab like any other download, and selecting that row does not make it a recogniser. The step writes one metadata-only voice-punctuation entry per dictation — no transcript text — so a decision is fact; under PI_VOICE_DEBUG it also logs one line per dictation with the character count, marks before and after, the gate, the reason when it did not run, and the elapsed time. The removed postProcess* keys from older releases are ignored when loading — never migrated, and left in place when the settings file is saved.

Privacy

No audio is uploaded to a remote speech service. Recording, recognition, punctuation and speech synthesis all run inside the Pi process — sherpa-onnx is loaded natively and works on buffers in memory. There is no endpoint to configure, no key to paste, no account to create.

That is the guarantee the code keeps, and it is narrower than "nothing ever leaves the machine". One configuration this README supports sends data off it, and it is your own configuration rather than something the extension does by itself.

System audio routing. Playback hands the synthesised WAV to the system player, and that process inherits your environment. Point PulseAudio at another host (PULSE_SERVER, an SSH audio tunnel, a remote desktop session) and the player sends the speech there — the audio tool honours the same variable, so a remote microphone means your voice crosses that link before this extension ever sees it.

That route is not a speech service this extension chose. If you need the audio kept on this machine, local playback is only half of it: capture honours the same variable, so both ends — the player and the microphone — have to stay local.

Everything you send to the agent — typed, or dictated and then submitted — is text, and it goes to whatever model Pi is configured with. That is not an exception to the claim above: the claim is about audio, and sending a message is the path every request takes.

What the gate checks

tests/no-cloud-egress.test.ts is the enforceable half of that claim. It parses every module under extensions/ and fails when one imports the removed WebSocket transport, names a retired vendor identifier or host, or calls fetch outside the two model-download modules (model-download.ts, tts-local-models.ts).

That is narrower than "this software can never open a socket", and it is worth saying so plainly: the gate binds the transport and the vendor this repository deleted, and it keeps ordinary code from growing a new call site. A future module could still build its own network client — which is why this is a test you can read, not a promise in prose.

The one outbound step this extension takes

Model files are downloaded from HuggingFace or GitHub releases: the speech models you pick, the TTS models you pick, the ~285 MB punctuation model on a machine that needs it, and the optional streaming preview model when you press ↵ on its row — never before. Downloads are resumable and checked when they finish: the speech model files must all be present at their expected sizes, and the punctuation model is verified file by file against the SHA-256 digests recorded in this repository. After a model is on disk, dictation and speech work with the network down.

What stays on the machine

• A dictation is held in a memory buffer and dropped after the transcript is written; no recording is stored. (/voice test does write a one-second sample to the temp directory to prove the microphone works — it is deleted as soon as its size is read.)
• Speech playback writes the synthesised audio to a temp WAV with mode 0600 and hands it to the local player (afplay on macOS, paplay/aplay on Linux, PowerShell's SoundPlayer on Windows), then deletes it. The player is local but it is a separate process that inherits your environment, which is why system audio routing is the one exception above instead of something this extension can promise about.
• /voice history keeps this session's transcripts in memory only.
• PI_VOICE_DEBUG=1 appends diagnostics — including a 100-character transcript preview — to $TMPDIR/pi-voice-debug.log. It is off unless you set it.
• No telemetry, analytics or crash reporting is transmitted: there is no crash reporter and no remote logging.

Upgrading from 0.5.x

0.6.0 removed the cloud backends, so the first start erases a stored provider API key and a localEndpoint from settings.json and tells you it did — the field names are listed in SECURITY.md. The rest of the file is left exactly as it was, byte for byte, including formatting JSON.parse/JSON.stringify would not preserve. An API key exported from a shell profile or ~/.env.secrets is no longer read at all; the extension does not edit those files, so delete the line yourself.

See SECURITY.md for vulnerability reporting.

Troubleshooting

Run /voice test inside Pi for full diagnostics.

More in docs/troubleshooting.md.

License

MIT — original by @baanditeagle, maintained by CyFeng16