owo-local

owo-local is a Pi package that connects Pi to OpenAI-compatible local and self-hosted model servers. It supports interactive login, bearer or keyless authentication, live /models discovery, offline fallback catalogs, streaming, reasoning toggles, and configurable model limits.

This repository also contains an example SGLang deployment for orcarouter/Qwen3.8-27B-Uncensored-FP8.

Use the Pi extension

Install dependencies and validate the package:

npm ci --ignore-scripts
npm run validate

Load the extension directly during development:

pi -e .

Or install this checkout as a persistent Pi package:

pi install /path/to/owo-local

Configure it interactively with /login owo-local, then enter:

• Base URL: your OpenAI-compatible server URL
• API key: your server API key, or leave it blank for a keyless server
• Model: qwen3.8-27b-uncensored-fp8

For a shell that reads the existing secret without printing or copying it:

./scripts/owo-sglang-shell.sh
pi -e .

Available Pi commands:

• /owo-local-status tests the connection and lists models.
• /owo-local-refresh refreshes the server model catalog.
• /owo-local-help shows setup instructions.
• /owo-sglang-start starts the user service and waits for model readiness.
• /owo-sglang-stop stops the user service and frees both GPUs.
• /owo-sglang-restart restarts it and waits for readiness.
• /owo-sglang-status reports the systemd state and restart count.
• /model selects a discovered model.

Client configuration

The extension reads these optional environment variables:

Variable	Purpose	Default
OWO_LOCAL_URL	OpenAI-compatible base URL	interactive setup
OWO_LOCAL_API_KEY	Bearer API key	keyless
OWO_LOCAL_MODELS_URL	Full or relative model-catalog URL	<base URL>/models
OWO_LOCAL_HEADERS	JSON object of additional request headers	none
OWO_LOCAL_BEARER_AUTH	Send Authorization: Bearer for the API key	true
OWO_LOCAL_MODEL(S)	Offline fallback model IDs	none
OWO_LOCAL_CONTEXT_WINDOW	Context advertised to Pi	server value or 32K
OWO_LOCAL_MAX_TOKENS	Maximum generated tokens	server value or 4K
OWO_LOCAL_REASONING	auto, true, or false	auto
OWO_LOCAL_VISION	auto, true, or false	auto
OWO_LOCAL_THINKING_FORMAT	Provider thinking parameter format	Pi default
OWO_LOCAL_DISCOVERY_TIMEOUT_MS	/models request timeout	5000 ms
OWO_SGLANG_SERVICE	User systemd unit controlled by Pi	qwen-sglang.service
OWO_SGLANG_HEALTH_URL	Readiness endpoint checked after start	local port 30000
OWO_SGLANG_START_TIMEOUT_MS	Maximum model startup wait	300000 ms

Custom base paths are preserved. Full endpoint URLs such as https://gateway.example/openai/v1/chat/completions are normalized to their base URL automatically.

For a gateway with a separate catalog route and X-API-Key authentication:

export OWO_LOCAL_URL=https://gateway.example/openai/v1
export OWO_LOCAL_MODELS_URL=https://gateway.example/catalog/models
export OWO_LOCAL_API_KEY=your-secret
export OWO_LOCAL_BEARER_AUTH=false
export OWO_LOCAL_HEADERS='{"X-API-Key":"${OWO_LOCAL_API_KEY}","X-Tenant":"personal"}'
pi -e .

OWO_LOCAL_MODELS_URL may also be relative to the base URL. Model discovery accepts OpenAI { "data": [...] }, { "models": [...] }, and raw array responses. Entries may be model objects, { "model": "id" }, or ID strings.

SGLang deployment

This project serves orcarouter/Qwen3.8-27B-Uncensored-FP8 through SGLang's OpenAI-compatible API. The included defaults target the following example configuration:

• 2 x NVIDIA RTX 3090 (24 GiB each), tensor parallel 2
• 128K maximum context and 128K total token pool
• One active request for private, single-user serving
• FP8 KV cache, FP32 GDN/SSM state
• 2K chunked prefill for smoother decode latency
• Decode-only CUDA graphs to keep TP workers synchronized with asymmetric VRAM headroom
• Triton FP8 GEMM on Ampere, where native Hopper FP8 kernels are unavailable
• Text-only serving to avoid spending VRAM on the vision path

The checkpoint occupies about 31 GB on disk. With TP=2, the weights and runtime state fit across the 48 GiB aggregate VRAM while retaining activation headroom. The five-slot GDN cache is derived from SGLang's low-latency extra_buffer strategy: 1 concurrent request x 5 state slots.

Install

1. Install the pinned environment:

   ./scripts/setup.sh
   

2. Open the model page, accept its access terms, and create a read token.

3. Authenticate without putting the token in this repository:

   .venv/bin/hf auth login
   

4. Optionally copy .env.example to .env and change the bind address, port, API key, context, or concurrency.

Run

./scripts/serve.sh

The first run downloads the model into the normal Hugging Face cache. When the server reports that it is ready, test it from a second terminal:

./scripts/smoke-test.sh

The endpoint is http://127.0.0.1:30000/v1 by default. For an OpenAI client, use qwen3.8-27b-uncensored-fp8 as the model name.

Terminal service control

Run SGLang in the background with the user systemd service:

systemctl --user start qwen-sglang
systemctl --user stop qwen-sglang
systemctl --user restart qwen-sglang
systemctl --user status qwen-sglang

Follow the server logs:

journalctl --user -u qwen-sglang -f

To run SGLang attached to the current terminal, stop the background service first and launch the server directly:

systemctl --user stop qwen-sglang
cd /path/to/owo-local
./scripts/serve.sh

Press Ctrl+C to stop a foreground server. Do not run the foreground command while qwen-sglang.service is active because both processes will compete for the GPUs and port 30000.

Reverse proxy or tunnel

To expose the server through a reverse proxy or tunnel, point it at http://127.0.0.1:30000 and configure authentication at the server or proxy. Keep credentials in environment variables or a local .env file that is not committed to source control.

Tuning boundaries

The defaults target stable interactive use. The native model context is 262K, but that is not a sensible default on two 24 GiB cards. Keep these relationships when tuning:

• The Mamba cache size is derived automatically as SGLANG_MAX_RUNNING_REQUESTS x 5 for the low-latency strategy.
• If increasing context, raise SGLANG_MAX_TOTAL_TOKENS with it and watch both GPUs during a worst-case prompt. Lower concurrency first if startup or prefill runs out of memory.
• Keep 2K prefill chunks on these cards. Larger chunks increase peak activation memory and make active decoding less smooth.
• FP8 KV is intentional here. Switching to BF16 KV roughly doubles KV memory.
• Do not enable the bundled MTP head until the base profile is measured; it adds memory pressure and is not needed for a reliable first deployment.

Extra SGLang flags can be appended without editing the script:

./scripts/serve.sh --log-level debug

This configuration follows the upstream Qwen3.8-27B cookbook, with TP=2 and Ampere-compatible kernels substituted for the H200 recipe.