English | 한국어

Sidehand

Your AI agent works in your Windows apps. You keep the keyboard.

Sidehand lets Claude Code, Codex, pi or any agent with a shell read and operate Windows desktop apps in the background. It fills in forms, clicks, right-clicks, drags, scrolls, presses keys and shortcuts, starts apps and arranges windows without bringing anything to the front, so whatever you are doing (a game, a browser, a document) keeps your keyboard. It works from Windows or from inside WSL, and it needs nothing beyond the PowerShell that ships with Windows.

Unedited take on a real Windows 11 desktop, recorded with Recordly by demo/record.ps1. Keystrokes go into Notes for the whole run: 145 of 145 arrived, and the foreground window never changed. MP4

Why Sidehand

• It does not take your focus. On a real desktop, 20 of 20 background actions in WinForms, WPF, Edge and VS Code landed while the window you were typing in stayed in front. With the other tools we measured, a different window jumped to the front every time.
• It is cheap to read. A window comes back as one compact line per control: 5 to 12 times fewer tokens than the JSON the other tools returned for the same window.
• It is small and transparent. No daemon, no telemetry, no downloaded binary. About 1,400 lines of PowerShell, C# and JavaScript, compiled on your PC into %LOCALAPPDATA%\sidehand on first use. Reading it takes an afternoon, removing it takes one command.

Benchmark

Same Windows 11 PC and test apps for all three tools. Focus: each action ran against a fresh background WinForms or WPF window while Notes held the keyboard (demo/record.ps1). Tokens: one read of the same window, o200k_base tokenizer; Cua Driver with include_screenshot: false. Size: npm unpacked size, or the binary for Cua Driver. Chart source: demo/benchmark_chart.py.

What works while you type

Every ✅ took effect without changing the foreground window, on a real desktop (demo/measure.ps1, demo/record.ps1) or on the hidden test desktop (test/isolated). Where the cursor was tracked, it never moved.

Starting an app and moving, resizing, minimizing or restoring a window use the same Windows calls for every app (tested with WinForms). WPF ignores mouse messages that do not come from the real mouse, and WPF and browser content take keys only while their window is focused, so those need the user's consent (--allow-foreground).

Compared with other tools

⚠️ and ❌ in the first row were measured for Cua Driver and agent-desktop and follow from the documented design for the other two; agent-desktop documents drag, hover and mouse commands as --headed only. A ❌ for hiding private info means we found no such feature in the docs or the source (November 2026).

Real mouse input is left out on purpose: Windows has one cursor per session, so an agent that moves it takes it from you. We also tried a second, synthetic pen pointer; Windows still moved the cursor and WPF ignored it, so it did not ship. Pick Cua Driver or agent-desktop if you need macOS or Linux, real mouse input or full browser automation. Pick Sidehand if you work on Windows, want the agent to stay out of your way, and prefer a tool small enough to read.

Install

# CLI, for Claude Code, Codex or any agent that runs shell commands (Windows or WSL, Node 18.3+)
npm install -g sidehand

# pi
pi install git:github.com/Blue-B/sidehand

For a CLI agent, add this to AGENTS.md or CLAUDE.md:

Use the `sidehand` CLI to read and operate Windows desktop apps without taking the user's focus.
Read with `sidehand tree "<window title>"`, act with `sidehand act <ref> <action>`, see `sidehand --help`.
Never pass `--allow-foreground` or `--bring-to-front` unless the user agreed.

Usage

$ sidehand tree "Contoso Orders"
window: Contoso Orders  (18 elements)
  e4 Edit "Customer"
  e6 Edit "Quantity"
  e8 ComboBox "Region"
  e11 CheckBox "Express shipping" [Off]
  e12 Button "Submit order"

$ sidehand act e4 type "Blue-B Studio"
ok set text via window message: e4 Edit "Customer" value="Blue-B Studio"

$ sidehand act e8 select Jeju
ok select "Jeju" via window message: e8 ComboBox "Region" value="Jeju"

$ sidehand act e12 click
ok click via window message: e12 Button "Submit order"

$ sidehand wait "Contoso Orders" "Order submitted"
ok found in Contoso Orders
  e13 Text "Order submitted: Blue-B Studio, 3 units to Jeju, express"

In pi the same operations are tools named sidehand_ui_tree, sidehand_ui_act, sidehand_capture and so on.

Limits

• focus and capturing a minimized window would take your foreground, so they are refused unless the agent passes --allow-foreground or --bring-to-front.
• Games and other apps that draw their own UI expose no controls. Sidehand can still capture them.
• Minimized windows: sidehand window <title> restore brings one back behind your windows first.
• WPF apps ignore background mouse gestures (right-click, double-click, drag, wheel), and HTML <select> drop-downs ignore background selection; Sidehand reports a failure instead of pretending.
• Drag and drop between two apps (files into a folder) needs the real mouse.
• Keys go to a control with its own window handle; WPF and browser content need --allow-foreground.
• Apps running as administrator only accept messages from an agent that also runs elevated.

How it works

• scripts/SidehandUia.cs is a native UI Automation client, the COM interface screen readers use. Windows PowerShell compiles it once into %LOCALAPPDATA%\sidehand together with an interop assembly generated from Windows' own UIAutomationCore.dll.
• For each action it first looks for a classic control with its own window handle (EDIT, BUTTON, LISTBOX, COMBOBOX) and sends it WM_SETTEXT, BM_CLICK or LB_/CB_SETCURSEL. A window message cannot activate a window.
• Other controls get the UI Automation pattern call while the target's top-level window carries WS_EX_NOACTIVATE, so the app's own Focus() call cannot bring it forward. The idea comes from Cua Driver's no-raise guard. If an app still comes forward, the result says so.
• Mouse gestures and keys are posted to the window under the element (WM_RBUTTONDOWN, WM_MOUSEWHEEL, WM_KEYDOWN and so on). They go through the app's own message loop but never move your cursor. For shortcuts, Ctrl, Shift and Alt are marked as held in the target thread's key state only, for the moment of the keystroke.
• Everything runs in physical pixels, so coordinates and screenshots are exact on scaled displays.

Reproduce the numbers

npm test                          # hidden desktop, nothing appears on screen, a few minutes
powershell -File demo/measure.ps1 # real desktop: focus and cursor for WinForms, Edge and VS Code, about 3 minutes idle
powershell -File demo/record.ps1  # real desktop: the demo video and the three-tool table, 5 to 10 minutes idle

npm test runs test apps and a decoy "user" window on a separate, never-shown Windows desktop inside a kill-on-close job, and fails if any Sidehand action activates its window. It also runs Cua Driver and agent-desktop when they are installed. demo/record.ps1 -Recordly re-creates the demo video and stops by itself if someone uses the PC.

License

MIT