TaskScribe

A Windows 11 desktop app that watches you perform a task once and produces a Markdown how-to guide in an Obsidian vault.

Type a title → click Start → do the task → click Stop → review the redacted contact sheet → click Generate. Within a few minutes a step-by-step note appears in Documents\TaskScribe-Vault\Tasks\, frontmatter-tagged with the software detected and backlinked to a per-application hub note. Over time the vault's graph view becomes a self-organising procedure library.

Status: v0.1.0 beta. Architecture, GUI, and 367-test suite are complete. The end-to-end capture pipeline currently writes empty image_path entries — a known wiring bug being tracked for fix.

---

Why this exists

Knowledge workers run dozens of small bespoke procedures every week — renewing a cert, configuring a tool, walking a billing dashboard. Each one is almost worth documenting, but the friction of writing it up afterwards is too high. Three months later you do it again, slowly, from scratch.

Existing tools get the tradeoffs wrong:

• Scribe / Tango / Guidde — SaaS, cloud-first, monthly subscription, screen content leaves your machine.
• Microsoft Recall — always-on, captures everything, recurring security panic.
• Windows Steps Recorder — raw click log, no semantic extraction.
• Macro recorders — replay tools, not documentation tools.

TaskScribe is personal-first, text-first, graph-linked, and cost-capped at $0.50/session (typical usage 5–20¢). The vision-LLM call to Anthropic is the one outbound touchpoint, gated by an explicit review of a redacted contact sheet.

---

How it works

┌─────────────┐    ┌──────────┐    ┌──────────┐    ┌──────────────┐    ┌────────────┐
│  Recorder   │───▶│ Redactor │───▶│ Analyser │───▶│  Coordinator │───▶│   Vault    │
│ (dxcam/mss) │    │  (UIA +  │    │  (Haiku  │    │  (Sonnet     │    │  Writer    │
│  + pynput   │    │   regex) │    │  per-    │    │   synthesis) │    │  (atomic   │
│   + UIA     │    │          │    │  frame)  │    │              │    │  Markdown) │
└─────────────┘    └──────────┘    └──────────┘    └──────────────┘    └────────────┘
       │                                                                      │
       ▼                                                                      ▼
  Frames + input events                                          Obsidian vault, frontmatter-
  + foreground window                                            tagged, backlinked to per-app
                                                                  hub notes

1. Capture — dxcam for DXGI frame grabs (fallback: mss), pynput for input events, uiautomation for UI tree snapshots, psutil for the foreground process.
2. Redact — UIA IsPasswordValue controls and regex-matched secrets (sk-, AKIA, ghp_, xoxb-, IBAN, ≥32-char high-entropy blobs) are Gaussian-blurred at their bbox before any pixel is written to disk. Denylist of password managers / banking windows drops frames entirely.
3. Per-frame analysis — Claude Haiku 4.5 extracts UI actions from each kept frame in parallel.
4. Synthesis — Claude Sonnet 4.5 stitches the per-frame actions into a numbered procedure.
5. Write — atomic Markdown write (temp file + os.replace) into the vault, frontmatter-tagged with detected software, with a backlink injected into the software hub note.

---

Engineering highlights

• Hexagonal architecture. Domain code (taskscribe/domain/, taskscribe/recorder.py, taskscribe/redactor.py, taskscribe/analyser.py) depends only on ports in taskscribe/ports/. Real Windows adapters (pywin32_accessor, dxcam via capture/_grab_worker, keyring_backend, anthropic_sdk) live behind those ports and are swapped for fakes in tests.
• Test-first. 367 tests across 42 files: unit, integration, and an adversarial/ suite that hits path traversal, YAML injection, concurrency races, recovery corner cases, secret handling, and pathological redactor inputs. Hypothesis is used for property-based regex tests.
• Defence in depth on secrets. API key in Windows Credential Manager (refuses to start if keyring falls back to plaintext). Logs carry only session_id/frame_id, never frame content or keys. Redactor runs before disk write, not after.
• Cost-capped AI. Hard $0.50 ceiling per session, enforced inside the analyser (test_analyser_cost_cap.py). Per-frame dedup via perceptual hashing (imagehash) cuts redundant Haiku calls.
• Crash-safe store. SQLite WAL with explicit recovery on boot (store/recovery.py); interrupted sessions resume cleanly. Retention sweeper purges aged frames to a configured cap.
• Async-aware GUI. PySide6 with qasync to bridge Qt's event loop to asyncio so the analyser can stream without freezing the window.
• Pre-flight doctor. taskscribe doctor verifies keyring backend, API-key vision entitlement (via a 1×1 PNG probe), vault writability, dxcam init, models manifest parse, and SQLite integrity — surfaces broken installs before the user records anything.

---

Tech stack

Language / runtime: Python 3.12, packaged with Hatchling, locked with uv.lock.

Capture & OS integration: dxcam (DXGI desktop duplication) → mss (fallback), pynput (global hooks), pywin32 + uiautomation + comtypes (Windows UI Automation), psutil (process introspection).

GUI: PySide6 (Qt 6), qasync (asyncio ↔ Qt bridge), Pillow for contact-sheet composition.

AI: anthropic SDK — Claude Haiku 4.5 for per-frame action extraction, Claude Sonnet 4.5 for procedure synthesis. Prompts in taskscribe/prompts/.

Persistence: aiosqlite (WAL), python-ulid for session IDs, tomli-w for config, PyYAML for frontmatter, keyring (WinVaultKeyring) for secrets, atomic Markdown writes for vault output.

Privacy primitives: imagehash for perceptual dedup, tldextract for URL normalisation, regex pattern bank for secret detection.

Tooling: pytest (+ pytest-asyncio, pytest-qt), hypothesis, ruff.

---

Repository layout

taskscribe/
  domain/          # Pure data classes — frames, steps, analysis, errors
  ports/           # Protocol interfaces (anthropic_client, clock, filesystem,
                   #   secret_backend, windows_accessor)
  adapters/        # Real implementations of the ports
  capture/         # Recorder, screen/input/foreground grabbers
  store/           # SQLite writer, reader, migrations, crash recovery
  gui/             # PySide6 windows, contact sheet, setup wizard, tray
  cli/             # `taskscribe doctor` entry point
  prompts/         # Versioned Haiku / Sonnet prompts
  resources/       # Default config, models manifest, seed software list
  analyser.py      # Per-frame Haiku calls + dedup + cost cap
  analysis_coordinator.py  # Sonnet synthesis pass
  redactor.py      # UIA-driven + regex-driven blurring
  recorder.py      # Capture FSM
  vault_writer.py  # Atomic Markdown writes + hub-note backlinking
  retention_sweeper.py
  software_resolver.py
  doctor.py
  settings.py
  app.py           # Boot wiring
tests/
  unit/            # 28 modules
  integration/     # 6 modules — boot recovery, end-to-end record→note, etc.
  adversarial/     # 7 modules — path traversal, YAML injection, secrets, ...
  fakes/           # In-memory fakes for every port

---

Running locally

# Requires Python 3.12 on Windows 11.
uv sync                       # install pinned deps from uv.lock
uv run taskscribe doctor      # pre-flight health check
uv run python -m taskscribe   # launch GUI
uv run pytest                 # 367 tests

The first launch opens a 4-page setup wizard that collects the Obsidian vault path, validates an Anthropic API key with a 1×1 PNG probe, and writes privacy defaults to %APPDATA%\TaskScribe\config.toml.

See README.txt for end-user install instructions (SmartScreen, Defender exclusion, uninstall).

---

License

MIT