Skip to content

About

Voice-first personal assistant for people with low vision (tunnel vision): notes, email, daily-life help and options trading by voice, with a large-text, big-button dashboard.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Voice AI Notetaking Agent

A local, voice-driven assistant with two modes:

  • Conversation mode (default) — talk to it like you talk to Claude. Ask about your saved notes ("what's my latest note?", "what did I say about the budget?"), switch which Claude model answers ("switch to Opus"), or ask how the agent itself works ("how does barge-in work?").
  • Notetaking mode — a silent recorder. It listens without speaking, handles long silences efficiently (an hour-long session with only a few minutes of speech does almost no transcription work), and on stop it saves a transcript + an AI summary to disk and reads the summary back to you.

Everything runs locally except Claude (the brains + summaries): transcription (faster-whisper), speech (SAPI on Windows, NSSpeechSynthesizer on macOS, Piper on Linux), and semantic note search (Chroma + sentence-transformers) are all on-device. No UI — just your voice and hotkeys — but everything is logged to logs/.

Who this is for

This assistant is designed for people with low vision. Its users aren't totally blind, but they have tunnel vision: a narrow field of view that makes reading dense screens, small text, and wide tables slow and error-prone. It helps them manage daily life (notes, email, reminders, to-dos) and trade options, mostly by voice. That design goal shapes everything:

  • Voice comes first. Every task can be done by speaking and listening; the screen is optional.
  • Spoken answers are accurate and current. The user can't easily check a number against a busy screen, so positions, prices, and fills come from the live data, never from a remembered guess.
  • The dashboard is built for a narrow field of view. It uses one narrow column, large text, big buttons, and puts each status message right next to the control that produced it.

For a detailed technical walkthrough — module map, data flows, the tool registry, and extension points — see PROJECT.md. (You can also just ask the agent: "tell me about this project.")

Setup

Runs on Windows, macOS, and Linux. requirements.txt carries platform markers, so the same pip install works everywhere and only pulls what your OS needs (pywin32/winrt on Windows, pyobjc on macOS, piper-tts/dbus-fast on Linux).

  1. Install Python dependencies:

    Windows — install into your regular Python (the Store Python works):

    pip install -r requirements.txt

    On Windows this uses webrtcvad-wheels (a precompiled build of webrtcvad) so you don't need Visual C++ Build Tools. It imports as webrtcvad either way.

    macOS / Linux — install into a project venv named .venv; the launch scripts pick it up automatically, so you never need to activate it:

    python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
  2. Provide your Claude API key. Easiest: copy .env.example to .env and put your key in it — it's loaded automatically on startup and is gitignored so it never gets committed:

    cp .env.example .env       # (Windows: copy) then edit .env, set ANTHROPIC_API_KEY

    Alternatively, set it as an environment variable instead of using .env:

    $env:ANTHROPIC_API_KEY = "sk-ant-..."   # current PowerShell session only
  3. First run downloads model weights once: faster-whisper small.en (~0.5 GB) and the embedding model all-MiniLM-L6-v2 (~90 MB).

Run

One launcher per OS family, same behavior: a plain launch first ingests anything new in knowledge/, then starts the agent; passing any argument skips the ingest pass and hands the argument through.

Agent Standalone dashboard
Windows run.bat (double-click works) dashboard.bat
macOS / Linux ./run.sh ./dashboard.sh

Useful arguments (any launcher, or python voice_agent.py directly):

./run.sh --selftest   # check mic, STT, TTS, Claude, and note search
./run.sh --miccheck   # meter the microphone without starting the agent
./run.sh --ingest     # absorb new knowledge/ files (video included), then exit
./run.sh --kb-list    # list ingested knowledge sources, then exit

Per-environment notes:

  • Windows — everything works out of the box: SAPI speech (pause/resume for the mute gesture), the SMTC media session for Bluetooth headset buttons, and the winsound thinking cue.

  • macOS — two permission prompts to expect, both granted to the terminal you launch from: Microphone (asked automatically on first run) and Input Monitoring / Accessibility (System Settings → Privacy & Security) for the headset button — without it, button presses silently never arrive. Speech uses NSSpeechSynthesizer, so barge-in and the mute gesture behave exactly as on Windows. The launch scripts prefer .venv/bin/python when it exists and fall back to python3 otherwise.

  • Linux — speech is Piper (piper-tts, pulled in by requirements.txt), a local neural voice in place of the OS's robotic espeak-ng. Its voice model (PIPER_VOICE in config.py, ~60 MB) is downloaded once into ~/.cache/piper the first time the agent talks — expect a pause then. Want a different voice? Fetch it and name it:

    .venv/bin/python -m piper.download_voices en_US-ryan-medium --data-dir ~/.cache/piper

    then set TTS_VOICE = "ryan" (or a persona's tts_voice) — voices match by name substring, and the dashboard's voice dropdown lists the downloaded ones. Without piper-tts the agent falls back to speech-dispatcher (speech-dispatcher python3-speechd system packages), then to espeak-ng through pyttsx3, where a reply can only be stopped, not paused. Headset buttons arrive through an MPRIS media player the agent registers on D-Bus (dbus-fast, also in requirements.txt), which is how Bluetooth headsets reach any Linux app and the only route that works under Wayland; it needs a desktop that forwards media keys to MPRIS players (GNOME and KDE do; on a bare window manager run playerctld). Under X11 wired headsets are also heard through the keyboard hook. If a Bluetooth headset's microphone is in use, PipeWire/PulseAudio switches it to the low-bandwidth headset profile (HFP), which makes everything it plays sound like a phone call — that is the Bluetooth profile, not the voice; use another microphone or enable the mSBC/LC3 codecs in your Bluetooth stack.

Only one agent can run at a time: a second launch detects the first (via a lock on data/agent.lock) and exits immediately with a spoken notice, so two instances can't talk over each other or corrupt your history and note index. The lock is released automatically when the agent stops — even on a crash — so there's nothing to clean up.

Controls

All controls work globally (even when another window is focused).

Headset button (play/pause)

Action Gesture
Toggle mute (stop listening) Single click
Toggle notetaking Double click
Quit Triple click

Mute stops the microphone, not the agent. Click it while a reply is playing and the reply finishes — you just stop being heard. The confirmation ("Muted." / "Listening.") is spoken over the reply on a second, quieter voice, so it lands the moment you press the button instead of waiting for the reply to end; on a machine with more than one voice installed it deliberately uses a different one, so you can tell the two apart. There's a brief hitch where the click landed: playback pauses the instant the button goes down, because the wireless dongle only passes on the next click of a double/triple if the host really stops (see docs/MEDIA_CONTROL.md), and the reply picks up again as soon as the click turns out to be a single.

The agent starts muted — it greets you with "Muted, click once to talk" and waits. Starting deaf is what stops a forgotten agent, running with no headset plugged in, from hearing the room, answering it, and spending tokens all afternoon. Turn it off with Start muted on the dashboard's settings page (START_MUTED) if you would rather it listen from the first second.

Muting while the agent is still thinking doesn't cancel anything either — the question you already asked is answered and spoken as usual; you just aren't being listened to while it happens. Double-click (notetaking) and triple-click (quit) do end a reply immediately, and to cut one short without changing anything, just start talking (barge-in, below).

On Windows and Linux, button presses are listened for on two channels at once (see docs/MEDIA_CONTROL.md): a keyboard hook (how wired headsets and USB wireless dongles deliver presses) and an OS media session (Windows SMTC, or an MPRIS player on Linux — how Bluetooth-native headsets like AirPods deliver them; those never appear as key events). A press arriving on both channels is counted once. On macOS the keyboard hook is the only channel (it also swallows the keys so a press doesn't launch Music.app). Multi-click detection uses a 450 ms window — clicks within that window count together. Headsets that decode multi-press in firmware (e.g. AirPods) send Next/Previous instead; those map to the same double/triple actions. A silent keepalive stream runs continuously (MEDIA_KEEPALIVE in config.py) so the headset's audio link never spins up from silence — wireless dongles drop presses during those first seconds — and every click briefly pauses it so state-tracking dongles stay in sync (see docs/MEDIA_CONTROL.md).

Dashboard controls (mute button, chat box)

The same mute, without the headset: the dashboard sidebar carries a mute button on every page, showing whether the microphone is live (🎙 Mic on) or deaf (🔇 MUTED, in red — and the browser tab title says so too, so a glance at the taskbar answers "am I muted?"). Clicking it does exactly what a single click on the headset does, spoken confirmation included; a reply already playing carries on. Mute from the headset and the dashboard follows within a couple of seconds.

Typing instead of speaking lives on the Conversation page, as a chat box under the transcript: type a message and the agent answers aloud, exactly as if you'd said it — persona addressing included ("Tom, …" typed works like "Tom, …" spoken). The message is answered at the next gap in conversation, never cutting off speech in progress, and typing works while muted — that's rather the point of typing. During note-taking it waits until the note ends. The exchange then appears in the transcript above like any spoken turn; the page watches for it, since the agent writes history only once it has finished answering.

How it works: the agent serves the dashboard itself (localhost:8765, from inside its own process — web/server.py's serve_embedded), so a button click is a direct method call on the running agent. A standalone dashboard (dashboard.bat, for browsing/config/ingest while the agent is off) answers control clicks honestly instead: "agent not running", or "the agent is running — use its dashboard" when one is alive in its own process. Adding a future control is one handler in web/server.py's CONTROL_ACTIONS table, one public method on the Agent, and a button; the routing doesn't change.

Barge-in (interrupt the agent)

While the agent is speaking a reply, just start talking — it will stop and listen. Say "continue", "go on", or "keep going" to resume where it left off.

Pausing mid-sentence

You don't have to say everything in one breath. When your utterance ends, the agent waits a short settle window (CONTINUATION_SETTLE_MS in config.py, default 600 ms) — listening, not yet answering. If you keep talking within it, the continuation is captured and merged and the window restarts; only once you've truly finished (the window passes in silence) does it call the model, once, with your complete question. So a multi-part question with pauses costs exactly one reply, not one per pause. The trade-off is a little latency before each reply (CONVO_ENDPOINT_MS + CONTINUATION_SETTLE_MS of silence): lower the settle window to answer sooner, raise it if your natural pauses get cut off mid-thought.

Conversation memory

Each persona keeps its own conversation. Alice, Bob, and Tom each have their own thread (data/conversation_history/<name>.json), saved after every turn and restored on the next start. What you tell one, the others cannot see: ask Tom what you discussed with Alice and he'll say he doesn't have access — and offer to ask her. Accept, and Alice answers from her own memory as a spoken interjection. Each thread's transcript is kept on disk for the dashboard (HISTORY_MAX_MESSAGES in config.py).

A fresh copy of Tom. Say "Tom, create an agent called Noah" and Noah appears: Tom's role, tools, accounts, model and voice, with an empty thread and no memory, like starting a new session. Say "Noah, …" or "switch to Noah" to switch over; Noah keeps a separate conversation from then on and survives a restart. The name must be one word that no persona already answers to. There is no delete yet.

What the model actually sees each turn is chosen by relevance, not by a window. Every user/assistant exchange is embedded into that persona's own conversations_<name> collection in Chroma as the turn ends, and — after the reply is already being spoken — a second model reads the exchange and updates the persona's facts: what is true now (a list's current contents, a decision, a preference), each fact naming the exchanges it came from. Before each reply a small model first reads your words against the last few exchanges — so "what about the other one?" becomes a question memory can answer, "this morning around 9:40" becomes a time filter, and "my list" names the thing whose facts should come along. The agent then sends only the last couple of exchanges verbatim (CONTEXT_RECENT_EXCHANGES) and retrieves a Background block for the rest: the facts about what you named, then the most relevant past exchanges and facts — found by embedding similarity and a BM25 keyword index, so tickers, names and numbers match exactly, with recent ones weighted up — plus the best-matching knowledge chunks, fitted to a character budget (CONTEXT_CONVO_CHARS, CONTEXT_KB_CHARS). So "how did that butterfly do?" finds the exchange from three weeks ago without you saying "remember when", and "what's on my list?" gets the whole list, not whichever items happened to rank. The recall tool is there for digging deeper: a follow-up the Background raised, the exact exchanges behind a fact, or a stretch of time ("what did we talk about this morning around 9:40?", "yesterday") read back in order. Conversations from before the per-persona split are kept in a shared archive on disk that no persona reads. Every turn is logged (query, context pull and remembered exchange lines in the session log, with the full candidate table when CONTEXT_DEBUG_LOG is on) so the thresholds can be tuned from what was actually retrieved; scripts/rebuild_facts.py regenerates a persona's facts from its exchanges whenever the extractor changes. scripts/seed_agent_memory.py (run once, agent off) backfills each persona's index from the session logs.

Knowledge splits the same way: the dashboard's ingest has a target selector, so a document can go into the common knowledge base (all personas) or one persona's private collection — private material never shows up in another persona's searches. Tom additionally has focus mode: "focus on my double diagonals" narrows his retrieval to that strategy until you say to clear it.

Each persona also keeps a working record of what it's tracking for you — Tom a journal of the campaigns you're running (the plan's rules, decisions and the reasons behind them), Linda her Coaches Notebook (your check-ins and the behaviour patterns she's tracking), Alice your to-do list (items stay open, and carry over to the next day, until you say they're done). Unlike the rest of memory, it isn't searched for: it's in front of the persona on every turn, so Tom knows what you're running without you naming it. They write it themselves as things happen and mention it in a few words ("noted in the journal"); "what's in your record?" reads it back, and "that's wrong, it's 40 points" corrects it.

Anything meant for every persona belongs in a note or the common knowledge base — those are shared on purpose.

You can also turn part of a conversation into a note without switching to note-taking mode: ask something ("what did we talk about trading?"), then say "save that as a note". The agent writes the note from the conversation and runs the usual folder dialogue to ask where to file it.

If you're talking to a persona that doesn't own notes (say, Tom mid-trading discussion), ask for Bob by name — "ask Bob to save this as a note" — and the save is delegated: Tom hands the task to Bob, who works in the background while your conversation continues uninterrupted. When the note is ready, Bob speaks up in his own voice at the next pause. If the request named a folder, he files it there and says so; otherwise he asks which folder. You never leave the conversation you were in. This works while muted too: muting stops the agent listening, not speaking.

Answering a folder question by typing. While Bob (or whoever is saving) waits for your folder answer, the dashboard says so in words: a box right above the Conversation page's message box reads "Bob is asking: … Type your answer here, or say it.", and a link in the sidebar points there from any page. Your next typed message is that answer, not a new message to the assistant you were talking to.

A persona contacts another one only when you name them in that message. Asked "do you remember…", it answers from its own working record and conversations; it may remember asking Bob for something, but it never goes to Bob to look it up on its own.

Where things are saved

Notes are sorted into category folders. Each finished note lives in its category folder as two files: the AI summary (<id>.md, with title/date/category frontmatter) and the raw transcript (<id>.transcript.md).

data/notes/Trading/       notes filed under "Trading"  (<id>.md + <id>.transcript.md)
data/notes/TherapyBooks/  notes filed under "Therapy book"
data/notes/General/       everything else
data/notes/_pending/      transient: live transcript while a session is recording
data/notes/index.json     ordered record of every note (title, date, category)
data/conversation_history/<name>.json  each persona's conversation
data/memory/chroma/       semantic search index (notes, knowledge, conversations,
                          plus per-persona private collections)
data/memory/working_state.db  each persona's working record (Tom's journal, Linda's notebook, Alice's list)
data/settings/            dashboard edits: personas, tunables, the active persona
data/gmail/               Gmail OAuth client and token
data/trading/             trade log, statements, per-broker tickets and tokens
knowledge/           reference PDFs/text/video you ingest + manifest.json (see below)
logs/                dated session logs of everything that happened

When a note-taking session ends, the agent suggests the best-fitting category and talks it through with you — you can just agree, name a different folder, or ask questions first ("what folders do I have?", "how many notes are in General?") before deciding. It files the note only once you commit. Queries ("what's my last note", "what did I think about X") search across all categories by default, and can be scoped to one folder by naming it ("what's the latest note in my General folder", "search my Trading notes for spreads").

The built-in categories are defined in categories.py under NOTE_CATEGORIES — each entry has a folder name and a description of what belongs there. You can also manage folders by voice in conversation mode:

  • Create — "create a folder called Recipes".
  • Rename — "rename Ideas to Brainstorms". Existing notes stay filed under it (the slug is preserved) and the folder is moved on disk.
  • Delete — "delete the Recipes folder". Notes are never lost: they move to General by default, or to a folder you name ("delete Recipes and move its notes to Ideas"). The General folder can't be deleted.
  • Move a note — "move my last note to Ideas", "put the grocery note in Recipes". The agent looks the note up, then moves its files.

Voice-created and renamed folders are persisted to data/notes/categories.json and overlaid on the defaults at startup.

Trading knowledge base (PDFs and course videos)

You can give the agent reference material to draw on — a trading book, or a recorded course — so it can answer questions from it without you pasting anything into the conversation.

Two ways in — the dashboard, or the folder plus a command:

From the dashboard (easiest): open the Knowledge page, which walks through three numbered steps in a single narrow column:

  1. Add files — press Upload files (or drag them onto the box below it).
  2. Waiting to add — everything uploaded but not yet in the knowledge base. Each file has a Remove button for undoing a wrong upload; it asks once before deleting, since there's no undo. Press Ingest to process the list.
  3. In the knowledge base — what the agent can already search.

A file that's already been ingested has no Remove button, and the server refuses to delete it even if asked directly: its chunks live in Chroma, so deleting the file alone would leave the agent citing a source that no longer exists. Ask the agent to forget it instead.

From the folder:

  1. Drop files into the knowledge/ folder (at the project root, next to run.bat). Documents: .pdf, .txt, .md. Video/audio: .mp4, .mkv, .mov, .webm, .m4a, .mp3, .wav.
  2. Just launch with run.bat. A plain launch ingests everything new — video included — before starting the agent, printing progress as it goes. Launches with nothing new cost a couple of seconds.

run.bat --ingest still does the ingest alone. Note that the pre-launch pass is skipped whenever you pass arguments, so run.bat --selftest and friends start immediately.

Each file is chunked and embedded once into a persistent knowledge collection in data/memory/chroma. Ingestion is idempotent: files are identified by content hash and recorded in knowledge/manifest.json, so re-scanning an unchanged folder is near-instant and never re-embeds.

Video and audio are transcribed on the way in by the same local Whisper model the agent uses for dictation (decoding is handled by PyAV, bundled with faster-whisper — no ffmpeg install needed). Each chunk keeps the timestamp it was spoken at, so a citation points you at the moment to rewatch.

Transcription is slow — on this machine small.en runs about 3.3× faster than real time, so budget roughly 20 minutes per hour of recording (more with a bigger KB_MEDIA_MODEL). You pay it exactly once per file, and progress is printed every few seconds so a long run never looks like a hang.

That is far too long to hold the agent's own startup open, so the agent process never transcribes: its boot scan takes documents only and just names any video still waiting. run.bat does the media pass first, as a separate --ingest process, then launches the agent. Same end result — everything searchable by the time the agent is listening — but the wait is visible and attributable rather than a half-started agent that looks wedged.

An ingest and a running agent are mutually exclusive. Embedding writes the same Chroma index the agent has open, and two writers corrupt it — so the dashboard's ingest takes the agent's single-instance lock. If the agent is running, close it, ingest, then start it again. (The same lock is why --ingest refuses to run beside a live agent.) While an ingest is in flight, leave the dashboard running; closing it cancels the job. Cancelling is safe — the manifest is written per file and chunk ids derive from the file hash, so a re-run overwrites rather than duplicates.

Two knobs in config.py: KB_MEDIA_MODEL (default small.en; medium.en is noticeably better on jargon and, as a one-time cost, usually worth it) and KB_MEDIA_EXTS. KB_MEDIA_MODEL is also settable from the dashboard.

After that, ask trading questions in conversation ("what does my course say about iron condors?"). The agent retrieves from it every turn, digs deeper with the recall tool on demand, and cites the source — a page for books, a timestamp like 14:32 for video. run.bat --kb-list shows what's been ingested, with page count or running time. The content stays local and, like the rest of data/, is gitignored.

One limitation worth knowing: only the spoken audio is captured. Whatever is drawn on a chart or slide is lost, so an instructor saying "as you can see here" ingests as exactly that.

Email (Gmail)

Alice and Tom can search your Gmail, read threads, save drafts, and — only after you've heard the recipient/subject read back and explicitly said to — send. Setup is one-time, per Google account:

  1. In the Google Cloud console (Google Auth Platform → Clients), create a Desktop OAuth client with the gmail.readonly and gmail.compose scopes, download its JSON, and save it as data/gmail/client_secret.json.
  2. Start the agent: with no token it opens the browser consent flow before the voice loop (or run it standalone: python -m lib.gmail_auth). It writes data/gmail/token.json, which then refreshes itself silently forever (and syncs to your other machines with the rest of data/).

The agent never waits for this. It starts straight away and does the sign-in in the background, so a missing or expired token costs nothing at startup. Approve the page whenever you like and Gmail connects itself to the running agent, mid-conversation, with no restart. Until it does, the email tools answer "Google account not authenticated, finish the authentication process, or restart the app and log in" rather than hanging the conversation.

Real trading by voice (Charles Schwab and tastytrade)

The agent can build and place real multi-leg option orders on two brokers — Tom trades the Charles Schwab (thinkorswim) account, Bonny the tastytrade one — with live quotes, a mandatory spoken review before any submit, order cancel, positions, and realized/unrealized P&L. Each persona reaches only its own broker: ask Tom for something on tastytrade and he sends you to Bonny. Linda reviews both accounts (positions, orders, P&L; never a ticket) and narrows to one when asked; the two accounts are stored and calculated separately and only ever added up in a sentence. The dashboard's Trading page works on one broker at a time, chosen at the top (same ticket as the voice, review/submit/cancel, P&L reports).

The display window shows what the latest trading answer was about — positions, orders, P&L, the trades behind a position, the ticket being built or reviewed — in large white-on-black type, so you needn't hold every leg in your head while listening. Every card has the same shape: a heading ("Open positions — Tom, Charles Schwab"), an "as of" time, the one figure that matters in a big box at the top (total unrealized, realized P&L, the limit price), then one row per position, leg, order or trade — a word badge (LONG, SHORT, WORKING, FILLED), a readable contract ("SPX 7750 Put · Fri Nov 6"), its value on the right, and small detail lines. The figures are the ones the agent answered from. Only profit and loss is coloured, and it keeps its "+" or "-"; cash, prices and quantities stay white. After two minutes the card says in words that the numbers aren't current. The window sits at the top-right, as tall as the screen allows. The agent opens it at start (it needs pip install pywebview; DISPLAY_WINDOW = False turns it off). It comes to the front only when a new answer arrives, never takes the keyboard, and if you close it the next trading answer reopens it. When the agent stops — quit, crash, or closed console — the window closes with it.

Charles Schwab (Tom)

Register an app on the Schwab Developer Portal (Trader API – Individual, with both Accounts and Trading and Market Data; callback URL https://127.0.0.1), wait for approval, then add to .env:

SCHWAB_CLIENT_ID=...        # the app's App Key
SCHWAB_CLIENT_SECRET=...    # its Secret
SCHWAB_CALLBACK_URL=https://127.0.0.1   # optional; must match the app
SCHWAB_ACCOUNT_ID=...       # optional: which linked account, when several

Then sign in from the dashboard's Trading page (Schwab → Sign in to Schwab): Schwab's login opens, you land on an error page at the callback address, and you paste that whole address back into the page. The sign-in lasts 7 days (Schwab's rule, not ours): the status warns a day ahead, and after it lapses every Schwab tool says to sign in again. Schwab has no sandbox — every token is a live one — and its API cannot trade futures, so /ES and /MES stay with Bonny. Quotes come from REST snapshots rather than a stream (Schwab allows one stream per user, which the agent and the dashboard would fight over), and option quotes carry the greeks directly.

Several accounts. If your login shares more than one account, name them once ("Tom, call 699 my IRA") and pick one ("move to my IRA"). Tom keeps using it — across sessions — until you switch, and every answer says which account it read ("IRA (…699): net liquidating value …"). Ask about another one ("how's red doing?") or all of them ("positions across all my accounts") without changing the choice; the Trading page's account buttons are the same choice. "Check my balance" gives net liquidating value, cash and buying power. Bonny works the same way on tastytrade.

tastytrade (Bonny)

Add your tastytrade OAuth credentials to .env:

TASTY_CLIENT_SECRET=...   # my.tastytrade.com -> My Profile -> API -> OAuth app
TASTY_REFRESH_TOKEN=...   # "Create Grant" on the same page (never expires)
TASTY_ACCOUNT_ID=...
TASTY_ENV=sandbox         # orders go to the cert sandbox until you set: live

If you already run the Tasty-Web project, point at its .env instead of copying keys: TASTY_ENV_FILE=C:\Home\Proj\Tasty-Web\.env (its CLIENT_SECRET/REFRESH_TOKEN/TASTYWORKS_ACCOUNT_ID names are accepted). The safety gate is TASTY_ENV: anything other than exactly live uses the sandbox, so real money always requires that explicit opt-in.

Then, in conversation with either persona: "set up an iron condor on SPX" → "move the put side down ten" → "price it at mid" → "review it" (the agent speaks cost, buying-power effect, fees, warnings) → "submit it". Submission only works after a review of that exact ticket and your explicit go-ahead; any edit invalidates the review. "Cancel the order", "what are my positions?", and "how much did I make this week on SPX?" work as expected. Design and API research: TRADING_PLAN.md, TRADING_RESEARCH.md.

Trade log for thinkorswim (Tom)

Tom keeps a log of your thinkorswim/Schwab fills in data/trading/trade_log.db (SQLite), and prices your positions from it.

  • Straight from Schwab. Ask "Tom, update your trade log." He pulls every fill (with exact fees) and expiration since the log's last update from the Schwab API, for every account you shared with the app — the same rows a statement would give, so no CSV export is needed. Repeating it is harmless. Needs the Schwab sign-in (see "Charles Schwab (Tom)" above).
  • Fill alerts. Ask "Tom, log my latest thinkorswim fill." He reads the alert email and logs it. The cash is exact, but fees stay pending until the next sync or statement.
  • Statements — now only for history older than a year, or an account Schwab doesn't share with the app. Export the account statement CSV from thinkorswim and drop it on the Statement for Tom box on the dashboard's Trading page. The import result shows right under the box, and if the agent is running Tom says it out loud and keeps the result in his thread, so you can ask him about it. A statement is the authoritative record: it adds exact fees and expirations, and it replaces the alert-logged fills of the span it covers. Nothing outside that span is touched, and importing the same statement twice is harmless. It is refused if its rows don't add up to its own TOTAL row. Imported files are kept in data/trading/statements/. By voice, "Tom, import my latest statement" imports the most recently saved .csv there, whatever its name (or in TRADE_STATEMENTS_DIR, if you set one in .env). "Import IRA.csv" imports that file from the same folder.
  • Positions. A position is the minimal set of trades whose legs net to zero. Ask "what was my P&L on calendars in August?" or "list my open positions on the IRA." Every figure is summed from the log. If you have several accounts and don't name one, Tom asks which. Linda can read the log too, for post-trade reviews.
  • Export. "Tom, export my IRA trades from October" saves the trades the same question would read out, every one of them, as a spreadsheet (.csv) in data/exports/, named trades_<account>_<date>_<time>.csv. Tom says how many trades went in and the file name. Any other table Tom shows exports the same way — "export my position table", "save my open positions as a spreadsheet", orders, P&L: the file holds exactly what the display window showed, totals and notes included. Exports have their own folder so one can never be taken for "my latest statement".
  • Campaigns. When Tom logs a fill that looks like part of a campaign in his journal, he says so and asks; say yes and the fill is linked, rolls included. "How is the diagonal campaign doing?" then sums exactly that campaign's trades — and the links survive a statement import.

Switching the model by voice

Conversation defaults to Haiku 4.5 for low latency. Ask for a different model mid-conversation and it switches from that reply onward:

  • "switch to Opus" / "use the smartest model" → Opus 5.5 (most capable, slowest)
  • "use Sonnet" → Sonnet 5.5 (stronger reasoning, a little slower)
  • "go back to the fast one" → Haiku 4.5
  • "use the cheap one" / "switch to DeepSeek" → DeepSeek V4 Flash (external, by far the cheapest)
  • "use DeepSeek pro" → DeepSeek V4 Pro (DeepSeek's strongest)

The DeepSeek options talk to DeepSeek's Anthropic-compatible API through the same client code; they require DEEPSEEK_API_KEY in .env (see .env.example) and are refused out loud when it's missing. Your conversation history goes to DeepSeek's servers while one is active.

The choice lasts for the session and resets to the fast default on restart (so you never get silently left on an expensive model). Note summaries always use SUMMARY_MODEL regardless. The models are defined in config.py under CONVO_MODELS.

When a provider fails, the agent moves over and tells you. If DeepSeek or Anthropic runs out of credit, or stops responding to the small background calls that power memory, that work moves to the other provider and you hear one sentence about it — "DeepSeek isn't responding, so memory lookups switched to Anthropic." Every two minutes the agent quietly tries the original again, and tells you when it's back. Your model choice itself never changes.

Each persona remembers its own model. Leave Tom on DeepSeek V4 Pro, switch to Bob, and Bob answers on his own model; switch back and Tom is still on DeepSeek. The switch announcement says which, since the model is the one thing you can't hear:

"Bob here, running on Haiku 4.5." "Tom here, running on DeepSeek V4 Pro."

Asking "what model are you on?" reads the real setting. A conversation history fills up with model talk that no longer applies — a choice from an hour ago, an old switch. Left to answer from that, the agent guesses, and it has guessed wrong (claiming Opus while on Haiku, and DeepSeek while on Opus). Every persona now has a get_current_model tool and a hard rule to call it before saying anything about models, so the answer is a live read rather than a recollection — and it shows up as a tool_use line in logs/ if you want to check.

Asking the agent about itself

The agent can answer questions about its own design — "how does barge-in work?", "where are my notes stored?", "what tools do you have?", "how do I switch models?". It reads PROJECT.md (via the describe_project tool) and answers from it, so its self-knowledge stays in sync with the documentation.

Project layout

voice_agent.py   entry point + Agent orchestration (main loop, modes, say/barge-in)
config.py        shared constants (paths, audio params, models, system prompt)
speech/          audio (mic/VAD), stt, tts, sound (thinking cue), barge_in
media_control/   headset button: gestures (click decode) + per-OS channels (main.py wires them)
brain/           llm (Claude loop), agents (personas), history, memory
stores/          notes, knowledge, categories, discord_data, chroma_store
lib/             atomic_io, single_instance, frontmatter — leaf utilities
web/             server.py (the dashboard, embedded in the agent) + static/
display/         the display window for trading answers (its own process; per-OS raise)
docs/            PROJECT.md, MEDIA_CONTROL.md, TODO.md, TRADING_*.md
tools/           tool registry — one file per domain (notes, discord, model, ...)
tests/           unittest suite over the pure logic (no hardware needed)
scripts/         manual hardware probes used while developing button handling

Adding a capability is one decorated function under tools/ — see PROJECT.md §5. Run the tests with:

python -m unittest discover tests

Tuning

All settings live in config.py:

  • WHISPER_MODEL — base.en (faster) ↔ small.en (default) ↔ medium.en (more accurate).
  • CONVO_MODELS / CONVO_MODEL / SUMMARY_MODEL — Claude models. Conversation defaults to Haiku for low latency and can be switched by voice (see above); summaries use Sonnet for quality.
  • CONVO_ENDPOINT_MS / NOTE_ENDPOINT_MS — how much trailing silence ends an utterance.
  • VAD_AGGRESSIVENESS — 0–3; raise it if background noise is being picked up as speech.
  • TTS_RATE / TTS_VOICE — speech speed and voice selection.

About

Voice-first personal assistant for people with low vision (tunnel vision): notes, email, daily-life help and options trading by voice, with a large-text, big-button dashboard.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages