Give your AI coding agent the exact slice of the repo it needs — not the whole file.
Local code index · Tree-sitter symbol graph · Surgical reads · Deterministic output filters · Native agent hooks
-dea584?style=flat-square&logo=rust){kind=icon}
⚡ Quick start · ✨ Highlights · 🧭 Guide · 🔌 Agent guides · 🖥️ Dashboard · 📖 Commands · ❓ Troubleshooting
⚡ tokenix is a high-performance Rust CLI that sits directly between your AI coding agent and your repository. It indexes your codebase locally, and it never changes what the agent reads. It trims verbose command output (a 10,000-line test run keeps its errors) and offers a symbol graph and outlines the agent can call when it wants them.
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Native Read(src/hook.rs) → 1,518 lines → 13,498 tokens (always allowed, untouched) │
│ tokenix read src/hook.rs → outline → 2,395 tokens (opt-in; omits usages/tests) │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
| ⚡ Local Code Index | 🌲 Tree-sitter Symbol Graph | ✂️ Deterministic Output Filters |
|---|---|---|
A SQLite index of every file, chunked by symbol. It is not a search engine: it only feeds the symbol graph, outlines and pack (who calls what, where a symbol lives). No model, no background process, zero external services. |
Deep AST symbol parsing with PageRank importance, bidirectional caller/callee tracing, and diff blast-radius impact analysis. | 534 bundled filters stripping noisy build/test logs without masking real errors or exit codes. |
| 🛡️ Zero-Trust Security | 📉 Measured Token Savings | 🔌 Universal Agent Compatibility |
|---|---|---|
| In-flight credential masking on hooks and failure tees. Repo-controlled filters are isolated behind SHA-256 trust gates. | 40% to 90% real payload reduction before prompt-cache writes. 100% fail-open contract (always exit 0 on errors). | Native hook support for Claude Code, OpenAI Codex, GitHub Copilot, Antigravity, OpenCode, and any MCP client. |
Note
Four simple steps, about three minutes. Nothing is downloaded besides the binary.
# macOS / Linux: pick the asset for your platform (table below)
curl -fsSL https://github.com/juninmd/tokenix/releases/latest/download/tokenix-linux-x86_64 -o tokenix
chmod +x tokenix && sudo mv tokenix /usr/local/bin/tokenix# Windows
irm https://github.com/juninmd/tokenix/releases/latest/download/tokenix-windows-x86_64.exe -OutFile tokenix.exe
.\tokenix.exe install-binary # copies itself to a per-user bin dir on PATHcd your-project
tokenix index .tokenix install-hook --tool claude-code # or codex · copilot · antigravity · opencode · allEach agent is wired a little differently. The agent guides cover what gets installed, what gets intercepted and how to undo it.
tokenix doctor # binary, filters, recordings
tokenix # dashboard → Stats tab: which agents are wired
tokenix gain # after a session: hook calls, intercepts, tokens removedThen work with your agent as usual. You do not need to change your prompts or habits.
All install options (platform assets, crates.io, source, verification)
Every release ships one binary per platform. The version-less
releases/latest/download/<asset> URL always resolves to the newest release.
| Platform | Asset |
|---|---|
| Linux x86_64 | tokenix-linux-x86_64 |
| Linux arm64 (aarch64) | tokenix-linux-aarch64 |
| macOS (Apple Silicon / aarch64) | tokenix-macos-aarch64 |
| Windows x86_64 | tokenix-windows-x86_64.exe |
(Intel macOS x86_64 users can install from source using cargo install tokenix --locked)
Each release also publishes sha256sums.txt and SLSA build-provenance attestations.
SECURITY.md explains how to verify a download.
cargo install tokenix --locked # from crates.io: the lean core
cargo install tokenix --locked --features full # everything the release binaries have
git clone https://github.com/juninmd/tokenix && cd tokenix
cargo install --path . --locked --features full # from sourceUse --locked. It builds against the committed Cargo.lock; without it, cargo install re-resolves dependencies. Building from source needs a recent stable
Rust toolchain (MSRV 1.90).
Lean core or full build. The default build is what an agent needs: the hook,
the MCP server, the index, the symbol graph (symbols, callers, callees,
impact, flow, blast, modules, deps, graph, tokenmap, pack), the
output filters, scan-secrets and install-hook --tool claude-code. Release
binaries are built with --features full. Cargo features add the rest:
| Feature | Adds |
|---|---|
tui |
The terminal dashboard (bare tokenix). Implies forensics and authoring |
forensics |
usage, discover, conversation-audit, prompt-audit, session-audit, egress-audit, benchmark |
authoring |
filter generate and filter record |
agents-extra |
install-hook / remove-hook for Copilot, Codex, OpenCode, Antigravity and mcp |
full |
All of the above |
[!TIP] For AI agents installing tokenix: Use the prebuilt binary, not
cargo install. It needs no toolchain and no compile step. Fetch the version-less URL and never hard-code a version. Detect the platform, download the matching asset, mark it executable, then runtokenix doctor.
After install-hook, tokenix sees each tool call before it runs and decides:
| The agent tries to… | tokenix does | Result |
|---|---|---|
| Read any file | Nothing: Read is never intercepted |
The agent gets the file. A measured A/B showed outlines hiding usages during renames |
| Grep, any pattern | Nothing (only the head_limit cap below) |
The agent's own grep runs as asked |
| Grep with unbounded content output | Adds head_limit (100 by default) |
Output is capped |
Run a noisy command (cargo test, terraform plan, git status …) |
Reruns it through tokenix run and the matching filter |
Failures and summaries stay, noise goes, and the exit code is the real one |
Anything else passes through untouched. Nothing is lost for good: see getting the full output back.
The same commands the agent uses work in your terminal. Here they run against a two-file demo project:
$ tokenix symbols apply_tax
## Symbols matching `apply_tax`
- src/billing.rs:5-7 [function] apply_tax
$ tokenix callers apply_tax
## Callers of `apply_tax`
- src/billing.rs:1 [function] invoice_total -> src/billing.rs:5 [function] apply_tax via `apply_tax` (references)
$ tokenix read src/billing.rs --symbol apply_tax
# L5-7 [function] apply_tax
fn apply_tax(amount: u32) -> u32 {
amount + amount / 10
}| You want to… | Run |
|---|---|
| Find a symbol by name | tokenix symbols apply_tax |
| See a big file's structure | tokenix read src/billing.rs |
| Read exactly one function | tokenix read src/billing.rs --symbol apply_tax |
| Find who calls something, and what it calls | tokenix callers apply_tax · tokenix callees main |
| See everything a change could break | tokenix impact apply_tax · tokenix flow main |
| Hand a budgeted bundle to a tool without hooks | tokenix pack --budget 8000 > context.md |
blast maps your diff to the symbols it touches and walks the call graph up from
them:
$ tokenix blast # vs HEAD; use --since origin/main for a branch
# Blast radius vs HEAD — 1 changed file(s), 1 changed symbol(s)
## Changed symbols
- apply_tax (function) src/billing.rs:5
## Impacted (up to 2 hop(s) of callers)
- [1] invoice_total (↑1) src/billing.rs:1
- [2] main (↑0) src/main.rs:11- Edits are picked up on their own. Before answering, every index command
re-chunks the files you changed since the last index. That takes milliseconds
so a function you just wrote shows up in
symbols/callersright away. Runtokenix indexnow and then (ortokenix index --if-stalein a script) for larger changes. - Branch switches are detected. The index records the branch and
HEAD. When they change, the hook passes calls through until you re-index, so it never answers from the wrong branch. - Draw it. Every graph command takes
--format mermaid|dot|json(andhtmlfor call graphs), so an answer can go straight into a PR description, a doc or a Graphviz render:tokenix blast --format mermaid,tokenix modules --format mermaid,tokenix impact SYMBOL --format html -o impact.html.
- A clipped successful run ends with
[tokenix: N bytes not shown — tokenix retrieve <key> …].tokenix retrieve <key>prints the exact original bytes. - A failed run tees its full output, with credentials masked, to
~/.tokenix/tee/, and prints the path. tokenix run --raw "cmd"(orTOKENIX_RAW=1) passes output through byte for byte. Use it when a script parses the output.TOKENIX_DISABLED=1 <cmd>bypasses the hook for one command.
Put a .tokenix.toml in the repository root. Unknown keys are reported, not
silently ignored.
[index]
exclude = ["fixtures", "vendor"] # extra directories to skip
extensions = ["proto"] # extra extensions to index
data_files = false # index .json/.yaml/.yml too
redact_secrets = true # mask secret-shaped strings in stored chunks
max_file_bytes = 1500000 # files above this are skipped wholeA filter is a TOML rule, never a model. Start from your real output:
tokenix discover # which commands in your history cost the most
tokenix filter record "npm test" # capture real output (redacted, git-ignored)
tokenix filter generate npm # draft a filter with an AI CLI you have installed
tokenix filter verify # run its golden tests through the real pipelineUser filters go in ~/.tokenix/filters/. Filters committed to a repo in
.tokenix/filters/ apply only after tokenix trust, and editing one revokes
trust until you approve it again. Output filters covers the
format.
| Agent | Install | Integration | Guide |
|---|---|---|---|
| Claude Code | tokenix install-hook --tool claude-code |
PreToolUse hook in settings.json |
docs/agents/claude-code.md |
| OpenAI Codex CLI | tokenix install-hook --tool codex |
PreToolUse hook for commands + tx-read/tx-symbols helpers |
docs/agents/codex.md |
| GitHub Copilot | tokenix install-hook --tool copilot |
PreToolUse + PostToolUse hooks; --local commits them to .github/ |
docs/agents/copilot.md |
| Antigravity | tokenix install-hook --tool antigravity |
Native plugin | docs/agents/antigravity.md |
| OpenCode | tokenix install-hook --tool opencode |
MCP server only | docs/agents/opencode.md |
| Any MCP client | tokenix mcp |
MCP server with six structure tools: tokenix_symbols, tokenix_graph (callers · callees · impact · flow), tokenix_blast, tokenix_modules, tokenix_deps, tokenix_tokenmap. They return locations and diagrams, never code, and never search contents |
tokenix install-hook --tool all wires every agent except OpenCode. OpenCode's
registration lives in the repository, so you have to ask for it explicitly.
tokenix remove-hook --tool <agent> undoes the install.
Run bare tokenix (a build with the tui feature, which the release binaries have) for a terminal dashboard with sixteen tabs and no flags,
grouped in four sections: 1·Overview, 2·Code, 3·Savings, 4·Health.
1–4 jump to a section, ←/→ walk the tabs, ↑/↓ move, ? lists the keys
of the current tab, y copies a report tab's text to the clipboard, q quits.
On a terminal, a report command opens its own tab: tokenix doctor lands on
Doctor, tokenix scan-secrets on Secrets. Piping, --json, --statusline,
--format, --output and --no-tui (or TOKENIX_NO_TUI=1) keep plain text
output. Agent-facing commands (hook, run, mcp, read, pack) never
open a UI.
| Tab | What it shows |
|---|---|
| Stats | Version, a “next steps” list (not indexed, stale index, agents without a hook, tokenix off PATH), per-agent hook status, index summary, one-key actions: index repo · install hooks · install binary on PATH |
| Symbols | Browse the symbol graph: symbols ranked by PageRank, / filters by name or path, t cycles the kind. The right side shows callers, callees and the source; Tab + Enter jump into a relation, ⌫ walks back, c copies path:line, y copies the source, m / d copy the impact graph as Mermaid / DOT, h opens it as an HTML page |
| Modules | Functional modules discovered from the symbol graph (tokenix modules) |
| Blast | Symbols that depend on your uncommitted diff (tokenix blast). s picks the base (HEAD · HEAD~1 · origin/main), d the caller hops |
| Pack | The budgeted repo map (tokenix pack), ready to paste: b budget · p profile · c changed files only · y copy |
| Gain | Tokens removed with a reduction bar, split by source and by command. c adds a ≈USD table at list input rates · a all projects · r refresh |
| Usage | Absolute token spend and ≈USD cost from agent transcripts. s cycles daily · model · 5-hour blocks · project · session |
| Filters | All 534 bundled filters by tool, with a live input → output preview and a per-filter token gauge |
| Studio | Record → preview → generate filters. Ranks the biggest unfiltered token sinks first (⚠), marks filtered commands (✓) and recordings (●) |
| Secrets | Credentials found in agent transcripts, grouped by rule, attributed to repo + branch. v reveal · c copy · x redact |
| Egress | External DNS/IP destinations in transcripts, checked against local reputation lists |
| Graph | Repo-wide symbol graph: god nodes, bottlenecks, blast-radius leaders |
| Tokenmap | The repository as a tree weighted by token count, heaviest paths first |
| Discover | Replays the current filters over past agent output: savings you could have had, plus commands no filter covers |
| Audit | MCP/tool weight of each agent's effective system prompt, plus always-on instruction files and skills |
| Doctor | Bundled-filter inventory, user/local filter config issues, active recording session |
 |
 |
 |
 |
 |
 |
Every number below counts tokens removed from a payload before it reached the model. The measurements are reproducible. They are not a claim about your bill.
| What | Baseline → tokenix | Tokens removed | Reproduce |
|---|---|---|---|
| Real sessions (7,807 hook calls, one developer's machine) | 475,360 → 169,175 | 67.4% | tokenix gain |
tokenix read outline vs the whole file, 31 real files (opt-in, not applied by the hook) |
346,892 → 58,154 | 83.2% | tokenix benchmark |
| Outline + targeted symbol workflow | 55,020 → 17,384 | 68.4% | tokenix benchmark |
| Command filters, verbose output | 1,891 → 369 | 80.5% | cargo test verbose_real_output -- --nocapture |
| Command filters, full golden corpus (1,167 cases) | 46,804 → 27,728 | 40.8% | cargo test filters_deliver_aggregate_token_savings -- --nocapture |
Baselines assume the agent would have read the whole file; there is no holdout group, so these are token measurements, not a causal effect. The benchmark rows use a small synthetic set from this repo and benchmark/samples, not a foreign codebase.
Golden filter cases reproduce their expected output byte for byte (1,167/1,167).
Why there is no dollar figure, and how to read these numbers
Published work measured provider-billed cost for hook-based compressors on Claude Code. It found token reduction and cost reduction close to uncorrelated (arXiv:2607.12161, 2,848 paired runs). One arm removed 38.4% of tool-output tokens and billed 6.8% more, with a per-task correlation of r = 0.15. Prompt-cache traffic dominates a real bill, so tokens removed from a fresh payload are mostly not the tokens you pay for. An independent 425-trial study of a competing tool measured a +7.6% cost increase while that tool's analytics reported 96 million tokens saved.
We have not yet measured tokenix against provider-billed cost. Until then this
README reports tokens and calls them tokens. Cache-aware cost accounting is the top
roadmap item. docs/research/2026-08-token-economy.md
has the full evidence review.
- The 40.8% corpus figure is pessimistic on purpose. Half the golden corpus is failure-path cases a filter must pass through unfiltered, so errors are never masked as success.
tokenix benchmark's command arm reports about 26% because its sample commands print only 27–148 tokens each, which leaves little to compress.- Runs that saved nothing are in the denominator. The 67.4% row was measured before that rule existed, so it reads high. It will be re-measured before the next release.
- Your mileage depends on your codebase and your agent.
tokenix gainshows what happened on your machine.
Tip
Run tokenix --help for the full catalog with usage examples; tokenix <command> --help displays all flags and parameters for any specific command.
| Command | Description |
|---|---|
tokenix read FILE |
Smart reader: outline for large files, full text for small ones (--symbol, --lines, --mode full|outline|signatures|diff|density:X) |
tokenix symbols QUERY |
Find indexed symbols by name or path (--kind filters by symbol type) |
tokenix callers SYMBOL |
Symbols that call or reference a symbol (--format text|json|mermaid|dot|html) |
tokenix callees SYMBOL |
Symbols a symbol calls or references |
tokenix deps FILE |
File-level import dependencies (--reverse, --transitive, --format text|json|mermaid|dot) |
tokenix impact SYMBOL |
Bidirectional impact graph (--format text|json|mermaid|dot|html) |
tokenix flow SYMBOL |
Forward call-flow trace (--depth, --format text|json|mermaid|dot|html) |
tokenix graph |
Repo-wide symbol-graph overview: god nodes, bottlenecks, blast-radius leaders, modules (--format text|json|dot|mermaid, --top N; includes circular dependencies) |
tokenix modules |
Functional modules found by community detection over the symbol graph (--top, --format text|json|mermaid|dot) |
tokenix blast |
Blast radius of the current diff: changed symbols and everything that calls them (--since REF, --depth, --json) |
tokenix pack |
Budgeted repo pack for tools without hooks (--mode/--profile, --changed, --token-map) |
tokenix retrieve KEY |
Print the exact original output behind a [tokenix: ...] marker |
| Command | Description |
|---|---|
tokenix (no args) |
Open the dashboard; piped or non-TTY falls back to help |
tokenix index [PATH] |
Index the repo at PATH (default .) |
tokenix install-hook / remove-hook |
Install or remove agent hooks and instructions (default --tool all) |
tokenix install-binary |
Copy the running executable to a per-user bin dir and put it on PATH |
tokenix update |
Check for, install, or toggle automatic updates from GitHub releases (--check, --auto, --enable-auto, --disable-auto) |
tokenix doctor |
Diagnose the install: filter inventory, filter config and recording state |
tokenix gain |
Tokens removed, split by source (command filters vs Grep cap), plus session shape. Dollar estimates only with --cost-estimate / --economics |
tokenix discover |
Replay current filters over past agent output: recoverable savings plus uncovered commands (--agent, --top, --json) |
tokenix trust / untrust |
Approve (SHA-256 pinned) or revoke this repo's executable inputs (--status) |
tokenix usage |
Absolute token spend and ≈USD from agent transcripts (daily|weekly|monthly|session|model|project|blocks, --all-projects, --statusline, --json) |
tokenix stats |
Index statistics (files, chunks, tokens, age) |
tokenix tokenmap |
Directory tree weighted by token count, heaviest paths first (--format html) |
tokenix benchmark |
Reproducible token-reduction benchmark, vanilla vs tokenix (--json) |
tokenix filter list|active|generate|record|verify |
Browse, generate, record and golden-test output filters |
| tokenix prompt-audit | Audit MCP/tool token weight across agents (--agent, --recommend, --profile-impact, --json) |
| tokenix session-audit | Health check: index, hook events, MCP/tool weight, cache hygiene |
| tokenix conversation-audit | Scan local conversation histories for token-waste patterns (--generate prints ready-to-run filter commands) |
| tokenix scan-secrets | Scan agent transcripts for exposed credentials, attributed to repo + branch (--group, --reveal, --json) |
| tokenix egress-audit | Scan agent transcripts for external DNS/IP destinations, checked against local reputation lists |
| tokenix artifacts list\|show | Context artifacts from .tokenix/artifacts.json |
| tokenix generate-ignores | Write .gitignore entries for tokenix artifacts |
When you open tokenix in a terminal, it checks GitHub releases in the background
on a 24-hour cache and automatically installs a newer release by default.
The current command keeps running; the new version is used on the next launch.
Downloads use the release's SHA-256 checksums; a GitHub token, when set, is sent
only to the releases API, never to asset downloads.
Checks are skipped in CI, piped commands, hooks, and agent-facing commands. Set
TOKENIX_AUTO_UPDATE=notify to receive update hints without installing, or run
tokenix update --disable-auto to turn off automatic installation.
| Command | Description |
|---|---|
tokenix hook |
PreToolUse handler: caps unbounded Grep output and filters noisy commands (never touches Read) |
tokenix hook-post |
PostToolUse handler: redacts secrets and compresses tool output (agents whose install wires it, see the agent guides) |
tokenix run "CMD" |
Run a command and compress its output (--shell, --path/-p, --raw) |
tokenix mcp |
MCP server with six structure tools: tokenix_symbols, tokenix_graph (callers · callees · impact · flow), tokenix_blast, tokenix_modules, tokenix_deps, tokenix_tokenmap. They return locations and diagrams, never code, and never search contents |
tokenix mcp-proxy -- CMD |
Wrap another stdio MCP server and compress only its tools/call text results |
Flags and environment variables
Global: --no-tui (or TOKENIX_NO_TUI=1)
prints plain text instead of opening the dashboard.
Environment:
| Variable | Effect |
|---|---|
TOKENIX_HOME |
Absolute path that replaces ~/.tokenix (indexes, hook log, trust store, recall stash). Relative values are ignored. Useful for CI and throwaway runs |
TOKENIX_DISABLED=1 |
Bypass the hook for one command |
TOKENIX_RAW=1 |
Same as tokenix run --raw |
TOKENIX_MAX_OUTPUT_TOKENS |
Global output ceiling (default 8000, 0 disables) |
TOKENIX_BRANCH_AWARE=true |
One SQLite DB per git branch |
TOKENIX_GREP_HEAD_LIMIT |
head_limit injected into unbounded Grep (default 100, 0 disables) |
TOKENIX_AUTO_REFRESH=0 |
Turn off the inline refresh before index commands |
TOKENIX_AUTO_REFRESH_MAX |
Dirty-file count above which the inline refresh is skipped (default 25) |
TOKENIX_DEDUP=0 · TOKENIX_DEDUP_MIN_TOKENS · TOKENIX_DEDUP_TTL |
Cross-call dedup of identical output (defaults: on, 200 tokens, 3600 s; scoped to the project) |
TOKENIX_READ_DEDUP=0 · TOKENIX_READ_DEDUP_TTL · TOKENIX_READ_DEDUP_MIN_TOKENS |
Re-read suppression (defaults: on, 900 s, 1500 tokens; scoped to the agent session) |
TOKENIX_TEE=0 |
Do not tee failed-command output to ~/.tokenix/tee/ |
TOKENIX_AUTO_UPDATE=1|0|notify |
Control auto-update behavior (1/install auto-updates in background; 0 disables checks) |
TOKENIX_NO_UPDATE=1 |
Disable all update checks and auto-updates |
tokenix index: --force/-f, --cpu-profile <low|default|max>,
--jobs N, --if-stale, --path/-p, --no-low-priority (indexing runs at below-normal priority by
default).
tokenix symbols: --limit/-l (20), --kind/-k, --json, --path/-p
tokenix deps: --reverse, --transitive, --json, --path/-p
Graph output: callers, callees, impact, flow, deps, graph, blast
and modules share --format and --output/-o (write to a file). text is the
default, --json is the old shorthand for --format json, and a format a command
cannot draw is an error listing the ones it can. mermaid prints a fenced block
ready for Markdown; dot is Graphviz; html is a standalone page for call
graphs. modules --format mermaid draws the modules with the number of calls
between them, and blast --format mermaid draws the hop layers of a diff.
tokenix impact: --depth/-d (2), --limit/-l (50), --path/-p
tokenix flow: --depth/-d (3), --limit/-l (50)
tokenix blast: --since REF (default HEAD, e.g. origin/main),
--depth/-d (2), --limit/-l (50), --path/-p
tokenix modules: --top (12), --path/-p
tokenix pack: --mode/--profile <plan|debug|audit|security|review>,
--budget N (8000), --format <markdown|xml|json>, --changed, --since REF,
--token-map, --output/-o
tokenix install-hook / remove-hook:
--tool <claude-code|copilot|codex|mcp|opencode|antigravity|all> (default all),
--local
tokenix scan-secrets: --agent <claude|gemini|copilot|antigravity|all>,
--filter <substr>, --group <none|value|rule|agent|file|repo>, --reveal,
--json. Output is redacted by default, and the exit code is 1 when findings
exist. Rules are TOML [[rules]] (id, pattern, optional capture /
min_entropy). Bundled defaults live in assets/secret-rules/.
~/.tokenix/secret-rules/*.toml may add or override rules. A repo's
<repo>/.tokenix/secret-rules/*.toml loads only after tokenix trust and may only
add: an id that collides with a bundled rule is dropped with a warning.
tokenix egress-audit: --agent, --filter, --group <host|rule|agent|file>,
--safe, --json. Local reputation lists live in ~/.tokenix/safe-hosts.toml and
~/.tokenix/dangerous-hosts.toml.
tokenix conversation-audit: --agent, --min-chars N (5000), --limit N
(30), --json, --generate
tokenix prompt-audit: --agent, --json, --recommend, --profile-impact
tokenix benchmark: --json, --refresh-index
534 bundled filters, 1,167 golden cases. A filter matches a command and shapes its output:
[filters.cargo-test]
match_command = '^cargo test'
strip_lines_matching = ['^\s*Compiling ', '^\s*Finished ']
priority_lines = ['(?i)^error', '^test result:']
tail_lines = 40
on_empty = "cargo test: all tests passed"
passthrough_when_emptied = true
[[tests.cargo-test]]
name = "all pass"
input = """..."""
expected = """cargo test: all tests passed"""Filters resolve in this order: <repo>/.tokenix/filters (only after tokenix trust) → ~/.tokenix/filters → bundled. Every bundled filter ships at least two
embedded golden cases, and CI also enforces the rules below.
- Failures are never masked. A non-zero exit suppresses success messages, and a filter that could empty a failure payload must keep the failure markers.
- Never worse. A filtered result never costs more bytes than the raw output.
- Every regex compiles. An invalid pattern would silently disable its rule;
CI fails on one in a bundled filter, and
tokenix doctorlists them in yours. - stderr is opt-in per filter (
filter_stderr = true). The cargo filters set it, because rustc writes every diagnostic there.block_capsthen keeps each warning's message and location and drops its snippet; errors keep theirs. - Line endings are preserved. CRLF comes back as CRLF, so text the agent quotes into an exact-match edit still matches the bytes on disk.
- Compression happens at write time, before output enters the conversation, so an already-cached prompt prefix is never rewritten.
| Support | Languages |
|---|---|
| Symbol-aware chunking + call graph (tree-sitter) | Rust, Python, JavaScript, TypeScript, Go, C, C++ |
| Symbol-aware chunking (line-based) | Visual Basic, SQL |
| Indexed as text (line chunks, searchable) | Shell (.sh, .bash), Markdown, TOML, plain text; JSON/YAML with data_files = true |
Anything else (Java, C#, Ruby, PHP, Kotlin, Swift, PowerShell, HTML, CSS …) is not indexed by default. Opt in per project and those files are indexed as text:
[index]
extensions = ["java", "cs", "rb", "php", "kt", "swift", "ps1", "inc"]
[languages]
inc = "cpp" # parse an extension with a supported grammar: rust, python, typescript,
# javascript, go, cpp/c, vb, sql (anything else is chunked as text)Files above max_file_bytes (1.5 MB) and binary files are skipped whole, never
truncated.
┌─────────────┐ tool call ┌──────────────┐ pass (exit 0) / intercept (exit 2)
│ AI agent │ ───────────▶ │ tokenix hook │ ─────────────────────────────────▶
└─────────────┘ └──────┬───────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ SQLite │ │ Symbol │ │ Output │
│ chunks │ │ graph + │ │ filters │
│ │ │ PageRank │ │ (TOML) │
└────────────┘ └────────────┘ └────────────┘
- Storage: one SQLite DB per project under
~/.tokenix/: chunks and the symbol graph. What the local index is for: answering structural questions an agent cannot get from grep (where a symbol is defined, who calls it, what a diff breaks, how modules depend on each other) and building outlines andpackmaps within a token budget. It never replaces the agent's own search or reads. There is no search engine, no embeddings, no model and no background daemon: the agent's own grep stays the way to find text. Indexes built by older versions drop their vector and full-text tables on the nexttokenix index. - Fail-open contract: the hook exits
0(pass) on any error, a missing or stale index, or even a panic. It exits2only to intercept, and never exits1.
- Your code stays local. Indexing, search and interception make no network
calls. The network is used for:
- the self-updater, which talks to this repo's GitHub releases only;
tokenix filter generate, which sends your recorded (redacted) output to an AI CLI you have installed, and asks before it re-runs any command.
- Repositories are untrusted by default. A repo's
.tokenix/filters,.tokenix/secret-rules,.tokenix/egress-rules,.mcp.json,opencode.jsonand.vscode/mcp.jsonare ignored untiltokenix trustpins their SHA-256, and any edit revokes that trust. A cloned repo cannot change what your agent sees, blind the secret scanner or makeprompt-auditspawn a server of its choosing. User-scoped configs (~/.claude.json,~/.codex/) are yours and are not gated. - Credentials are masked before anything is saved. The hook log, the failure
tee and recordings go through one redactor, and
~/.tokenixis owner-only (0600). The recall stash is the deliberate exception:tokenix retrievemust return the exact original bytes, so it relies on file permissions alone. - Secrets are not indexed.
.env,.pem, keystores and similar files are excluded, andtokenix scan-secretsredacts its output by default. - Supply chain: releases publish
sha256sums.txtand SLSA provenance. Workflows are SHA-pinned and checked bycargo-deny,zizmorand OpenSSF Scorecard. See SECURITY.md.
| Symptom | Fix |
|---|---|
No index found |
Run tokenix index . in the repository root. The nearest indexed directory counts as the project root, even when a parent directory has a package.json. |
| The agent still reads whole files | By design: tokenix never intercepts Read. The agent can call tokenix read or tokenix symbols itself. |
| A symbol I just wrote is missing | It should not: edits are re-chunked before each command. If more than 25 files changed, run tokenix index (or raise TOKENIX_AUTO_REFRESH_MAX). |
| A repo filter is ignored | Run tokenix trust --status. Repo filters need tokenix trust, and an edit revokes it. |
| Output was cut and I need all of it | Run tokenix retrieve <key> from the [tokenix: ...] marker, or tokenix run --raw. |
| First index is slow or the machine stalls | Use tokenix index --cpu-profile low. |
| Offline or air-gapped | Nothing to download: the only network use is the self-updater (TOKENIX_NO_UPDATE=1 disables it). |
| Remove everything | tokenix remove-hook --tool all, then delete ~/.tokenix (or your TOKENIX_HOME). |
See CONTRIBUTING.md. ./scripts/verify.sh runs every CI gate
locally, in CI order: fmt, clippy, unit, golden and end-to-end tests, release build
and the homologation scripts. It never
touches your real ~/.tokenix. New filters need at least two golden cases, and
AGENTS.md lists the engine rules a change must not break.
MIT. See LICENSE.