A sophisticated Retrieval-Augmented Generation (RAG) system specifically designed for personal Obsidian knowledge bases. Implements state-of-the-art techniques from 2024-2025 research including semantic chunking, hybrid retrieval, graph-based search, self-correcting retrieval, iterative research mode, and advanced reranking.
- ๐ฌ Research Mode Enhancements: Exhaustive queries (
@all), dual-model architecture, progressive retry - ๐ถ Euro Currency Display: Costs shown in EUR with VAT for Google Cloud billing alignment
- ๐ LLM Token Tracking: Per-day cost aggregation with detailed breakdown
- ๐ Citation Filtering: Show only cited sources in research mode results
- ๐ Source Excerpts: Increased display limit to 1500 characters
See CHANGELOG.md for full version history.
- โ Smart Document Loading: Parses Obsidian markdown with frontmatter, wikilinks, and tags
- โ Advanced Chunking: Markdown-aware semantic chunking with configurable strategies
- โ Late Chunking: 10-12% better retrieval accuracy by preserving document-level context
- โ
Multiple Embedding Options:
- voyage-3.5-lite (default, 200M free tokens/month)
- voyage-3-large (highest quality)
- qwen3-8b (best open-source, self-hosted)
- โ Vector Database Support: LanceDB (embedded) or Qdrant (scalable)
- โ Advanced Reranking: Voyage rerank-3 (200M free tokens)
- โ Hybrid Retrieval: Vector + query fusion for better results
- โ Query Transformation: HyDE, Multi-Query expansion, or both for significantly better retrieval
- โ PTCF Prompting: Research-backed prompt engineering for Gemini 3 Flash
- โ Wikilink Graph: Builds knowledge graph from note connections
- โ AI Conversations RAG: Federated search across your vault AND past AI conversations (ChatGPT, Claude, Gemini)
- โ
Inline Citations: Clickable
[1],[2],[3]citations that link to sources
- โ Streamlit Web Interface: Full-featured web UI with federated search
- โ Query Caching: LRU cache for faster repeated queries
- โ Incremental Indexing: Checkpoint-based recovery for large vaults
- โ Wikilink Graph Retrieval: Traverse note connections for related content
- โ Research Mode: Iterative multi-step retrieval with LLM-powered gap analysis (Khoj-inspired, 141% accuracy improvement)
- โ Self-Correction: Self-RAG/CRAG patterns with relevance grading and query refinement
- โ RAGAS Evaluation: Automated evaluation framework with faithfulness, relevancy, precision, and recall metrics
- โ RAPTOR Summaries: Hierarchical document summaries for better context
- โ Temporal Filtering: Filter by creation/modification date
- ๐ Neo4j graph database (deferred - in-memory graph sufficient for most use cases)
- Python 3.13+ (uv installs it if missing)
- uv (
brew install uvorcurl -LsSf https://astral.sh/uv/install.sh | sh) - Your Obsidian vault path
- API keys (optional, depending on model choice)
- Clone and setup
cd /path/to/UltraRAG
uv sync- Configure environment
cp .env.example .env
# Edit .env with your settings:
# - OBSIDIAN_VAULT_PATH=/path/to/your/vault
# - VOYAGE_API_KEY=your_key (for Voyage embeddings/reranking)
# - GOOGLE_API_KEY=your_key (for Gemini LLM)- Run the system
uv run python main.pyEdit .env to customize:
# Default - best value (200M free tokens/month)
EMBEDDING_MODEL=voyage-3.5-lite
VOYAGE_API_KEY=your_key
# Highest quality (paid)
EMBEDDING_MODEL=voyage-3-large
VOYAGE_API_KEY=your_key
# Best open-source (free, self-hosted, requires 16-32GB VRAM)
EMBEDDING_MODEL=qwen3-8b# Embedded (no setup required)
VECTOR_DB=lancedb
LANCEDB_PATH=./data/lancedb
# Production (requires Qdrant server)
VECTOR_DB=qdrant
QDRANT_HOST=localhost
QDRANT_PORT=6333CHUNK_SIZE=512 # Optimal for mixed content
CHUNK_OVERLAP=75 # 15% overlap
TOP_K=75 # Initial retrieval candidates
RERANK_TOP_N=100 # After reranking (UI max_sources controls display)
ENABLE_HYBRID_SEARCH=true # Use query fusion
# Query transformation for better retrieval
QUERY_TRANSFORM_METHOD=hyde # Options: hyde, multi_query, both, none
QUERY_TRANSFORM_NUM_QUERIES=3 # Number of query variations (for multi_query/both)
# Self-correction (Self-RAG/CRAG patterns)
USE_SELF_CORRECTION=true # Enable self-correcting retrieval
SELF_CORRECTION_MAX_RETRIES=2 # Max retry attempts with refined queries
# Research Mode (iterative retrieval)
ENABLE_RESEARCH_MODE=true # Enable @research prefix and ๐ฌ checkbox
RESEARCH_MAX_ITERATIONS=3 # Max retrieval iterations
RESEARCH_CONFIDENCE_THRESHOLD=0.8 # Stop when confidence exceeds this
RESEARCH_MAX_SUBQUERIES=3 # Sub-queries per iterationChoose your chunking strategy based on accuracy vs. speed requirements:
# Available strategies:
# - obsidian_aware: Structure-preserving (recommended for Obsidian)
# - markdown_semantic: Markdown + semantic splitting
# - late_chunking: Best accuracy (+10-12%), slower indexing
# - semantic: Pure semantic chunking
# - simple: Fast, basic sentence splitting
CHUNKING_STRATEGY=obsidian_aware # Default
# For best retrieval accuracy, use late_chunking:
CHUNKING_STRATEGY=late_chunking
LATE_CHUNKING_ALPHA=0.7 # 0.7 = 70% local, 30% global contextLate Chunking (NEW):
- 10-12% better retrieval accuracy than standard chunking
- Preserves document-level context in each chunk embedding
- Combines local chunk semantics with global document context
- Trade-off: 2x slower indexing (requires embedding both document and chunks)
- Recommended for: High-accuracy retrieval when indexing time is not critical
See docs/features/LATE_CHUNKING.md for detailed documentation.
Query transformation significantly improves retrieval by bridging the query-document vocabulary gap:
HyDE (Hypothetical Document Embeddings) - Default, recommended
- Generates a hypothetical answer to your question
- Embeds the answer instead of the query
- Since answers resemble documents more than queries, this improves matching
- Best for: Most queries, especially complex questions
Multi-Query Expansion
- Generates 3-5 variations of your query from different perspectives
- Retrieves with all variations and combines results
- Uses reciprocal rank fusion for score aggregation
- Best for: Broad topics, exploratory search
Both (HyDE + Multi-Query)
- Combines both techniques for maximum recall
- Generates query variations, then creates hypothetical documents for each
- Most comprehensive but slower and uses more API credits
- Best for: Critical queries where you need best possible results
None/Disabled
- Direct query embedding without transformation
- Fastest but lower quality retrieval
- Best for: When speed matters more than quality
Research mode enables iterative, multi-step retrieval with LLM-powered gap analysis. Inspired by Khoj's research mode (141% accuracy improvement on benchmarks).
How It Works:
- Initial retrieval with your query
- LLM analyzes gaps in retrieved content ("What information is still missing?")
- Generates refined sub-queries for missing information
- Retrieves again with sub-queries (up to
max_iterations) - Aggregates and deduplicates results across all iterations
- Synthesizes comprehensive answer from all retrieved content
Usage:
- CLI: Prefix query with
@research- e.g.,@research what are all the productivity techniques in my notes? - Web UI: Check the ๐ฌ "Research Mode" checkbox before querying
Configuration:
ENABLE_RESEARCH_MODE=true # Enable research mode (default: true)
RESEARCH_MAX_ITERATIONS=3 # Max iterations (default: 3)
RESEARCH_CONFIDENCE_THRESHOLD=0.8 # Stop when confident (default: 0.8)
RESEARCH_MAX_SUBQUERIES=3 # Sub-queries per iteration (default: 3)Best For:
- Comprehensive research queries: "What are ALL the X in my notes?"
- Topic synthesis: "Everything I know about habit formation"
- Cross-reference queries: "How do my notes on X relate to Y?"
Self-correction implements Self-RAG and CRAG patterns to improve retrieval quality through automatic query refinement.
How It Works:
- Initial retrieval with your query
- LLM grades relevance:
CORRECT,AMBIGUOUS, orINCORRECT - If not
CORRECT: LLM refines query and re-retrieves - Repeats up to
max_retriestimes - Returns best results from all attempts
Configuration:
USE_SELF_CORRECTION=true # Enable self-correction (default: true)
SELF_CORRECTION_MAX_RETRIES=2 # Max refinement attempts (default: 2)See docs/features/SELF_CORRECTION.md for detailed documentation.
UltraRAG generates responses with inline citations that link to sources:
- Citations appear as
[1],[2],[3]in the response text - In the web UI, clicking a citation scrolls to that source
- Sources are numbered consistently between response and source list
Example:
"Atomic habits work because small changes compound over time [1]. The habit loop consists of cue, routine, and reward [3]."
UltraRAG can index and search your past AI conversations alongside your Obsidian vault using federated retrieval. This means you can query both your personal notes AND your ChatGPT/Claude/Gemini conversation history in a single search.
-
Export your AI conversations using AI Conversation Toolkit
-
Configure UltraRAG to use your exports:
# In your .env file
CONVERSATIONS_ENABLED=true
CONVERSATIONS_PATH=/path/to/ai-conversation-toolkit/output
CONVERSATIONS_WEIGHT=0.8 # Score weight vs vault (vault=1.0)- Index and search:
- CLI: Type
convto index conversations, then use@vault,@conv, or@allprefixes - Web: Click "Index Conversations" in sidebar, then use the search scope toggle
- CLI: Type
| Prefix | Scope | Description |
|---|---|---|
| (none) | Both | Federated search across vault + conversations |
@vault |
Vault only | Search only your Obsidian notes |
@conv |
Conversations | Search only AI conversation history |
@all |
Both | Explicit federated search |
Results are tagged with ๐ (vault) or ๐ฌ (conversation) so you know the source.
uv run python main.pyThis will:
- Load your Obsidian vault
- Index all notes (one-time process)
- Start interactive query loop
from main import UltraRAG
# Initialize system
rag = UltraRAG()
# Index your vault (one-time)
rag.index_vault()
# Query the system
result = rag.query("What are my thoughts on machine learning?")
print(result['answer'])
# View sources
for source in result['sources']:
print(f"{source['title']}: {source['score']}")
# Search without generation
notes = rag.search_notes("project ideas", top_k=5)โโโโโโโโโโโโโโโโโโโ
โ Obsidian Vault โ
โโโโโโโโโโฌโโโโโโโโโ
โ
โโโโโโผโโโโโโ
โ Loader โ Extracts wikilinks, tags, metadata
โโโโโโฌโโโโโโ
โ
โโโโโโผโโโโโโ
โ Chunker โ Markdown-aware semantic splitting
โโโโโโฌโโโโโโ
โ
โโโโโโผโโโโโโโโโโ
โ Embeddings โ voyage-3.5-lite (default) / qwen3-8b
โโโโโโฌโโโโโโโโโโ
โ
โโโโโโผโโโโโโโโโโโ
โ Vector Store โ LanceDB / Qdrant
โโโโโโฌโโโโโโโโโโโ
โ
โโโโโโผโโโโโโโโโโ
โ Retrieval โ Hybrid vector + graph search
โโโโโโฌโโโโโโโโโโ
โ
โโโโโโผโโโโโโโโโโ
โ Reranking โ Voyage rerank-3
โโโโโโฌโโโโโโโโโโ
โ
โโโโโโผโโโโโโโโโโโ
โ Generation โ Gemini 3 Flash + PTCF prompts
โโโโโโโโโโโโโโโโโ
- voyage-3.5-lite: Free (200M tokens/month)
- voyage-3-large: $13-40 (API pricing)
- qwen3-8b: Free (self-hosted, requires 16-32GB VRAM)
- Time: 10-30 minutes depending on model
- Per query: $0.001-0.01 (with reranking)
- Monthly (moderate use): $5-20
- Self-hosted: $0 after hardware investment
Expected metrics on a 1,650-note vault:
| Metric | Target | Notes |
|---|---|---|
| Retrieval Accuracy | 85-95% | vs 40-50% for naive RAG |
| Latency (simple) | <1s | Single hop retrieval |
| Latency (complex) | <3s | Multi-hop + reranking |
| Index Time | 10-30min | One-time operation |
| Scale | 10K+ notes | No architecture changes needed |
- Document loading and parsing
- Semantic chunking
- Vector indexing
- Basic retrieval
- LLM integration
- Inline citations with clickable links
- Streamlit web interface
- Query caching
- Incremental indexing with checkpoints
- Wikilink graph retrieval
- AI conversations federated search
- Research Mode (iterative retrieval with gap analysis)
- Self-Correction (Self-RAG/CRAG patterns)
- RAGAS evaluation framework
- RAPTOR hierarchical summaries
- Temporal filtering
- Neo4j graph database (deferred - in-memory graph sufficient)
Get a free API key from Voyage AI
- Use smaller embedding model (Qwen3-1.5B variant)
- Reduce batch size in chunking
- Use LanceDB instead of Qdrant
- Enable reranking (
RERANK_TOP_N=10) - Reduce
TOP_K(try 50 instead of 75) - Use faster embedding model
- Increase
TOP_K(try 100) - Adjust
CHUNK_SIZE(try 768) - Enable hybrid search
- Add reranking
This is a personal project implementing research from the compass_artifact document. Feel free to adapt for your own use case.
MIT License - See LICENSE file
Full documentation is available in the docs/ folder:
- Quick Start Guide
- Architecture Overview
- Testing Guide
- RAGAS Evaluation Guide
- Feature Guides: Late Chunking | Query Transformation | Self-Correction | Graph Retrieval | Research Mode | File Exclusions
Based on cutting-edge RAG research from 2024-2025:
- RAPTOR (recursive abstractive processing)
- Late Chunking (Jina AI)
- Self-RAG and CRAG (corrective retrieval)
- Khoj Research Mode (iterative retrieval)
- Voyage AI embeddings and reranking
- RAGAS evaluation framework
- Gemini 3 Flash
- LlamaIndex framework
Built with โค๏ธ for Obsidian power users
