Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.
-
Updated
Sep 4, 2026 - Python
Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.
Ultra-low-latency LLM gateway with microsecond caching, dynamic routing, budgets, analytics, and forecasting.
Asynchronous Multi-Modal RAG Platform with HNSW Indexing, Redis Semantic Caching, and Telemetry across 15+ Relational Tables.
Official implementation of "SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching" (COLM 2025). A novel KV cache compression method that organizes cache at sentence level using semantic similarity.
MorrowCache: open-source semantic cache for OpenAI-compatible LLM APIs. Local proxy that reuses answers for paraphrased prompts and AI agent retries. npm: @kushalicious/jevcache
This app leverages Semantic Caching to minimize inference latency and reduce API costs by reusing semantically similar prompt responses.
Semantic caching demo with real-time streaming and a cost & sizing calculator, powered by Azure Managed Redis and Azure OpenAI.
LLM Observability & Evaluation Platform — Live Frontend: https://llm-obs-web-app-9664.web.app
Semantic caching system using Meta Llama 3.3, Groq, FAISS, and Sentence Transformers to reduce LLM latency and API costs through meaning-based query matching.
Provider-agnostic LLM observability & reliability SDK — semantic caching, retries, rate limiting, and real-time dashboard via a single @track() decorator.
Evaluate how a semantic cache performs on your dataset by computing key KPIs over a threshold sweep and producing plots/CSVs:
High-performance AI gateway & token optimizer in pure Go. Cut LLM costs & latency with zero code changes.
Enterprise AI infrastructure platform: distributed vLLM inference, agentic workflows, hybrid multimodal RAG, CNN vision, QoS-tiered FastAPI gateway, guardrails and Prometheus/Grafana observability, fully Dockerized.
Rust Local Token Compression Proxy for coding agents, built solo for GenAI Genesis 2026. 🏆 1st Google Sustainability Hack
Research behind CacheVerifier (cacheverifier.com): verifier gating for semantic cache hits vs. static and adaptive thresholds, on ~210k requests across three public benchmarks plus real Twitter support traffic (4 datasets). Fine-tuning vs. off-the-shelf, label noise, drift, conformal risk control under shift, online ACI from randomized audits.
An intelligent gateway for Claude APIs that dynamically routes requests to the most cost-efficient model, caches responses, and escalates based on confidence signals — reducing LLM spend without sacrificing quality.
Semantic caching for LLM responses using Redis Vector DB, LangChain, and HuggingFace embeddings, parses PDFs, generates FAQs with Groq, and serves similarity-based answers without redundant LLM calls.
Semantic memory and caching for LLM agents with classifier-validated equivalence instead of naive cosine thresholds.
A retrieval-augmented generation pipeline in Python with a rigorous offline evaluation harness. Chunks and embeds documents, retrieves by vector similarity, and generates grounded answers — with pluggable LLM providers (including a deterministic local fake for tests) and metrics for retrieval quality and answer faithfulness. No API key required.
To associate your repository with the semantic-caching topic, visit your repo's landing page and select "manage topics."