I build AI tools, then measure how they fail.
- I'm finishing my M.S. in Data Science at Northeastern University (December 2026) and I'm a teaching assistant for DS 3500, Advanced Programming with Data.
- Most of my recent work measures how AI systems fail: retrieval poisoning, stale agent memory, A/B tests that weren't what they seemed.
- Before this: Data Scientist at Launch Plan (LangGraph and MCP agents, RAG on Pinecone, an Airflow data layer) and Data Engineer / Full Stack Developer at Soverin.
- Looking for forward-deployed engineer, data scientist and software engineer roles. Portfolio and case studies: srikarmk.dev
| Project | What it is |
|---|---|
| rag-poisoning | One bad passage in 10,000: how far a planted false fact or instruction spreads through a RAG system, on three open LLMs, and what defences cost. Case study |
| memory-rot | Do agents with long-term memory answer with stale facts? Seven memory designs, four open models. Case study |
| llm-judge-bias | What sways an LLM judge: answer order, padding, stated confidence, markdown and its own authorship, on pairs with a known better answer. Case study |
| long-context-recall | Whether a local LLM finds one fact in a long document as it grows, as the fact moves, and among near-miss look-alikes; plus missing facts and two-hop questions. Case study |
| tool-call-reliability | How reliably local LLMs call tools: right tool and arguments, multi-step chains, recovery from tool errors, and how much the tool interface matters, varied one factor at a time. Case study |
| prompt-brittleness | How often a harmless rewording changes an LLM's answer, and why the best-looking prompt is usually a lucky draw. Three open models. Case study |
| whygit | A read-only Git explainer, as a CLI and an MCP server. On PyPI and in the official MCP registry. uvx whygit |
| Keystone | What a research paper builds on, disputes and assumes, read straight from its LaTeX with no LLM; 197 papers, with its error rate published. Live site |
| experiment-forensics | A pre-registered re-analysis of 22,743 real A/B tests: peeking, the winner's curse, power and CUPED. Case study |
| GitGotchi | A pet that lives in your Git repo and grows with how you commit; AI-written commits slowly possess it. On PyPI. Case study |
| Duskline | A browser racing game made almost entirely in code: shader skies, procedural cars, synthesised engine and music. 33 to 85 fps once I measured the frame. |
More in the series Measuring LLM Systems.
- Case studies and posts on my site, including what went wrong
- Vectorized Thoughts, my LinkedIn newsletter