🏥 CareFlow

Voice-First Patient & Appointment Assistant

Book appointments, track a live queue, and talk to CareBot — a multi-agent voice assistant that speaks English, Hindi and Kannada.

▶ Try the live demo · sign in as ananya@careflow.example / careflow-demo-1 · may take a minute to wake

Features • CareBot • Architecture • Latency • Guardrails • Hardest problem • Quick Start • Roadmap

---

Warning

Demo system — do not enter real health information. CareFlow uses synthetic data only. The Gemini free tier allows Google to use submitted content to improve its products, so no real patient data or real voice recordings belong here.

Note

Project status: all five milestones are built. Patients book appointments, follow a live queue and talk to CareBot by text or voice in English, Hindi or Kannada; doctors run their queue; admins see department load and CareBot's latency on an analytics page. It runs in Docker Compose, and a lite version is live on Render. Measured results are under Benchmarks, including the targets it misses; see the Roadmap for what is left.

💡 The Problem

Outpatient departments run on front-desk phone calls and paper tokens. Patients don't know which department to visit, can't see real wait times, and call the front desk for information a system could answer.

CareFlow replaces that with real-time queue state, self-service booking, and a voice/text assistant for routine requests — while keeping clinical judgement strictly with doctors.

The engineering focus is the voice agent: two interchangeable voice pipelines, measured per-stage latency, graceful failover when free-tier quotas run out, and guardrails enforced in code.

---

✨ Features

🧑‍⚕️ Patient Book, reschedule and cancel by department or doctor Live queue position — "3 patients ahead of you" Doctor delay broadcasts with updated ETA Symptom-based department routing (not diagnosis) Visit & prescription history (last 12 months) Talk to CareBot by voice or text

👨‍⚕️ Doctor Today's queue at a glance Mark patients in_consultation, done or no_show Broadcast a delay — pushed live to every waiting patient Seeded schedules

📊 Admin Department load: queue depth, doctors on duty Voice latency P50 / P95 per stage, per pipeline Time to first audio Intent mix and fallback rate Gemini quota errors

Languages: English · हिन्दी · ಕನ್ನಡ · code-mixed speech ("kal ka appointment cancel karo")

---

🤖 CareBot — the voice agent

Four specialist agents

Each agent is bound to a narrow tool set, so it can only do its own job.

Agent	What it does	Hard limit
📅 Booking	Checks availability, books, reschedules, cancels — real database tool calls	Acts only for the logged-in patient
🩺 Triage	Maps symptoms to a department ("chest pain" → Cardiology)	Never diagnoses, prescribes or gives dosage
📋 Records	Reads visit and prescription history	The authenticated patient's own records only
ℹ️ Support	Timings, department locations, emergency contact	—

Routing without an LLM call. A fast local intent classifier (keywords + a small multilingual embedding model) handles most turns on CPU. Only low-confidence turns go to the LLM, which removes one model round-trip from the typical turn.

Two voice pipelines, switchable in the UI

	🅰️ Pipeline A — Real-time	🅱️ Pipeline B — Cascaded, open source
Role	Primary	Fallback + measurable baseline
Flow	Browser mic → FastAPI WebSocket → Gemini Live API → audio streamed back	Silero VAD → faster-whisper → intent classifier → Gemini Flash-Lite → sentence-chunked TTS
Strength	Lowest latency; speech-to-speech in one model	Every stage measurable and swappable; works when Live quota is exhausted
Routing	One Live session holds all tools, with server-side guardrail checks on every tool result	Per-turn router dispatches to specialist agents

Why routing differs: the Live API keeps a single session per conversation, so per-turn agent hand-off isn't possible there without restarting the session and losing latency. This is a deliberate trade-off.

Shared behaviour

• 🧠 Conversation memory — follow-ups like "what time was that again?" resolve
• 🔄 Server-driven state machine — LISTENING → TRANSCRIBING → THINKING → TOOL_EXECUTION → SPEAKING, streamed over Socket.IO so the UI never guesses timing
• ✋ Barge-in — speaking during playback stops the bot and starts a new turn
• 🧾 Workflow tracker — each booking moves through received → intent_classified → slot_checked → awaiting_confirmation → confirmed / failed in Redis, readable while it runs at GET /api/chat/{session_id}/workflow
• ⚡ Response cache — frequent questions ("OPD timings") are answered from Redis with no LLM call
• 📈 Per-turn analytics — pipeline, intent, stage timings and fallback tier

---

🏗 Architecture

flowchart LR
    subgraph Browser["🖥️ Browser — React 18"]
        UI["Dashboards<br/>Patient · Doctor · Admin"]
        MIC["🎙️ Mic + audio playback"]
    end

    subgraph Server["⚙️ FastAPI — single container"]
        REST["REST API"]
        SIO["Socket.IO<br/>queue + voice state"]
        WS["Voice WebSocket"]
        ROUTER["Intent classifier<br/>+ orchestrator"]
        AGENTS["Booking · Triage<br/>Records · Support"]
        GUARD["🛡️ Guardrails"]
    end

    subgraph Voice["🗣️ Voice pipelines"]
        PA["🅰️ Gemini Live API"]
        PB["🅱️ VAD → STT → LLM → TTS"]
    end

    subgraph Data["💾 Data"]
        MONGO[("MongoDB")]
        REDIS[("Redis")]
    end

    UI --> REST
    UI <--> SIO
    MIC <--> WS
    WS <--> PA
    WS <--> PB
    PB --> ROUTER
    REST --> ROUTER
    ROUTER --> AGENTS
    PA --> GUARD
    AGENTS --> GUARD
    AGENTS --> MONGO
    ROUTER --> REDIS
    REST --> MONGO

Loading

Failover chain

Free-tier quotas run out. CareFlow degrades step by step instead of failing.

flowchart TD
    START(["🗣️ User turn"]) --> CACHE{"Redis cache hit?"}
    CACHE -- Yes --> DONE(["✅ Answer"])
    CACHE -- No --> A["🅰️ Pipeline A<br/>Gemini Live"]
    A -- OK --> DONE
    A -- "429 · timeout · session drop" --> B["🅱️ Pipeline B<br/>Gemini Flash-Lite"]
    B -- OK --> DONE
    B -- "429 · timeout" --> O["🦙 Ollama<br/>local model"]
    O -- OK --> DONE
    O -- Unavailable --> KB["📖 Rule-based<br/>knowledge base"]
    KB --> DONE

Loading

• Circuit breaker — after 3 failures in a row (quota errors, timeouts or server errors), a provider is skipped for 60 seconds with no request made, then tried again
• Timeout — a model request slower than 8 seconds counts as failed, and the next tier answers
• Every turn records which tier answered (cache, gemini, ollama or rules); the fallback rate will appear on the analytics page
• Status: the text and cascaded-voice tiers are built. Checked live with Gemini pointed at a model that does not exist: three failed requests opened the circuit, and every later turn was answered by rules in about 0.05 s. The Ollama tier is tested against a stand-in server only; the development laptop does not have the memory to run it
• When a Live session hits its duration limit, the client reconnects and conversation memory carries the context across

---

⏱ Latency targets

The headline number is time to first audio (TTFA): the user stops speaking → the first audio plays.

Metric	🅰️ Pipeline A	🅱️ Pipeline B
TTFA P50	< 1.5 s	< 2.5 s
TTFA P95	< 2.5 s	< 4.0 s

These are targets. What was measured, on a laptop CPU over a home connection, is under Benchmarks: Pipeline A reaches 2.29 s at P50, Pipeline B 4.03 s.

Pipeline B latency budget — stage by stage

Stage	Measured as	Main levers
End-of-speech detection	Last speech frame → VAD end event	VAD silence threshold
STT	VAD end → final transcript	Model size, int8 quantisation, beam size
Routing	Transcript → agent selected	Local classifier first; LLM only on low confidence
LLM time-to-first-token	Request → first token	Flash-Lite, short prompts, cache hits skip this stage
TTS first chunk	First full sentence → first audio bytes	Sentence-level chunking, streaming while the LLM still generates

Pipeline A is measured end to end, since its stages happen inside one model.

📊 Benchmarks

All numbers below were measured on one 8-thread laptop CPU, with the app in Docker Desktop on Windows over a home connection, using the free Gemini tier. Questions are synthesised speech. Every row links to its raw results.

At a glance

What	Before	After
🅱️ Time to first audio, P50 (19 spoken questions)	7.94 s	4.03 s
🅱️ Speech to text, P50 (shorter Whisper window)	4.12 s	1.02 s
🅱️ LLM stage, P50 (facts looked up before the call)	2.12 s	1.20 s
A repeated hospital question, by text (Redis cache)	5.7 s	0.03 s
🅰️ Time to first audio, P50 / P95 (Gemini Live)		2.29 s / 3.29 s
Hindi word error rate (Whisper small for Hindi)	104.7%	48.8%
Intent, best guess right on unseen phrasings (embeddings added)	35%	85%
Load test, 50 patients at once / 10 with real Gemini		0 failures in both

• Neither voice pipeline meets its P50 target. Pipeline A is 0.8 s over its 1.5 s target, and Pipeline B 1.5 s over its 2.5 s target.
• The free Gemini tier is the least predictable part. Its latency varies by more than some optimisations save, and with 10 patients at once the rule-based fallback answered most turns. The failover chain is what kept every patient answered.
• Hindi by voice is the weakest result, and Kannada is not measured. Pipeline A, where Gemini hears the speech itself, is the better path for both.

Pipeline A against Pipeline B (measured)

The same 19 spoken English questions through each pipeline, on the same hardware. Pipeline A is measured end to end, because its stages happen inside one model; its figure includes the 300 ms for which reply audio is held so its transcript can be checked first. Raw numbers: pipeline-a-live.json, preload-facts.json.

Time to first audio	P50	P95	Target P50	Target P95
🅰️ Pipeline A — Gemini Live	2.29 s	3.29 s	1.5 s	2.5 s
🅱️ Pipeline B — cascaded, after the optimisations below	4.03 s	22.36 s	2.5 s	4.0 s
🅱️ Pipeline B — baseline	7.94 s	13.59 s

• Pipeline A is the faster of the two by about 1.7 s at the median, and it misses its own target by about 0.8 s. Every one of its 19 turns took between 1.5 and 3.3 s.
• Pipeline A is far steadier. Pipeline B's P95 is one turn in which a Gemini text request hung to its timeout; that run predates the 8 s timeout and the circuit breaker.
• All 19 Pipeline A turns were answered and spoken by Gemini Live, with no fallback.
• With 19 turns, each P95 is that run's slowest turn. The questions are synthesised speech, in English only.

Voice pipeline B — baseline (measured)

19 spoken English questions, streamed to the app over the voice WebSocket at speaking speed, before any optimisation. Run with python eval/latency_bench.py; the raw numbers are in backend/eval/results/baseline.json.

Stage	P50	P95
End-of-speech detection	0.55 s	0.65 s
Speech to text (Whisper base, 8 threads)	4.12 s	7.75 s
Routing	0.01 s	4.20 s
LLM, tools included (Gemini Flash-Lite)	2.04 s	4.00 s
TTS first chunk (Piper)	0.77 s	3.24 s
Time to first audio	7.94 s	13.59 s

• The targets are 2.5 s (P50) and 4.0 s (P95), so this is about three times too slow. It is the "before" column for the optimisation work that follows.
• Speech to text is the largest stage, at about half the wait. Whisper small, the planned model, was slower still on this laptop (10 to 20 s per utterance), so base is used.
• With 19 turns, P95 is the slowest turn. The 4.2 s routing figure is the one turn the local classifier handed to the LLM; the other 18 were routed locally in about 10 ms.
• Every reply was spoken by Piper. Gemini TTS was out of free quota (HTTP 429) for the whole run.
• The LLM is not steady on the free tier. In an earlier three-turn check, plain Gemini requests took 4 to 11 s and time to first audio was 18 to 28 s.
• The questions are synthesised speech, not recorded voices, and all in English. Hardware: an 8-thread laptop CPU, with the app in Docker Desktop on Windows.

Optimisation 1 — give Whisper only the audio it needs (measured)

Whisper pads every utterance to 30 seconds, and its encoder's cost follows the padded length, so a 2-second question cost as much as a 30-second one. The encoder now gets the utterance plus 5 seconds of quiet, and never less than 8 seconds. Same 19 questions, same hardware; raw numbers in stt-short-window.json.

Stage	Before P50	After P50	Before P95	After P95
Speech to text	4.12 s	1.02 s	7.75 s	5.11 s
Time to first audio	7.94 s	4.48 s	13.59 s	12.71 s

• Accuracy did not change on these clips: 18 of 19 transcribed exactly, with either window.
• The window must not be tight. With only 1 second of quiet after the speech, Whisper looped and repeated itself (word error rate above 100%), which is why the margin is 5 seconds.
• P95 barely moved, because it is one turn. The first turn of each run is the slowest, in both runs; the other 18 turns took 2.6 to 8.3 s.
• What did not help: thread count (4, 6 or 8), dropping timestamps, fixing the language, and a single decoding temperature all left the stage at about 4 s. Whisper tiny was faster but less accurate; small took about 20 s.
• Hiss, hum and clicks that get past the voice-activity detector are now dropped by a confidence check, where Whisper used to invent a sentence for them.
• Still about twice the 2.5 s target. The LLM, at about 2 s, is now the largest stage.

Optimisation 2 — answer repeat hospital questions from Redis (measured)

A reply about the hospital itself (timings, a department's location, the emergency contact) is the same for every patient, so the LLM's answer is kept in Redis for an hour and reused. The benchmark was run twice: once with an empty cache, then again with the cache the first run had filled. Raw numbers: response-cache-cold.json, response-cache-warm.json.

The 4 hospital questions	Cache empty	Cache hit
LLM stage	2.1 – 2.6 s	0.01 – 0.2 s
Time to first audio	4.4 – 5.1 s	2.4 – 3.7 s

• All 4 hospital questions hit the cache on the second run, and two of them came in under the 2.5 s target.
• The other 15 turns do not benefit, and the overall median did not improve: it was 4.72 s on the first run and 5.40 s on the second, because the LLM happened to be slower during the second (2.86 s against 2.12 s at P50). Free-tier latency varies by more than this optimisation saves across a mixed set of questions.
• By text, a repeated question took 5.7 s the first time and 0.03 s the second.
• Never cached: anything about a patient (records, bookings, symptoms), and questions that lean on the conversation ("Where is it?"). Keys are hashes, so a patient's wording is not stored.
• A first-time question still pays for the LLM. The cache only helps what has been asked before, in the same words and language.

Optimisation 3 — look the facts up before asking the model (measured)

The Triage, Records and Support agents made two LLM calls per turn: one in which the model asked for a lookup, and one to phrase the answer. None of those lookups needs the model's judgement (hospital information, the patient's own visits, symptoms to a department), so the server now does them first and hands the results over. Those turns take one call. Booking keeps its tools, because there the model has to choose dates and slots. Same 19 questions, cache empty in both runs; raw numbers in preload-facts.json.

P50	Before	After
LLM stage, the 12 triage / records / support turns	2.12 s	0.98 s
Time to first audio, those 12 turns	4.64 s	3.70 s
LLM stage, all 19 turns	2.12 s	1.20 s
Time to first audio, all 19 turns	4.72 s	4.03 s

• P95 got worse, and not because of this change: one Gemini request hung until the 20 s timeout and that turn fell back to a rule-based reply (22.4 s to first audio). A shorter timeout and a failover chain are the next piece of work.
• The 5 booking turns still make two or more calls (2.1 s at P50).
• The guardrails are unchanged: identity still comes from the session, the model is simply never offered a records tool to aim at anyone, and the triage filter still checks every reply.

Phone-quality audio and Hindi (measured)

Phone calls carry 8 kHz audio limited to about 300–3400 Hz, usually through a G.711 codec. eval/stt_wer.py transcribes each labelled utterance three ways, as the browser sends it, as an 8 kHz phone line, and as a phone line with the G.711 codec, and scores each against its text. Raw transcripts and scores: stt-wer-dev.json.

Language (utterances)	Word error rate, 16 kHz	8 kHz	8 kHz + G.711
English (19)	2.8%	4.9%	4.9%
Hindi (11), Whisper base only	104.7%	104.7%	102.3%
Hindi (11), Whisper small for Hindi	48.8%	48.8%	48.8%

• Phone audio costs English about two points of word error rate, and they land on the words triage needs: "I have had chest pain" became "I have had just pain", and "ear pain" became "your pain". Over a phone line, a caller with chest pain would lose the emergency advice and the Cardiology suggestion.
• Whisper base cannot do Hindi here. It wrote 9 of 11 Hindi questions in Urdu script, which no Hindi keyword can match, and dropped 3 as too uncertain. Whisper small writes Devanagari, and with the short encoder window it takes about 2.2 s. The cascaded pipeline now listens with base and hands Hindi and Kannada to small (WHISPER_MULTILINGUAL_MODEL).
• Even then, Hindi is weak: about half the words are wrong, and only 6 of the 11 Hindi transcripts were routed to the right agent by the keyword rules alone. Pipeline A, where Gemini Live hears the speech itself, is the better path for Hindi.
• By voice in the container, Hindi on Pipeline B is slow and silent: over the 11 Hindi questions, speech to text took 6.7 s at P50 (Whisper small inside Docker), every question was answered by Gemini, and none of the answers could be spoken, because Gemini TTS was out of quota and the Piper fallback voice is English only. Raw numbers: stt-hindi-small.json.
• Kannada and code-mixed speech are not measured yet. Their audio could not be synthesised: the free Gemini TTS quota ran out after 11 Hindi clips.
• All questions are synthesised speech, and the Hindi references were written by the project author.

Load test — many patients at once (measured)

backend/loadtest/locustfile.py registers simulated patients, each of whom chats with CareBot and checks availability and their appointments, with 1–3 s between actions. All requests come from one laptop, so the app runs with its per-IP rate limits raised (backend/loadtest/docker-compose.loadtest.yml). Raw numbers: backend/loadtest/results/.

Run	Requests	Failures	Chat P50	Chat P95	Chat P99
50 patients, no LLM, 2 min	2,574	0	0.09 s	1.1 s	4.6 s
10 patients, real Gemini, 1 min	232	0	0.03 s	4.2 s	12 s

• Nothing failed in either run.
• With the real model, the free tier could not keep up. Of 147 chat turns, Gemini answered 9%, the Redis cache 31% and the rule-based replies 61%: the circuit breaker opened and every patient still got an answer. The P95 and P99 are the turns that waited on Gemini before it gave up.
• The load test found two ways one request could stall everyone else's. Password hashing (bcrypt, about 0.26 s per hash) ran on the server's single event loop, so 50 sign-ups at once held every request; it now runs in a worker thread, and a test fails if it moves back. And a chat request arriving before start-up had loaded the embedding model would load it itself on the event loop, which froze the server for 2.7 s once; until the models are ready, requests now classify with keywords alone.
• Measure only once the models have loaded. Straight after a cold start, while the speech models are still loading, the same 50-patient test gave a chat P95 of 7.6 s. /api/health now reports "models": "loading" until they are ready.
• Two warm runs of the 50-patient test gave a chat P95 of 0.52 s and 1.1 s: the laptop also runs Locust itself, so treat these as approximate.

Intent classifier (measured)

How often the local classifier picks the right agent, and how often it is sure enough to skip the LLM. Run with python eval/run_eval.py --classifier --set all.

Set	Stages	Best guess correct	Settled without the LLM	Wrong when confident
Dev (75)	Keywords only	75 / 75	72 / 75	0
Dev (75)	Keywords + embeddings	75 / 75	73 / 75	0
Held-out (34)	Keywords only	12 / 34 (35%)	9 / 34 (26%)	0
Held-out (34)	Keywords + embeddings	29 / 34 (85%)	16 / 34 (47%)	0

• The dev score is not a quality claim. That set was written together with the keyword rules, so it only guards against regressions. The held-out set was written before the rules were first run, and is the honest number.
• Keywords alone do not generalise: on unseen phrasings they recognise about a third. The embedding stage is what lifts the best guess to 85%.
• Kannada is the weak spot: 5 / 7 held-out with embeddings, against 7 / 7 for Hindi. The embedding model (paraphrase-multilingual-MiniLM-L12-v2) was not trained on Kannada.
• About half of unseen turns still go to the LLM. The thresholds are cautious: no confident local answer was wrong on either set, at the cost of handing off more often.
• Both sets are small and were written by the project author, not collected from patients.

Script	What it measures
eval/run_eval.py --classifier	Intent accuracy of the local classifier on labelled utterances (EN / HI / KN / code-mixed) — results above
eval/run_eval.py --live	End-to-end tool-call correctness with the real LLM
eval/stt_wer.py	faster-whisper word error rate at 16 kHz vs 8 kHz phone audio, with and without the G.711 codec, per language — results above
eval/latency_bench.py	P50 / P95 per stage for the voice pipeline — baseline above
loadtest/locustfile.py	P95 response time for 20–50 concurrent text sessions — results above

---

🛡 Healthcare guardrails

Guardrails are enforced in code, not only in prompts, and each one has a test. Guardrail tests run as their own required CI job.

Rule	How it is enforced
🎙️ The same rules in speech-to-speech	In Pipeline A the model's own transcript of what it is saying is checked as it arrives, and reply audio is held 300 ms so the check comes first; a violating reply is cut off and the safe fallback is spoken instead. Tools still run on the server as the session's user
🩺 Triage never diagnoses, prescribes or gives dosage	System prompt plus a pattern-based output filter that replaces violations with a safe fallback — also applied to Live API tool results and transcripts
⚠️ Every triage response shows a disclaimer	Added server-side and rendered by a dedicated UI component
🪪 Identity comes from the session, never the model	Every tool receives the user ID and role from the verified JWT; tools accept no patient ID from the LLM
🚫 No privileged self-registration	The register endpoint creates patients only; doctors and admins are seeded
🔌 Sockets require auth	JWT checked on connect; patients receive only their own queue position
📅 No double booking	Unique index on (doctor_id, slot_start); a duplicate insert returns 409
✋ No change without a confirming turn	Booking, cancelling and rescheduling tools only take effect in the turn after the one that proposed them, so the patient always hears the change first
🗃️ Only public answers are cached	The Redis response cache holds replies about the hospital only; a reply to a records, booking or symptom question is never stored or shared
🚦 Abuse and quota protection	Rate limits on auth, chat and voice endpoints
🧪 Mock DB never used outside tests	The app refuses to start with the mock database unless running in the test environment

Data handling: raw audio is processed in memory and never stored. Analytics store timings and intent labels, not transcripts.

---

🧩 The hardest problem

Applying the guardrails to a model that speaks for itself.

In Pipeline B the guardrails sit in an obvious place. The LLM writes text, the server checks it, and only then is it turned into speech. Pipeline A has no such step. Gemini Live hears the patient and answers in audio directly, so the reply exists as sound before the server has any text to check. A rule such as "never name a dosage" has nothing to inspect.

What the server does get is the model's own transcript of what it is saying. That transcript streams in beside the audio, but not in step with it: it can arrive a moment after the words it describes. The solution has four parts:

1. Hold every audio chunk for a moment. The server keeps each chunk for 300 ms (LIVE_GUARD_HOLD_MS) before passing it on, so the transcript of those words usually arrives first.
2. Check the transcript as it grows. The same filter used in Pipeline B runs over everything the model has said so far in the turn.
3. Cut and replace. On a violation, the held audio is dropped and the browser is told to stop playing. The safe reply is spoken in the patient's language instead, and the turn is recorded as blocked.
4. Keep the tools on the server. Bookings and record lookups run as the signed-in patient, so whatever the model says, it cannot reach another patient's data.

What it costs and what it does not cover:

• Every Pipeline A reply starts 300 ms later. That is included in its measured 2.29 s.
• The hold is a delay, not a guarantee. If the transcript ever trails its audio by more than 300 ms, a few words could be heard before the cut. The tests prove the mechanism with a scripted Live server, where a blocked reply is never heard at all, but how far the real transcript lags has not been measured.
• The check reads the model's transcript of its speech, not the speech itself. If the two ever differ, the check follows the transcript.
• Reading a patient's own prescription note back to them is exempt, because medicine names belong there.

Bringing Pipeline B from 7.9 s to 4 s took more hours, but each step was ordinary: measure every stage, then fix the largest. The guardrails needed a design that a speech-to-speech model does not offer, and that design trades some latency for safety on every reply.

---

🛠 Tech stack

Every component is free-tier or open source.

Layer	Technology
Frontend	React 18 · Vite · React Router 6 · Socket.io-client · Axios
Backend	Python 3.12 · FastAPI · Uvicorn · python-socketio
Database	MongoDB (Motor + Beanie) — Atlas free tier for the demo
Cache / workflow	Redis — Upstash free tier for the demo
Auth	PyJWT · bcrypt
LLM	Gemini Flash / Flash-Lite · Ollama (offline fallback)
Real-time voice	Gemini Live API (native audio)
Intent classifier	Keyword rules + a multilingual sentence-transformers model run through fastembed (ONNX, local CPU)
VAD / STT / TTS	Silero VAD · faster-whisper · Gemini Flash TTS · Piper (local, English; GPL-3.0, optional)
Observability	structlog · Langfuse (optional, self-hosted)
Testing	pytest · httpx · Vitest · React Testing Library · Locust
Quality	ruff · mypy · ESLint · Prettier · pre-commit · gitleaks
DevOps	Docker Compose · GitHub Actions · Dependabot

---

🚀 Quick start

Try it now — no Docker or database needed

This runs the whole app against an in-memory database that is already seeded. Everything is lost when you stop it.

git clone https://github.com/RahulBailur/CareFlow.git
cd CareFlow

cd frontend && npm ci && npm run build && cd ..

cd backend
python -m venv .venv
.venv\Scripts\activate          # macOS / Linux: source .venv/bin/activate
pip install -e ".[dev]"
python scripts/dev_server.py

Then open http://localhost:8000. The server prints a patient, a doctor and an admin login when it starts.

To see the live queue, open the patient in one browser window and the doctor in another, then start a consultation or broadcast a delay from the doctor's side.

With Docker Compose

This runs the app with a real MongoDB and Redis. Data is kept in a Docker volume between restarts.

cp .env.example .env            # then set JWT_SECRET and SEED_DEFAULT_PASSWORD in .env
docker compose up --build -d    # app :8000 · mongo :27017 · redis :6379
docker compose exec app python scripts/seed_data.py

Then open http://localhost:8000 and log in with a seeded account (for example ananya@careflow.example) and the password you set as SEED_DEFAULT_PASSWORD.

docker compose down stops everything; add -v to also delete the data.

Public demo (Render, free)

Live at https://careflow-uam3.onrender.com. Every demo account's password is careflow-demo-1:

Role	Email
Patient	ananya@careflow.example
Doctor	asha.rao@careflow.example
Admin (analytics page)	admin@careflow.example

The repository includes a Render Blueprint (render.yaml) that deploys the app with MongoDB Atlas and Upstash Redis, all on free tiers.

Render's free instance has 512 MB of memory and a tenth of a CPU, too little for the local speech and embedding models, so the demo runs lite:

	Demo on Render	Full app (Docker Compose)
Text chat	Gemini, with cache and rule-based fallbacks	Same, plus the embedding classifier
Voice A (Gemini Live)	✅	✅
Voice B (cascaded, local Whisper + Piper)	Switched off	✅
Memory used	about 110 MB	about 1.3 GB

Checked before deploying, with the same image held to 512 MB and a tenth of a CPU against the real Atlas and Upstash databases:

• A cold start's first login took 2.1 s (19 s before the database connection moved to start-up).
• Chat took 1–3 s, and a repeated hospital question 1.1 s from the cache.
• Voice A's first audio arrived 2.8 s after speaking.
• The very first voice session after a start is much slower.
• Render's free plan has no fixed IP address, so the Atlas IP access list allows 0.0.0.0/0; the database user's password is what protects it.
• The free instance also sleeps after 15 minutes without visitors and takes about a minute to wake: open the demo a minute before showing it.

Optional — fully offline LLM fallback

docker compose --profile offline up -d ollama
docker compose exec ollama ollama pull <small-instruct-model>

Key environment variables

Variable	Purpose	Example
LLM_PROVIDER	Which LLM backend to use. mock means no model: CareBot answers from rules only	gemini · mock
GEMINI_API_KEY	Free key from Google AI Studio	—
VOICE_PIPELINE_DEFAULT	Pipeline used on first load	live · cascade
STT_PROVIDER / TTS_PROVIDER / VAD_PROVIDER	Which speech components to use; the mock ones need no model	faster_whisper / gemini / silero
WHISPER_MODEL	faster-whisper model size	small · base
VAD_SILENCE_MS	Silence before end-of-speech	500
GEMINI_BREAKER_THRESHOLD	Failures in a row before the breaker opens	3
GEMINI_LIVE_MODEL	Live model for Pipeline A; empty means voice always uses Pipeline B	—
LIVE_GUARD_HOLD_MS	How long Pipeline A holds reply audio so its transcript is checked first	300
GEMINI_BREAKER_COOLDOWN_S	Seconds Gemini is skipped	60

Model IDs live only in .env, never in code — free-tier model names change.

---

👥 Roles

Role	How created	Can do
🧑‍⚕️ Patient	Self-registration	Book / reschedule / cancel, live queue, CareBot, own history
👨‍⚕️ Doctor	Seeded only	Queue controls, delay broadcast
📊 Admin	Seeded only	Department load, latency & analytics page

🌐 API endpoints

Method	Endpoint	Auth	Description
POST	/api/auth/register	—	Register a patient
POST	/api/auth/login	—	Login (all roles)
GET	/api/auth/me	JWT	The logged-in user
GET	/api/appointments/availability	JWT	Open slots by department / doctor / date
POST	/api/appointments/book	Patient	Book a slot (409 if taken)
PUT	/api/appointments/{id}	Owner	Reschedule / cancel
GET	/api/appointments/me	Patient	Own visit & prescription history
GET	/api/appointments/queue/{doctor_id}	JWT	Queue status
PUT	/api/appointments/{id}/status	Doctor	Update consultation status
GET	/api/doctors/{doctor_id}/schedule	JWT	View schedule
POST	/api/doctors/status	Doctor	Broadcast delay / availability
GET	/api/admin/stats	Admin	Department load
GET	/api/admin/analytics	Admin	Latency, intent mix, fallback rate
GET	/api/hospital/config	—	Public hospital info
POST	/api/chat	JWT	CareBot text turn (rate-limited)
GET	/api/chat/{session_id}/workflow	JWT	Where the caller's booking requests in that conversation have got to
WS	/ws/voice?pipeline=live|cascade	JWT	CareBot voice stream

---

🗺 Roadmap

	Milestone	Scope	Tag
✅	M1 — Foundation	Repo, Docker Compose, CI, pre-commit, models, auth, seed data, demo banner	v0.1.0
✅	M2 — Appointments & real-time	Availability, unique-slot booking, history, authenticated Socket.IO queue, delay broadcast, patient and doctor screens	v0.2.0
✅	M3 — Agents & Pipeline B	Intent classifier, guardrails + tests, eval set, specialist agents, text chat in the app, cascaded voice pipeline with barge-in, voice in the app ✅	v0.3.0
✅	M4 — Pipeline A & reliability	Gemini Live pipeline, failover chain + circuit breaker, Redis workflow tracker, latency bench, 8 kHz WER test, load test	v0.4.0
✅	M5 — Analytics & demo	Analytics page ✅ · live demo ✅ · benchmarks write-up ✅ · "hardest problem" write-up ✅ · demo GIF ✅	v0.5.0

🔭 Out of POC scope

• Telephony channel (Twilio / Exotel) with outbound reminder calls
• Real-time Indic TTS on GPU and Indic-specific STT models
• Multiple Uvicorn workers with the Socket.IO Redis manager
• Refresh tokens, audit log of records access, appointment reminders
• Doctor schedule editing, complaints, password reset
• Compliance work (DPDP Act, HIPAA-style controls) and a paid LLM tier

---

CareFlow is a proof of concept, not a medical device. It does not diagnose, prescribe or replace a doctor.

📄 Full specification: PRD.md

Built by Rahul Bailur