English | 繁體中文

Medical EHR Agentic AI (MedAgent-EHR)

A Self-Hosted Clinical Agent That Plans, Acts and Checks Itself in a FHIR EHR

Unlike "one question, one answer" chat AI, this is an agentic AI: the LLM plans its own steps, decides which tool to call, performs clinical tasks in a standard FHIR EHR, and self-checks before writing to the medical record — fully self-hosted on a local GPU, with no cloud LLM API.

GitHub Repo

Design Concept

Project Overview

Architecture

MedAgent-EHR architecture: Planner → Executor → Verifier → FHIR Web demo run: Plan and step-by-step ReAct timeline, including a blocked direct commit

An actual web-demo run (record a blood pressure): a two-step Plan → find_patient locates the patient → stage_write passes Verifier review and writes to FHIR → the executor's attempt to call commit_write directly is blocked by the program-side gate (red block) → done. Both protection layers — the LLM Verifier and the program-side gate — are demonstrably active.

Tech Stack

CategoryStack
LLM (self-hosted)Gemma-4-26B-A4B-it (QAT Q4, Google) via llama.cpp on an RTX 4090; also validated on Qwen3.6-35B-A3B
Agent architecturePlanner / Executor (ReAct) / Verifier
Tools & dataMCP (fhir-mcp-server), HAPI FHIR R4
Write safetyTwo-phase stage → commit, Verifier gate, medical-code / dosage validation
EvaluationMedAgentBench official refsol grader (300 tasks, 10 categories)
Deployment / toolsFlask (web demo), Docker, GGUF quantization, Python

Key Results

MedAgentBench score progression 124 → 265 Overall success rate comparison across models Gemma final score 263/300 with per-category results

Actual benchmark output: the deployed Gemma model over the full 300 tasks, official grade 263 pass (write tasks task3 / task8 both 30/30; read task task4 also 30/30).

Qwen score 265/300 with per-category results

Pre-migration Qwen3.6 score of 265 pass — compared with the chart above, the model swap kept the score nearly flat (265 → 263), evidence that the system is decoupled from the model.

Local llama-server serving the Gemma model

Fully self-hosted: a local llama-server loads gemma4-26b-a4b — no cloud LLM API involved.

Honest note: this 88% reflects the performance of an agent system aligned to this benchmark, not a pure comparison of raw model capability; being able to distinguish "system performance" from "model capability" is itself part of this project's engineering value. task10 (conditional + [value, timestamp] special format) is a known output weakness of the deployed Gemma model; after investigation confirmed it is a model-level limitation, the choice was to keep the honest score rather than patch a single benchmark task.

Serving Performance (single RTX 4090)

MetricMeasured
Generation throughput (single stream)82 tokens/s (median of 5 runs)
Aggregate throughput (4 concurrent)119 tokens/s (4/4 succeeded)
Time to first token (TTFT)2.4 s
Total latency, 256-token response5.5 s (median)
GPU memory while serving20.1 / 24 GB (whole card, incl. KV cache)
Model footprint~14 GB (26B MoE, QAT Q4 GGUF)

Measured against llama.cpp llama-server (OpenAI-compatible API) with a streaming probe included in the repo (benchmark/llm_perf_probe.py). The takeaway: a single consumer GPU serves a 26B model to multiple concurrent users at interactive latency — deployable where medical records must never leave the premises.

Task efficiency (full 300 tasks): 3.6 steps per task on average (median 4) — read/query tasks ~3.4 steps, write tasks 4.2, at most 10 steps for a single task. The agent converges well inside the max_steps=25 guardrail, so its bounded autonomy has a controllable cost.

llm_perf_probe measured output: 82 tok/s, 119 tok/s at 4 concurrent, VRAM 20.1/24GB

Measured output: 5 streaming runs + one 4-concurrent batch — the source of the table above.

Write-Safety Evidence (the Verifier gate)

Every resource written to the record must pass Verifier review before commit. The excerpt below is from a real execution trace (task8: an orthopedic referral order, completed in two steps) — the write is only released after all five structured checks pass; any failure sends it back for the agent to correct and resubmit:

"verdict": {
  "staged_id": "stg_209141be",
  "verdict": "approved",
  "reason": "The ServiceRequest correctly uses the provided SNOMED code for
             orthopedic surgery referral, targets the correct patient, includes
             the exact free-text note requested in the task, and sets the
             priority to 'stat' as specified ...",
  "checks": {
    "required_fields": "pass",
    "code_resolved":   "pass",
    "value_sanity":    "pass",
    "intent_match":    "pass",
    "safety":          "pass"
  }
}

The full trajectory (Plan → tool call → generated FHIR resource → verdict → commit) is reproducible with benchmark/run_single_task.py in the repo.

Verifier review log across four write tasks: every stage_write is judged before commit

Actual execution log: four consecutive tasks (two referral orders, one blood-pressure record, one conditional order) — every write passes Verifier judgment (with a concrete reason) before commit; task10_1, whose condition was not met, correctly produced no write at all.

Scope & Next

Development Journey

Not a straight line of successes — a chain of "hypothesis → verify → honest correction". Key nodes:

① All write tasks failed: the FHIR shape did not match the grader

The FHIR resources the agent produced did not match the shape the official grader expected. The fix: let tools accept the codes / values / times given in the task context and carry them verbatim into the FHIR resource (code passthrough) — task3 went 0 → 30.

② The Verifier rejected correct writes because it could not see them

The safety gate only received a one-line summary, not the full staged resource, so it wrongly rejected correct writes. Including the full FHIR resource in the staged summary let the Verifier genuinely review the content — task8 went 0 → 30.

③ Read tasks found no data: forced LOINC resolution

The tool force-resolved terms to LOINC codes while the dataset used local codes (GLU / MG / A1C / K). Adding a code parameter to get_observations to query with the task-given code verbatim, skipping resolution, improved read tasks sharply.

④ A score illusion: re-running only the failed tasks

Evaluating by "re-running only failed tasks" silently skipped tasks that had regressed, inflating the score (274 apparent vs ~249 real). From then on, only full 300-task runs count as the single source of truth.

⑤ After the Gemma migration, reasoning leaked into task10 answers

The model leaked reasoning / markdown into task10 answers, contaminating the output. Thinking was disabled the documented way and it still leaked, confirming a model-level limitation of Gemma-4-26B. Patching a single benchmark task would be gaming, so the honest score was kept.

task10 answers contaminated by model reasoning text

Evidence: task10 final_answer contaminated by leaked reasoning / markdown (e.g. '''''json way, reasoning_summary…), which breaks the official grader's exact match — a model output-layer limitation, not a system logic error.