AI in clinical research: assist with discovery, never decide alone

A researcher types a sentence into a search box: women over forty with bladder pain syndrome, no concurrent endometriosis diagnosis, at least two years of records. Twelve seconds later a draft cohort definition appears — inclusion criteria mapped to standard codes, a washout window, an estimated count.

The demo is genuinely impressive. But the question that decides whether this is progress or malpractice never appears in the demo: what, exactly, is the language model allowed to decide?

Where language models genuinely help

Language models are good at one thing that clinical research has always been starved for: translation between human language and structured artifacts. That single capability covers a surprising share of research drudgery.

• Cohort discovery. A plain-English question becomes a draft computable definition — concepts mapped to vocabularies, time windows made explicit. The drafting is the help; the executing is not. How a question becomes a dataset is its own discipline, covered in what is a research cohort.
• Coding messy text. Clinical notes and patient-reported histories arrive as prose. Models are strong at proposing candidate codes — this phrase looks like a SNOMED concept, that one like a medication — for a human or a validated mapping to confirm.
• Protocol and document drafting. First drafts of study summaries and plain-language materials, where a person reviews every sentence before it matters.
• Literature triage. Sorting a thousand abstracts into probably-relevant and probably-not, a task where errors are cheap and recoverable.

Notice the common shape: high volume, language-shaped, and always followed by verification. A wrong draft costs minutes of a reviewer's time and nothing else. That is the right blast radius for a probabilistic tool — and the test to apply to any proposed use: ask what the worst plausible output costs, and who absorbs the cost.

Where AI must not decide alone

Anything that changes what happens to a person or their data is not a drafting task. Eligibility determinations, dataset releases, consent interpretation, privacy gating — these are decisions, and language models are the wrong instrument for decisions, for four compounding reasons.

First, nondeterminism: the same question can return different answers on different days, and a decision that cannot be reproduced cannot be governed, appealed, or even meaningfully described in a protocol. Second, confabulation: models produce fluent wrong answers with the same confidence as right ones. Third, no replay: an audit must be able to re-run a decision a year later and get the identical result, which a sampled output cannot promise. Fourth, accountability: "the model decided" is not an answer an IRB, a regulator, or a patient should ever have to accept.

Regulators have been converging on the same conclusion from different directions. The WHO's guidance on ethics and governance of AI for health puts human oversight and accountability at the center; the FDA has been working through how AI fits into drug development and device oversight with a consistent emphasis on transparency and a defined human role; the OECD's AI principles make accountability a first-class requirement rather than a nice-to-have. The vocabulary differs. The direction does not: someone — or something deterministic — must own the decision.

The pattern: AI proposes, engines dispose

The architecture that resolves the tension is old-fashioned and unglamorous: let the model propose, and let versioned deterministic logic dispose.

The dividing rule fits in one sentence: model output may be the input to a decision, never the decision. A deterministic engine — explicit logic, versioned like code, tested like code — gives research the three properties models cannot: the same answer every time, a test suite that proves it, and a record that lets an auditor replay it. This is the same argument for computable phenotypes over ad-hoc queries, extended to everything in the decision path. And every proposal, execution, and refusal should land on the same append-only audit trail, so the assist itself is inspectable.

Why locally hosted models matter

Sending a patient record to a third-party cloud API is not a technical detail — it is a disclosure, with its own contracts, its own cross-border questions, and its own per-query exhaust accumulating in someone else's infrastructure. Every prompt that contains protected health information widens the boundary that has to be defended.

A locally hosted model collapses that boundary back to where it belongs. The prompt and the record never leave the platform; no vendor sees the queries; nothing about a patient's history transits a network on its way to being summarized. Locally hosted models are smaller and less capable than frontier cloud models — and for the assist role, that is an acceptable trade, because drafting queries and suggesting codes does not require frontier capability. It requires competence inside a boundary. (Development and testing of the surrounding software, meanwhile, can run on synthetic records and touch no one at all.)

This is how Mithras Health draws the line, as architecture rather than policy: locally hosted AI assists discovery, and deterministic engines decide. A researcher can explore in plain language, but what executes is versioned cohort logic over consented records; what gates release is a hard k ≥ 5 floor that no model — and no person — can configure away, part of the platform's non-configurable invariants; and every assist, execution, and refusal is written to an append-only, cryptographically anchored audit trail with six-year retention.

The interesting question about AI in clinical research was never whether the model is smart enough. It is whether the decision can be replayed: same inputs, same rules, same answer, any year an auditor asks. Models that draft and engines that decide can pass that test together. Neither can pass it alone.