Hieu "Ray" Bui

Can a language model predict how doctors actually prescribe medication, just from a patient's basic demographics and diagnosis? I've been testing this question against real clinical data from OCHIN, a network of community health centers, across two benchmark tasks covering 896+ patient demographic groups, three open-source models spanning different architectures and parameter scales, and three tiers of prompt guidance.

The core finding: even with detailed clinical context, LLMs consistently fall short of a naive baseline that simply predicts the population average. Traditional machine learning models, trained directly on the real data, do better. But the gap isn't about clinical reasoning; the models correctly identify which drugs are generally more or less common. It's about grounding: without access to the actual dataset, general medical knowledge alone isn't enough to predict fine-grained, group-specific patterns.

Along the way, I built and debugged the full inference pipeline from scratch on SCU's WAVE HPC cluster, running open-source models through Ollama, and caught a subtle class-mapping bug that was silently corrupting results before it reached evaluation.

This work is ongoing — the next phase expands testing to an 18-model roster spanning different architectures, sizes, and reasoning capabilities, and moves toward individual patient-level prediction.