Every CDSS post-mortem we’ve done in the last eighteen months has the same shape. The model passed evaluation. The integration shipped on time. Six weeks in, alert acceptance is below 12% and the clinical lead is asking whether to pull it.
The instinct is to retrain. Almost always, the right move is to fix the prompt surface — when the alert fires, what it asks the clinician to do, and how easy it is to dismiss versus engage.
Three patterns we keep seeing
- Asking for confirmation when you should be asking for a reason. A binary “Confirm / Cancel” trains the clinician to dismiss. A two-tap reason code (“Wrong patient” / “Already addressed” / “Disagree, see note”) gives you a feedback signal without adding clicks.
- Firing on the wrong screen. An alert that interrupts an open chart note has a different cost than one that lives in a side panel. We measure interruption distance — how far the clinician’s attention has to travel — and weight alert frequency against it.
- No acknowledgment of dismissal. If the system fires the same alert four times in a shift, you’ve built a low-grade adversarial relationship. Persist dismissals. Show the clinician you heard them.
The fix is mostly product, not science
This isn’t a model problem. The model is right often enough; the system around it is asking too much for too little. The fastest wins we’ve shipped have been:
- Reducing alert frequency by 60% by adding a 24-hour suppression window per patient/finding pair.
- Replacing modal dialogs with inline suggestions in the order entry sidebar.
- Adding a one-click “don’t show this for this patient” that the clinician’s lead can audit weekly.
None of those need an ML team. They need a product designer who has watched a clinician work through a busy clinic on a Tuesday afternoon.