Smart Software, DRGs, Treatment Chains, and Money. Yep, Money.
June 12, 2026
Another dinobaby post. No AI unless it is an image. This dinobaby is not Grandma Moses, just Grandpa Arnold.
I read “Inside the Accountability Vacuum: Why Clinical AI Errors Have No Owner.” This is an “inside baseball” type of write up. However, if you are into medical fraud innovations, you may find the article food for thought. I want to hit the highlights of the report from my dinobaby perspective, of course. Then I want to outline how the baked in methods of a medical software might allow someone to put billing before patient well being.
The main point of the story is that when an AI module does something as part of the diagnostic process, it may be difficult to pinpoint why something went off the rails. One example: An X-ray interpretation of granny’s lungs output a probability that suggested she had a problem and provided probabilities for treatments. Ooops. Granny died after a series of “actions” were implemented by humans relying on the probabilistic outputs of the smart software in the diagnostic and treatment chain.
Good enough, Midjourney. Hey, do you output DRG treatment chains too?
Yeah, it happens. And, because I am a dinobaby, I think it will happen more often.
The cited article says:
Clinical AI is no longer speculative, no longer the next thing, no longer a topic for a panel discussion at a digital health conference. It is already embedded in care. The question is no longer whether it will arrive but whether the institutions that deploy it can evaluate it honestly once it has.
The long article asserts:
By the early months of 2026, the United States Food and Drug Administration had authorized more than 1,350 AI-enabled medical devices, roughly double the figure from 2022. The technology is propagating into clinical workflows on three continents simultaneously, and the institutions tasked with policing it are still drafting the rulebook in public.
After a romp through some baseball type information, the article reports:
The Stanford-Harvard report’s central anxiety is not that clinical AI is bad. It is that nobody yet knows how to tell when it is….A model that performs flawlessly at one teaching hospital can quietly degrade at a community hospital ten miles away because the patient population is different, the equipment is older, or the implementation team configured the alert thresholds in a slightly different way.
I have a healthy skepticism for information from both of these estimable certifying institutions. However, I do want to mention that both outfits have been linked with made up information. Yep, research fraud is everywhere folks.
Two examples of smart software fancy dancing appear in the cited article. The first is the old chestnut about Watson’s cancer foibles. The other is from Epic Systems. I think the anecdote is indeed epic. I quote:
The Epic Systems sepsis prediction model is the more instructive one. Documented in a series of investigations published from 2021 onwards, the Epic Sepsis Model had been deployed across hundreds of American hospitals when an independent external validation by researchers at the University of Michigan, including the work of Karandeep Singh, found that the model missed sixty-seven percent of sepsis cases and that eighty-eight percent of its alerts were false positives. Epic had claimed accuracy of between seventy-six and eighty-three percent. The independent figure was closer to sixty-three. What made the Epic story matter was less the performance gap than the institutional dynamics it revealed. Hospitals had bought a tool, in some cases under financial incentives that included payments of up to a million dollars to use the algorithm, without seeing an external validation study. Clinicians had spent months responding to alerts that turned out to be wrong most of the time, building up the very automation fatigue that ECRI now warns about. By October 2022, Epic had overhauled the model and was recommending that hospitals retrain it on their own patient data before clinical use, which is itself an admission that the original product was not fit for the purpose for which it had been sold.
The cited article tiptoes into the question of killing granny as malpractice. I quote:
Talk to a medical malpractice plaintiff’s lawyer about AI cases, and the conversation eventually arrives at a particular kind of frustration: the audit trail that does not exist. … When AI sits in the chain of decisions, that reconstructibility starts to break down. The first is technical: many of the models in clinical use, particularly those based on deep neural networks, do not produce outputs whose reasoning can be inspected after the fact in any meaningful sense. There is no chart of inferences. The model produced a probability, and the probability turned into a flag, and the flag turned into a recommendation, and the recommendation either was or was not heeded.
I am certainly no medical professional. I am definitely not a legal eagle. I am 82, and I view health care with some skepticism. My mother told me when I was in college, “Stephen, you know you could have lost your leg when you had osteomyelitis.” Okay, that was a surprise. I still have two legs and both seem to work okay.
I do have some observations:
- Smart software embedded in “diagnostic workflows” can be incorrect. Busy people may miss the bad outputs. People can lose their leg. (See above.)
- The outputs from embedded software may recommend high payoff treatments; that is, the incentive to fiddle with data can translate into more money.
- Overworked or under trained staff can just accept what the smart software outputs. Fatigue, problems at home, doom scrolling, whatever can distract the health care human. Granny? Hasta la vista.
- Regulations in the US and elsewhere are not in step with smart software applied to health care. When a fraud or treatment issue arises, the rules output will, by definition, be reactive and inapplicable to those who create smart systems and are quite adaptable to guard rails.
Net net: When did IBM Watson make its capabilities known to cancer docs in Houston? I think it was 2012 or 2013. That was more than a decade ago. That tells you something about regulations, guard rails, oversight, and medical fraud opportunities.
Stephen E Arnold, June 12, 2026
Comments
Got something to say?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
