Clinical practice, deliberate practice, and “big data”
A couple of weeks ago I was carving the Sunday roast when the knife slipped and I cut the knuckle of my left forefinger. No big deal. As a chronic do-it-yourselfer, I’m always cutting myself. I bandaided it and carried on.
The next day we went up to the cottage for a few days. Two days later the finger was in serious trouble. The cut was continuously oozing pus, the finger was swollen, and it was Viagra stiff. After a day of this, I decided to go into town and get help. The local pharmacist recommended Epsom salts and topical antibiotic. Figuring that’s all she could do, and being somewhat suspicious of the healing power of Epsom salts, I decided to take the big step and go to the local ED at the Cottage Country Hospital (not to be confused with Cook County Hospital). After a couple of hours, I saw the doctor, who recommended Epsom salts too. When I asked why, he said, “It draws the poisons out”. Well, “draws the poisons out” is something my mother in law might say, but is unlikely to appear in a microbiology text. So over his objections, I persuaded him to write a script for Keflex, but he extracted a promise that I would go with just Epsom salts for a day.
Two days later, I was back home, having faithfully soaked 6 times a day, (and popped the pills). The finger looked the same. Fearing flesh-eating disease, I phoned my family doc (on a Sunday morning) and made an appointment with the doc on call. She took a look, and immediately her expression went very serious (kinda like a judge pronouncing a death sentence). She said she had experience with this kind of injury and informed me that I had “bacterial flexor tenosynovitis”, that if it was not controlled by medicine quickly they may have to operate, that it could travel up the tendon into my arm, and that I could permanently lose movement in that finger. I was discharged with a prescription of a new antibiotic 450 mg. 4 ×/day with the admonition that if it was not dramatically improved by the next morning to go straight to the ED. Amazingly, the weeping stopped in 2 h, and movement began to return in 5 h. However the next morning some seeping had started again.
So does this improvement make it to “dramatic”? Being a complete chicken, I went to the ED. The Emergency physician took a look, said, “You don’t have tenosynovitis. Not even close” but then gave me a 3 day course of ambulatory IV antibiotics.
All is well now. I still have my finger. It works again. And I’ll live happily every after—at least for a while. I ended up with 9 health professional contacts in 7 days and a cost to the system in excess of a thousand dollars. But I have no illusion that things went a whole lot better as a result of effective drugs, and ultimately good care.
So what does all this have to do with medical education? Well, I’ve spent a lot of time studying medical errors, and suddenly I found myself experiencing it firsthand. As I reflected on the experience, it seemed to me that there are a number of possible lessons to be derived from the experience, ranging from micro to macro.
First, a forensic look at the process itself. Three physicians were involved. Their approach was drastically different, and led to both different diagnostic and management strategies. We do not know what doc 1 was thinking diagnostically, but his management could be interpreted as sub-optimal. He did the right thing in prescribing an antibiotic, but only at my insistence. As I understand it, Keflex is the appropriate first line antibiotic in this case. But how did he get the impression that Epsom salts was the right treatment?
For the second physician, the opposite is the problem. Her management was appropriate, although if she was that concerned it might have been better to send me direct to the ED without the step of judging a “dramatic” improvement, and also to get me on IV antibiotics the sooner the better. But for reasons I don’t understand, she overdiagnosed the problem.
So how do we frame these errors and what can we do about it? Well, if you look at the literature, there are only two possible mechanisms—knowledge deficits and cognitive biases (Norman et al. 2014).
How does this fit with what happened?
For the emergency physician, it is tempting to immediately come to the conclusion that, in advocating Epsom salts for a bacterial infection, he had a serious knowledge deficiency. But is it really possible that a practicing physician is unaware of basic concept in infection. That seems implausible.
One could, I suppose advance some bias or other (with 130 or so to choose from something must come close). But so what? Is it possible that his direct experience with infections and Epsom salts (and he commented several times on how he has used it himself in these situations) would overcome his formal training? Perhaps. But would coming up with a label for the bias have changed anything about the process?
For the family physician, the question is why she would overcall a cellulitis. Certainly the “Not even close” comment of the second emergency physician suggests a severe overcall. When speaking to me, she did emphasize that she had had experience with these injuries. I had the sense that she had a very dramatic experience in the past—for better or worse I know not. Perhaps the misdiagnosis could really be a result of a cognitive bias. One that comes to mind is not on the standard list; the “vividness” hypothesis described by Nisbett and Ross (“a single vivid instance has more weight than pallid statistics of far greater evidential power”) (1980). But there are two problems. First, we have to make some sweeping assumptions abut her cognitive processes. We are speculating that she had a very good or bad experience in the past. But we have no way of knowing whether this is the case. Indeed, there is a good chance that she herself could not articulate why she thought of tenosynovitis (Norman 2018). Second, give the imprecise way that cognitive biases are defined, there are likely half a dozen candidates for the primary bias—availability, premature closure, anchoring—and no way to rule in or out one or the other. Finally, even is we could achieve agreement on a most likely label, it is unclear if this would change things. Do we believe that if we persuaded her to run down a mental list of biases, she would have second thoughts? Some evidence suggests otherwise (Sibbald et al. 2018). However, in any case, her overcall and aggressive management certainly started the wheels in motion to prevent further serious complications. I do wonder what consequences would have ensued if I had left it a couple of days more.
In the end, I am left wondering if our present theories of diagnostic errors are incomplete. The errors I observed are hard to explain away as knowledge deficits. And while we might find a cognitive bias that sort of fits in hindsight, that has its own problems (Zwaan et al. 2016). And I have difficulty envisioning the physician, at the time, running down a mental checklist along the lines of “Did I commit availability bias?”, Did I commit representativeness bias?” And even if she did, would this lead her to consider cellulitis? It is perhaps not surprising that current strategies to reduce errors based on either mobilization of knowledge or identification of cognitive biases show very small benefits in most circumstances (Norman et al. 2014).
So how could we use these episodes of care more effectively as an educational intervention? One thing that occurred to me as the process unfolded was that each physician was acting in isolation. No feedback was provided. The first doc likely went home satisfied that he had cured another patient with Epsom salts and done his bit to stave off development of superbugs. The family physician had no reason to doubt her diagnosis, even though a careful examination (or as happened, a cursory examination by an expert) shows that the criteria for a tenosynovitis were not present—it was simple cellulitis. And her management, although perhaps too aggressive for a cellulitis, was perhaps too conservative if it really was a tenosynovitis.
In short, these practitioners, although highly experienced, were not provided the opportunity to maintain deliberate practice, and as Ericsson (2004) shows, this is a prerequisite to maintain expertise. While as described by Ericsson, deliberate practice has a number of elements, one critical feature is feedback. And feedback is notable by its almost complete absence in medical practice.
As my personal odyssey began, I had the naïve idea that in fact all my various contacts with health professionals had left a digital paper trail for all to see. After all, Ontario should be ideally placed to do so. We have universal health care for all 7 million citizens; we have all billing data on a central site—indeed many health care researchers have access to this huge data base. And I was dimly aware that the government had spent considerable sums (pushing a billion dollars) to create a comprehensive and standardized e-health record. So how hard could it be to send the entire transcription of this episode of care to the practitioner involved so they could see where their judgment fit into the overall picture and learn from the feedback?
Well, I was living a pipe dream. It is not hard; it’s impossible. After spending all those millions, the government walked away from the system and it’s now a dead duck. Individual institutions have various e-health platforms; the hospital where I visited the ED has a state of the art system that cost them $70 million for just one hospital.
I assumed that at least my family doc would get all the documentation, and it would just be a case of distributing it to all actors. Not so. While he did get a note from the family doc and from the ED, nothing arrived from Cottage Country Hospital. And to add insult to injury, the communications are done via Fax machine, so are not readable or easily indexable, and often arrive once a month in a stack.
So, like fusion power, quantum supercomputing and artificial intelligence, the idea of a seamless electronic system to track everybody’s health remains a fantasy of journalists. We don’t have access to “big data”; all we have are piles and pile of unconnected and incompatible “little data”.
By the way, although Canada is at the forefront in implementation of competence-based education thanks to initiatives of the Royal College, it too shares similar problems. As I understand it, they attempted to develop a common data base to collect assessment data from all programs, but the product was unworkable. Faced with tight timelines for implementation, individual programs have gone their separate ways. So not only is there no way to access uniform data across institutions and programs, even within McMaster different programs are using different systems (including things like Google Forms), and there is no uniformity.
But let us now enter into the risky game of futurology. Perhaps one day, not in my professional lifetime, it really will be possible to have a patient’s experience with the health care system accessed electronically and easily. This really does present a unique educational opportunity. Just suppose we could make the entire record machine-readable and then get the computer to identify episodes of care where there were important changes along the way in diagnosis or management. And suppose these were selected and then sent to all practitioners involved, highlighted with red ink, with the appropriate privacy checks so no one ended up in a malpractice court. Seems to me this would be the kind of feedback that could really alter behaviour.
And are all these “suppose’s” unrealizable? Maybe not. There are at least two obstacles: (1) extracting information from free text (which is very different from the keystrokes and clicks that Facebook uses to violate our privacy). But qualitative researchers already have software that is pretty good at dealing with verbal information. (2) Identifying episodes of care containing potentially problematic issues. But it seems to me this could be a not very difficult application for AI approaches.
So there you go. Maybe, just maybe, an opportunity to extract educationally meaningful information economically from actual care, and actually impact on medical errors. Finally! And more important, a strategy to provide meaningful and timely feedback based on real clinical performance.
Gotta run. I’m off to the patent office.
- Ross, L., & Nisbett, R. (1980). Human inference: Strategies and shortcomings of social judgment. Upper Saddle River: Prentice-Hall.Google Scholar
- Sibbald, M., Sherbino, J., Ilgen, J.S., Zwaan, L., Blissett, S., Monteiro, S.D., & Norman, G. R. (2018). Debiasing versus knowledge retrieval checklists to reduce diagnostic Error in ECG interpretation. Submitted to Academic Medicine, 2018.Google Scholar