Chirp chirp.
Did you hear that?
It’s the sound of crickets chirping. Other than the vapid and generic official statement, that’s about the only sound that came out of the USMLE’s Invited Conference on USMLE Scoring (InCUS), held earlier this week in Philadelphia.
For a conference addressing an issue that will affect all of us in medicine, why the silence?
It’s not accidental. Attendees have been asked to keep the conference workings confidential.
On the one hand, I get that. The folks at the conference should focus on communicating with and listening to each other first, and too much external discussion on such a polarizing issue may not further productive debate.
On the other hand, there’s a thin line between being confidential and being clandestine, or between being discreet and being muzzled. And in the context of InCUS – and the legitimate questions it raises about financial conflict of interest – secrecy is not a good look.
The fox is guarding the henhouse
Not long ago, I received a sincere and well-meaning e-mail from a faculty member who’d read some of my posts. After complimenting my research, he urged me to be careful with my language when discussing the finances of the NBME.
He meant well. He knew some of the NBME executives personally, and was concerned that my discussion of financial conflicts would be seen as a personal insult to their moral character. He suggested using phrasing such as “potential” or “possible” conflict of interest when discussing their personal or organizational finances.
I declined to do so. Here’s why.
A conflict of interest occurs whenever there are circumstances in which a professional judgment (like determining the best testing policy for society) may be unduly influenced by a secondary interest (like financial gain).
Note that financial conflict of interest is a condition, not a behavior. A conflict of interest exists regardless of the decision that’s ultimately made. And at InCUS, a financial conflict of interest exists. We don’t need to mince words.
Look, I’ve already discussed the significant financial COI for the sponsor organizations – especially the NBME and AAMC.
Approximately half of InCUS attendees were executives or employees (or former executives) of the sponsor organizations. Among these were 11 executives for the NBME, FSMB, or AAMC whose financial compensation was listed on their organization’s IRS Form 990 in 2017 (Schedule J, Part II, Column E).
The total compensation for just those 11 in 2017? $7,211,414.
And please bear in mind that for organizations like the NBME, the percentage of revenue devoted to executive/employee compensation is constant. Since 2001, executive compensation has been fixed at ~4-5% of NBME program revenue; employee compensation is ~30%. The bigger the revenue grows, the bigger everyone’s paycheck.
The other half of InCUS attendees represented other groups: deans, faculty, even a few medical students and residents. Early this week, I reached out to the USMLE to inquire about whether these attendees were responsible for their own travel expenses, meals, and lodging, or if those were covered the the conference’s sponsors. So far, they have declined to respond.
Folks, money matters. Financial incentives matter. Even just being wined and dined matters. (If you think it doesn’t, you haven’t read the Big Pharma playbook.) Pointing out a financial conflict of is not making an accusation of bribery – it’s highlighting the likelihood of unconscious bias.
Power corrupts – and there is nothing more corrupting than power exercised in secret.
In many areas of medicine, the remedy for a conflict of interest is disclosure. Be honest about what secondary incentives might bear on your primary professional obligation, and let others make up their minds.
Say, for instance, that I go to a scientific meeting and listen to a presentation about a new drug from an eminent researcher. If the speaker discloses that she has a financial interest in the sales of the drug, I’ll probably look more critically at their conclusions to be sure that I interpret the data the same way.
However, disclosure alone won’t work here. (And it’s not just because the NBME refuses to do it.)
The problem with disclosure is that it depends on the availability of a viable alternative.
See, when I listen to the pharmaceutical company’s presentation, I’m free to decide whether their secondary incentives may have biased their interpretation of the data. And if I think that they have, I can choose not to prescribe that medicine.
But the USMLE has a monopoly on licensure examinations for the students I teach, and participation is compulsory. There is no alternative.
The USMLE’s sponsors have total control over the medical licensing (i.e., “residency admission”) testing in the United States. They also have total control over who was invited to InCUS, and now, even what those people can say about it.
The “key stakeholders” invited to InCUS are not the only ones with a stake in these issues. I am a stakeholder. You are, too. And yet, the configuration and execution of the conference seem deliberately designed to maximize the influence of a select few. Closing the windows of the meeting hall so that sunlight can’t shine upon the workings within is both predictable – and deeply disconcerting.
So what’s next?
We wait for the InCUS conclave to show us a puff of smoke. Recommendations from the conference will be released in 6-8 weeks. There will then be a period for public comment before final recommendations are submitted to the USMLE governance, likely at the end of the summer or early fall.
My predictions
My guess is that we should look for the initial recommendations in mid-May – right when the collective attention of medical schools will be focused on graduation. (Wait, did you think InCUS was scheduled during Match Week by coincidence?)
And I expect that the recommendation will be to maintain the status quo.
This recommendation will be issued in the context of some strongly-worded statements of concern over the harms of Step 1 Mania, some somber-faced head nodding, and a call for “more research” into alternative methods of resident selection. Folks, we’ve seen it all before.
Thirty years ago, before the USMLE even existed, NBME executives issued a statement of concern about inappropriate use of NBME Part I scores in residency selection. For a sneak preview of what’s likely to come, I’d recommend this vintage editorial from former NBME President Robert Volle in 1988, which concludes thusly:
Whew! Good thing we got those special meetings convened to address the inappropriate use of standardized examination results! Things might have really gotten out of hand otherwise! I’d sure hate to be discussing this issue 30 years later... (snapping back to reality). Oh yeah.
Am I being overly pessimistic? I hope so. I’d be happy to be proven wrong by thoughtful and productive “deliverables” from InCUS that advance solutions instead of just legitimizing the status quo. Unfortunately, things seem to be playing out as I predicted earlier this week.
For those who care about this issue, I will do my part to keep you posted.
The Sheriff of Sodium 