If you have been following the USMLE Pass/Fail debate, you should know that the final battle is upon us. The InCUS is nigh.
WHAT THE HECK IS THE InCUS?
It’s the Invitational Conference on USMLE Scoring (InCUS), to be held in Philadelphia, PA, on March 11-12, 2019.
WHAT’S AT STAKE?
Why, nothing less than the very future of the USMLE.
Seriously. As has been done in the past, the score reporting policy for the USMLE exam is being reviewed. Should the test be scored, or results reported pass/fail?
Those attending the meeting will consider this issue and make a recommendation that will be presented to the USMLE Management Committee this fall.
WHY DO YOU CARE?
Because I care deeply about medical education – and feel strongly that we’re headed down the wrong path.
Step 1 Mania is rotting medical education from the inside out. In an era when medical information is more readily accessible than it’s ever been, we prize the results of a multiple-choice memorization test more than ever before. Why?
For the first few years that I was involved in medical education, I assumed that the best thing I could do for my students was to help them succeed within the broken system, so I focused my efforts on helping them excel on Step 1. Without any analysis or introspection, I just assumed “Step 1 Mania” was something we were stuck with.
I had a Popeye moment a few months ago, right after I read the infamous “Netflix/Instagram” commentary from the presidents of the NBME and FSMB. The more I investigated the drivers and harms of Step 1 Mania, the more I felt that something has to be done about it.
The InCUS is our chance to do that something.
SO WHO’S GOING TO InCUS?
Only people who were invited. Remember, the “In” in InCUS stands for “invitational.”
OKAY SMARTY-PANTS, WHO’S INVITED?
Beats me. The invitation list, or how invitees were selected, has not been publicized by the conference’s sponsors. (UPDATE 3/11/19: A list of conference attendees is available on the InCUS website.)
(You all may not believe this, but I was not invited.)
WHO’S SPONSORING InCUS?
The conference is sponsored by:
- The National Board of Medical Examiners (NBME)
- The Federation of State Medical Boards (AMA)
- Association of American Medical Colleges (AAMC)
- Educational Commission for Foreign Medical Graduates (ECFMG)
- American Medical Association (AMA)
DO ANY OF THE SPONSORS HAVE FINANCIAL CONFLICT OF INTEREST?
Yes.
WOULD YOU CARE TO ELABORATE?
Sure.
The NBME. No one has benefitted more from the rise in Step 1 Mania than the test’s primary sponsor. The NBME now generates more profit from its ancillary services than it does from the USMLE itself. Without pressure for higher and higher Step 1 scores to drive the market, the demand for those lucrative product lines is likely to shrink considerably.
The FSMB. Co-sponsor for the USMLE program. When a pass/fail USMLE was considered in 2000, it was an issue with state medical boards that ultimately sank the proposal. Turned out that several states could not accept a non-scored USMLE without a change in their medical practice acts – so making the test pass/fail risked opening the market for a competitor. And nothing would be more financially devastating to the USMLE’s sponsors as loss of their monopoly. So far, I have been unable to confirm whether this may still be an issue today. According to the FSMB website, all states require only USMLE passage – not scores. However, I have written before about the potentially profound implications should even a few state boards refuse to accept pass-fail scoring.
The AAMC. The AAMC owns the Electronic Residency Application Service (ERAS). As the number of residency applications per student have grown rapidly in recent years, ERAS has become the AAMC’s major revenue stream. However, Application Fever needs a scored Step 1. Without an across-the-board metric to use for screening, the increases in applications can’t be sustained, and growth in the AAMC’s largest revenue stream will cease.
The ECFMG. In 2017, the ECFMG had program revenues of $72.8 million dollars. Of that, $62.2 million came from the USMLE and ERAS. I don’t know the details of those contracts… but when 85% of your revenue comes from the NBME and AAMC’s products, it’s hard to imagine the ECFMG intentionally biting the hands that feed it.
The AMA. Even the AMA – the group intended to advocate for all physicians – has some skin in the Step 1 Mania game. According to their IRS Form 990, the AMA received $14.8 million in revenue from advertising in 2017. (Not much compared to the other organizations, but to put it in perspective, that’s almost twice as much as they received from “educational programs” and journal reprints combined.) Some of those advertising dollars come from Kaplan, who get product placement in all of the AMA’s medical student articles, such as the one below.
COME ON, THESE ARE NON-PROFIT ORGANIZATIONS! WHY DO THEIR FINANCES MATTER?
Non-profit corporations don’t distribute profits to shareholders, and they don’t pay income tax. But non-profit and for-profit corporations have one extremely important thing in common: the executives and employees of non-profits are human beings.
Non-profit corporations aren’t staffed by some disinterested and altruistic hominid superspecies – they’re they same kind of Homo sapiens as the rest of us. They care about regular things, like getting a raise, pleasing their boss, getting a promotion, keeping their job, acquiring prestige, etc.
Money matters.
SO YOU DON’T TRUST OUR BENEVOLENT OVERLORDS?
I trust them to act in their own interests.
BUT SURELY THE INVITED GUESTS WILL PUSH BACK AGAINST THE SPONSORS’ CONFLICT OF INTEREST AND ENSURE A FAIR PROCESS, RIGHT?
I honestly do not know even a single person who has been invited to InCUS. So what I am about to say is directed at no one in particular – but I’m sure that everyone was invited for a reason.
If you were invited to InCUS, maybe it’s because of important research you’ve done your the field. Maybe it’s because of a leadership position you hold, or a group you represent. Maybe it’s because you have a well-earned reputation as a key opinion leader in medical education or residency selection.
These are all good things, and the organizers are right to include you for such reasons.
But…
…there may be other reasons that you were invited. Maybe it’s because the sponsors perceive you as being more likely to nod your head and endorse their arguments than to ask tough questions. Maybe they want to use your name and reputation to legitimize the policies that line their pockets.
If that suggestion offends you – it should.
Don’t kill the messenger – but please remember how much money is at stake. You’ll be rubbing elbows with executives who earn million-dollar salaries. If InCUS were a card game, the sponsors have a lot of chips on the table. And in the words of Paul Newman, “If you’re playing a poker game, and you look around the table and can’t tell who the sucker is, it’s you.”
SO WHAT SHOULD INCUS ATTENDEES DO?
Don’t accept outright the trite old arguments about why we need Step 1 scores. Demand to see the data. Ask the tough questions. Insist that the sponsors declare their financial conflicts when they argue for the status quo. Do your part to ensure that the debate is honest and data-driven and representative – regardless of the final decision.
Look, I think we know how the debate will unfold. We’re gonna hear the same arguments that have been put out for years.
Below is how the conference organizers frame the issues in advance of the debate:
(For the moment, let’s ignore the subtle psychology in play here: instead of framing the issue as “Scored vs. Pass/Fail”, it’s framed as “Numeric Scores,” and the alternative is only suggested by the “Cons” subheading.)
For now, let’s just focus on the arguments.
DO THE ARGUMENTS IN FAVOR OF A SCORED USMLE STEP 1 HOLD UP UNDER SCRUTINY?
You be the judge.
Argument #1
Not everything that is highly reliable and objective is meaningful. Where are the data that show that Step 1 scores predict the things we really care about? As Step 1 Mania consumes more and more time and focus, this becomes a question that can’t be ignored.
I’ve made the argument that if we asked medical students to memorize digits of pi, that test would take on the same meaning and importance as Step 1 does in the current system (and would be even more reliable and objective!).
Argument #2
Reliability is an important feature of a test… but it is not the most important goal in and of itself. I’ll take meaningful or important over reliable.
I understand the concerns over medical schools’ assessments. Many medical schools have changed to pass/fail curricula in response to Step 1 Mania, or now use only NBME assessments (like CAS) for student testing. But if the problem is the inadequacy of medical school assessments, is the solution to rethink about the way we’re doing assessments, or is it for students to spend more time memorizing basic science trivia because it’s more reliable?
There are legitimate concerns here that aren’t solved by just by outsourcing everything to the NBME.
Argument #3
Totally concede that one. You can drop the word “may” – a scored Step 1 absolutely stimulates more preparation.
The real question is, what value does increased student preparation bring?
Much of the knowledge gained in Step 1 prep is transient. And yet, students spend more time in Step 1 prep than they do in any course, clerkship, or rotation in all of medical school.
What is the return we are getting on our massive investment in Step 1? Whatever it is, it needs to be something greater than just an easy screening metric for residency applications.
If you’re interested, I calculated the intellectual opportunity cost of Step 1 here. (Spoiler alert: it’s large.)
Argument #4
Of all of the arguments advanced in support of maintaining a scored Step 1, this is the most disingenuous.
What are these other “valued measures”? Please, don’t buy this argument hook, line, and sinker. Ask to see the data. You may be underwhelmed. I was.
Here is a post where I pulled the references for the NBME’s claims that Step 1 correlates with “improved practice” and state disciplinary board actions. They’re flimsy.
And here is one where I looked at the data on the relationship between Step 1 scores and board certification.
Argument #5
Does Step 1 “level the playing field”? I doubt it.
I’ve reviewed how Step 1 scores likely disadvantage under-represented minorities in medicine. I’ve also suggested that Step 1 scores do more to perpetuate disadvantage than “level the playing field” for IMGs and ‘lower tier’ U.S. graduates.
Argument #6
In considering this argument, I would like to remind everyone of one thing. InCUS will consider the United States Medical Licensing Exam. It is not the United States Residency Selection Exam, even though we use it that way.
Read the mission statements of the NBME, FSMB, AAMC, ECFMG, and AMA. Know how many have residency selection policy as a core mission?
Zero.
No voice ever called out in the wilderness for the NBME to come and bestow upon us gifts of residency selection metrics. It just kind of ended up that way.
But now that Step 1 has become our de facto residency entrance and vocational aptitude exam, these organizations have a vested interest in maintaining their position. Just remember, this is not what the test was designed to do – and even though it may function better than nothing, to act as though that’s our choice here creates a false dichotomy.
Why are we so sure that Step 1 is the best we can do? What makes us so certain that an instrument that we specifically designed to facilitate residency selection would function worse than a test that got grandfathered into such a role? There are people doing good work in this space now – and the biggest impediment to their work is our collective addiction to Step 1 scores.
Honestly, what makes me saddest about InCUS is the fearmongering.
To residency program directors: “You know applications are rising… how will you ever screen your applicants without our test?” To the most disadvantaged residency applicants: “You’ll be even more disadvantaged without Step 1.” At best, these are short-sighted arguments. At worst, they’re transparent attempts by those with financial interest to engage others to fight a proxy war.
Why does the fear of “what fills the gap” hold such sway?
Am I wrong? Am I too optimistic? It is that far-fetched to believe that our best and brightest could come up with a better system for residency selection? I don’t think so.
If you’re like me, and you feel like we’re on the wrong track – then now is the time to do something about it.
There is no natural end to Step 1 Mania. We are in a self-amplifying loop. If we choose to do nothing, then what do you think medical education will look like in 5 years?
I say the time to act is now – and it starts tomorrow at the InCUS.
The Sheriff of Sodium 