In 2018, 21,611 examinees from MD-granting medical schools in the U.S. and Canada took USMLE Step 1. They didn’t have any choice. See, the USMLE is a licensing exam – so if you want to practice as an allopathic physician in the United States, you’ve got to take it.
But there were also 4,136 osteopathic medical students who took the USMLE. Exactly zero of those students were required to take the exam. In fact, they’re required to take a different test – the Comprehensive Osteopathic Medical Licensing Examination, or COMLEX – to graduate from medical school and obtain a medical license.
These students were taking the USMLE to improve their application for residency. Because, despite the consolidation of osteopathic and allopathic residency programs under a single accreditation system run by the ACGME, many residency program directors (PDs) are not comfortable using COMLEX scores to evaluate applicants. This is especially true for more competitive programs and specialties, and DO students know it: in one survey, 70% of osteopathic medical students recommended that future students take the USMLE to maximize their prospects of finding a residency.
Accordingly, osteopathic medical students have been taking the USMLE in increasing numbers, as shown below.
The number of DO students taking USMLE Step 1 has increased 13-fold since 1997. This rise is only partially explained by the increased number of DO students.
Taking two sets of licensing examinations isn’t cheap. Most DO students who take the USMLE take both Step 1 and Step 2 CK – meaning that they pay $1260 for the privilege of taking a licensing exam that won’t even help them get licensed.
Last year, the American Medical Association passed a resolution saying that DO and MD licensing exams should be viewed equally. This is an admirable move that seems likely to change nothing. So what should be done? How can we prevent osteopathic students from being fleeced on their licensing exams?
Let’s take a look at five options.
1. Make #USMLEPassFail
Do away with USMLE scores, and the number of DO students who choose to take the USMLE will instantly drop to zero. Trouble is, I don’t see it happening.
The idea of a pass/fail USMLE was considered at the March 2019 InCUS meeting, and although the final determination on score reporting won’t be publicly released for another few weeks, my reading of the tea leaves is that Step 1 scores aren’t going anywhere.
There are multiple reasons for this – including the one I pointed out here. But it’s also important to remember that the National Board of Medical Examiners has significant financial incentives to maintain score reporting. And this is one of them.
Last year, the NBME took in $4,399,320 from osteopathic students who took Step 1 and Step 2 CK. Think they’re eager to get by without that revenue?
Me neither.
In other words, if we want to keep DO students from getting fleeced, we’ve got to do something different. We’ve got to make COMLEX scores more interpretable for PDs who aren’t used to seeing them.
There are a few ways this could be done.
2. “Convert” COMLEX to USMLE scores.
Several authors have analyzed the relationship between COMLEX and USMLE scores. Probably the most authoritative analysis is this one, which included 1016 students at the College of Osteopathic Medicine of the Pacific who took both the COMLEX and the USMLE.
Unsurprisingly, there was a linear relationship between scores on COMLEX and the USMLE.
There is a strong linear relationship between COMLEX and USMLE scores (R2 = 0.696). Figure from Lee AS et al., J Grad Med Educ 2014. PubMed
The authors derived the following formula to “convert” COMLEX to USMLE scores:
USMLE Step 1 = (0.2392)*(COMLEX Level 1) + 82.563
Using this equation, a 600 on COMLEX Level 1 predicts a USMLE Step 1 score of 226.
Still, there are obvious limitations with this approach.
First, even though the correlation between COMLEX and USMLE is strong, it’s not perfect.
Look closely at the spread of data the scatterplot above. For a student who scores a 600 on COMLEX, the range of observed USMLE Step 1 scores goes from around 200 to over 250. (This is not surprising – even a student who takes USMLE Step 1 two times on consecutive days may have scores that differ by 24 points. But for all the PDs out there who mistakenly believe that the USMLE provides a perfectly precise prediction of an applicant’s aptitude for residency, such a wide range is problematic.)
Second, the data in the Lee paper were collected from 2006 to 2012. Although they provide a useful snapshot, they probably won’t produce valid predictions for contemporary test-takers. Why? Not only is the population of COMLEX/USMLE test-takers changing, but the mean score for USMLE Step 1 is also increasing by around 0.9 points/year.
USMLE “score creep”. Note the rightward shift of the score distributions from 2015-2017 to 2016-2018.
Score creep means that, the older the formula, the more likely it is to under-predict USMLE scores.
Take, for example, this paper from 2006, which also found a linear relationship between COMLEX and USMLE scores. But using their formula, a 600 on COMLEX Level 1 predicts a USMLE Step 1 score of just 211.
Instead, to ensure that PDs are evaluating current data, we could try something different. We could…
3. Encourage PDs to calculate COMLEX percentiles.
The COMLEX score reports that students receive do not include a percentile. However, the National Board of Osteopathic Medical Examiners (NBOME) does make available an online tool that PDs could use to calculate the percentile for a given COMLEX score. (A link to this tool now appears in the ERAS Program Directors’ Work Station for osteopathic applicants.)
Using the NBOME tool, we can see that an applicant with a COMLEX Level 1 score of 600 did pretty well – scoring at the 80th percentile of all test-takers.
Current distribution of COMLEX Level 1 scores, based on the NBOME percentile calculator.
But there are at least two major problems with this approach, too.
The first is practicality. Do we really think that PDs are going to take the time to look up a COMLEX percentile? Remember, the very reason most PDs use licensing examination scores is for screening – to make a quick, yes/no determination about whether to interview a candidate or not. The idea that these PDs will take the time to go to a separate website – especially a clunky one that requires several inputs to spit out the data you want – seems hopelessly naive.
The second problem is meaning. Some PDs like USMLE scores because they provide a universal measuring stick. Knowing that an applicant has a 80th percentile COMLEX is nice – but it still doesn’t tell you how that applicant compares to those who took the USMLE.
But maybe we could partially get around these concerns if we…
4. Convert COMLEX percentiles to USMLE percentiles.
Before getting into the nuts and bolts of this, let’s be clear: COMLEX and USMLE are different tests.
They have different content specifications and formats, and are taken by groups of test-takers whose demographics and prior standardized test scores differ systematically.
But all that aside – what if PDs could see, at a glance, what a COMLEX score would be if it were a USMLE score at the same percentile? What if we put the conversion on a single table so that PDs wouldn’t have to go into the NBOME website and look up a percentile every time they come upon a DO applicant?
It would be easy to make this kind of thing. So easy, in fact, that I went ahead and did it.
Comparison of COMLEX and USMLE percentile scores. (Data extracted from here and here.)
Is this unscientific? Sure.
But so is using USMLE scores to select residents. So if we’re gonna be unscientific, let’s just have students pay once for their licensing exams while we’re at it.
Still, there’s an easier solution – one that doesn’t rely on the NBME to act against its financial best interests, and one that doesn’t involve PDs using a clunky website or printing out the table above on a laminated card.
5. Change the COMLEX score reporting scale.
The scale for COMLEX goes from 0 to 1000. The scale for the USMLE goes from 0 to 300.
Why?
PDs have an intuitive sense of what’s a ‘good’ USMLE score for their specialty. Why not capitalize on that intuition by reporting COMLEX results using the numbers that most PDs are more familiar with? Make me president of the NBOME, and the first thing I’ll do is start reporting COMLEX scores on the same scale used by the USMLE.
Make me the King of Medical Education, though, and the first thing I’ll do is demand that we stop relying so much on any one-time, point-estimate of basic science memorization that measures so little of what PDs and patients truly care about. But until then…
___
ADDENDUM: Several readers have asked why I didn’t discuss the idea of getting rid of COMLEX and having all osteopathic medical students take the USMLE instead. It’s a good idea – but one that seems unlikely to occur. For one thing, in some states, COMLEX is protected by law, so getting rid of it would require amending the medical practice act. Many states do allow DOs to be licensed by taking the USMLE – but this seems to be an option few DOs pursue. In 2018, only 21 DOs took USMLE Step 3, despite the thousands who took Steps 1 and 2 CK. Let’s also not forget that the NBOME has a vested interest in maintaining COMLEX – with revenue of over $30 million and a CEO who took in $635,943 in reportable compensation in 2017. It’s hard to imagine them willingly allowing themselves to be legislated out of existence. Getting rid of COMLEX and providing a single licensure exam for DOs and MDs is a laudable goal – but it won’t be easy.
YOU MIGHT ALSO LIKE:
A non-MBA’s Guide to NBME revenue in 9 simple charts
USMLE Step 1: Leveling the playing field – or perpetuating disadvantage?
The Sheriff of Sodium 