The USMLE Cheating Scandal

On January 31, 2024, the USMLE announced that it was invalidating the scores of 832 examinees due to “a pattern of anomalous performance associated with Nepal.” This news flooded my inbox in a way I’ve never seen before, dwarfing even the surge of correspondence I received after the USMLE announced their decisions to make Step 1 pass/fail or cancel USMLE Step 2 CS.

Such extraordinary times demanded an extraordinary amount of analysis and commentary, spawning not just one, but an unprecedented two Emergency Mailbags, as well as a third video yesterday to update and cover the factual revelations that emerged from one examinee’s lawsuit against the NBME.

Part I

Here, I discussed…

what are “USMLE recalls”
how test-making authorities can identify cheaters by comparing performance on compromised vs. non-compromised items (with a case study from the Canadian MCCQE Part 1 exam)
an overview of USMLE scoring, and whether USMLE scores will be adjusted for non-cheaters because cheaters raised the curve (hint: two-letter answer)
how much the scandal will hurt IMGs who didn’t cheat
whether everyone with invalidated scores is screwed… or whether some will land on their feet
what the NBME can do to make the USMLE more resistant to cheating in the future

Part II

This one covers…

whether there was a security breach at Prometric centers in the United States
how you can mathematically detect cheating when examinees are scoring very highly (i.e., answering almost all questions correctly)
a conceptual introduction to agreement analysis
why people defending those with invalidated scores don’t understand math
whether it’s really wrong to use USMLE recalls… and how legitimate test prep companies sanitize recalls
just how high many test scores from Nepali examinees had become (with a bonus extended analogy comparing to the steroid era in baseball)
my suspicion that the USMLE’s announcement occurred more rapidly than it otherwise would have due to questions from a reporter
whether the scandal will lead to USMLE Step 2 CK becoming pass/fail

Part III

This one discusses, among other things…

the methods the NBME disclosed using to catch the cheaters, revealed in their court filings
how agreement analysis works (using a real-life example from a surgical certifying examination)
the data that explain why the USMLE’s investigation centered on Nepal
how answer response times damned some of the cheaters
whether there was a security breach at a Prometric center – or not
whether the examinees with invalidated scores knew they were cheating or not
the chance that the ringleaders of this scheme could face legal consequences (using the infamous Optima University case as a case study)
whether we should bring back USMLE Step 2 CS
whether we should restrict USMLE Step 1 and Step 2 CK administration to the centers in the United States
whether the USMLE will invalidate scores for examinees from other countries in the future (hint: three letter answer)
the status of the lawsuit against the NBME
the GoFundMe to support legal fees to “stop defamation of Nepali doctors”
a lesson from a Jedi master of pediatrics