If you’ve followed me for any length of time, you’ve probably seen me share this graphic on USMLE score creep.
The average USMLE Step 2 CK score has risen by nearly 50 points over the past 30 years.
But it’s not just the average USMLE score that’s rising – the minimum passing score has also increased steadily over time.
What’s going on?
Give me a few minutes of your time, and all will be revealed:
Running time – 49:16
For those of you who think you don’t have the time to listen to me explain, in detail, how the USMLE passing score is set and how that process has been configured to justify progressive increases in the standard, lemme just give you just a taste of some of the things you could learn about if you just click on the video above.
–
Why does the USMLE Step 1 minimum passing score increase by a few points every year?The NBME uses a modified Angoff procedure to determine the minimum content necessary to pass the exam.The key operation in an Angoff standard setting exercise is to imagine the universe of test-takers and develop a profile of the minimally competent examinee.How often would the minimally competent examinee answer a question like this?USMLE standard-setting experts not share their opinions on question difficulty with other judges, but also receive access to real performance data for each question.Once you collect the experts’ judgments about the difficulty of each question, you can determining the cut score for the exam by taking the average.However… the modified Angoff cut score doesn’t automatically become USMLE policy. The USMLE’s Management Committee considers other data when deciding where to set the passing score. Among other things, they conduct surveys about whether current pass rates are too low/high, and what the “ideal” fail rate should be.Students have strong incentives to perform well on the USMLE, and more efficient test prep resources to do it. But rising USMLE performance doesn’t just enable increases to the passing standard – it necessitates them.There are real risks for the maker of a high stakes exam that fails too many – or too few – examinees. For the USMLE, the market expects (and will tolerate) failure rates of 3-9%.The faster scores rise, the more rapidly the USMLE’s Management Committee has had to step in to increase the passing standard.But the maker of a criterion referenced test can’t just dial up or down the pass rate on a whim. Instead, they have ensure that the process they’ve developed comes up with the right numbers. To find out where you can ever-so-lightly put your fingers on the scales… you’re gonna have to watch the video.
The Sheriff of Sodium 