Mailbag: Second Looks, the USMLE Step 1 “Curve,” and Residency Placement Rates for DOs

It’s been a while, and my inbox has once again filled up with more good questions that deserve public explanations. Time for another Mailbag.

(As always, these are real questions from real readers, edited for anonymity and clarity.)

The 2021-2022 Ophthalmology Residency Match timeline.

Did you hear that ophthalmology is going to allow their applicants to take in person “second look” visits this year? Applicants can visit programs after the programs have submitted their rank lists, but before applicants have to submit theirs – so there’s no pressure on applicants to attend unless they’re really interested! Why can’t the NRMP move their calendar to allow this for other specialties, too?

In theory, they could… and maybe in future cycles, they will. But the 2021-2022 NRMP Match calendar was finalized months ago, and it doesn’t include a staggered rank order list submission deadline. So it ain’t happening for anyone other than ophthalmology this year.

In fact, some specialties – like pediatrics – have come out in opposition to second look visits. And for NRMP-matched specialties, I support that position.

As your question suggests, when applicants know that their ranking hasn’t been finalized, receiving an invitation for an in person second look becomes an offer they can’t refuse. Declining to visit would be perceived as a lack of interest – and for some programs, will result in the applicant being ranked lower.

However, even though ophthalmology programs will submit their rank order lists a couple of weeks before applicants do, this system may still leave opportunities for gamesmanship.

Remember, the time allotted for ophthalmology second looks is short – just 12 days. Programs can’t necessarily wait till the last second to invite applicants – the practical realites of arranging travel may require may that invitations be sent before the program submits their rank order list.

Should that comes to pass, applicants who know how to play the game will certainly know that the right answer to any second look invitation is “yes.” (They know they can cancel once the program has submitted their rank order list and it’s safe to do so.)

Every Ophtho Chad knows that the right answer to “Would you like a second look?” is “Yes.”

Of course, if insincere applicants accept every second look invitation just to protect their rank order list position, it may limit opportunities for other students (who would have eagerly accepted and followed through). After all, programs may not be able to accommodate everyone that they interviewed for a second look… so not only will the invitations they choose to send tip their hand about their most preferred applicants, but it could also end up leading to a ‘second look hoarding’ phenomenon in which these preferred applicants consume the bulk of second look offers to the exclusion of others.

I am an international medical graduate who will take the USMLE Step 1 examination this fall. I have done well on my practice tests (projected score in the 240s), so I wanted to take the test before it goes pass/fail in January. However, now I am concerned that the curve on the test will be very unfavorable because only the top test-takers will want to take the test this fall; the rest will wait until January when it goes pass fail. Should I reschedule my test?

No.

If you believe that having a Step 1 score will enhance your residency application, go for it. You don’t need to worry about the “curve” or the mix of other applicants taking the test when you do.

Remember, your USMLE score is generated by comparing your exam performance to the performance of a reference group of examinees who answered those test items previously. I’ve written before about the general method involved in USMLE scoring, but the important thing to remember here is that that reference group consists only of first-time test-takers from U.S. MD schools.

In other words, although you’re probably right that a more polished group of IMG test-takers will take the Step 1 in the final months before pass/fail scoring, those examinees will not impact your score, because you’ll be graded relative to the group of US MDs who took those test items previously (i.e., as experimental items on their test last year). Although the average score for US MDs has been steadily rising over the years, I really don’t think there will be a substantial difference among the recent test-takers that will make up your reference group.

A 98% residency placement rate in the Match for osteopathic applicants? Could that be right?

I saw your Tweet to the American Association of Colleges of Osteopathic Medicine (AACOM). Did you ever figure out what’s going on with their numbers?

Sort of. At least, I figured out enough to confidently assert that a few osteopathic medical schools are fudging their numbers to inflate their residency placement rates.

For those who missed it, the issue is that AACOM’s numbers don’t line up with those in the NRMP’s 2021 Match report.

Specifically, Table 4 of the NRMP report shows that there were 7101 osteopathic seniors who were active applicants in the NRMP Match (i.e., they registered for the Match, did not withdraw, and submitted a rank order list). Of these, 6327 matched (89.1%). According to Table 19, 484 DO seniors accepted offers in the SOAP, which would give a residency placement rate of 95.9%. That’s a nice success rate… just not quite as gaudy as the AACOM report claims.

Residency placement outcomes, from AACOM’s Report on Osteopathic Medical School GME Placements in 2021 Matches.

In contrast, notice how two of AACOM’s figures above differ from those in the NRMP.

The first discrepancy is the number of participants in the NRMP Match, which AACOM lists as 7001 – 100 applicants fewer than listed in the NRMP report.

The second is the “Number matched in NRMP/SOAP,” which AACOM reports as 6907. But that’s 96 more than we can account for in the NRMP report (with 6811 placements in the Match and 483 in the SOAP).

There really is no justifiable explanation for these discrepancies. These differences are not explained by the fact that some DOs match in the military or San Francisco Matches – those matches are long gone by the time of the NRMP Match, and successful applicants are required to to be withdrawn from the NRMP Match (it’s a Match violation for the school if they’re not).

So what’s going on?

To their credit, AACOM has been very transparent about their data and how they’re collected, and they responded to me promptly on Twitter and in subsequent DMs.

The issue is that AACOM’s figures are self-reported by individual medical schools. That is, they’re not verified by the NRMP (or military/San Francisco matches). So if a school chooses to fudge their numbers a little bit… well, AACOM is none the wiser.

So yeah, I think certain osteopathic schools are cooking their books. They’re reclassifying applicants who failed to Match as if they’d never sought residency training in the first place (removing them from the denominator of the placement calculation), and by classifying applicants who found research fellowships or scrambled into post-SOAP positions as if they’d successfully received a spot in the Match/SOAP (thus inflating the numerator).

Does this matter? I say it does. Before a student invests $200-400k in a particular medical school, s/he has a right to accurate data on how that school’s graduates fared in terms of residency placement. Fudging those figures – even a little bit – is deceptive and dishonest.

My hope here is that AACOM will be able to work with the NRMP to audit the data they receive from their members, so that students have access to accurate information – and schools who fudge their numbers can be called to account.

YOU MIGHT ALSO LIKE:

Mailbag: ERAS Secondary Applications and Preference Signaling

The 2021 Match, By the Numbers

The Residency Selection Arms Race, Part 2: Anatomy of an Arms Race

The Lecture that Never Got to Be