A few weeks ago, I got interested in financial conflict of interest at the National Board of Medical Examiners (NBME) – particularly as it relates to their opposition to a pass/fail USMLE.
At this point, I’ve spent more time reviewing the NBME’s Annual Reports and IRS Form 990s than any person should. I’ve learned a lot about the organization, its history; its mission; and its philosophy. However, if you think I’m just going to give up that hard-won knowledge in an easy-to-read essay, you are sorely mistaken.
Instead, I’m going to make you work for it, one USMLE-style question at a time.
__
Question 1
The USMLE was unveiled in the early 1990s, replacing two other examinations that provided parallel pathways to physician licensure.
Which of the following was the proximate reason for creating a single pathway to physician licensure?
A. Data suggested a single examination would better screen out incompetent practitioners
B. Consolidation of previous exams would result in lower costs to medical students
C. The NBME just thought it was the right thing to do
D. The NBME’s business interests were threatened by political action
__
The correct answer is D.
In the decades before the USMLE, there were two pathways to physician licensure in the United States. You could take the NBME certifying examinations, or you could take the Federation Licensing Exam, or FLEX.
Around 75% U.S. medical students chose to take the NBME. However, a few state medical boards – such as Texas and Louisiana – did not accept the NBME’s exam. So some U.S. students chose to take the FLEX.
On the other hand, international medical graduates (IMGs) were not permitted to register for the NBME exam – so they had to take the FLEX (as well as an additional test called the Foreign Medical Graduate Examination in the Medical Sciences, or FMGEMS).
Although these “separate but equal” pathways were functionally equivalent, there was a perception that the FLEX was the more difficult exam. Students – especially IMGs, who didn’t have a choice in which test to take – began to complain.
In a 2006 article in Academic Medicine, former NBME President Donald Melnick recalled the situation:
In 1987, political action by U.S.-citizen medical graduates from international medical schools resulted in pressure from several state legislatures to eliminate the dual system of examination then in place. . .
Although both FMGEMS and FLEX were designed to have content and standards similar to the NBME certifying examinations, U.S. citizen international graduates asserted that the ECFMG/FLEX requirements were made intentionally more difficult in order to limit entry of international graduates into the United States. Legislators in New York and California were sympathetic to this view and were poised to enact legislation requiring a single examination for all licensure applicants. . .
Specifically, the proposed bills in New York and California would have disallowed the use of the NBME exam for making licensure decisions in those states.
This was a serious threat. If the two largest medical boards in the country stopped accepting the NBME exams, would other states follow?
After noting the current political issues, the NBME’s 1987 Annual Report included this warning from the President of the NBME, Dr. Robert Volle:
According to present projections, revenues over the next three to five years will decline. Staff will, therefore, contain costs, improve productivity and efficiency, and obtain extramural funding for research.
So what happens next?
Wagons get circled. Deals get cut. And the medical licensure kingdom gets divided. Leaders from the NBME, the FSMB, and the Educational Commission for Foreign Medical Graduates (ECFMG) agree on a single, three-part test for medical licensure to be called the USMLE.
The NBME will stop offering its certifying exams; in return, they get to run Steps 1 and 2 of the USMLE. The FSMB will stop offering the FLEX, but will get to take over USMLE Step 3. (And the ECFMG? They get to handle registering international applicants and overseeing test administration at international sites – and later, they’ll split the revenue from the Step 2 CS test centers.)
Learning Objective: When its business interests are threatened, the NBME acts decisively.
__
Question 2
Each year, the President of a major medical licensing examination corporation authors a message that appears in the organization’s Annual Report. Typically, this message is used to highlight previous progress and set forth the organization’s agenda for the year to come.
Which of the following phrases is MOST likely to have appeared in the President’s Message in the most recent Annual Report?
A. “Developing better tests to make better doctors”
B. “Improving medical licensure to defend the safety of the public”
C. “Investing heavily in R&D to better measure the things that matter”
D. “Reconsidering our test costs in an era of spiraling student loan debt”
E. “Putting our customers first with a new customer relationship management system”
—
The correct answer is E.
Yup, those other things do sound like nice things for the NBME to do… though they are not mentioned in the President’s Message. But hey, at least we’ve got a new customer relationship management system!
Think I’m cherry-picking quotes? Read the President’s message in the 2017 Annual Report yourself. It starts with the statement about “targeted strategic growth” and wanders on from there. The concepts mentioned in answer choices A through D are not touched upon at all.
Learning Objective: The non-profit NBME has been infected with a for-profit corporate mindset.
Question 3
To communicate its strategy, mission, and purpose, the NBME has devised graphics which appear in its internal publications.
Which of the following graphics did NOT appear in the NBME Annual Report over the past decade?
__
The correct answer is D.
Yeah, I made that one up. (Although if you read my last post on NBME revenue, a logo like that would not have been inappropriate.)
The other graphics are real, and have been used by the Board to discuss various aspects of their corporate strategy. Precisely what aspects are being communicated in answers A-C remains a bit of a mystery to me. I highlight them here simply to paint a picture of the C-Suite mentality in today’s NBME.
Former NBME President Dr. Donald Melnick about to enjoy a slice of the inscrutable graphic shown in answer choice A.
Learning Objective: The non-profit NBME has been infected with a for-profit corporate mindset.
And yes, I know that this was also the learning objective for Question 2. But this is my test, and I get to decide the content specifications. And this is an important concept that underlies a lot of what’s gone wrong at the NBME.
Look, I’m not anti-business. We want our nonprofits to have some business sense. It does the public a disservice if an organization like the NBME can’t balance its budget or keep up with their revenues.
But what we need is the same kind of garden variety financial stewardship you’d expect from a school or a church. What we don’t need is the leaders of our non-profits playing dress-up as Fortune 500 CEOs. Left unchecked, this breeds predatory business practices and a toxic culture of of greed that undermines the very non-profit missions that these leaders were once called to serve.
__
Question 4
A nonprofit organization that provides test instruments used to inform decisions on medical licensure develops a new assessment tool that appears to be better than the one currently in use.
Which of the following factors will be explicitly considered in determining whether this product is put into use?
A. Confirmation that the new product is objectively better than the test currently in use
B. Cost-benefit analysis, to ensure that any extra cost to students is more than offset by benefits to society
C. Ensuring the new test is acceptable to a majority of score users
D. Evidence that introduction of the new product will not threaten the financial stability of the sponsor organization
__
The correct answer is D.
I am going to quote directly from “The Message from the Chair and President” in the 2015 Annual Report (emphasis added):
The concept of a sustainability index was introduced as part of the establishment of institutional metrics in 2014. Staff designed the index during the first half of 2015 and demonstrated an initial model of the index at the June 2015 Finance Committee and July 2015 Executive Board meetings. The proposed model focuses on projected inflows and outflows of cash over a five-year period. . . The purpose of the sustainability index is to provide evidence that our development of better or new assessments to contribute to our mission fulfillment does not threaten the stability of the NBME.
I’d like to let that last line sink in for a moment.
In other words, if the NBME were to come up with a new test that was better than what we have now, they might put it into practice. Or they might not. It would depend in part on the sustainability index score, and whether the development might “threaten the stability of the NBME.”
Learning Objective: The NBME is explicit in considering financial implications of all of its products.
__
Question 5
A non-profit organization is founded with a specific mission to ease the process of interstate physician licensure. The organization achieves this mission and delivers a valuable public good.
Which of the following represents the most likely pattern of financial growth in this non-profit organization?
A. Negative; services will be reduced since the mission has been achieved
B. Flat; since similar revenue will be required to ensure the mission continues to be met
C. Increasing slightly; since existing services may be improved or become more expensive over time
D. Increasing rapidly; the organization must ask, “Can we do more?”
__
The correct answer is D.
Ummm… wait a sec… that answer doesn’t make sense. I’m going to need former NBME President Melnick to explain this one:
The NBME was founded to solve a significant yet rather simple problem: can we ease interstate reciprocity of medical licenses by creating an examination of such high quality that all U.S. medical licensing authorities will accept its results in lieu of their own examinations, thereby eliminating the requirements for re-examination? With a growing reputation for excellence and commitment to purpose, the NBME ultimately achieved success in solving the problem it was founded to solve. Today, all U.S. licensing authorities accept USMLE as the examination that meets requirements for medical licensure. Through our centennial year look back and from our 21st century perspective, we can certainly say, “Mission accomplished,” but we also must ask, “Can we do more?”
(emphasis added; quoted from the 2015 NBME Annual Report)
__
Okay, I’ll admit – that logic still doesn’t make sense to me. Why must the NBME do more? Why isn’t achieving a useful goal enough?
Medical licensure serves a valuable purpose in protecting patients and instilling confidence in our profession. And if each individual state still required its own board exam, it would present a needless barrier to physician movement in the U.S., leading to marketplace inefficiencies and hurting patient care. So we should all be grateful to the NBME for that.
So, again, why must the NBME do more? Why can’t it focus on doing what it does better? Why have NBME revenues tripled since 2001? Why is any of this logical – and why do we accept is as being necessary?
NBME revenue more than tripled from 2001 to 2017. But we must ask: can they do more?
Learning Objective: The NBME has set an agenda for growth.
__
Question 6
In a free market economy, various economic conditions and market forces may present barriers to growth of a business.
Which of the following constrains growth for the National Board of Medical Examiners?
A. Consumer preferences
B. Competition from other testing agencies offering physician licensure products
C. General economic downturns
D. None of the above
—
The correct answer is D.
The NBME operates in an unusual business space. Since the initiation of the USMLE in 1992, the NBME has had a monopoly on licensure testing for allopathic medical graduates. Even better, its “customers” have no choice about whether to buy their product or not, at whatever price the NBME chooses to sell it.
Say what you want about the for-profit USMLE test prep industry – but at least they operate in the regular free market.
If I decide that I want to make a new Step 1 test prep book, I can. But if I expect customers to actually buy it, my book needs to be better than First Aid for the USMLE. And if my book isn’t better, then it at least needs to be cheaper. The Step 1 prep market space is cutthroat – but that ends up lowering prices and improving quality for everyone.
The NBME executives act like they are running a real business. But real businesses compete. In the NBME’s market space, if you set an agenda for growth, you’re gonna achieve it. In spite of all the pretentious business language and self-congratulatory back-slapping about “growth” in their Annual Reports, the NBME executives have more in common with the warden running the prison canteen than they do with Jack Welch or Steve Jobs. It would almost be laughable if it weren’t so predatory.
Learning Objective: The NBME has a monopoly in a market space with captive customers. If the NBME charts a course for growth, they will achieve it.
__
Question 7
A survey is conducted regarding whether the results of a national licensing examination should be reported pass-fail, or as a three-digit score.
A majority of which of the following groups preferred pass/fail USMLE score reporting?
I. Residency program directors
II. Medical school deans of education and student affairs
III. Medical students (for score reporting to residency program directors)
A. I only
B. II only
C. III only
D. I and II
E. II and III
F. I, II, and III
__
The correct answer is E.
The data above are from a real survey conducted by the NBME, published in Academic Medicine in the year 2000.
Both medical students and deans of student affairs/education preferred a pass/fail USMLE. Additionally, a majority of state medical boards responded that pass/fail score reporting would not impact their licensure decisions. Only residency program directors preferred numeric score reporting.
So if there was such broad-based support for a pass-fail USMLE in 2000, why does the NBME still vigorously defend a scored test in 2019?
Well, there was one other interesting response to the study. Quoting directly from the article:
[Fourteen] state boards indicated that they could not continue to use the USMLE if pass-fail reporting were implemented unless a change in their medical practice acts occurred. Few states are anxious to reopen their medical practice acts for legislative review.
__
Take, for instance, Kentucky. Here is the relevant section of the Kentucky Administrative Regulations.
201 KAR 9:031 stipulates the examination requirements for physicians who wish to be licensed in Kentucky.
Note that the law specifies a minimum passing score for the USMLE of 75. (These two-digit USMLE scores are no longer provided to examinees, but are still reported to state boards.)
In other words, if the USMLE were to report results only as a pass/fail designation, the Kentucky Board of Medical Licensure could no longer accept the test’s result until or unless the state legislature amends the medical practice act. (By the way, this is not speculation – I did confirm this with the Board’s counsel.)
And if medical licensing boards can’t accept the results of the USMLE, it could potentially open up the market for another licensing examination – which would break the NBME’s monopoly.
Think that’s farfetched?
I don’t. If my post on the NBME revenues taught you nothing else, it should have at least demonstrated that there is good money to be made by peddling tests to medical professionals. And if 14 states simultaneously needed a new licensing exam, are you really so sure that another group wouldn’t step in to fill the void and get a piece of that action?
Predicting what any state legislature is going to do is tricky. And you can bet the NBME isn’t going to chance losing their monopoly in a high-stakes game where they have nothing to gain.
Let’s go back to our first Learning Objective. The NBME acts decisively when its business interests are threatened. We should therefore interpret the NBME’s lack of action similarly – as a way to protect its business interests.
Nothing could be more threatening to the NBME’s profitability and long-term stability than market competition. Do you think that the NBME would risk losing its market monopoly by playing roulette with legislative action in 14 different states? How do you think that works out on the sustainability index?
(Postscript: The InCUS public commentary showed less support for a pass/fail USMLE than the 2000 survey… though these data are somewhat suspect. And importantly, there was significant opposition to a pass/fail USMLE from state boards. I reviewed both of these things on Twitter.)
Learning Objective: Beyond loss of revenue from ancillary services, moving to a pass/fail USMLE could threaten the NBME’s business monopoly.
The Sheriff of Sodium 