Criteria for selecting students in the Match

We are fast approaching a very important day in the academic calendar. On February 23, 2011 residency programs around the country have to enter their Rank Order List. This day is the culmination of a lot of work by the Program Director, faculty and residents from the program, and from students applying to that program. The actual day that the results of the Match are released is about a month later on March 17. But the work is all done once the lists are in on February 23.

You may not understand this process, so let me walk you through it. Medical students around the country decide what specialty they are interested in applying to at some point during the third year of medical school. Students gather letters of recommendation from faculty over the next several months. Frequently, they will do a fourth year elective rotation in their specialty of interest. They also have to decide if they are going to try to go to another school and do an “away” rotation. For most students, they are obligated to enter the National Resident Matching Program.  If they are a fourth-year student at an allopathic medical school in the US, they have no choice but to enter the Match.

Beginning in October, students begin their job interviews. We call these residency interviews, but honestly the students are trying to land a job as a resident in a particular program. Students will have anywhere from 10 to 40 interviews depending on the competitiveness of the specialty that they are applying to enter. These interviews may be anywhere across the country, but are mostly in larger cities (that is where the teaching hospitals are located).

So, on February 23 the students enter their Rank Order List. The program that they like best is Number 1. Their least favorite is last. Residency programs do the same. They rank all of the students that they interviewed from 1 to however many they want to rank.  The programs don’t have to rank all of the students that they interview, and the students don’t have to rank all of the programs.  But the Match is a binding contract,¹ if they rank someone (student or program) they are legally bound to that ranking.

The question for today is how do programs decide how to rank the students that they interview? There are several ways, some good, some really bad! Let’s start with the good ways.

Letters of recommendation can be very helpful, if they are written by honest faculty physicians, who know the student, and have personally worked with a student. These letters can be a great assessment of a student’s global performance. An old study by Keynan, et al ² done in 1987 compared objective faculty ratings to other types of assessment. This study compared a global faculty rating, a multiple choice question (MCQ) test, and an oral examination.  They found that the “the 'subjective' expert assessment of performance through global rating scales is comparable to that of 'objective' evaluation through written MCQ.” They also found, using a stepwise regression analysis, that the ratings of 'reliability', 'knowledge', 'organization', 'diligence,' and 'case presentation' were the most predictive of the overall global rating. Chair’s letters which are often written by the Chair of a department (who probably does not know the student very well) are generally not much help.

Another good way to rank students is through an interview. Skilled interviewers can pick up on many communication and personality issues that probably don't show up on a paper application. Maybe the applicant is very introverted and has difficulty talking during the interview. Or maybe they are a jerk or a racist or a sexist. A personal interview can pick up these problems (not always, but often).

Unfortunately, there are also some bad ways to rank students.  Commonly, grades and boards are used. Frequently, medical school grades and USMLE board scores are the screens that decide whether a program invites a student to interview.

I want to focus on USMLE scores. Grades are quite variable from school to school. Some schools have an A to F scale, some have Pass/Fail, and others have Satisfactory to Superior. Preclinical grades don't have a lot of predictive value for clinical grades and neither are very predictive for performance in residency.

Board scores are just as bad. They seem to be an objective way to compare students. Everyone, across the country takes the same test. There is one big problem. The USMLE is designed to measure knowledge and application of knowledge. It was created to be used by the State Licensing agencies as a common evaluation for licensure. There are statistical problems when you try to interpret the scores that are given with a pass/fail based test. There have been several studies that all show basically the same thing about board scores. Performance on the boards does not correlate to performance as a physician.

In 2005, Rifkin and Rifkin³ compared the performance of all the first year Internal Medicine residents at a large academic medical center on standardized patient encounters to their scores on the USMLE Step 1 and 2. They found very low correlations. For Step 1, the correlation was 0.2 (df=32, p=0.27) and for Step 2 it was 0.09 (df=30, p=0.61). Remember a higher number means that the two measures are more strongly related.

A more recent study is very critical of the use of USMLE scores for selection of residents. This study by McGaghie and colleagues,⁴ was a research synthesis using a critical review approach.⁵ They collected and reported correlations between USMLE Step 1 and 2 and several reliable measures of clinical skills. These skills included auscultation of the heart, performance of ACLS (Advanced Cardiac Life Support), communication with patients, thoracentesis, and central line placement. They found correlations from -0.05 to 0.29 to Step 1 and -0.16 to 0.24 for Step 2.

Their conclusion sums it all up. "Use of these scores for other purposes, especially postgraduate residency selection, is not grounded in a validity argument that is structured, coherent, and evidence based. Continued use of USMLE Step 1 and 2 scores for postgraduate medical residency selection decisions is discouraged."

I couldn't agree more. If I need a neurosurgeon to operate on my brain, I want to know that he has a very steady hand, not the highest board score. If I need a radiologist, I want to know that her visual pattern recognition is outstanding, not that she scored well on a multiple-choice question test. And if I need a family doctor, I want to know that his clinical reasoning and communication skills are excellent, not that he scored well on the boards.

References

1. http://www.nrmp.org/res_match/policies/map_main.html

2. Keynan A, Friedman M, and Benbassat J.  Reliability of global rating scales in the assessment of clinical competence of medical students. Med Educ  1987;21(6):477-81.

3. Rifkin WD, Rifkin A. Correlation between house staff performance on the United States Medical Licensing Examination and standardized patient encounters. Mt Sinai J Med. 2005;72(1):47-9.

4. McGaghie WC, Cohen ER, and Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med  2011;86(1):48-52.

5. Eva KW. On the limits of systematicity. Med Educ. 2008;42:852–853.

Christmas gift

On Christmas morning as we were opening presents with our kids and family, I got to thinking about how lucky I am. I really don't know how I got into medical school.

I grew up in a pretty modest home. These days you would have called us poor. At the time I thought we were middle class but, I didn't know that middle class families were not on WIC. But we never went hungry and my mom kept us in clean clothes (though they were rarely new clothes). We lived in several places growing up but my grandparents lived on a farm in rural SW Missouri, so I thought of myself as a country (rural) boy. We didn't have any doctors in my immediate family, although my grandmother was a nurse and my great ,great uncle was a country doc (more about him some other time).

So, how did I get into medical school???

These days my chances would be pretty slim at most medical schools. The vast majority (65%) of our current medical students come from the top quintile of wage-earning families.(1)  From 1997 to 2004, students admitted to medical school with a family income of greater than $100,000 increased over 60%, while those from the middle and lower income quintiles declined. The median income for the parents of a medical student is over $110,000 per year. 59.7% of students' parents make over $100,000. (2)

Many of our students at Kansas come from urban counties. In Kansas that means basically one of five counties/metropolitan areas (KC, Overland Park, Wichita, Topeka, and Lawrence). We have to work very hard to get and keep qualified applicants from small towns and the rural counties. On the 2009 AAMC Matriculant survey, only 2.8% of matriculating students were planning to practice in an area with a population of 2,500 or less.(3)

We have seen an increase in students whose parents are physicians. Some of these are physicians from the community and some are kids of our own faculty. Most medical students' parents have a high level of education. 52% of the fathers of medical students have a graduate degree compared to 12% in the general population. For moms, it is 35% and 10%, respectively. (4)

So what?

Several things in my opinion.
First, as a state school, we have a contract with the state of Kansas. We are the only medical school in Kansas we have to provide doctors for the state. All of the state. Not just the urban/suburban parts of the state.

We also need to produce doctors that look like the population of the state. If all of our students are from the urban counties, they are going to be much more likely to be white or Asian and not underrepresented. If they are all from the highest income brackets, we are neglecting a large part of our population.

And they will be much less likely to practice in the places that we want/need doctors. Rural underserved counties. Urban underserved areas. Primary care practices. Community health centers. Rural health clinics.

Right now we are doing pretty well in this area when you compare us to the rest of the country. The problem is that the rest of the country is doing really badly. So, even though we look good in comparison, we aren't doing that great. A recent analysis done by Fitzhugh Mullan (5),  found that many of the "top" medical schools in the nation did pretty poorly when you measured them on social mission (Vanderbilt #141, Duke #124, Boston U #131, Mayo #103). Social mission is a construct that uses number of primary care physicians produced, graduates practicing in HPSAs, and numbers of underrepresented minorities. KU ranked 5th in the nation on that scale.

We also do pretty good at turning out primary care doctors, at least in Family Medicine. Over the last 10 years (1998-2009 graduating classes), KU and the University of Minnesota were ranked number 1 and 2 when you combine number and percentage of students that graduate and choose a residency in Family Medicine.

But, we have to continue to work hard to maintain what we have. It is a constant struggle to convince people on the admissions committee that we shouldn't just take the students with the highest GPAs. And it is easy for the Dean to say, "we need to raise the MCAT scores of our incoming students". And it is easy to decrease funding for programs that support students from underserved backgrounds.  But those are the battles that have to be fought.  As someone who is really smart told me, "we don't need programs to increase the number of white kids from Johnson County".

References
(1) Bowman B. personal communciation
(2) Jolly P. Diversity of US Medical Students by Parental Income. AAMC Analysis in Brief. 2008;8(1).
(3) AAMC Matriculating Student Questionnaire (MSQ) 2009
(4) Grbic D, et al. Diversity of US Medical Students by Parental Education.  AAMC Analysis in Brief  2010;9(10).
(5) Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The Social Mission of Medical Education: Ranking the Schools. Annals Intern Med  2010;152:804-811.

The clinical enterprise is to a medical school as the athletic department is to an undergraduate university

I love analogies and while no analogy is perfect, I find this one fascinating.  If you are like us, you probably watch a lot of college sports this time of year. I was watching a basketball game the other day (KU vs Arizona) and I was thinking about the athletic department at our school.  Like many large state schools, we have a pretty large budget for the athletic program. (more than $50 million in 2006) (1)  All of that money may translate to some success , at least on the basketball court, not so much on the football field.  A report last year from the Knight Commission on Intercollegiate Athletics detailed athletic department spending increases at major colleges and universities. Increases to the tune of 11% annually, and this was during the worst recession since the Great Depression.

Why do we spend so much money on the athletic department? And what does this have to do with medical education, which after all is what this blog is supposed to be about? It struck me that in a medical school system the clinical enterprise has some similarities to the athletic department at the undergraduate school.

Both the clinical enterprise and the athletic department are askew from the primary mission of the university at large. Each has a significant portion of the budget. Both generate significant revenue. Both attract philanthropy for their own use. World famous specialists and top athletic teams attract media attention. Attention leads to more patients and more fans and more money and higher rankings and goodwill from the local community. The highest paid people are the stars in these areas: the super subspecialists and the coaches for the money-making sports. They are arguably the highest paid because they bring in the most money. And, because of their ability to generate revenue, they seem to have a different set of expectations for their behavior. (2)

But here is where the analogy breaks down. It seems that few undergraduate universities have as part of their mission statement, “to have the best athletic department” or “to generate the most money on athletics.” And if US News and World Report has done a ranking on “most lucrative athletic departments,” I haven’t seen it. On the other hand, medical schools invariably have something about the clinical enterprise in the mission statement. And, that statement is not, “create a clinical enterprise that is sufficient to meet the educational needs of our students while providing excellent patient care.”

Perhaps the most important difference is that while the athletic department has some opportunity to impact the education of students, the clinical enterprise profoundly affects student education. The athletic department probably pulls some philanthropy away from other areas and definitely pulls attention away from other areas. But if the university decides to focus on football instead of swimming, the science majors are probably not affected by that decision. However, if the clinical enterprise focuses on higher revenue specialty practices and decreases lower paying primary care or generalist practices, the medical students are at risk of receiving inadequate or inappropriate medical training. If the clinical enterprise adopts policies that limit medical students' use of the electronic health record or participating in procedures, their education suffers. If the clinical enterprise needs the physicians to make more money by seeing more patients and spending less time teaching students, their education suffers.

Generally, about this time in the argument, someone pipes in with No Money, No Mission. And they go on to describe how the clinical mission funds the other missions of the medical school. Some people believe that the clinical enterprise and the athletic department works hard to make money for the university, so that the university can attend to its other, less interesting mission: education. And perhaps it does work that way in some places. Perhaps, but mostly they fund themselves. They make decisions based on revenue and they keep most of the profit. In fact, an article in USA Today says that for every dollar the university spends on the athletic department, they only realized one dollar in revenue. (3)  Not a great return on the university's investment.  At least the athletic department pretends that they aren’t trying to make money. 

Medical student education at allopathic schools in the United States is clearly in a marriage with the clinical enterprise – sometimes an abusive marriage and maybe sometimes a partnership, but nonetheless a marriage. So what can be done? Perhaps that is a good topic for another blog – what would a counselor say about the marriage of the clinical enterprise and medical education?

(1) King, Jason. "Hawk Market", The Kansas City Star (June 11, 2006), pp. C1

(2) http://bleacherreport.com/articles/11863-from-caron-butler-to-chris-andersen-how-should-we-handle-troubled-athletes

(3) http://www.usatoday.com/sports/college/2009-04-29-college-athletic-spending-report_N.htm

The beginning...

Why should I write a blog?

My wife said: "what are you going to give up if you start writing a blog?"
Hmmm, good question.
(The answer was nothing)

My chair said: "John, I would read it." 
Now, that is high praise coming from Joshua Freeman, MD.  Josh writes a fascinating blog on the state of the US healthcare system (or lack there of) called Medicine and Social Justice.

Others said, who would care what you have to say?
My answer is maybe nobody.
But I hope that there are others out there that are interested in medical education. If so, maybe this blog will be of value to them.

If nothing else, I get to say my piece. And isn't that the point of a blog, after all?