Using interviews to select medical students

This is the fourth in a series of posts about entry of students into medical school. In the last post, I blogged about personality traits that may be better suited to being a medical student. Unfortunately, most schools in the US do not use personality profiles as a screen for incoming medical students.

At most schools, the closest that we come to this is the admissions interview. Many believe that a 30 minute interview is a good way to weed out bad apples. I am not sure that this is true. Just on the surface, it seems like an experienced interviewer may be able to identify highly dysfunctional people. By dysfunctional, I mean traits that would be obviously detrimental to their function as a physician. These obvious dysfunctional traits are things like: students who have difficulty talking to others, students who have flaws in their ethical approach to life, and students who have problems with their reasons for entering medical school. But what does the literature say?

Powis, et al (1) used a case-control design to study students who were admitted to medical school but did not graduate. They retrospectively analyzed 56 paired cases and controls. The cases were students who had left medical school due to failure or withdrawal, while the controls were students who had completed medical school. The controls were all students who had excelled in the their academic performance. The students who left medical school had all been rated lower at their admission interview. Effect sizes were statistically significant in the Overall rating (ES=2.17), self-confidence (ES=2.59), perseverance (ES=2.98), and tolerance of ambiguity (ES=1.04).

The Powis study used a objective and structured interview and they compared the students who left or were dismissed from medical school to those who received Honors in medical school. It is not clear that the admission interview would distinguish between failing students and anyone who would not fail. Admission interviewers have widely variable reliability. Powis found the inter-rater reliability varied from .23 to .63 for seven different qualities assessed by two faculty members. Other studies have found that reliability data is better for interview programs that use a structured interview process (.82 to .84) while with unstructured interviews the reliability is .61 to .75. (2)

So, reliability is not great, but seems to be better with more structure. Part of what provides structure is giving interviewers training and giving them types of questions to ask. But (and this is a big but)...I think that the interviewer has to be experienced as an interviewer. They have to be able to sort through the information presented by the student. They have to be willing to ask probing questions and be willing to make the student uncomfortable. Questions about ethical grounding or hypotheticals about decision-making are difficult. Interviewers can be blinded by other characteristics. Like MCAT scores．

For instance, I have heard interviewers say, well they didn't interview very well but they have great MCATs, so they will do fine. I am not kidding, I really heard a faculty member say that. And they were being serious! I know that is not supported by the data, but you still hear it a lot. The interviewer has to be experienced enough to ask tough questions. Not just, "tell me about your fraternity activities in college" but hard questions about ethics, hypotheticals, and dilemmas. They have to ask about motivation, why do they want to come to medical school. They have to get beyond the pat and prepared responses that students practice during their mock interviews and really push the student to get at internal motivations and thought processes.

And what about medical students? In my experience the medical students that we ask to interview are pretty good at sniffing out the bull. But their problem is that they feel so happy to actually be in medical school and almost finished (our interviewers are fourth year students) that they tend to be a little easy on the score sheets. Gutowski and colleagues,(3) looked at current medical student interviewers. They found that when compared to faculty interviewers, students wrote more about applicants' motivation, personality, communication skills, and interests. Student wrote more in the overall evaluation sections (p<0.001) and gave more examples on the motivation section (p<0.0011) and communication skills section (p<0.0035).

So, I guess the bottom line is that there is no easy way to figure out who are the right students to admit to medical school. We should push for multi-dimensional models that minimize the MCAT and utilize personality characteristics. We should ask the admissions committee (and the dean) to define what they think are the qualities and characteristics of the students that should matriculate to our medical school. And maybe most importantly the admissions committee should be held responsible for the results of their work.

References
(1) Powis DA, Neame FLB, Bristow T, Murphy LB. The Objective Structured Interview for Medical Student Selection. BMJ. 1988;296:765-768.
(2) Albanese M, et al. Assessing Personal Qualities in Medical School Admissions. Acad Med 2003;78:313–321.
(3) Gutowski CJ, et al. Current medical student interviewers add data to the evaluation of medical school applicants. Medical Education Online 2010;15:5245.

Posted using BlogPress from my iPad

Location:Common St,New Orleans,United States

Personality traits that predict success in medical school

This is the third in a series of posts about my concern with the students that we are bringing into medical school. I am constantly reminded that not everyone believes that there is a problem. Some think that we are getting the right students into medical school. My question. The burning question for this series of posts is: if we are getting the right students in, why is the end product not meeting the needs of America? Or the needs of our individual states? There was a recent op-ed in the New England Journal of Medicine by Stephen R. Smith, MD, MPH from the Warren Alpert Medical School at Brown University.(1) Dr Smith said "medical schools must recognize the current factors that discourage medical students from pursuing primary care careers and then devise ways to overcome these barriers."

My first post in this series, Why do we put so much import on the MCAT? was about the MCAT and why (in my opinion) we need to de-emphasize the MCAT as a criterion for entry into medical school. I am not the only one saying this. Dr. Smith said, "The first test of this commitment will come in the way in which admissions are handled. The little evidence that is available on factors predicting career choice indicates that students who express a desire to serve underserved populations, who demonstrate altruism, and who are committed to social responsibility are more likely to go into primary care." Dr. Smith went on to say that "admissions criteria need to be broadened beyond scores on the Medical College Admission Test (MCAT) to include these personal attributes. The school should adopt an “MCAT-blind” admissions policy, dictating that students whose MCAT scores are at or above a predefined minimum that predicts a likelihood of success in medical school should then be considered further for admission without the reporting of their MCAT scores to the admissions committee."

In my last post, Characteristics of Future Physicians, I blogged about characteristics that I believe that we should want in graduates of our medical schools. There are several factors that medical school faculty have rated as extremely important but very hard to teach. My argument is that we should select students who already have those traits prior to matriculation. Some of the traits listed are obvious: (1) is emotionally stable; (2) is a person of unquestionable integrity; (5) is unusually intelligent; and (6) has sustained genuine concern for patients during their illness.  Some are less obvious but seem really important: (9) is motivated primarily by idealism, compassion, and service; (14) is able and willing to learn from others; (17) is observant; and (18) is adaptable.

So the question for today is: are there personalities that do better in a medical curriculum and those that do worse? If you read the article by Dr Sade (2) I am sure that it is no surprise that there are some personalities that do better in medical school and some that seem to struggle more. Medical school is an extremely stressful environment. Many of the students that come to medical school, have never struggled academically in their lives. I have heard folks say, medical school is a pressure cooker.

The pressure cooker brings out all of the problems. It stresses them. It pushes them in ways that many of these students have never been pushed. 28 percent of physicians report that they have stress that affects their ability to provide clinical care.(3) A recent systematic review found that medical students perceive similar levels of stress to physicians.(4)

I see students struggle because they have never been in this kind of situation. Students are young, they are often coming straight from undergraduate school. Students that come to medical school have often not had any real life experiences. They have not had a job, or had extensive life experiences. Then we put them into an academically stressful environment that is extremely competitive and at some schools even cutthroat. The good news is that some personalities may be better at handling the stress of medical school, and then potentially life as a physician. McManus (5) studied stress in UK students over a period of 12 years. The authors found that stress could exacerbated or even caused by personality factors, specifically by high levels of neuroticism, low levels of extraversion and low levels of conscientiousness. These traits are dimensions that are part of the Five Factor model of personality. This model is well accepted and validated. The model is used by many of the studies of personality contributing to success in academic settings.

Another study from the UK (6) looked at Goldberg's "Big Five dimensions of personality" to study 176 medical students at the Nottingham Medical School. The authors compared the students' personality scores and academic assessments in Years 1 through 5. The conscientiousness dimension was a significant predictor of academic performance in the pre-clinical years but interestingly in the clinical years (Years 4 and 5) it correlated with poorer performance.

So, there are personalities that may be more suited for medical school. And even more importantly, these personality traits can affect success as physicians. The Aussies looked at this a few years ago. In this study, Knight, et al (8) studied whether personality characteristics, measured by the Hogan Developmental Survey (HDS), were associated with academic performance in 139 medical students. They found that borderline/schizoid and narcissistic/antisocial characteristics were negatively correlated with academic success. That one seems pretty obvious. One of the subsections of the HDS, the ‘Diligent syndrome’, were found to be positively correlated to higher exam scores. The Diligent syndrome is students who have a tendency to be attentive and good with details, orderly, rational, careful and well organized.

What about here in the US? The folks at Jefferson Medical College are the only ones who have studied this. Hojat, et al(8) studied six personality measures. They also asked about the students' relationships with their parents and their general health. They compared these assessments with global faculty ratings of competence in the clinical clerkships (family medicine, internal medicine, obstetrics and gynecology, pediatrics, psychiatry and surgery). The ratings used a 4-point scale (‘high honours’, ‘excellent’, ‘good’, ‘marginal competence’). The students in the lowest group had significantly lower levels of self-esteem and sociability, they were lonelier, and had less satisfactory relationships with their parents.

So, what should we do? Conscientiousness seems to be an important predictor of success in the preclinical years, but it may also lead to vulnerability to stress. And it is related to worse performance in the clinical years. Extraversion, self-esteem, and sociability seem to be more important in the clinical years. Maybe we should do like the Australians did and have every applicant fill out a personality profile. I am not sure how this was used, but it does not seem that it was used to choose students. Next time, I will write about some of the ways to screen incoming students.

References
(1) Smith S. A Recipe for Medical Schools to Produce Primary Care Physicians. New Eng J Med 2010; 364(6).
(2) Sade M, et al. Criteria for selection of future physicians. Ann Surg. 1985 February; 201(2): 225–230.
(3) Firth-Cozens J. Doctors, their well-being, and their stress. BMJ 2003;326:670–1.
(4) Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among US and Canadian medical students. Acad Med 2006;81 (4):354–73.
(5) McManus IC, et al. Stress, burnout and doctors’ attitudes to work are determined by personality and learning style: a 12-year longitudinal study of UK medical graduates. BMC Med 2004;2:29.
(6) Ferguson E, et al. Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. BMJ 2003;326 (7386):429–32.
(7) Knights JA, Kennedy BJ. Medical school selection: impact of dysfunctional tendencies on academic performance. Med Educ 2007;41 (4):362–8.
(8) Hojat M, Callahan CA, Gonnella JS. Students’ personality and ratings of clinical competence in medical school clerkships: a longitudinal study. Psychol Health Med 2004;9 (2):247–52.

The characteristics of future physicians

In my blog from 2/4/11 "Why do we put so much import on the MCAT?", I discussed some of the negative characteristics that can be associated with a higher MCAT. The conclusion of the article by Dr. Gough¹ was that students with higher MCATS and a scientific orientation were found to "less adept in interpersonal skills, less articulate, narrower in interests, and less adaptable than their fellows". 

Wow!  I don't think those are characteristics that I want in my doctor.  What about you?

What are the characteristics that we want in our medical students? We want them to be great at science, right?  On average, academic performance in undergraduate classes only predicted about 9% of the variance in medical school performance.² What about MCAT? We want them to have a high MCAT, right? Well, a high MCAT is good at predicting performance on the USMLE step 1 and preclinical grades,³ but as someone who is really smart once told me "we are not trying to make step 1 passers".

Sade and colleagues asked this same question a few years ago. Their specific question was to identify the specific characteristics that are important qualities of a superior physician. They also asked which of these qualities are hardest to teach in the medical curriculum. They based their work on a study by Price, et al⁴ who had previously generated a list of positive traits associated with a superior physician.  

Dr. Sade took this list of traits and showed them to the faculty of the College of Medicine at the University of South Carolina. The faculty were asked to rate the personal qualities on a scale of 1-10, where 1 is non-teachable and 10 was easily teachable. The survey was sent to all of the faculty at the college of medicine. They also asked a select group of experienced medical educators to take the survey. There was remarkable agreement between the faculty, greater than 80% inter-rater reliability. There was also a high correlation between the basic science faculty's ratings and the clinical faculty's ratings of the importance of characteristics (r=0.87, p<0.001) and the teachability of characteristics (r=0.93, p<0.001). 

The outcome of this survey was a list that ranked the characteristics from 1 to 87.  Each characteristic was given a rank for importance and for difficulty in teaching. The authors converted the rankings to a Z-score. (***Note: This was my favorite line in the manuscript...)  "The teachability Z-score was subtracted from the importance Z-score, and the combined Z-scores were multiplied by 10 and added to 50."  This gave a combined score that they called the NonTeachable-Importance Index (NTII). The NTII gives you a list of characteristics that are ranked from highest to lowest based on importance and the difficulty of teaching it to medical students.

That sounds like a good list of pre-matriculant variables to me.  If we can't easily teach it but it is important then obviously we should select students that have these characteristics before coming to medical school.

Using the NTII ranking, some of the characteristics are obvious: (1) is emotionally stable; (2) is a person of unquestionable integrity; (5) is unusually intelligent; and (6) has sustained genuine concern for patients during their illness.  Some are less obvious but seem really important: (9) is motivated primarily by idealism, compassion, and service; (14) is able and willing to learn from others; (17) is observant; and (18) is adaptable. The list goes on from 1 to 87. 

The characteristic that was ranked as the most difficult to teach was: is unusually intelligent. The next four were: (2) is naturally energetic and enthusiastic; (3) is imaginative and creative; (4) has a warm, friendly, outgoing personality; and (5) is motivated by sheer liking of people.

So why are we still choosing medical students based on the MCATs and GPAs? Maybe, we should be looking at these factors.

Next time, I am going to write about personality factors that influence medical student performance.

References

(1) Gough HG. Some Predictive Implications of Premedical Scientific Competence and Preferences. J Med Educ  1978; 53: 291-300.

(2) Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ  2002; 324: 952–7.

(3) Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med  2007; 82(1): 100-6.

(4) Price PB, et al. Measurement and predictors of physician performance: two decades of intermittently sustained research. Salt Lake City: Aaron Press, 1971; 121-149.

Why do we put so much import on the MCAT?

Every year thousands of potential medical students spend a lot of time and money to study for the MCAT. Students spend a lot of money, for example,  $1,749 to take the Kaplan Complete MCAT Preparation and $1,000 for the Princeton Review. Why?? They spend this money preparing themselves to sit for this life changing examination. The test itself is actually comparatively inexpensive-- $235 dollars. 

The worst part is that the MCAT is probably not measuring any of the most important variables for our matriculating medical students. I have been on the admissions committee at two different medical schools.  Both were State supported medical schools with a strong commitment to graduating students interested in primary care, rural practice, and practice in underserved areas. But at the Admissions Committee level there is a serious lack of understanding of the importance of pre-matriculant noncognitive data and the variables used to select students to enter the school. Many of my colleagues (physicians and basic scientists) over the years have held the belief to some extent or another that higher pre-admission scores lead to better medical students which leads to better doctors.

Nothing could be further from the truth. In fact, for most of the variables there is very little correlation. And even more worrisome for many of the important characteristics of being a physician, there is an inverse relationship.  At this point, many of you are thinking, "Delzell is completely off his rocker!". In fact, one of my basic science colleagues said as much last year.  Well, that may be true, but I do have evidence to back up these statements.

Let's go back. Way back, to 1978. Harrison Gough, PhD, a psychologist at the university of California-Berkley, published one of the most fascinating studies (1) that I have seen in the medical educational literature. It is also one of the best written papers that I have ever read, and as an editor for a major medical journal I get to read a lot of manuscripts. Dr. Gough collected data on medical students from the University of California-San Francisco from 1951 to 1977. Wow! That is like the Framingham study of medical students. A longitudinal study of a medical school and its medical students.  This study reports on 1,195 UCSF students from 1972 to 1977. Data collected included MCAT scores, undergraduate GPA, and a measure to assess scientific preferences.  He created a composite index score by adding the "measures of scientific talent" (MCAT Science subtest and Undergraduate Science GPA) to Science Preference.

Science Preference is a fascinating concept that was developed by Goldstein (2) and modified by Dr. Gough. Students rated the three subjects from college that they liked best and the three that they liked least. The average score for the three least liked subjects was subtracted from the average score for the three subjects liked best. This gives an overall score, which was termed the students' preferences for science.  

The next step was to measure correlations between these measures and performance during medical school. He looked at grades in all four years of school. As you would expect, there was a significant correlation between the composite index and GPA in year 1 (Pearson Product- Moment correlation .34; p < 0.01) and year 2 (Pearson Product- Moment correlation .21; p < 0.01). But, there was no significant correlation with GPA in years 3 and 4. More importantly medical school faculty ratings of clinical competence and general competence were not significantly correlated (Pearson Product- Moment correlation .01; p = ns).

Ok, so maybe MCAT and GPA are not great at predicting things after the first two years but those years are important. Right???  Someone very smart once told me, "they would be really important if we were trying to make Step 1 passers, but we are not trying to make Step 1 passers, we are trying to make doctors."

Fortunately, Dr. Gough didn't stop there, there was another part to his study. He selected 70 students for an intensive study of personality at the UC- Berkley Institute of Personality Assessment and Research. The students were evaluated by 10-15 trained assessment staff members. Students were observed closely for an entire day and the staff members described them using a 300 item Adjective check list. These descriptors were then correlated with the four previously evaluated science predictors for each student.  

There was no statistical correlation between MCAT scores and personality descriptors. But, Science GPA was correlated with "painstaking"(r =.26) and "silent"( .26). There were negative correlations with adjectives such as "poised" (r = - .32), "self-controlled" (- .30), and "interests wide" (- .27).  When compared to the composite index there were also several significant correlations, such as "awkward" (r = .27), "cautious" ( .23), and "conservative" ( .27).  The composite index was negatively related to several descriptors, such as "stable" (r = - .28) and "relaxed" (- .28).  

Each of the students was also judged by all of the observers. The reliability of their judgment is striking. The inter-rater reliability was greater than .80.  The students with higher composite index were rated lower in their ability to communicate (r = - .28), breadth of interests (r= - .35), self-acceptance (r = - .26), and verbal fluency (r= -.29). All of these correlations were significant. By the way, the MCAT by itself also had a negative correlation with every measure, with r values between - .11 and - .20.  Dr Gough's conclusion was that scientifically oriented students were "less adept in interpersonal skills, less articulate, narrower in interests, and less adaptable than their fellows.". 

Wow! That is amazing. Why didn't anybody tell about this when I joined the admissions committee? For my next blog, I am going to write about some important characteristics of future physicians and how we can better select students that will have those characteristics.

References

(1) Gough HG. Some Predictive Implications of Premedical Scientific Competence and Preferences. J Med Educ  1978; 53: 291-300.

(2) Goldschmid ML.  Prediction of College Major by Personality Tests. J Counseling Psychol  1967; 14: 302-308.

Dr Jerry Kruse's Seussian rhyme

The following Seussian rhyme was imagined and written by Dr Jerry Kruse, Professor and Chair, Department of Family & Community Medicine, Southern Illinois University School of Medicine.
It was given during the final plenary session at the 2011 STFM Conference on Medical Student Education.

The Saga of Michael Klein

Or…..

 Ein Kleiner Schnitt

Or…..

The Triumph of Reason Over Power, Finally!

By Dr. Kreuss*

In that faraway land to the North, in Quebec,

Lived a doctor whose practice was very low tech.

A family doc, accoucher Michael Klein,

Who didn’t like forceps or women supine

Or ‘lectronic monitors, stirrups or sections

Or enemas, shaving or IV injections.

He hated electrodes and IUPC’s

And treatment of labor as if a disease.

And one of the worst – epidural blockade –

A stab in the back to start the cascade

Of catheters, tubing, Nubain and pit

And Sulfate of Mag so she won’t have a fit.

Blood pressure cuffs and punctures of veins,

Cesarean Sections and Tucker-McLains

Retained placenta, post partum metritis

I’ll bet you a buck she’ll come down with mastitis.

“I don’t like these women to all be strapped down.

Stand up and walk!”  he cried with a frown.

Michael knew in his heart, way deep down inside

That obstetrical knowledge was not well applied. 

“Technology’s great, for those who are ill,

But for those who are healthy it’s really no thrill

To be strapped down and poked, and scared stiff as a board.

This just isn’t right!”, his fervent voice roared.

One thing more than others, did gnaw at his heart,

Made his blood boil, and stung like a dart

He just couldn’t stand it, to see a long slice,

A cut, an epis - what a terrible vice,

Disruption of skin for no reason at all,

A snip with the scissors that starts very small

But rips and extends as the baby comes through

Tears into the sphincter and up the wazoo.

A third, then a fourth, oh my what a mess

“They must like to sew, is my only guess.”

So Michael jumped up, and he raised his right hand

And opened his mouth, and he took a firm stand

“I’ll study this problem,” he said with a shout

“And when I am finished there won’t be a doubt

That these cuts are no good…the whole world will see….

This idea’s a good one, they’ll have to agree.

I’ll start up a randomized, single blind study

And I’ll work with Michel who’s my very good buddy

And we’ll put ole’ McGill right here on the map.

This study of perineal trauma’s a snap.

“We’ll put in a grant, we’ll get recognition.

We send all this stuff to a good statistician.

Our alpha will be less than point zero five

And beta point two will let us derive

The number of women we’ll need.  It’s a slew.

We’ll enroll ‘bout a thousand five hundred and two.”

He worked and he toiled, he felt quite convicted—

The results were exactly the ones he predicted.

“Midline epis, when routinely done

For women in labor, is not very fun.

Our EMG’s show that sliced muscles get weak,

They heal up quite slowly and let urine leak.

And that isn’t all that comes out that should not.

She’ll find our real quick when she sits on the pot.

“The relative risk for a fourth degree rip

Is greater than twenty, with each little snip.

And all of us know, if the rectum is torn

That a permanent hole will often be born

That connects the vagina and rectal mucosa

And where it comes out then will make you nervosa.

So my warning to you, who practice OB

Is to use the epis quite conservatively.

Don’t be in a rush, and don’t interfere

Throw down the scissors, they’re not needed here.”

Now Michael was proud, and really excited

With his results he was very delighted.

He wanted to spread the good news he had found

He wanted to broadcast this stuff all around,

But a funny thing happened.  He couldn’t believe it

No one would listen, they just couldn’t perceive it.

The doctors in charge of the medical journal

The ones who are experts in issues maternal

Did not want to hear about data that’s new.

It was hard to convince that conservative crew.

“No one will believe it,” the editors cried

And they wrote down “REJECTED” with feelings of pride

And one after another they all did the same.

They suppressed this great knowledge – oh my what a shame!

But this story’s not over, he didn’t back down

He battled the towers who sport cap and gown.

Then once, then again and a third time, it’s true

His papers were granted another review.

And though eight years late and quite overdue

All three ended up in the publishing queue.

And now the world knew, both up in the North

And in states to the South, the new message went forth.

The paradigm shifted, the good word was this:

“The epis is archaic, and you’ll be remiss

To ignore this great knowledge that newly exists

And to squander this chance to cause perineal bliss!”

The rēsearch of Michael had ended the reign

Of procedure and practice we all thought mundane,

Of cuts and incisions most surely inane

And the scissors were thrown to the floor with disdain.

In just over a decade, the rate of epis,

Of pelvic dysfunction and fourth gaping degrees

Had tumbled to levels that went far below

The figures observed just a few years ago.

From sixty percent of all getting cleaved

To just nine in a hundred a cut to receive.

“Never give up!”  Michael’s voice still rings clear.

The moral today is to be of good cheer

To persistently fight, in the face of all the odds.

To battle ideas of conventional gods

To fight for new facts, and new evidence find

To give power to reason and sight to the blind.  (THE END).

*Dr. Kreuss                 Jerry Kruse, MD, MSPH

                                    Professor & Chair

                                    Department of Family & Community Medicine

                                    Southern Illinois University School of Medicine

                                    Springfield, Quincy, Carbondale and Decatur, Illinois

Bibliography:

1.      Klein MC, Gauthier RC, Jorgensen SH, Robbins JM, Kaczorowski J, Johnson B, et al.  Does episiotomy prevent perineal trauma and pelvic floor relaxation?  Online J Curr Clin Trials, Doc 10, July 1, 1992.

2.      KIein MC, Gauthier RJ, Robbins JM, Kaczorowski J, Jorgensen SH, Franco ED, et al.  Relationship of episiotomy to perineal trauma and morbidity, sexual dysfunction, and pelvic floor relaxation.  Am J Obstet Gynecol 1994; 171:591-8.

3.      Klein MC, Kaczorowski J, Robbins JM, Gauthier RJ, Jorgensen SH, Joshi AK.  Physician beliefs and behavior within a randomized controlled trial of episiotomy; consequences for women under their care.  Can Med Assoc J 1995; 153:769-79

4.      Huston P.  The pursuit of objectivity [editorial].  Can Med Assoc J 1995; 153:735.

5.      Schultz KF.  Unbiased research and the human spirit: the challenges of randomized controlled trials [editorial].  Can Med Assoc J 1995; 153:783-6.

6.      Klein MC.  Studying episiotomy:  When beliefs conflict with science.  J Fam Pract 1995; 41:483-8.

7.      Frankman EA, Wang L, Bunker CH, Lowder JL.  Episiotomy in the United States:  has anything changed?.  Am J Obstet Gynecol 2009; 200: 573.e1-573.e7

8.      Hyer R.  ACOG 2009:  Steep decline in episiotomy rates credited to research, peer pressure.  ACOG 57^th Annual Clinical Meeting, Medscape Medical News.  http://www.medscape.com/viewarticle/702541

The rhyme contained in this blog is the intellectual property of Dr. Kruse and cannot be copied without his express consent.  Thanks. JED

The STFM Conference on Medical Student Education

Last week I had the privilege of chairing the 37th annual STFM Conference on Medical Student Education. Until 2010, the conference was known as the STFM Predoctoral Education Conference. We changed the name to the Conference on Medical Student Education. You may not know much about STFM. The Society of Teachers of Family Medicine is my academic and professional home. All of my mentors, my teachers, my peers, and my colleagues are in STFM. It is a great organization. The Conference on Medical Student Education is a premier educational meeting that includes most of the family medicine educators from around the country. 

Let me give you some highlights of the meeting. 

We started the meeting with an amazing plenary speaker. Dr Kevin Eva, Senior Scientist from the Centre for Health Education Scholarship (CHES) at the University of British Columbia in Vancouver, Canada. Dr Eva gave an invigorating talk about medical decision making. My favorite concept from the talk was that we have to make errors in order to get better, and maybe more importantly, we as educators have to provide safe environments that allow students to make those mistakes. His talk is posted on FMDRL.

There was a great talk by Stacy Brungardt, CAE (Executive Director of STFM) about the alphabet soup of family medicine. She described several of the organizations that make up the "family" of family medicine (AAFP, CAFM, COGME, etc...). There was an excellent peer session describing a study of teaching students about the Four Habits model of patient-centered communication, by Dr Hannah Maxfield and colleagues. (full disclosure here, Drs. Maxfield, Zaudke and Chumley are my colleaguesy at KU)

Dr. Chumley and I presented some of our data about using Artificial Neural Networks to classify students' information gathering patterns to make a diagnosis. We looked at 200 students' performance on a standardized patient case, with a 22 item checklist. We used the first 100 patients to train the ANN, and then we tested the neural network with the second 100 cases.  We found that the ANN was able to predict whether the student got the right or wrong answer/diagnosis with a 85% accuracy.  This was better than two other standard classifiers called Bayesian and KNN (K Nearest Neighbor).

There was an awesome dance party on Friday night that brought together faculty (old and young) with medical students.  

The Saturday morning plenary was by Dr Cathy Pipas from the Dartmouth medical college. Dr. Pipas is the Vice Chair of Community and Medicine. She gave a stimulating talk about the transformation of the Dartmouth practices to patient centered medical homes. The scary part of that talk was that the senior administration at Dartmouth have still not aligned the financial incentives with the clinical practices that are transforming to PCMHs.

Drs. Jana Zaudke and Hannah Maxfield presented an interesting randomized trial of giving feedback about the Four Habits model of communication after watching the students perform on a standardized patient.

On Sunday morning Dr. Joshua Freeman moderated a special session on social justice and family medicine. There's were several medical students at the session and we had a great discussion after his talk.

The final plenary for Sunday morning was Dr. Jerry Kruse. Dr. Kruse is the Chair of the Department of Family and Community Medicine at Southern Illinois University School of Medicine. I asked Dr. Kruse to talk about his views of health care reform. He said that there are two different and divergent views of healthcare reform and its importance to the nation's progress toward the future.  He called the passage of the health care reform bill last year, "the triumph of reason over power". Dr Kruse is famous amongst his friends for his poetry. He gave the most amazing Seussian rhyme describing the saga of Dr Michael Klein, the Canadian doctor that studied the routine use of episiotomy. Dr. Kruse gave me permission to post the lyrics of this poem for your edification. Look for it coming in a couple of days.

Dr. Kruse also presented the new COGME report, "Advancing Primary Care" and its recommendations. The most important recommendation from COGME was that the percentage of primary care physicians should be at least 40% of all physicians.

Overall, this was a great meeting.  Thanks to all of the presenters for your great work. Thanks to all the attendees, including over 200 students attending the national student-run free clinic forum. Thanks to the STFM staff for your hard work, in particular Ray Rosetta, the hardest workin' man in the conference business.  Next year, the meeting will be February 2-5, 2012 in sunny Long Beach, California. The Call for Papers opens in March, so get ready!

Criteria for selecting students in the Match

We are fast approaching a very important day in the academic calendar. On February 23, 2011 residency programs around the country have to enter their Rank Order List. This day is the culmination of a lot of work by the Program Director, faculty and residents from the program, and from students applying to that program. The actual day that the results of the Match are released is about a month later on March 17. But the work is all done once the lists are in on February 23.

You may not understand this process, so let me walk you through it. Medical students around the country decide what specialty they are interested in applying to at some point during the third year of medical school. Students gather letters of recommendation from faculty over the next several months. Frequently, they will do a fourth year elective rotation in their specialty of interest. They also have to decide if they are going to try to go to another school and do an “away” rotation. For most students, they are obligated to enter the National Resident Matching Program.  If they are a fourth-year student at an allopathic medical school in the US, they have no choice but to enter the Match.

Beginning in October, students begin their job interviews. We call these residency interviews, but honestly the students are trying to land a job as a resident in a particular program. Students will have anywhere from 10 to 40 interviews depending on the competitiveness of the specialty that they are applying to enter. These interviews may be anywhere across the country, but are mostly in larger cities (that is where the teaching hospitals are located).

So, on February 23 the students enter their Rank Order List. The program that they like best is Number 1. Their least favorite is last. Residency programs do the same. They rank all of the students that they interviewed from 1 to however many they want to rank.  The programs don’t have to rank all of the students that they interview, and the students don’t have to rank all of the programs.  But the Match is a binding contract,¹ if they rank someone (student or program) they are legally bound to that ranking.

The question for today is how do programs decide how to rank the students that they interview? There are several ways, some good, some really bad! Let’s start with the good ways.

Letters of recommendation can be very helpful, if they are written by honest faculty physicians, who know the student, and have personally worked with a student. These letters can be a great assessment of a student’s global performance. An old study by Keynan, et al ² done in 1987 compared objective faculty ratings to other types of assessment. This study compared a global faculty rating, a multiple choice question (MCQ) test, and an oral examination.  They found that the “the 'subjective' expert assessment of performance through global rating scales is comparable to that of 'objective' evaluation through written MCQ.” They also found, using a stepwise regression analysis, that the ratings of 'reliability', 'knowledge', 'organization', 'diligence,' and 'case presentation' were the most predictive of the overall global rating. Chair’s letters which are often written by the Chair of a department (who probably does not know the student very well) are generally not much help.

Another good way to rank students is through an interview. Skilled interviewers can pick up on many communication and personality issues that probably don't show up on a paper application. Maybe the applicant is very introverted and has difficulty talking during the interview. Or maybe they are a jerk or a racist or a sexist. A personal interview can pick up these problems (not always, but often).

Unfortunately, there are also some bad ways to rank students.  Commonly, grades and boards are used. Frequently, medical school grades and USMLE board scores are the screens that decide whether a program invites a student to interview.

I want to focus on USMLE scores. Grades are quite variable from school to school. Some schools have an A to F scale, some have Pass/Fail, and others have Satisfactory to Superior. Preclinical grades don't have a lot of predictive value for clinical grades and neither are very predictive for performance in residency.

Board scores are just as bad. They seem to be an objective way to compare students. Everyone, across the country takes the same test. There is one big problem. The USMLE is designed to measure knowledge and application of knowledge. It was created to be used by the State Licensing agencies as a common evaluation for licensure. There are statistical problems when you try to interpret the scores that are given with a pass/fail based test. There have been several studies that all show basically the same thing about board scores. Performance on the boards does not correlate to performance as a physician.

In 2005, Rifkin and Rifkin³ compared the performance of all the first year Internal Medicine residents at a large academic medical center on standardized patient encounters to their scores on the USMLE Step 1 and 2. They found very low correlations. For Step 1, the correlation was 0.2 (df=32, p=0.27) and for Step 2 it was 0.09 (df=30, p=0.61). Remember a higher number means that the two measures are more strongly related.

A more recent study is very critical of the use of USMLE scores for selection of residents. This study by McGaghie and colleagues,⁴ was a research synthesis using a critical review approach.⁵ They collected and reported correlations between USMLE Step 1 and 2 and several reliable measures of clinical skills. These skills included auscultation of the heart, performance of ACLS (Advanced Cardiac Life Support), communication with patients, thoracentesis, and central line placement. They found correlations from -0.05 to 0.29 to Step 1 and -0.16 to 0.24 for Step 2.

Their conclusion sums it all up. "Use of these scores for other purposes, especially postgraduate residency selection, is not grounded in a validity argument that is structured, coherent, and evidence based. Continued use of USMLE Step 1 and 2 scores for postgraduate medical residency selection decisions is discouraged."

I couldn't agree more. If I need a neurosurgeon to operate on my brain, I want to know that he has a very steady hand, not the highest board score. If I need a radiologist, I want to know that her visual pattern recognition is outstanding, not that she scored well on a multiple-choice question test. And if I need a family doctor, I want to know that his clinical reasoning and communication skills are excellent, not that he scored well on the boards.

References

1. http://www.nrmp.org/res_match/policies/map_main.html

2. Keynan A, Friedman M, and Benbassat J.  Reliability of global rating scales in the assessment of clinical competence of medical students. Med Educ  1987;21(6):477-81.

3. Rifkin WD, Rifkin A. Correlation between house staff performance on the United States Medical Licensing Examination and standardized patient encounters. Mt Sinai J Med. 2005;72(1):47-9.

4. McGaghie WC, Cohen ER, and Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med  2011;86(1):48-52.

5. Eva KW. On the limits of systematicity. Med Educ. 2008;42:852–853.