In the Nation's Interest

Choosing Quarterbacks and Hiring Teachers

Apparently it is very hard to predict which college quarterbacks will succeed in the NFL.  Malcolm Gladwell wrote about this in the New Yorker in 2008, and the Atlantic published a more recent article on the topic last year.  Academics even weighed in with an analysis in 2011.

What does this have to do with education?  Well, as Gladwell pointed out, K12 education faces a similar problem: teachers are really important (perhaps even more important than quarterbacks?!) and like quarterbacks, we have a hard time predicting which ones will be effective when we hire them.  That was true when Gladwell wrote on this issue in 2008, and since then we haven’t progressed a whole lot in our ability to predict who will be a good teacher.

I was reminded of all this last week at the Society for Research on Educational Effectiveness (SREE) conference, where I attended a fascinating presentation on how to identify effective teachers at the time of their initial hire.

First let me highlight why teacher selection is so important.  Teachers have consistently been found to be the most important within-school factor affecting student outcomes.  (Within-school is researcher-speak for all the factors a school controls – in contrast to factors affecting outcomes that the student “brings to the schoolhouse door,” such as a student’s socio-economic status.)  Raj Chetty et al. of Harvard University found that having a teacher in the top 5% of teachers for one year raises a student’s lifetime earnings by $80,000 on average, as well as increasing the likelihood that the student will attend college and other positive outcomes.  If you multiply this impact by a classroom of 20 students, as Eric Hanushek of Stanford University did in a separate study, a teacher in the top 15% of teachers increases one classroom’s discounted lifetime earnings by over $400,000.  Multiply that by the number of years the teacher spends in the classroom, and we’re talking near-NFL money.

So good teachers definitely matter.  And we can identify who are good teachers “after the fact,” so to speak, through their impact on student test scores (which predict student earnings and outcomes in early adulthood).  But how do we get more of those good teachers in our classrooms?

Well, as employers know, there are at least four ways to have more effective staff:
   • Identify and hire the applicants who are most likely to be effective
   • Train and develop staff post-hire to maximize performance
   • Retain effective performers
   • Let go of ineffective performers, if necessary

Focusing on just the first of these four strategies in the context of teaching, the difficulty is that we’ve not been particularly good at identifying top candidates before we hire them.  It turns out that most of the information available at time of hire – including being certified, having a graduate degree, the teacher’s knowledge about how to teach the subject, and even teachers’ socio-emotional skills – doesn’t predict teachers’ effectiveness in their first year of teaching very well.

Once a teacher has been hired, her effectiveness tends to be rather stable, with her past year’s performance being quite predictive of the following year’s performance.  So getting the right teachers into the classroom initially is especially important.

That’s why the talk I saw by Alejandro Ganimian, a graduate student at Harvard, was so intriguing.  Ganimian’s presentation described a small study of Enseñá por Argentina (ExA), the Argentinian version of Teach for America.  (Teach for America is an alternative pathway into teaching that aims to encourage graduates of top universities to enter the profession.)

Would-be ExA teachers go through three stages before becoming a regular classroom teacher, with some applicants filtered out at each stage: 1) an online application, 2) interviews, a demonstration lesson, and other exercises at an assessment center, and 3) a summer training institute, including two weeks of “clinical” practice teaching with real students.

The Ganimian study was interesting because it examined how well the information collected at each of these stages predicted actual classroom performance during the teacher’s first year of teaching, as measured by classroom observations and assessments by the school principal.

As it turned out, among individuals who were actually chosen for ExA and made it to the classroom, neither the applicant’s online application nor his or her overall score at the assessment center were predictive of his or her subsequent first year effectiveness.

Now before you conclude that the online application was worthless, note that we’re looking at the performance of applicants who made it all the way through a highly selective ExA application process, which was only 32 of 1,400 applicants.  All of those 32 applicants presumably had very strong online applications, so their online application scores may not distinguish between those 32 individuals very well – just like everyone who makes it as far as Olympic trials has an incredibly high level of fitness, and minor differences in their stats may not tell you much.

Nevertheless, despite the impressive credentials of the ExA teachers, they did differ in their first-year effectiveness in the classrooms, as measured by classroom observations, student surveys, and assessments by their school principals.  As it turns out, their effectiveness during their two weeks of “clinical practice” teaching during the summer did predict their first-year effectiveness in a regular classroom, as measured through classroom observations and principal surveys.

What’s more, performance during the two-week summer “clinical practice” was most useful in predicting effectiveness (or lack of effectiveness) among the least effective ExA teachers.  That is, among applicants who scored below the 20th percentile during the summer clinical practice, the lower their summer-score then the lower their effectiveness during the school year.  For applicants above the 20th percentile in summer performance, their summer performance did not predict their subsequent performance on average.

The predictive value of the applicant’s performance during the 2 weeks of summer teaching remained even after taking into account his scores on the online application and the assessment center.  That is, the applicant’s clinical teaching performance added useful information about his future effectiveness beyond what was collected in the online application and at the rather extensive assessment center process.

Why does this matter?  Well, teachers matter, as we established earlier.  Initial performance tends to predict subsequent performance, at least among teachers.  Hiring teachers – like hiring any employees – is expensive.  And while more than 42% of teachers leave teaching within their first 5 years (with 12% or more leaving after just 1 year, depending on the subject), once teachers acquire tenure (typically within 3 years), it becomes very difficult to remove them.  So hiring the right teachers in the first place is essential.

I searched to find what proportion of school districts require job candidates to teach even a demonstration lesson as part of the interview process, and beyond some allusions to the fact that this practice was “increasing,” I couldn’t find any hard figures.  In any case, it appears that school districts do not universally require teaching applicants to teach even a demonstration lesson. 

And recall that in the Argentinian study, even the demonstration lessons had limited predictive value.  It was the applicant’s performance during the full 2 weeks of summer clinical practice that really added the predictive “oomph” regarding their first year performance.

I don’t want to overstate the significance of this study.  It’s just one study in one site with very small sample sizes (24 ExA teachers in one year and 32 ExA teachers in a second year).  So it’s best to view these results as worthy of further research and exploration, rather than definitive.

But given how important teachers are and how little progress we’ve made in even the last 6 years in knowing how to select those who will be most effective in the classroom, this study gave me a little hope.  Sure, using a 2-week clinical teaching exercise as a pre-hire screening device would be expensive. But given the stakes of getting it wrong, wouldn’t it be worth it if it helps schools identify effective teachers before they hire them?

Now if we could just figure out which quarterbacks to draft…

 

For more details on the Ganimian et al. presentation, click here.