All Questions are Good Questions

On Friday, I did the keynote address for the Connecticut Association of Student Councils' state conference.  It was a blast.  The energy from the kids gave me a tremendous boost in my stride.  As part of my day, the fantastic Todd Burlingham and I presented "Do the Math: Let X Equal Student Activities," the new presentation from the Alliance for Student of Activities.

In brief, "Do the Math" is a presentation that aims to promote the expansion of student activities at the middle school and high school levels.  Scholarly evidence of the highest quality and published in over 100 prestigious peer-reviewed journals show that student activities have an independent, statistically significant positive effect on almost any outcome you can choose.  You want a kid to have a better GPA, better test scores, better chance of graduating high school and college?  Put that kid in student activities, because student activities make all of those things happen and more.

The early stages of the presentation's history have been an unqualified success.  The presentation debuted in December of 2011, and there's a very good chance that 5,000 or more people will see it before the end of 2012.  Moreover, we're winning the hearts and minds of our audience faster than we thought possible.  In particular, principals seem to be paying attention.

The people behind "Do the Math" are mostly professional educators.  I am the lone social scientist in the group, so my role has been to translate these compelling results into terms that our audience of teachers, principals, school board members, superintendents, and policymakers can understand.  I have done a good enough job in this task for our presentation to be credible and professional, and I would stake my professional reputation that the information in the presentation is accurate.

However, I am not beyond being phased by a particularly good question about how statistics work.  After our presentation, a very nice teacher approached me  and asked what she thought was a really stupid question.  To make the point about independent effects in a setting with controls, our presentation asserts that the studies we cite create, through statistics, sets of identical twins, where everything about two people is exactly the same--same parents, same demographics, same level of achievement--except that one participates in student activities, and the other does not.

This teacher asked how statistics did that.

And I was stumped.

This Connecticut teacher, being all apologetic, asked a question about how multivariate statistical analysis actually worked, how it actually determined the size of the effect of each independent variable on the dependent variable, how we know that putting any tenth grader into student activities will result in an almost 6 percent gain in his/her scores on standardized math tests.  Keep in mind that I explain this stuff for a living.  I taught my department's methods courses and explained regression analysis more times than I can count.  Student evaluations and faculty review said I was really, really good at it.  And yet the right question at the right time threw me for a loop, at least for a few seconds.

The point to this long, only kinda focused blog entry: this statistics stuff is complicated.  Because I am immersed in stats pretty consistently, I can forget that the logic isn't intuitive.  It's a good experience for me to remember my limitations.

Also, I hope this story gives heart to the rest of the Alliance Ambassadors, who don't have the statistical training that I do.  Before our presentation this week, Todd said he was glad I was there, because the research findings confuse him.  Yet I saw Todd do an exceptional job explaining every statistical concept in the presentation.  One need not be able to explain every sophisticated nuance of statistics to understand and use them.  In Todd's case, and in the case of the rest of the trained ambassadors, they understand the notion of controls, longitudinal data, independent effects, and everything else well enough to explain them competently to educated audiences.  They may not be able to stand up at an academic conference and explain everything perfectly, but I am so proud and so impressed  by how much they have learned and are able to share.

Besides, if Todd and the rest of the Ambassadors could explain statistics as well as I could, I would have taught myself right out of a job.  It doesn't pay, but the fringe benefits are fantastic.

Lest the point of "Do the Math" get lost in this discussion of statistics, let me reiterate that we know, beyond all reasonable standard of doubt, that student activities make kids better.  Participants don't have better scores just because activities attract better students.  These things make kids smarter, more motivated, more compassionate, and whatever else you can think of, and we have empirical evidence that will withstand any standard of scholarly scrutiny.