First in a series on education.
When I set out to write this post, I was expecting a relatively simple story line, something along the lines of “technology saves education, data proves it.”
The actual story line has proved to be quite a bit more complicated, with a lot of strong and contradictory opinions out there and a surprising lack of published, quality data. In the interest of improving the general quality of discourse, I have emphasized results supported by quality data while keeping opinions about what should work and what can’t possibly work to a minimum.
One of the most compelling studies of the potential for improvements actually predates not only online learning but the world wide web itself. In 1984 Benjamin S. Bloom, a Professor at the University of Chicago, published a paper entitled The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. The paper was based on the findings in the PhD dissertations of two of his students, Anania and Burke, who had conducted experiments with elementary school students of various grades, testing relative performance under differing protocols. Not surprisingly, the most successful standard was one-to-one tutoring. What is surprising is the magnitude of the improvement: a full two standard deviations. Translated from statistical speak, that means the average tutored student outperformed 98% of the students in the “control” (i.e. regular classes) group. Take average kids, tutor them instead of sending them to class and they turn into academic stars – or at least they perform like academic stars.
Bloom and his students considered the cost of tutoring to be prohibitive, so the dissertations investigated other approaches that might achieve most of the benefits of tutoring for a fraction of the cost. Tutoring aside, the most effective single technique for improving outcomes is known as mastery learning, which was found to produce a 1 standard deviation improvement. The figure below, reprinted from Bloom’s 1984 paper shows the relationship among the three approaches.
Through combining various complementary techniques Anania and Burke were able to achieve improvements of as much as 1.6 standard deviations, but even in combination nothing could match high quality tutoring.
Although left largely unexplored in Bloom’s original paper, tutoring has a second highly desirable attribute: tutoring narrows the dispersion of outcomes. The graphic above clearly depicts a narrowing of the distribution. In other words, not only did the average performance improve, but the gap between the best and worst performing students closed. This attribute further gilds the reputation of tutoring as the gold standard, as it appears to benefit the weakest performers the most. Tutored students not only average excellence, but they do so more uniformly. Consistency of results is sometimes overlooked by proponents of various online learning initiatives, but from a public policy perspective greater consistency is often a primary goal, as evidenced by the prevalence of metrics that measure the number of students that surpass a performance threshold.
For purposes of this note and those that follow, tutoring, and the 2 standard deviation improvement it produces, is the gold standard in educational improvement. Perhaps there is a higher standard out there that will someday be achieved, but for the time being a technique that takes average kids and turns their performance to magna cum laude levels is a worthy standard. Sadly, of course, this gold standard is also priced like gold. If only there were a way to achieve that kind of efficacy without the prohibitive cost…
