Survivorship Bias in Education
“Don’t armor the places that sustained the most damage on planes that came back. By virtue of the fact that these planes came back, these parts of the planes can sustain damage.” -Abraham Wald
An Interesting Example
During World War II, Abraham Wald worked for the New York company Statistical Research Group. The American military approached him with a problem. War planes were coming back from air battles covered in bullet holes, and the military was interested in protecting future planes by using minimal armour placement. The question was where to best place this armour so as to protect the planes and pilots. The bullet holes were roughly distributed as such:Where would you choose to place the minimal armour?
If you said any section other than the engine section, then you would be absolutely… wrong. Wald’s explanation is as follows: the statistics that were presented were gathered from planes that had survived battle – the planes that didn’t survive battle were not included in the numbers. Thus, the planes that returned were not a random sample and conclusions could not be made by only reflecting on the surviving planes. Wald recommended adding armour to the sections of the plane that returned home relatively unscathed (here, the engine section) would be wise, as the surviving planes tell us the regions of the plane that can take damage and still return home safely.
Suggesting to give armour to regions of the plane that are damaged the most reflects a statistical bias known as survivorship bias. This is an error that arises when we only consider objects that “survive” a particular process, ignoring the objects that do not survive. For example, in In Search of Excellence, Peters & Waterman observe 43 “excellent” companies and determine eight traits that have propelled these companies to the top. However, as of 2016, 20 of the 35 public traded companies were under market average, and five of these companies have since gone bankrupt. So what’s going on here? Are these companies actually “not excellent”? Well, there are other factors at play here, such as luck, that may have helped these companies propel themselves to the top. And since Peters & Waterman only chose companies they deemed as “excellent”, their sample reflected survivorship bias: maybe some of the poorest-performing businesses also used these eight traits – we don’t know since only “excellent” companies were analyzed.
Survivorship Bias in Education
The above aforementioned is why I typically don’t care too much for the advice of celebrities. Why would I listen to their advice on how to be successful when they are a part of the small minority that have survived the process? How do we know there aren’t other factors such as luck or socioeconomic status that have helped advance their careers? What about all of the people who have tried to become a celebrity by following this advice, but haven’t survived the process?
So why is it that we listen to similar advice of celebrities in mathematics education?
Arguments such as “I never memorized my times tables and it’s never held me back.” reflect survivorship bias. Maybe the statement is true, and that’s fine, but what is missing are the countless number of individuals who did not memorize their times tables and have been held back in some form. We need to see and hear those stories to get a full picture on what to conclude. In addition to the previous argument, award-winning mathematicians who “think slowly” and “visually” are often given as key examples in pedagogical arguments. What about all the mathematicians who are not visual thinkers, or who don’t think slowly? Are there other factors, such as the field of study, that cause these traits? Again, more examples are needed to get a better sense of the typical traits of a mathematician.
As educators, I think we get so excited when we hear about the stories of progressive-style pedagogy that works, that we often forget that there may be just as many stories in which progressive-style pedagogy doesn’t work. Traditional-style pedagogy isn’t safe from survivorship bias either – how many times have you, within a traditional-style lesson, received a correct answer from one student and proceeded through the lesson, oblivious to the fact that there may be several students confused? You are making an assumption based on the “survivors” and your “non-survivors” are potentially being left behind. We need to be mindful of this, and not allow survivorship bias to elude us into thinking something is there when it is not.
I want to ask a favour of all the teachers out there: I want to know of a time when you utilized progressive-style pedagogy and it failed. Personally, I had one class in which I flipped my lessons and it was a disaster – with pass rates being abysmally lower than normal. This came after I was on a high from the last semester, in which I flipped my class and pass rates were much higher than normal! There were definitely differences between the two cohorts of students that I had failed to see until post-reflection.
Interestingly enough, Peters & Waterman, the authors of In Search of Excellence, later came out and said that they had made up some of their data. I firmly believe that both sides of the math debate have valuable advice to offer, which is why I am interested in hearing the different stories out there. I do get concerned, however, when teachers/researchers begin to fall into the Peters & Waterman trap: “… go find smart people who are doing cool stuff from which you can learn the most useful, cutting-edge principles … [and] worry about proving the facts later.” Rather, let’s observe all sides of our pedagogy (both the good and the bad) and use this information to perfect our craft.