Survivorship Bias in Education

“Don’t armor the places that sustained the most damage on planes that came back. By virtue of the fact that these planes came back, these parts of the planes can sustain damage.” -Abraham Wald

An Interesting Example

During World War II, Abraham Wald worked for the New York company Statistical Research Group. The American military approached him with a problem. War planes were coming back from air battles covered in bullet holes, and the military was interested in protecting future planes by using minimal armour placement. The question was where to best place this armour so as to protect the planes and pilots. The bullet holes were roughly distributed as such:UntitledWhere would you choose to place the minimal armour?

If you said any section other than the engine section, then you would be absolutely… wrong. Wald’s explanation is as follows: the statistics that were presented were gathered from planes that had survived battle – the planes that didn’t survive battle were not included in the numbers. Thus, the planes that returned were not a random sample and conclusions could not be made by only reflecting on the surviving planes. Wald recommended adding armour to the sections of the plane that returned home relatively unscathed (here, the engine section) would be wise, as the surviving planes tell us the regions of the plane that can take damage and still return home safely.

Suggesting to give armour to regions of the plane that are damaged the most reflects a statistical bias known as survivorship bias. This is an error that arises when we only consider objects that “survive” a particular process, ignoring the objects that do not survive. For example, in In Search of Excellence, Peters & Waterman observe 43 “excellent” companies and determine eight traits that have propelled these companies to the top. However, as of 2016, 20 of the 35 public traded companies were under market average, and five of these companies have since gone bankrupt. So what’s going on here? Are these companies actually “not excellent”? Well, there are other factors at play here, such as luck, that may have helped these companies propel themselves to the top. And since Peters & Waterman only chose companies they deemed as “excellent”, their sample reflected survivorship bias: maybe some of the poorest-performing businesses also used these eight traits – we don’t know since only “excellent” companies were analyzed.

Survivorship Bias in Education

The above aforementioned is why I typically don’t care too much for the advice of celebrities. Why would I listen to their advice on how to be successful when they are a part of the small minority that have survived the process? How do we know there aren’t other factors such as luck or socioeconomic status that have helped advance their careers? What about all of the people who have tried to become a celebrity by following this advice, but haven’t survived the process?

So why is it that we listen to similar advice of celebrities in mathematics education?

Arguments such as “I never memorized my times tables and it’s never held me back.” reflect survivorship bias. Maybe the statement is true, and that’s fine, but what is missing are the countless number of individuals who did not memorize their times tables and have been held back in some form. We need to see and hear those stories to get a full picture on what to conclude. In addition to the previous argument, award-winning mathematicians who “think slowly” and “visually” are often given as key examples in pedagogical arguments. What about all the mathematicians who are not visual thinkers, or who don’t think slowly? Are there other factors, such as the field of study, that cause these traits? Again, more examples are needed to get a better sense of the typical traits of a mathematician.

Concluding Remarks

As educators, I think we get so excited when we hear about the stories of progressive-style pedagogy that works, that we often forget that there may be just as many stories in which progressive-style pedagogy doesn’t work. Traditional-style pedagogy isn’t safe from survivorship bias either – how many times have you, within a traditional-style lesson, received a correct answer from one student and proceeded through the lesson, oblivious to the fact that there may be several students confused? You are making an assumption based on the “survivors” and your “non-survivors” are potentially being left behind. We need to be mindful of this, and not allow survivorship bias to elude us into thinking something is there when it is not.

I want to ask a favour of all the teachers out there: I want to know of a time when you utilized progressive-style pedagogy and it failed. Personally, I had one class in which I flipped my lessons and it was a disaster – with pass rates being abysmally lower than normal. This came after I was on a high from the last semester, in which I flipped my class and pass rates were much higher than normal! There were definitely differences between the two cohorts of students that I had failed to see until post-reflection.

Interestingly enough, Peters & Waterman, the authors of In Search of Excellence, later came out and said that they had made up some of their data. I firmly believe that both sides of the math debate have valuable advice to offer, which is why I am interested in hearing the different stories out there. I do get concerned, however, when teachers/researchers begin to fall into the Peters & Waterman trap: “… go find smart people who are doing cool stuff from which you can learn the most useful, cutting-edge principles … [and] worry about proving the facts later.” Rather, let’s observe all sides of our pedagogy (both the good and the bad) and use this information to perfect our craft.

5 Comments on “Survivorship Bias in Education

  1. Seems to me that the best we can do is be an active part of conversations about observing, teaching, and learning. As you point out, there is no one best method, but perhaps there’s just a best stance, which is responsive, curious, and thoughtful. My own involvement comes from thinking that I have something to add to the conversation that helps support others who are also wanting to be thoughtful We support each other not only by sharing our own experience, but also by responding to what we read from each other. Otherwise, oh, it can be so isolated, this relationship we have to teaching.

    I hope you get some responses to the favor you ask….sounds like a really interesting topic.

    Something that comes to mind that took me by surprise with a class I was teaching was the way in which they received me during our initial meeting. As in itinerant artist, I visit classrooms to do special bookmaking projects with students. Generally, since I am a new face, students are enthusiastic just to have this novel person in their midst. Occasionally, actually really rarely, students greet me with suspicion: without generosity or.enthusiasm they’re not going to be my best friend just because I happened to land in their room! When I get over the shock of the failure of my familiar set of opening remarks, I know have to shift gears with this kind of unaccepting group and have to prove myself. I have to respect and honor the aloof class, and also have to reassure the classroom teacher that I think everything will be just fine… and everything always has turned out well. With the class I’m thinking about, by our final meeting of the project they had fully bought into the project and we were just humming along in perfect sync, which. likely was mostly due to the fact that I was able to accept that they didn’t conform to my expectations, and responded appropriately.


  2. Enjoy thinking about ideas like this.

    It makes me think that, in terms of teaching in the K-12 sector, especially, perhaps (?), in K-8, the idea is to expand our repertoire of instructional practices not replace one with another. Is that conditional enough!

    But, at the same time, it is important to also expand our knowledge of the why of mathematics AND of the group in front of us this day/month/year to make better decisions around matching instructional practices with the strengths/weaknesses of the group we’e working with.

    I am a fan of small group instruction in class. Often opening a lesson with an interesting puzzle or idea, then having people work on it in small groups/pairs whilst I work with a group (sometimes directly related to the opening and sometimes its an opportunity to catch up on a missing idea or skill) and then gathering ideas and commenting on/demonstrating their mathematical effectiveness as a way to consolidate the learning before moving into some practice. This usually works well (unless it is windy or rainy!). However I have had classes where this has bombed completely and have returned to a more traditional teacher focused structure, or an independent model of the above. As you mentioned, on reflection it is usually that I didn’t know my students well enough *as a group* that led to me being less responsive.

    Oh well you live and learn 🙂



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: