Not all data is relevant

Astronaut Harrison H. Schmitt. During the mission, Schmitt unintentionally breathed in some lunar dust and for the rest of the day exhibited signs similar to hay fever or an allergy — sneezing and watery eyes. In 1972, the NASA doctors thought he was allergic to the Moon

Since there are only a dozen people who have walked on the moon, this is an example of “interesting, but not statistically relevant” data.  The small sample pretty much guarantees that we can’t project the data and accurately state “8.33% of people are allergic to moondust.”  It looks like good data – it is good data – but the sample size is too small.

This site shows some spurious correlations – the one I used most was the number of sociology doctorates correlation with commercial space launches, An important lesson from bullet holes in planes  offers another example –

The article is worth reading – but for those folks who don’t want to click the link, here’s an excerpt as a teaser:

“The Army was planning on adding armor strategically to the parts of the plane that tended to get most riddled with bullets and asked Wald to help them determine exactly where the armor should go.

Wald had a different view. He said the armor shouldn’t go where the bullet holes are. Instead, it should go where there aren’t holes. His insight was to realize that when considering all the planes flying missions, the bullet holes should be evenly distributed — there is no reason why anti-aircraft guns fired from thousands of feet away would hit just the wings and fuselage. It is the planes that didn’t return that were hit in the engines and that’s where the armor should go.

Wald’s insight was to identify a logical fallacy called “survivorship bias.”

Survivorship bias “is the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.” Source. Survivorship bias is commonly overlooked and can lead to bad decisions. Keeping in mind the bullet hole story is a powerful mental model for making better decisions.”

Read the whole thing – statistics and correlations are a lot easier to understand if you know the places they can’t be expected to provide answers.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s