Trust the Scientific Method

I believe that scientific method gives us the best chance of finding the truth through one variant of experiment or another.  That doesn’t mean that research flaws can’t slip in and give us incorrect conclusions that can be accepted for a while.

I’ve just noticed some articles about African Neanderthals – these guys may have ranged from the Med to Jo’berg.  It conflicts with all my training – that sub-Saharan Africans lack Neanderthal genetics has been an article of rather amusing faith. 

I researched the correlation between technology – cell phones, mostly – and Hutterite outmigration.  The data looked fine – the statistical probability of my research not being significant was extremely low – yet one interview with an old Hutterite minister, and an article in the Mennonite Quarterly Review brought forward a confounding situation.  The presence of a group called the Arnoldleut, and their earlier incorporation and eventual ejection from the colonies was far more significant than the technological changes I studied.  While that discovery was part of my research, it was also luck – a secretary with whom I had worked arranged the interview not so much to help my research but to create a situation where a runaway could spend an evening back on the colony visiting with her mother as I interviewed the minister.  I trust the scientific method – but without that last interview, I would have published nonsense . . . and my last interview occurred because of a secretary’s kindness in finding a way for a grandmother to meet her new grandchild. 

I believe that scientific method is the best way we have to get facts – but all of our results are subject to further examination.  The physical laws that Isaac Newton developed were all the results of observations that occurred within earth’s gravity and atmosphere – yet most of the universe is vacuum and free fall.  The ability to observe the very small increased after Newton – and we’ve moved into a time when Quantum became the word to describe a form of physics that was not available for Newton to observe. 

I trust scientific method.  Scientific method insists that all of our findings are tentative . . . I am unlikely to be the last person to research Hutterite outmigration.  My findings are correct (due to a secretary getting me one more interview) and I was saved the embarrassment of publishing an incorrect explanation of that outmigration.  Newton’s laws are in print, and useful – but later researchers till the fields of quantum physics.  The results of scientific inquiry are always tentative, they can always be questioned.  It’s worth remembering that Piltdown man spent nearly 40 years in mankind’s family tree before the hoax was conclusively proven in 1953. 

Science does not move through consensus nor certainty.  Trust the method – but question the results.  We do not prove with statistics – statistical methodology just quantifies the likelihood of something occurring due to random happenstance.  We take that statistical data, and infer causality.  I’m one of the lucky ones – Mary Kidwiler arranged an opportunity that kept me from the embarrassment and mockery that accompanies publishing a scientific blunder.  Follow the scientific method – but remember, all conclusions are tentative and subject to revision.

A Science for Everyone, Demography

Non-reproducible research

About 20 years ago, I realized that I had a fairly unique opportunity to test the hypothesis that 4-H was strongest where it was multigenerational – 4-H members grew up to be 4-H leaders, and the program was strongest where the multi-generational membership was the most common. 

I was working with 22 counties, and 4 of them had Extension secretaries with 30 or more years of experience, and full records.  Complete records is more challenging than you might think – when I worked as a County Agent, the records were in the basement, and a cracked sewer line helped me make the decision that they couldn’t be recovered.  Obviously, if I had only 4 counties out of 22, reproducing the research would be difficult at best.  On the other hand, if it didn’t get done in the next year, retirements would make it impossible to do once. 

In 1950, 18% of rural youth belonged to 4-H, with the membership plateau ending in 1976 (Putnam 2000, Bowling Alone), with a 26% decline in membership between 1950 and 1997.  And I was listening to folks who told me that the problem was a shortage of volunteer leaders.  It looked like I could find the numbers in those 4 counties with the oldest secretaries. 

I was on a roll – the secretaries showed that 151 4-H families had at least one parent who had been a 4-H member as a child, 78 families where neither parent had been a member, and the parents of 6 families could not be determined.  We defined 4-H members who had belonged to a club four years or more as persistent, and contrasted their statistics with first-year members.  None of the six families whose 4-H history couldn’t be determined had any persistent members, so the sample, while not particularly large, was clean.

Well, the stats were simple – Chi square was calculated at 45.03, the probability of the distribution occurring by chance was less than 0.001.  The data supported the hypothesis that parental involvement in 4-H (as a club member) is the greatest single predictor of member persistence in 4-H.  Two thirds of the persistent members (4 years or more) had parents who had been 4-H members in their youth, while two thirds of the first-year members had parents who had not been 4-H members.   The kids most likely to drop 4-H were kids whose parents had not been in 4-H and were not 4-H club leaders. 

The evidence was pretty solid that a multigenerational 4-H identity helped keep kids in 4-H – but it was equally solid that 4-H membership wasn’t random . . . it was hereditary, like the British nobility.    Still, making a conclusion about a national program from a sample of 334 people in 4 counties seems to be a stretch.  As I look at the Harry that was once an English prince, I wonder about researching the worldwide decline of royalty.

Non-reproducible research isn’t necessarily bad research, and it can provide some interesting conclusions – but it is better when you know it’s non-reproducible from the beginning.