A Criminologist Looks at the Reproducibility of Research
By the time you finish reading this post, you will have forgotten 90% of what you have read. Mind you, this is not a bug, but likely a necessary feature of a productive brain. It would make sense that we filter out most of what we perceive to stay the course toward valuable signals in a sea of noise.
This begs the question: “How do we figure out what to remember (the signal), and what to forget (the noise)?”
As mentioned in an earlier post, “we see what we understand”, see link below. We remember stories that make sense to us.
Did Your Mother Make You Eat Spinach?
Did your mother make you eat spinach because it was high in iron? Worse yet, are you making your children eat spinach because you believe it is high in iron?
In 2010, Mike Sutton, a criminologist in the Department of Social Sciences at Nottingham Trent University wrote a 34-page paper on the topic.
Source: Sutton, Mike (2010), SPINACH, IRON and POPEYE: Ironic lessons from biochemistry and history on the importance of healthy eating, healthy skepticism and adequate citation, Internet Journal of Criminology
Before I read Mike’s paper, the story of “spinach, Popeye, and iron” felt like a validated fact worth repeating to change an audience’s perspective about non-validated facts. It was believable because it had the structure of an understandable, thus memorable story, a worthy signal in an ocean of noise.
Using one of the storyline templates presented in an earlier post for subscribers, the understandable and memorable structure of the story of “spinach, Popeye, and iron” looks like this:
Why: The dangerous situation and the crisis create the emotional appeal around the facts:
- Did your mother force you to each awful tasting spinach out of an erroneous belief that it was high in iron?
Who: Observe with the eyes of the hero to provide the proof statement of the situation:
- Everybody knows Popeye, and everybody knows he is made of iron and that he eats spinach
What: Orient the story against the villain to assert an opinion and to evaluate plans of action:
- The blame lies with Victorian-era German scientists for getting the numbers wrong, spinach is not high in iron.
How: Decide and make thoughtful requests to compound luck through the mastery of the comparative advantage:
- The story teller, assuming the mantle of authority for telling the real story rather than the good story: “Trust me, I am not like the others”, as Howie Carr still says on the radio. Stop eating spinach.
When: Achieve transcendence through the cumulative effect of completed actions and resolved promises:
- “And now, you know the rest of the story” as Paul Harvey used to say on the radio, even Popeye says that he eats spinach for vitamin A, and not iron: “Spinach is full of vitamin “A” an’ tha’s what makes hoomans strong an’ helty”. Throw the spinach can, and buy this bottle of multi-vitamin…
This structure made it a valuable morality play to emphasize doing good research, disbelieving prior authorities, being mindful of false positives in classification testing, etc.
However, the story of “spinach, Popeye, and iron”, as researched and told by Mike Sutton, will challenge your mind, help you understand the changing contexts of history, and leave you hungry for more.
A Criminologist Looks for a Lie
Mike’s paper turns the once understandable, memorable, and valuable story into a confusing mess. He is a criminologist working to validate the reproducibility of research. He ends up obsessing, like a real life Columbo, to prove the existence of an intentional lie to cover-up a respectable researcher’s mistaken and careless assertions from authority.
By the time Mike is done, the clarity of the story is gone, in a way that echoes of Monty Python’s quip about African or European sparrows: “Was it a fresh spinach, or was it a dried spinach?”
The arc of the story has gone upside down. Some of the German scientists may, or may not exist. The good guy in the original version may, or may not be the bad guy in the messy version.
The required level of research becomes mind-numbing: One must read all of Popeye newspaper cartoons ever-produced, and in ascending order of publication to perhaps, or perhaps not, answer one of four key questions. On the other hand, there may be several other important questions to ask.
Looking at this smoldering heap of ashes, the importance of asking the right questions is all that remains. The questions are more important than the answers because answers are always subject to change.
Trusting Trust
In his Turing Award lecture, published in 1984 in the Communications of the ACM, Ken Thompson addresses the issue of trusting trust in the context of computer code.
Source: Thompson, Ken (1984), Reflections on Trusting Trust, To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software? Communications of the ACM, August 1984, Volume 27 Number 8
He shows, with code examples, that we cannot trust code, ever, unless we write it ourselves, or we trust the person that wrote the code.
Consider that his statement applies to the “small-world” problems of models and algorithms. Looking at the “large world” of self-reflecting individuals prone to telling lies, including self-deception, his statement is even more useful to define practical actions in the context of our spinach story.
We cannot trust research that we did not do, or check deeply enough - if this is even doable. We cannot trust arguments from authority. We must become the authors of our own models, stories, and reference narratives.
An earlier post presents sources to build-up our own reference narratives, see link below:
However, something useful - because it helps us put these sources to work - comes from yet another messy story attributed to Hemmingway: “For Sale. Baby Shoes. Never Worn.”
Spinach, Widely Seen, Not Usefully Real
A short, understandable story becomes memorable. Hemmingway’s open-ended story also keeps your mind going in the form of a question: What happened to the baby?
However, its provenance, like the “spinach, Popeye, iron” story, is messy. How can we research, and prove that this was actually the work product of a bet Hemmingway placed with his friends during an informal, and likely drunken, conversation, that he could write a compelling story in six words?
In the end, we may be left with little more than short, open-ended stories as useful tools for “large world” decision-making. Stories that filter signal from noise with common sense, memorable, and understandable patterns that we then see in reality. We see what we understand, instead of understanding what we see.
Trust your own eyes, gut-feel, and experience. Write your own stories.
In this spirit, here is a sample of six-word stories from the discussion that took place after a presentation on this topic, at a retirement planning conference:
- “Spinach, Widely Seen, Not Usefully Real” to describe Type I error in classification testing, false alarms, and false positives
- “Invisible Gorillas, Usefully Real, Rarely Seen” to describe Type II error in classification testing, missed alarms, false negatives
- “Spend Less, Save More, Live Longer”, “Plan Now, Invest Smart, Be Happy”, and “No Plan, No Money, No Fun” to describe the importance of retirement planning
- “Function in Disaster, Finish in Style” to describe just about anything worth doing in real life.
What are your own, trusted stories written in six words?
At the end of each post, we ask ourselves the following question before we publish it:
“What is an individual member of the next generation supposed to do with this?”
Personal practice based on this post, include:
(i) Write your own stories in six-words
(ii) Send me the six-word stories you like best. All readers can e-mail me. In addition, subscribers can add comments to the posts.
Minds are closing down under stress. People are tired of ideas. They want to know how to survive. “CTRI by Francois Gadenne” connects the dots of life-enhancing practices for the next generation, free of controlling algorithms, based on the lifetime experience of a retirement age entrepreneur, and continuously updated with Wealth, Health, and Statistics research performed on behalf of large companies.