Every once in a while, I'll be posting excerpts from Evaluating Research Quality: Evaluating Research Quality - Guidelines for Scholarship (2012) by Todd Litman; Victoria Transport Policy Institute This clearly written, concise and accessible document “discusses the importance of good research, discusses common causes of bias, provides guidelines for evaluating research and data quality, and describes examples of bad research.” It should be required reading for anyone wanting to be a more discerning consumer of scientific evidence and arguments, especially as found in opinion pieces claiming solid scientific backing. Check it out – the pdf is available online.
One of my favorite parts is Litman's list of "methodological potholes" frequently encountered in science writing. These are based mostly on Huron (2000)* and include:
Hypocrisy: Holding others to a higher methodological standard than oneself.
Ad hominem argument: Criticizing the person rather than their argument.
Discovery fallacy: Criticizing an idea because of its origin (e.g., from a religious text).
Ipse dixit: Appealing to authority figures(e.g., “Research increasingly shows that...”).
Confirmation bias: The tendency to see events as confirming a theory while viewing falsifying events as “exceptions”.
Ad-hoc hypothesis: Proposing a supplementary hypothesis to explain why a favorite theory or interpretation failed a test.
Data neglect: Tendency to ignore available information when assessing theories or hypotheses.
Anti-operationalizing: The tendency to raise perpetual objections to all operational definitions.
Presumptive representation: The practice of representing others to themselves.
Magnitude blindness: Preoccupation with statistically significant results that have small magnitude effects.
Regression artifacts: The tendency to interpret regression toward the mean as an experimental phenomenon.
Demand characteristics: Any aspect of an experiment that might inform subjects of the purpose of the study.
Placebo effect: Positive or negative response arising from the subject’s expectations of an effect.
Litman also identifies the pothole of “Know-nothing, which implies that “because some issues are unknown, nothing is known”, to which I would add: “Predict-Nothing”: implies that since we can’t know the future for sure, we shouldn’t put any stock in any predicted scenarios.
*David Huron (2000), Sixty Methodological Potholes, Ohio State University (http://dactyl.som.ohio-state.edu/Music829C/methodological.potholes.html)