If your familiarity with science mostly comes from watching television scientists point to squiggly lines on computer screens and spout ingenious explanations like on the hit show "NCIS," you might not be aware of what for researchers is an embarrassing secret: A lot of science turns out to be wrong.
The Reproducibility Project recently reported that in a sample of studies published in recent years in major psychology journals, only 36 of 100 findings held up when retested. Dr. John Ioannidis, a director of Stanford University's Meta-Research Innovation Center, told the New York Times he suspected the rate of errant conclusions might be even higher in other scientific fields.
Advertisement
One big problem for researchers is that they're under a lot of competitive pressure to publish studies in order to win tenure at universities and secure government grants. Coming up with novel, attention-getting findings is the surest way to do those things. Additionally, though, in experiments that involve working with normal people as subjects, researchers can easily be tripped up by various flaws and shortcomings that are part of being human. Here are five of the most difficult obstacles that present challenges to coming up with the right conclusions.
1. The Placebo Effect: Medical researchers have long been aware of an unsettling phenomenon, in which some control-group patients in studies show improvement, even though they're only receiving sugar pills or some sham medical procedure instead of the actual treatment that's being tested. The effect can distort the results so much that scientists have actually done studies focused on figuring out why the placebo effect occurs. They've discovered a likely explanation: When ill patients agree to participate in a study, they are so hopeful that the treatment will help them that their bodies release dopamine, a neurotransmitter that's part of the brain's reward mechanism and helps create a feeling of well-being. To compound the problem, recent research shows that the placebo effect actually seems to have grown stronger than it was in the past, perhaps because people these days are subjected to more publicity and hype about medications' effectiveness.
2. Problems with Self-Reporting: A lot of studies depend upon subjects to self-report—that is, to keep track of data themselves. That can lead to big problems, as subjects either inadvertently or intentionally provide wrong information. According to the National Institutes of Health, numerous studies show, for example, that subjects in medical studies—for example, people who have to take medication on a schedule—tend to over-estimate their how closely adhere to the regimen. If they're reporting to a telephone survey taker, they also may misunderstand a question or simply remember past events incorrectly.
3. Confirmation Bias: Basically, the human brain is big on using pattern recognition, based on information that we already have stored in our memories, as a shortcut to solving problems and reaching conclusions. The problem, as science writer Leonard Mlodinow has noted, is that our brains "are focused on finding and confirming patterns, rather than minimizing our false conclusions." So if we go into an experiment expecting to find a certain result based on our existing knowledge, there's a risk that we're going to see that result in the data, no matter what. That's why some scientists recommend using a process called "blind analysis," in which researchers in a drug study, for example, aren't told which patients are taking a medication and which are getting the placebo, until after the data has been collected.
4. Memory and Perception Glitches: Another factor that stymies researchers is that experimental subjects may provide faulty data because of limitations in the human senses and memory. You may think that what you see and remember is an exact version of events, but as memory researcher Elizabeth Loftus has explained, the human eye only gathers information in fleeting bursts that last a fraction of a second, and then stitches together that incomplete series of images into the movie in your head. Worse yet, memory isn't static, but malleable (watch that video above to learn more). As you access a recollection again and again, it tends to decay, and exposure to outside information — or inadvertently suggestive questioning by researchers — can actually alter what you remember.
5. Lying: One of the most frustrating aspects of dealing with human research subjects is that they sometimes intentionally choose to either keep important information from researchers, or give them wrong information. In a 2013 study published in Clinical Trials, researchers found that about three quarters of participants in medical studies admitted to some sort of deception, including 32 percent who didn't reveal health problems and 28 percent who lied about what medications they already were taking. Worse yet, nearly a quarter of the subjects admitted to exaggerating their symptoms so that they could qualify to participate in studies, and 14 percent admitted to faking their conditions completely.
mean that science should be discounted? Absolutely not! It means, rather, that professional researchers need to take these potential factors into account, and that proper training in research procedures and protocols should cover human imperfections.
Advertisement
