Why is this interesting? - The Neuroscience Edition
On neuroscience, replication, and bad science
Noah here. On Monday the Royal Institution, who puts on amazing science lectures in the UK (and on YouTube), Tweeted a link to this story about tea drinking “boosting brain connectivity.” It seemed interesting enough and I read through it, becoming more and more skeptical by each paragraph. Finally, at the end, I encountered five paragraphs of warning under the header “A drop in the ocean of evidence.” Paragraph one began: “It is important to reiterate that this study compared the brains of just 36 individuals. From such a small sample, we cannot draw any solid conclusions.” Things continued from there.
The obvious question is why, if you need that many caveats, you even bother publishing the story at all. Beyond that, though, this brought to mind something that has been floating around my head for a while now: That I’ve lost faith in most brain studies as a real source of insight into human behavior.
Why is this interesting?
Brains are complex and studying them is worthwhile. Thanks to that complexity, however, it’s humongously difficult. On top of that, there’s a broader replication crisis that’s been occurring across science (at some point I’ll make this a WITI topic). To that end, last week The Atlantic ran a story debunking a piece of brain science that lots of you have probably heard about: “the notion that our brains make choices before we are even aware of them.” That idea, known as the Bereitschaftspotential, came from an experiment in 1964 with some inventive design. “... to collect data on what happened in the brain beforehand, the two researchers realized that they could record their participants’ brain activity separately on tape, then play the reels backwards into the computer.”
The clever design isn’t why we all know about it, though, that comes two decades later:
This momentous discovery was the beginning of a lot of trouble in neuroscience. Twenty years later, the American physiologist Benjamin Libet used the Bereitschaftspotential to make the case not only that the brain shows signs of a decision before a person acts, but that, incredibly, the brain’s wheels start turning before the person even consciously intends to do something. Suddenly, people’s choices—even a basic finger tap—appeared to be determined by something outside of their own perceived volition.
Lots of pop philosophers have been referencing Libet as a way to show we’re not entirely in control of our actions. The problem is, it turns out that the original findings were almost definitely an experimental error.
In 2010 a researcher named Aaron Schurger wanted to test the phenomenon against the regular blips of activity that happen in all of our brains. “This ongoing electrophysiological noise rises and falls in slow tides, like the surface of the ocean,” the article explains. His suspicion was that maybe the original scientists had misinterpreted those random peaks with the experimental action taking place: A regular case of correlation versus causation. He ran an experiment that controlled for the noise and this time found that the movement and the brain activity were happening at the same time. “In other words, people’s subjective experience of a decision—what Libet’s study seemed to suggest was just an illusion—appeared to match the actual moment their brains showed them making a decision.”
When you get to Functional Magnetic Resonance Imaging (fMRI) studies, things look even worse. Those are the ones where you can actually see blood flow into different brain regions as people react to various stimuli. While I’m sure they have valuable medical uses, you’ve got Harvard Business Review urging marketers to pay more attention to fMRI because “Data from fMRI scans has been shown to outperform behavioral data in predicting market-level music sales, charity donations, and even the relative persuasiveness of anti-smoking ad campaigns.” Got it.
While I’ve approached this kind of research with a raised eyebrow for some time, I didn’t have the words until I was listening to this episode of the always-excellent Rationally Speaking podcast with Razib Khan. Here’s part of his answer for why he’d become skeptical of fMRI:
Yeah, it was these small sample sizes of brain imaging. And they had to be small sample sizes, because if you know people who do fMRI, I mean, how many people are you going to get to sit in that little chamber to get scanned? And, the images were really, really captivating. And you would see the associations of images to particular regions and particular stimuli. And it was just… everything fit together in terms of what you would think. Because you know that this is in the brain somewhere and they're actually showing you where it is. Right? And so it was very attractive, in terms of being able to see the physical location of some sort of psychological phenomenon. But I mean, I think a lot of it has turned out to be just like small sample sizes, and spurious associations because of the small sample sizes using the traditional P values. They would try to make some functional sense – like, “This is in a region of the brain associated with this, that, and this” -- but obviously, there was a lot of data dredging, lack of multiple hypothesis testing going on there. And so, I just don't know ... I assume some of it is still valid, but a lot of it was obviously just wishful thinking. And, the sexiness of the technology really sold it for a lot of people, including me.
Some of this gets into our desire to understand ourselves and our thinking. Other parts go to fundamental philosophical questions about consciousness, behavior, and its relationship with the brain. But part of it is just bad science and the overextension of methods, especially ones that push out nice looking pictures like fMRI does. In the end there’s plenty of smart people working hard to slowly move forward our understanding of how brains work and anyone that offers too simple an explanation of that science should be approached with a heavy dose of skepticism. (NRB)
Mailbox of the Day:
In response to yesterday’s WITI on the drone attacks in Saudi Arabia, an anonymous subscriber responds:
Went down a bit of a drone rabbit hole yesterday…the Houthi fleet is not off the shelf commercial – at least not the stuff that carries the heavy ordinance. Mostly copies Iranian fixed-wing designs using parts from China and other countries (including possibly German-made engines!). HUGE questions about how Houthis could have done command & control and guidance on an attack like this – if you look at the satellite photos, they bullseyed the spheroid processing tanks one after the other. Same spot on each one. Separately, from a friend, check out this video of a Netanyahu bodyguard with an anti-drone jammer-gun.
We’re basically living in a William Gibson novel now, just with slightly more familiar contours – and no Turing Police.
Also: we have the smartest readership around. (CJN)
Quick Links:
Thanks for reading,
Noah (NRB) & Colin (CJN)
Why is this interesting? is a daily email from Noah Brier & Colin Nagy about interesting things. If you’ve enjoyed this edition, please consider forwarding it to a friend. If you’re reading it for the first time, consider subscribing (it’s free!).