• Skip to main content
  • Skip to primary sidebar

Tracking Health and Wellness Applications of Brain Science

Spanish
sb-logo-with-brain
  • Resources
    • Monthly eNewsletter
    • Solving the Brain Fitness Puzzle
    • The SharpBrains Guide to Brain Fitness
    • How to evaluate brain training claims
    • Resources at a Glance
  • Brain Teasers
    • Top 25 Brain Teasers & Games for Teens and Adults
    • Brain Teasers for each Cognitive Ability
    • More Mind Teasers & Games for Adults of any Age
  • Virtual Summits
    • 2019 SharpBrains Virtual Summit
    • Speaker Roster
    • Brainnovations Pitch Contest
    • 2017 SharpBrains Virtual Summit
    • 2016 SharpBrains Virtual Summit
    • 2015 SharpBrains Virtual Summit
    • 2014 SharpBrains Virtual Summit
  • Report: Pervasive Neurotechnology
  • Report: Digital Brain Health
  • About
    • Mission & Team
    • Endorsements
    • Public Speaking
    • In the News
    • Contact Us

BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

April 20, 2010 by Alvaro Fernandez

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing does­n’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting togeth­er a250px-ClintEastwood vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) did­n’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approach­es with­out sup­port­ing evi­dence. And this is pre­cise­ly the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only includ­ed peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

“But the improve­ment had noth­ing to do with the inter­im brain-train­ing, says study co-author Jes­si­ca Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and oth­er neu­ro­sci­en­tists have long sus­pect­ed: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teas­er used by many video games — improve dra­mat­i­cal­ly on that task, but the improve­ment does not car­ry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal tri­al, angeleyescleef1.thumbnailcon­tained obvi­ous flaws both in method­ol­o­gy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have start­ed to point out. Back to the TIME article:

“Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tif­ic stud­ies on the ben­e­fits of brain train­ing, and found­ed a com­pa­ny on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­plet­ed 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cif­ic test is need­ed to get a [gen­er­al improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­i­ty con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or oth­er dis­trac­tions around, is like­ly to result in bad qual­i­ty of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a crit­ic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle:

“I real­ly wor­ry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown Uni­ver­si­ty’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­o­ry and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An old­er test group, he adds, would have a low­er mean start­ing score and more vari­abil­i­ty in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­i­ty to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study peri­od, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

“In our meta-analy­sis of cog­ni­tive brain train­ing RCTs in healthy elder­ly*, dos­es of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cit­ed total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is giv­en as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vid­ed, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not tri­al a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­i­ty of training.

This study was seri­ous­ly flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­at­ic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Log­ic

Let’s ana­lyze by anal­o­gy. Aren’t thetucogbu1.thumbnail BBC-spon­sored researchers bas­ing their extreme­ly broad claims on this type of faulty logic?

  1. We have decid­ed to design and man­u­fac­ture our first car ever
  2. Oops, our car does­n’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
  5. Case closed, let’s all con­tin­ue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cif­ic train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence aimed to debunk the debunker (I have been crit­i­cal of sev­er­al Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

“This is a sur­pris­ing study method­ol­o­gy,” said Dr. Hen­ry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you test­ed was sug­ar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the val­ue and lim­i­ta­tions of scal­able approach­es to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to direct­ly con­tra­dict), mixed with many claims not sup­port­ed by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much need­ed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapid­ly chang­ing world.

Resources:

  • You can read the full paper Here (opens free 5‑page PDF)
  • TIME arti­cle: Here
  • Nature arti­cle: Here
  • Wall Street Jour­nal arti­cle: Here
  • My email to BBC six months ago: Here (we gave the BBC full access to the Jan­u­ary Sharp­Brains Sum­mit on Tech­nol­o­gy for Cog­ni­tive Health and Per­for­mance; they chose not to engage)
  • Book The Sharp­Brains Guide to Brain Fit­ness: Here

Pre­vi­ous Sharp­Brains articles:

  • Does Cog­ni­tive Train­ing Work? For whom? for what?
  • Are videogames good for you? It depends on who YOU are

Share this:

  • Tweet
  • Email
  • Print
  • More
  • Share on Tumblr
  • Pock­et

Filed Under: Brain/ Mental Health, Education & Lifelong Learning Tagged With: BBC, BBC brain training, Brain-games, Brain-Training, criticism, critique, jessica-grahn, making claims, nintendo, Peter Snyder

Reader Interactions

Comments

  1. Ed Batista says

    April 20, 2010 at 11:34

    Thanks, Alvaro–very help­ful clarification.

  2. Laura Fay says

    April 21, 2010 at 1:08

    Alvaro, You raise some very valid points regard­ing the BBC exper­i­ment. Bernard Croisile, Chief Sci­ence Office of Sci­en­tif­ic Brain Train­ing / HAP­PYneu­ron, makes addi­tion­al points on the con­struc­tion and inter­pre­ta­tion of the exper­i­ment. They can be read at http://www.brainfitnessforlife.com. — Laura

  3. Robyn says

    April 21, 2010 at 1:42

    thanks, very inter­est­ing. I’m inter­est­ed in design­ing apps and games that can have a trans­fer­able effect (www.timestableclock.com, for exam­ple). While I appre­ci­ate the attempts to bring sci­ence to a wider audi­ence, I wish they would­n’t make it sound like they were doing a valid exper­i­ment, and instead explained the lim­i­ta­tions and that some­thing like this can only be a demonstration.

  4. Kathy O'Brien says

    April 22, 2010 at 10:47

    I went to a non-com­put­er, cog­ni­tive train­ing work­shop about 5 years ago and have nev­er for­got­ten what the psy­chol­o­gist (Feuern­stein) said, “If we don’t believe we can change brain behav­ior, then why do we teach?” 

    My expe­ri­ence with a com­put­er-based, inten­sive, audi­to­ry train­ing pro­gram was that it enhanced the audi­to­ry sys­tem momen­tar­i­ly, but the pos­i­tive effects did­n’t last over time. There’s def­i­nite­ly room for advance­ment in the ‘brain train­ing’ world.

  5. Alvaro Fernandez says

    April 26, 2010 at 9:42

    Hel­lo every­one, thank you for your comments.

    Let’s hope some­thing good comes out all of this, such as more clar­i­ty into what “brain train­ing” is and isn’t. 

    We have decid­ed to pub­lish online close to the full con­tent of our book and con­sumer guide The Sharp­Brains Guide to Brain Fitness:
    https://sharpbrains.com/resources/

  6. Billiam James says

    April 26, 2010 at 7:59

    I agree. The show’s log­ic is ugly. And stu­pid. It’s like say­ing that since most diets don’t work, that all the research on healthy eat­ing is just a bunch of baloney. But then again, it’s a TV show. And on TV, stu­pid log­ic works. 

    So what do we need to do? Dumb down brain-fit­ness? I don’t think so… But if you are going to sell Brain Fit­ness to TV audi­ence, you need to use visu­al and emo­tion­al hooks to deliv­er the message.

  7. Luc P. Beaudoin says

    April 28, 2010 at 11:49

    Test­ing brain/cognitive train­ing prod­ucts is impor­tant. How­ev­er, in sci­ence, one can’t con­clude that an effect (e.g., cog­ni­tive improve­ment through soft­ware use) is impos­si­ble or unlike­ly based on obser­va­tion (even if the sam­ple size is very large). Owen and col­leagues could refer to the wikipedia arti­cle on the “null hypoth­e­sis” which most researchers under­stand. The range of pos­si­ble cog­ni­tive mech­a­nisms, strate­gies and skills that one can poten­tial­ly train through soft­ware, and the range of man­ners in which one could train them, are so broad as to call for much more the­o­riz­ing, soft­ware devel­op­ment and empir­i­cal test­ing of the effects of such soft­ware. (And of course many cog­ni­tive train­ing effects have already been documented.)

  8. Richard says

    April 29, 2010 at 5:09

    I’ve tried brain train­ing on-line and I’ve tried it on a Nin­ten­do DS and notice con­sid­er­able improve­ments. Try med­i­tat­ing. That’s REAL brain training.

  9. Mark A Smith says

    May 10, 2010 at 11:09

    Excel­lent points. But why did­n’t the study draw from the lit­er­a­ture and use brain train­ing pro­grams that have been sci­en­tif­i­cal­ly demon­strat­ed to have a broad trans­fer effect — such as the dual n‑back (e.g. http://www.highiqpro.com/high-iq-pro/scientific-basis-of-software)? It’s not just a dose prob­lem but a task prob­lem. Sin­gle n‑back train­ing gains have NOT been found to trans­fer to flu­id intel­li­gence gains, while the dual n‑back has. This kind of speci­fici­ty is important.

  10. Alvaro Fernandez says

    May 14, 2010 at 3:53

    Yes, BBC basi­cal­ly mud­dled the waters for con­sumers (even more, yes) by nam­ing “brain train­ing” some­thing that has noth­ing to do with the cog­ni­tive train­ing that has enabled trans­fer in pre­vi­ous stud­ies. If they want­ed to debunk Nin­ten­do, well, then, test Nintendo.

    Cog­ni­tive train­ing has been iden­ti­fied in the recent NIH inde­pen­dent pan­el report as the only pro­tec­tive fac­tor against cog­ni­tive decline based on the high­est degree of evi­dence. Cog­ni­tive engage­ment over­all and phys­i­cal activ­i­ties are also pro­tec­tive, based on low­er qual­i­ty evidence.

    It would seem as if the BBC may be inad­ver­tent con­tribut­ing to the cog­ni­tive decline of its viewers/ read­ers who trust its news and pro­grams with­out crit­i­cal judgment.

Primary Sidebar

Top Articles on Brain Health and Neuroplasticity

  1. Can you grow your hippocampus? Yes. Here’s how, and why it matters
  2. How learning changes your brain
  3. To harness neuroplasticity, start with enthusiasm
  4. Three ways to protect your mental health during –and after– COVID-19
  5. Why you turn down the radio when you're lost
  6. Solving the Brain Fitness Puzzle Is the Key to Self-Empowered Aging
  7. Ten neu­rotech­nolo­gies about to trans­form brain enhance­ment & health
  8. Five reasons the future of brain enhancement is digital, pervasive and (hopefully) bright
  9. What Educators and Parents Should Know About Neuroplasticity and Dance
  10. The Ten Habits of Highly Effective Brains
  11. Six tips to build resilience and prevent brain-damaging stress
  12. Can brain training work? Yes, if it meets these 5 conditions
  13. What are cognitive abilities and how to boost them?
  14. Eight Tips To Remember What You Read
  15. Twenty Must-Know Facts to Harness Neuroplasticity and Improve Brain Health

Top 10 Brain Teasers and Illusions

  1. You think you know the colors? Try the Stroop Test
  2. Check out this brief attention experiment
  3. Test your stress level
  4. Guess: Are there more brain connections or leaves in the Amazon?
  5. Quick brain teasers to flex two key men­tal mus­cles
  6. Count the Fs in this sentence
  7. Can you iden­tify Apple’s logo?
  8. Ten classic optical illu­sions to trick your mind
  9. What do you see?
  10. Fun Mental Rotation challenge
  • Check our Top 25 Brain Teasers, Games and Illusions

Join 12,562 readers exploring, at no cost, the latest in neuroplasticity and brain health.

By subscribing you agree to receive our free, monthly eNewsletter. We don't rent or sell emails collected, and you may unsubscribe at any time.

IMPORTANT: Please check your inbox or spam folder in a couple minutes and confirm your subscription.

Get In Touch!

Contact Us

660 4th Street, Suite 205,
San Francisco, CA 94107 USA

About Us

SharpBrains is an independent market research firm tracking health and performance applications of brain science. We prepare general and tailored market reports, publish consumer guides, produce an annual global and virtual conference, and provide strategic advisory services.

© 2023 SharpBrains. All Rights Reserved - Privacy Policy