Sharp Brains: Brain Fitness and Cognitive Health News

Neuroplasticity, Brain Fitness and Cognitive Health News


BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing does­n’t work” and sim­i­lar, based on the BBC “Brain Test Britain” exper­i­ment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are respon­si­ble.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting togeth­er a250px-ClintEastwood vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) did­n’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approach­es with­out sup­port­ing evi­dence. And this is pre­cise­ly the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only includ­ed peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

But the improve­ment had noth­ing to do with the inter­im brain-train­ing, says study co-author Jes­si­ca Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and oth­er neu­ro­sci­en­tists have long sus­pect­ed: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teas­er used by many video games — improve dra­mat­i­cal­ly on that task, but the improve­ment does not car­ry over to cog­ni­tive func­tion in gen­er­al.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal tri­al, angeleyescleef1.thumbnailcon­tained obvi­ous flaws both in method­ol­o­gy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have start­ed to point out. Back to the TIME arti­cle:

Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tif­ic stud­ies on the ben­e­fits of brain train­ing, and found­ed a com­pa­ny on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­plet­ed 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cif­ic test is need­ed to get a [gen­er­al improve­ment in cog­ni­tion].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­i­ty con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or oth­er dis­trac­tions around, is like­ly to result in bad qual­i­ty of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a crit­ic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle:

I real­ly wor­ry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown Uni­ver­si­ty’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­o­ry and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An old­er test group, he adds, would have a low­er mean start­ing score and more vari­abil­i­ty in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­i­ty to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study peri­od, which seemed mod­est to Sny­der.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature arti­cle:

In our meta-analy­sis of cog­ni­tive brain train­ing RCTs in healthy elder­ly*, dos­es of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cit­ed total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is giv­en as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vid­ed, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not tri­al a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­i­ty of train­ing.

This study was seri­ous­ly flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­at­ic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Log­ic

Let’s ana­lyze by anal­o­gy. Aren’t thetucogbu1.thumbnail BBC-spon­sored researchers bas­ing their extreme­ly broad claims on this type of faulty log­ic?

  1. We have decid­ed to design and man­u­fac­ture our first car ever
  2. Oops, our car does­n’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your mon­ey.
  5. Case closed, let’s all con­tin­ue rid­ing hors­es.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cif­ic train­ing study to cog­ni­tive train­ing in gen­er­al.”

Posit Sci­ence aimed to debunk the debunker (I have been crit­i­cal of sev­er­al Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are say­ing):

This is a sur­pris­ing study method­ol­o­gy,” said Dr. Hen­ry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you test­ed was sug­ar and not peni­cillin.”

We do need seri­ous sci­ence and analy­sis on the val­ue and lim­i­ta­tions of scal­able approach­es to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to direct­ly con­tra­dict), mixed with many claims not sup­port­ed by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much need­ed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapid­ly chang­ing world.


Pre­vi­ous Sharp­Brains arti­cles:

Leave a Reply...

Loading Facebook Comments ...

10 Responses

  1. Ed Batista says:

    Thanks, Alvaro–very help­ful clar­i­fi­ca­tion.

  2. Laura Fay says:

    Alvaro, You raise some very valid points regard­ing the BBC exper­i­ment. Bernard Croisile, Chief Sci­ence Office of Sci­en­tif­ic Brain Train­ing / HAP­PYneu­ron, makes addi­tion­al points on the con­struc­tion and inter­pre­ta­tion of the exper­i­ment. They can be read at — Lau­ra

  3. Robyn says:

    thanks, very inter­est­ing. I’m inter­est­ed in design­ing apps and games that can have a trans­fer­able effect (, for exam­ple). While I appre­ci­ate the attempts to bring sci­ence to a wider audi­ence, I wish they would­n’t make it sound like they were doing a valid exper­i­ment, and instead explained the lim­i­ta­tions and that some­thing like this can only be a demon­stra­tion.

  4. Kathy O'Brien says:

    I went to a non-com­put­er, cog­ni­tive train­ing work­shop about 5 years ago and have nev­er for­got­ten what the psy­chol­o­gist (Feuern­stein) said, “If we don’t believe we can change brain behav­ior, then why do we teach?”

    My expe­ri­ence with a com­put­er-based, inten­sive, audi­to­ry train­ing pro­gram was that it enhanced the audi­to­ry sys­tem momen­tar­i­ly, but the pos­i­tive effects did­n’t last over time. There’s def­i­nite­ly room for advance­ment in the ‘brain train­ing’ world.

  5. Hel­lo every­one, thank you for your com­ments.

    Let’s hope some­thing good comes out all of this, such as more clar­i­ty into what “brain train­ing” is and isn’t.

    We have decid­ed to pub­lish online close to the full con­tent of our book and con­sumer guide The Sharp­Brains Guide to Brain Fit­ness:

  6. I agree. The show’s log­ic is ugly. And stu­pid. It’s like say­ing that since most diets don’t work, that all the research on healthy eat­ing is just a bunch of baloney. But then again, it’s a TV show. And on TV, stu­pid log­ic works.

    So what do we need to do? Dumb down brain-fit­ness? I don’t think so… But if you are going to sell Brain Fit­ness to TV audi­ence, you need to use visu­al and emo­tion­al hooks to deliv­er the mes­sage.

  7. Luc P. Beaudoin says:

    Test­ing brain/cognitive train­ing prod­ucts is impor­tant. How­ev­er, in sci­ence, one can’t con­clude that an effect (e.g., cog­ni­tive improve­ment through soft­ware use) is impos­si­ble or unlike­ly based on obser­va­tion (even if the sam­ple size is very large). Owen and col­leagues could refer to the wikipedia arti­cle on the “null hypoth­e­sis” which most researchers under­stand. The range of pos­si­ble cog­ni­tive mech­a­nisms, strate­gies and skills that one can poten­tial­ly train through soft­ware, and the range of man­ners in which one could train them, are so broad as to call for much more the­o­riz­ing, soft­ware devel­op­ment and empir­i­cal test­ing of the effects of such soft­ware. (And of course many cog­ni­tive train­ing effects have already been doc­u­ment­ed.)

  8. Richard says:

    I’ve tried brain train­ing on-line and I’ve tried it on a Nin­ten­do DS and notice con­sid­er­able improve­ments. Try med­i­tat­ing. That’s REAL brain train­ing.

  9. Mark A Smith says:

    Excel­lent points. But why did­n’t the study draw from the lit­er­a­ture and use brain train­ing pro­grams that have been sci­en­tif­i­cal­ly demon­strat­ed to have a broad trans­fer effect — such as the dual n‑back (e.g. It’s not just a dose prob­lem but a task prob­lem. Sin­gle n‑back train­ing gains have NOT been found to trans­fer to flu­id intel­li­gence gains, while the dual n‑back has. This kind of speci­fici­ty is impor­tant.

  10. Yes, BBC basi­cal­ly mud­dled the waters for con­sumers (even more, yes) by nam­ing “brain train­ing” some­thing that has noth­ing to do with the cog­ni­tive train­ing that has enabled trans­fer in pre­vi­ous stud­ies. If they want­ed to debunk Nin­ten­do, well, then, test Nin­ten­do.

    Cog­ni­tive train­ing has been iden­ti­fied in the recent NIH inde­pen­dent pan­el report as the only pro­tec­tive fac­tor against cog­ni­tive decline based on the high­est degree of evi­dence. Cog­ni­tive engage­ment over­all and phys­i­cal activ­i­ties are also pro­tec­tive, based on low­er qual­i­ty evi­dence.

    It would seem as if the BBC may be inad­ver­tent con­tribut­ing to the cog­ni­tive decline of its viewers/ read­ers who trust its news and pro­grams with­out crit­i­cal judg­ment.

Leave a Reply to Ed Batista × Cancel reply

Categories: Education & Lifelong Learning, Health & Wellness

Tags: , , , , , , , , ,

About SharpBrains

As seen in The New York Times, The Wall Street Journal, BBC News, CNN, Reuters,  SharpBrains is an independent market research firm tracking how brain science can improve our health and our lives.

Search in our archives

Follow us and Engage via…

RSS Feed

Watch All Recordings Now (40+ Speakers, 12+ Hours)