Sharp Brains: Brain Fitness and Cognitive Health News

Neuroplasticity, Brain Fitness and Cognitive Health News

Icon

BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing doesn’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting together a250px-ClintEastwood vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) didn’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approaches with­out sup­port­ing evi­dence. And this is pre­cisely the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only included peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

But the improve­ment had noth­ing to do with the interim brain-training, says study co-author Jes­sica Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and other neu­ro­sci­en­tists have long sus­pected: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teaser used by many video games — improve dra­mat­i­cally on that task, but the improve­ment does not carry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal trial, angeleyescleef1.thumbnailcon­tained obvi­ous flaws both in method­ol­ogy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have started to point out. Back to the TIME article:

Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tific stud­ies on the ben­e­fits of brain train­ing, and founded a com­pany on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­pleted 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cific test is needed to get a [gen­eral improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­ity con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or other dis­trac­tions around, is likely to result in bad qual­ity of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a critic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle:

I really worry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown University’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­ory and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An older test group, he adds, would have a lower mean start­ing score and more vari­abil­ity in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­ity to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study period, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

In our meta-analysis of cog­ni­tive brain train­ing RCTs in healthy elderly*, doses of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cited total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is given as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vided, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not trial a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­ity of training.

This study was seri­ously flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­atic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Logic

Let’s ana­lyze by anal­ogy. Aren’t thetucogbu1.thumbnail BBC-sponsored researchers bas­ing their extremely broad claims on this type of faulty logic?

  1. We have decided to design and man­u­fac­ture our first car ever
  2. Oops, our car doesn’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
  5. Case closed, let’s all con­tinue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cific train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence dares to debunk the debunker (I have been crit­i­cal of sev­eral Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

This is a sur­pris­ing study method­ol­ogy,” said Dr. Henry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you tested was sugar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the value and lim­i­ta­tions of scal­able approaches to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to directly con­tra­dict), mixed with many claims not sup­ported by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much needed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapidly chang­ing world.

Resources:

Pre­vi­ous Sharp­Brains articles:

Leave a Reply...

Loading Facebook Comments ...

10 Responses

  1. Ed Batista says:

    Thanks, Alvaro–very help­ful clarification.

  2. Laura Fay says:

    Alvaro, You raise some very valid points regard­ing the BBC exper­i­ment. Bernard Croisile, Chief Sci­ence Office of Sci­en­tific Brain Train­ing / HAP­PYneu­ron, makes addi­tional points on the con­struc­tion and inter­pre­ta­tion of the exper­i­ment. They can be read at http://www.brainfitnessforlife.com. — Laura

  3. Robyn says:

    thanks, very inter­est­ing. I’m inter­ested in design­ing apps and games that can have a trans­fer­able effect (www.timestableclock.com, for exam­ple). While I appre­ci­ate the attempts to bring sci­ence to a wider audi­ence, I wish they wouldn’t make it sound like they were doing a valid exper­i­ment, and instead explained the lim­i­ta­tions and that some­thing like this can only be a demonstration.

  4. Kathy O'Brien says:

    I went to a non-computer, cog­ni­tive train­ing work­shop about 5 years ago and have never for­got­ten what the psy­chol­o­gist (Feuern­stein) said, “If we don’t believe we can change brain behav­ior, then why do we teach?”

    My expe­ri­ence with a computer-based, inten­sive, audi­tory train­ing pro­gram was that it enhanced the audi­tory sys­tem momen­tar­ily, but the pos­i­tive effects didn’t last over time. There’s def­i­nitely room for advance­ment in the ‘brain train­ing’ world.

  5. Hello every­one, thank you for your comments.

    Let’s hope some­thing good comes out all of this, such as more clar­ity into what “brain train­ing” is and isn’t.

    We have decided to pub­lish online close to the full con­tent of our book and con­sumer guide The Sharp­Brains Guide to Brain Fit­ness:
    http://www.sharpbrains.com/resources/

  6. I agree. The show’s logic is ugly. And stu­pid. It’s like say­ing that since most diets don’t work, that all the research on healthy eat­ing is just a bunch of baloney. But then again, it’s a TV show. And on TV, stu­pid logic works.

    So what do we need to do? Dumb down brain-fitness? I don’t think so… But if you are going to sell Brain Fit­ness to TV audi­ence, you need to use visual and emo­tional hooks to deliver the message.

  7. Luc P. Beaudoin says:

    Test­ing brain/cognitive train­ing prod­ucts is impor­tant. How­ever, in sci­ence, one can’t con­clude that an effect (e.g., cog­ni­tive improve­ment through soft­ware use) is impos­si­ble or unlikely based on obser­va­tion (even if the sam­ple size is very large). Owen and col­leagues could refer to the wikipedia arti­cle on the “null hypoth­e­sis” which most researchers under­stand. The range of pos­si­ble cog­ni­tive mech­a­nisms, strate­gies and skills that one can poten­tially train through soft­ware, and the range of man­ners in which one could train them, are so broad as to call for much more the­o­riz­ing, soft­ware devel­op­ment and empir­i­cal test­ing of the effects of such soft­ware. (And of course many cog­ni­tive train­ing effects have already been documented.)

  8. Richard says:

    I’ve tried brain train­ing on-line and I’ve tried it on a Nin­tendo DS and notice con­sid­er­able improve­ments. Try med­i­tat­ing. That’s REAL brain training.

  9. Mark A Smith says:

    Excel­lent points. But why didn’t the study draw from the lit­er­a­ture and use brain train­ing pro­grams that have been sci­en­tif­i­cally demon­strated to have a broad trans­fer effect — such as the dual n-back (e.g. http://www.highiqpro.com/high-iq-pro/scientific-basis-of-software)? It’s not just a dose prob­lem but a task prob­lem. Sin­gle n-back train­ing gains have NOT been found to trans­fer to fluid intel­li­gence gains, while the dual n-back has. This kind of speci­ficity is important.

  10. Yes, BBC basi­cally mud­dled the waters for con­sumers (even more, yes) by nam­ing “brain train­ing” some­thing that has noth­ing to do with the cog­ni­tive train­ing that has enabled trans­fer in pre­vi­ous stud­ies. If they wanted to debunk Nin­tendo, well, then, test Nintendo.

    Cog­ni­tive train­ing has been iden­ti­fied in the recent NIH inde­pen­dent panel report as the only pro­tec­tive fac­tor against cog­ni­tive decline based on the high­est degree of evi­dence. Cog­ni­tive engage­ment over­all and phys­i­cal activ­i­ties are also pro­tec­tive, based on lower qual­ity evidence.

    It would seem as if the BBC may be inad­ver­tent con­tribut­ing to the cog­ni­tive decline of its viewers/ read­ers who trust its news and pro­grams with­out crit­i­cal judgment.

Categories: Education & Lifelong Learning, Health & Wellness

Tags: , , , , , , , , ,

New market report on Pervasive Neurotechnology & Intellectual Property

Haven’t you read this book yet?

Follow Us…

twitter_logo_header