BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing does­n’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting togeth­er a vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) did­n’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approach­es with­out sup­port­ing evi­dence. And this is pre­cise­ly the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only includ­ed peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

“But the improve­ment had noth­ing to do with the inter­im brain-train­ing, says study co-author Jes­si­ca Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and oth­er neu­ro­sci­en­tists have long sus­pect­ed: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teas­er used by many video games — improve dra­mat­i­cal­ly on that task, but the improve­ment does not car­ry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal tri­al, con­tained obvi­ous flaws both in method­ol­o­gy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have start­ed to point out. Back to the TIME article:

“Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tif­ic stud­ies on the ben­e­fits of brain train­ing, and found­ed a com­pa­ny on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­plet­ed 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cif­ic test is need­ed to get a [gen­er­al improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­i­ty con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or oth­er dis­trac­tions around, is like­ly to result in bad qual­i­ty of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a crit­ic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle:

“I real­ly wor­ry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown Uni­ver­si­ty’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­o­ry and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An old­er test group, he adds, would have a low­er mean start­ing score and more vari­abil­i­ty in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­i­ty to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study peri­od, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

“In our meta-analy­sis of cog­ni­tive brain train­ing RCTs in healthy elder­ly*, dos­es of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cit­ed total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is giv­en as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vid­ed, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not tri­al a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­i­ty of training.

This study was seri­ous­ly flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­at­ic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Log­ic

Let’s ana­lyze by anal­o­gy. Aren’t the BBC-spon­sored researchers bas­ing their extreme­ly broad claims on this type of faulty logic?

1. We have decid­ed to design and man­u­fac­ture our first car ever
2. Oops, our car does­n’t work
3. There­fore, cars DON’T work, CAN’T work, and WON’T work
4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
5. Case closed, let’s all con­tin­ue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cif­ic train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence aimed to debunk the debunker (I have been crit­i­cal of sev­er­al Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

“This is a sur­pris­ing study method­ol­o­gy,” said Dr. Hen­ry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you test­ed was sug­ar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the val­ue and lim­i­ta­tions of scal­able approach­es to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to direct­ly con­tra­dict), mixed with many claims not sup­port­ed by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much need­ed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapid­ly chang­ing world.

Resources:

• You can read the full paper Here (opens free 5‑page PDF)
• TIME arti­cle: Here
• Nature arti­cle: Here
• Wall Street Jour­nal arti­cle: Here
• My email to BBC six months ago: Here (we gave the BBC full access to the Jan­u­ary Sharp­Brains Sum­mit on Tech­nol­o­gy for Cog­ni­tive Health and Per­for­mance; they chose not to engage)
• Book The Sharp­Brains Guide to Brain Fit­ness: Here

Pre­vi­ous Sharp­Brains articles:

• Does Cog­ni­tive Train­ing Work? For whom? for what?
• Are videogames good for you? It depends on who YOU are