BBC

Scientific critique of BBC/ Nature Brain Training Experiment

by

logo-bbcThere has been quite a bit of com­ment about the Owen et al study in Nature avail­able online on April 20, 2010. A quick syn­op­sis of the study is that the BBC show Bang Goes the The­o­ry worked with the study authors to pro­vide a test of the hypoth­e­sis that com­mer­cial­ly avail­able brain train­ing pro­grams trans­fer to gen­er­al cog­ni­tive abil­i­ties. The con­clu­sion was that, despite improve­ments on the trained tasks, “no evi­dence was found for trans­fer effects to untrained tasks, even when those tasks were cog­ni­tive­ly close­ly related.”

The exper­i­ment

The study was con­duct­ed through the show’s web site. Of 52,617 par­tic­i­pants who reg­is­tered, approx­i­mate­ly 20% (11,430) com­plet­ed full par­tic­i­pa­tion in the study, which con­sist­ed of two bench­mark­ing assess­ments 6 weeks apart with vari­ants of neu­ropsy­cho­log­i­cal tests and at least two train­ing ses­sions. Peo­ple were ran­dom­ly assigned to one of three groups that were asked to train for about 10 min a day three times a week for the 6‑week peri­od, though they could train either more or less fre­quent­ly. One of the two exper­i­men­tal groups was a “brain train­ing” group that com­plet­ed tasks includ­ing sim­ple arith­metic, find­ing miss­ing pieces, match­ing sym­bols to a tar­get, order­ing rotat­ing num­bers by numer­i­cal val­ue, updat­ing, and mem­o­ry for items. Most of the train­ing ses­sions were 90 sec each; the rotat­ing num­bers tasks was 3 min. These activ­i­ties are sim­i­lar to those used in “edu­tain­ment” pro­grams that can be played online or with a hand­held device. The oth­er exper­i­men­tal group was trained on rea­son­ing tasks that involved iden­ti­fy­ing rel­a­tive weights of objects based on a visu­al “see­saw”, select­ing the “odd” item in a con­cept for­ma­tion type task, a task involv­ing think­ing through the effects of one action on cur­rent and future states, and three plan­ning tasks includ­ing draw­ing a con­tin­u­ous line around a grid while ascer­tain­ing that the line will not hin­der lat­er moves, a ver­sion of the Tow­er of Hanoi task, and a tile slid­ing game. The con­trol group spent time answer­ing ques­tions about obscure facts and orga­niz­ing them chrono­log­i­cal­ly based on any avail­able online resource. Results indi­cat­ed that the two exper­i­men­tal groups per­formed bet­ter than the con­trol group on only one out­come test of gram­mat­i­cal rea­son­ing; there were no dif­fer­ences between either exper­i­men­tal group and the con­trols on the remain­ing test. The exper­i­men­tal groups had improved on the trained tasks but not on the trans­fer tasks.

Sci­en­tif­ic concerns

Although some news reports sug­gest that these find­ings are defin­i­tive, there are a num­ber of con­cerns, many of which have to do with whether the find­ings have been over­gen­er­al­ized to all forms of brain train­ing because only a few tests were used. Sec­ond, there have been ques­tions raised about the amount of time allo­cat­ed to train­ing and the issue of test­ing in the home envi­ron­ment. The study report­ed [Read more…] about Sci­en­tif­ic cri­tique of BBC/ Nature Brain Train­ing Experiment

BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

by

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing does­n’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting togeth­er a250px-ClintEastwood vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) did­n’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approach­es with­out sup­port­ing evi­dence. And this is pre­cise­ly the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only includ­ed peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

“But the improve­ment had noth­ing to do with the inter­im brain-train­ing, says study co-author Jes­si­ca Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and oth­er neu­ro­sci­en­tists have long sus­pect­ed: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teas­er used by many video games — improve dra­mat­i­cal­ly on that task, but the improve­ment does not car­ry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal tri­al, angeleyescleef1.thumbnailcon­tained obvi­ous flaws both in method­ol­o­gy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have start­ed to point out. Back to the TIME article:

“Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tif­ic stud­ies on the ben­e­fits of brain train­ing, and found­ed a com­pa­ny on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­plet­ed 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cif­ic test is need­ed to get a [gen­er­al improve­ment in cognition].”

Sec­ond, [Read more…] about BBC “Brain Train­ing” Exper­i­ment: the Good, the Bad, the Ugly

Brain Training @ BBC/ Nature: Fact, Hope, Hype?

by

Update (04/20/10): after read­ing the full BBC study in Nature, I wrote the arti­cle titled BBC “Brain Train­ing” Exper­i­ment: the Good, the Bad, the Ugly, say­ing that “you prob­a­bly saw the hun­dreds of media arti­cles titled “brain train­ing doesn’t work”, based on a BBC exper­i­ment. Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are respon­si­ble.” You can keep read­ing full updat­ed arti­cle Here.

Below is what I orig­i­nal­ly wrote before the paper itself was available.

—

Tomor­row we’ll prob­a­bly wit­ness brainpica lot of media cov­er­age about a exper­i­ment run by the BBC in the UK, to be pub­lished in Nature, on whether “brain train­ing” works.

The paper is still embar­goed, so we can­not com­ment on it, but what I can do is to share frag­ments of my email to a BBC reporter six months ago, dis­cussing impres­sions on what they had announced as the ulti­mate test of whether “brain train­ing” works.

Again, these were pure­ly my impres­sions based on lim­it­ed pub­lic infor­ma­tion. Once we can com­ment on the pub­lished paper we’ll be able to pro­vide a more informed perspective.

“Hel­lo XYZ,

Here go some of my thoughts based on my exter­nal per­cep­tion of your test:

  • I agree with many of the premis­es for the test
  • But “Does brain train­ing real­ly work” is a high­ly mis­lead­ing frame: the obvi­ous answer is, yes, it works as a cat­e­go­ry. If not, do you mean peo­ple can’t learn? med­i­tate? go through cog­ni­tive ther­a­py? cog­ni­tive retrain­ing? increase work­ing mem­o­ry and oth­er brain func­tions? All these are estab­lished beyond doubt through dozens of well-con­trolled stud­ies where the inter­ven­tion effect a) goes beyond place­bo, and b) remains there once train­ing is over. The 2009 report I sent you includes 10 Research Exec­u­tive Briefs by lead­ing sci­en­tists who ref­er­ence pub­lished papers in high-qual­i­ty jour­nals. None eval­u­ates Nin­ten­do — but should they be ignored, as a group?
  • Now, the key ques­tions are, “what spe­cif­ic brain train­ing are we talk­ing about”, “work for what?” and “work for whom?”. That’s where we could help edu­cate con­sumers sep­a­rate hope from hype.
  • …Right now you are invent­ing your own “brain game”, and the only thing you will test is whether that spe­cif­ic “brain game” you have devel­op “works” or not (not clear what out­come mea­sures you have). I would­n’t dare to man­u­fac­ture my own car now from scratch and claim, based on the results, that “cars” work or don’t.
  • I could­n’t agree more with “brain train­ing that is good for one per­son might not be good for you”, since one of “brain train­ing” prop­er­ties (both strength and weak­ness) is its high­ly tar­get­ed nature. The impli­ca­tion? we need bet­ter assess­ments to pin­point bot­tle­necks and direct appro­pri­ate inter­ven­tion. con­sumers need bet­ter edu­ca­tion and infor­ma­tion to know what is a waste of time and mon­ey and what may be wor­thy. Yet, your test seems to ful­ly ignore this, and test whether the same thing is good for everyone…you may be throw­ing out the baby with the water…”

Your thoughts?

(Will link to paper once pub­lished). Relat­ed articles: