Sharp Brains: Brain Fitness and Cognitive Health News

Neuroplasticity, Brain Fitness and Cognitive Health News


Scientific critique of BBC/ Nature Brain Training Experiment

logo-bbcThere has been quite a bit of com­ment about the Owen et al study in Nature avail­able online on April 20, 2010. A quick syn­op­sis of the study is that the BBC show Bang Goes the The­o­ry worked with the study authors to pro­vide a test of the hypoth­e­sis that com­mer­cial­ly avail­able brain train­ing pro­grams trans­fer to gen­er­al cog­ni­tive abil­i­ties. The con­clu­sion was that, despite improve­ments on the trained tasks, “no evi­dence was found for trans­fer effects to untrained tasks, even when those tasks were cog­ni­tive­ly close­ly relat­ed.”

The exper­i­ment

The study was con­duct­ed through the show’s web site. Of 52,617 par­tic­i­pants who reg­is­tered, approx­i­mate­ly 20% (11,430) com­plet­ed full par­tic­i­pa­tion in the study, which con­sist­ed of two bench­mark­ing assess­ments 6 weeks apart with vari­ants of neu­ropsy­cho­log­i­cal tests and at least two train­ing ses­sions. Peo­ple were ran­dom­ly assigned to one of three groups that were asked to train for about 10 min a day three times a week for the 6-week peri­od, though they could train either more or less fre­quent­ly. One of the two exper­i­men­tal groups was a “brain train­ing” group that com­plet­ed tasks includ­ing sim­ple arith­metic, find­ing miss­ing pieces, match­ing sym­bols to a tar­get, order­ing rotat­ing num­bers by numer­i­cal val­ue, updat­ing, and mem­o­ry for items. Most of the train­ing ses­sions were 90 sec each; the rotat­ing num­bers tasks was 3 min. These activ­i­ties are sim­i­lar to those used in “edu­tain­ment” pro­grams that can be played online or with a hand­held device. The oth­er exper­i­men­tal group was trained on rea­son­ing tasks that involved iden­ti­fy­ing rel­a­tive weights of objects based on a visu­al “see­saw”, select­ing the “odd” item in a con­cept for­ma­tion type task, a task involv­ing think­ing through the effects of one action on cur­rent and future states, and three plan­ning tasks includ­ing draw­ing a con­tin­u­ous line around a grid while ascer­tain­ing that the line will not hin­der lat­er moves, a ver­sion of the Tow­er of Hanoi task, and a tile slid­ing game. The con­trol group spent time answer­ing ques­tions about obscure facts and orga­niz­ing them chrono­log­i­cal­ly based on any avail­able online resource. Results indi­cat­ed that the two exper­i­men­tal groups per­formed bet­ter than the con­trol group on only one out­come test of gram­mat­i­cal rea­son­ing; there were no dif­fer­ences between either exper­i­men­tal group and the con­trols on the remain­ing test. The exper­i­men­tal groups had improved on the trained tasks but not on the trans­fer tasks.

Sci­en­tif­ic con­cerns

Although some news reports sug­gest that these find­ings are defin­i­tive, there are a num­ber of con­cerns, many of which have to do with whether the find­ings have been over­gen­er­al­ized to all forms of brain train­ing because only a few tests were used. Sec­ond, there have been ques­tions raised about the amount of time allo­cat­ed to train­ing and the issue of test­ing in the home envi­ron­ment. The study report­ed no rela­tion­ship between expo­sure to train­ing and out­come, sug­gest­ing that the amount of time was not crit­i­cal. How­ev­er, there may not have been suf­fi­cient train­ing in gen­er­al, and there may be a thresh­old of expo­sure to train­ing that may be need­ed before trans­fer is observ­able. Third, there are ques­tions about whether recruit­ment of par­tic­i­pants from a show web­site that is bla­tant­ly skep­ti­cal of var­i­ous claims pro­duced a sam­ple biased against find­ing pos­i­tive effects of train­ing on gen­er­al­ized out­comes.

1. Sub­stan­tial, selec­tive and unex­plained dropout rates

There was a sub­stan­tial dropout rate for the study, as best as we can tell, with 52,617 par­tic­i­pants reg­is­ter­ing for the tri­al and 11,430 com­plet­ing pre and post train­ing bench­mark­ing assess­ments and at least two 10-minute train­ing ses­sions. If there are no oth­er inclu­sion cri­te­ria, this indi­cates a sub­stan­tial dropout rate. There was equal prob­a­bil­i­ty of assign­ment to the exper­i­men­tal and con­trol groups, and it is not indi­cat­ed how many peo­ple actu­al­ly dropped out from the reasoning/planning group with its study sam­ple size of 4,678. There were high­er dropout rates for the “brain train­ing” group of 13% and the con­trol group of 32%. In a clin­i­cal tri­al, such selec­tive and high dropout rates would be con­sid­ered very prob­lem­at­ic. How­ev­er, on bal­ance, the con­trol group did not score as supe­ri­or to the exper­i­men­tal groups on the base­line tests.

2- Ques­tion­able out­come mea­sure­ment and inter­pre­ta­tion

I add to these ques­tions a seri­ous method­olog­i­cal con­cern about the mea­sure­ment of out­come data. Of the four trans­fer tests, only one (rea­son­ing) was scored as total cor­rect. The oth­er three tests (ver­bal short term mem­o­ry), spa­tial work­ing mem­o­ry, and paired asso­ciates) were scored as span tests (max­i­mum items cor­rect with­in a tri­al). Span tests are famous­ly insen­si­tive to change, because the max­i­mum of work­ing mem­o­ry span is 4 “chunks” of infor­ma­tion when rehearsal process­es are pre­vent­ed (see Cow­an, 2001). Ver­haeghen, Cerel­la, and Basak (2004) found that approx­i­mate­ly 10 hours of train­ing on one work­ing mem­o­ry span task pro­duced a broad­en­ing of span from one to four chunks when rehearsal was pre­vent­ed in young adults. Thus the use of span as the type of mea­sure­ment is ques­tion­able. Even when span mea­sures poten­tial­ly allow for rehearsal, as in the Owen study with a stan­dard dig­it span task, the range of per­for­mance is quite nar­row. For exam­ple, dig­it span is approx­i­mate­ly 7 items +/- 2. Total cor­rect scor­ing pro­duces a con­sid­er­ably wider range of per­for­mance, which allows for “growth”. This can be seen in the mean scores and the error bars of Fig­ure 1 in the Owens et al., (2010) arti­cle, by com­par­ing rea­son­ing to the each of the oth­er three out­come mea­sures.

The arti­cle indi­cates that the out­come mea­sures have been used to show effects of drugs on cog­ni­tive per­for­mance and this is indeed the case, but the cit­ed arti­cles use total cor­rect but not span mea­sures for paired asso­ciates and spa­tial work­ing mem­o­ry out­comes (e.g. Turn­er, et al., 2003). Notably, if we eval­u­ate the rea­son­ing mea­sure in the Owen et al. study, there was sig­nif­i­cant improve­ment in the 2 train­ing groups com­pared to the con­trols, with effect sizes of .17 and .22. These are con­sid­ered small effect sizes but are not much dif­fer­ent than those for the effect size of modafinil on back­wards dig­it span, stop-sig­nal response time, and visu­al mem­o­ry in a much small­er sam­ple of 60 adults (Turn­er et al., 2003). The authors of the modafinil study wrote that the results of their study “sug­gest that modafinil offers sig­nif­i­cant poten­tial as a cog­ni­tive enhancer, par­tic­u­lar­ly with respect to its effects on plan­ning, accu­ra­cy and inhi­bi­tion” (p. 268). We note that modafinil has a mod­er­ate effect size of .52 on spa­tial plan­ning.

In con­clu­sion

In con­clu­sion, in my opin­ion, the Owen et al. (2010) study con­tributes to the lit­er­a­ture on com­put­er­ized brain train­ing, by show­ing that a sub­stan­tial num­ber of indi­vid­u­als can be recruit­ed to par­tic­i­pate, with a wide range of actu­al amount of prac­tice, and that trans­fer as mea­sured did not occur in tasks mea­sured as spans, but did show small effects sim­i­lar to that of drug effects on the one test mea­sured as num­ber cor­rect. Trans­fer effects have been observed in stud­ies with old­er adults as well as younger ones in more con­trolled research envi­ron­ments; it remains to be seen whether the data col­lect­ed by the Nature study authors on old­er adults, which were not includ­ed in the pub­lished arti­cle, will show dif­fer­ent results. Obvi­ous­ly, few stud­ies in gen­er­al have been con­duct­ed on the role of auto­mat­ed cog­ni­tive train­ing in healthy adults, and more are need­ed before we can draw final con­clu­sions about its val­ue in tests of trans­fer from brain train­ing activ­i­ties. We also note that trans­fer is assumed to occur in edu­ca­tion­al envi­ron­ments; enor­mous sums of mon­ey are spent on train­ing young peo­ple not just so that they can do well in school, but so that they can lead pro­duc­tive lives after­wards.


  • Cow­an, N. (2001). The mag­i­cal num­ber 4 in short-term mem­o­ry: A recon­sid­er­a­tion of men­tal stor­age capac­i­ty. Behav­ioral and Brain Sci­ences, 24, 87–185.
  • Owen, A. M., Hamp­shire, A., Grahn, J. A., Sten­ton, R., Dajani, S., Burns, A. S., et al., (2010). Putting brain train­ing to the test. Nature, online advance pub­li­ca­tion
  • Turn­er, D. C., Rob­bins, T. W., Clark, L., Aron, A. R., Dow­son, J., & Sahakian, B. (2003). Cog­ni­tive enhanc­ing effects of modafinil in healthy vol­un­teers. Psy­chophar­ma­col­o­gy, 165, 260–269.
  • Ver­haeghen, P., Cerel­la, J., & Basak, C. (2004). A work­ing mem­o­ry work­out: How to expand the focus of ser­i­al atten­tion from one to four items in 10 hours or less. Jour­nal of Exper­i­men­tal P sychol­o­gy: Learn­ing, Mem­o­ry, and Cog­ni­tion, 30, 1322–1337.

Relat­ed arti­cles:

BBC “Brain Train­ing” Exper­i­ment: the Good, the Bad, the Ugly

Brain Train­ing: No Mag­ic Bul­let, yet Use­ful Tool

zelinskiProfile-150x150Eliz­a­beth Zelin­s­ki, Ph.D., is a Pro­fes­sor of Geron­tol­ogy and Psy­chol­o­gy at the Leonard Davis School of Geron­tol­ogy. Dr. Zelin­s­ki has joint appoint­ments in the Psy­chol­o­gy Depart­ment, Neu­ro­sciences and the Study of Women and Men in Soci­ety (SWMS) Pro­grams. Dr. Zelin­s­ki grad­u­at­ed sum­ma cum laude from Pace Uni­ver­si­ty and received her grad­u­ate degrees in psy­chol­o­gy, with a spe­cial­iza­tion in aging, from the Uni­ver­si­ty of South­ern Cal­i­for­nia. Dr. Zelin­s­ki is the prin­ci­pal inves­ti­ga­tor of the Long Beach Lon­gi­tu­di­nal Study, and was Co-PI in the IMPACT study.

Leave a Reply...

Loading Facebook Comments ...

Leave a Reply

Categories: Cognitive Neuroscience, Health & Wellness

Tags: , , , , , , , , ,

All Slidedecks & Recordings Available — click image below

Search for anything brain-related in our article archives

About SharpBrains

As seen in The New York Times, The Wall Street Journal, BBC News, CNN, Reuters, and more, SharpBrains is an independent market research firm and think tank tracking health and performance applications of brain science.