Scientific critique of BBC/ Nature Brain Training Experiment

There has been quite a bit of com­ment about the Owen et al study in Nature avail­able online on April 20, 2010. A quick syn­op­sis of the study is that the BBC show Bang Goes the The­o­ry worked with the study authors to pro­vide a test of the hypoth­e­sis that com­mer­cial­ly avail­able brain train­ing pro­grams trans­fer to gen­er­al cog­ni­tive abil­i­ties. The con­clu­sion was that, despite improve­ments on the trained tasks, “no evi­dence was found for trans­fer effects to untrained tasks, even when those tasks were cog­ni­tive­ly close­ly related.”

The exper­i­ment

The study was con­duct­ed through the show’s web site. Of 52,617 par­tic­i­pants who reg­is­tered, approx­i­mate­ly 20% (11,430) com­plet­ed full par­tic­i­pa­tion in the study, which con­sist­ed of two bench­mark­ing assess­ments 6 weeks apart with vari­ants of neu­ropsy­cho­log­i­cal tests and at least two train­ing ses­sions. Peo­ple were ran­dom­ly assigned to one of three groups that were asked to train for about 10 min a day three times a week for the 6‑week peri­od, though they could train either more or less fre­quent­ly. One of the two exper­i­men­tal groups was a “brain train­ing” group that com­plet­ed tasks includ­ing sim­ple arith­metic, find­ing miss­ing pieces, match­ing sym­bols to a tar­get, order­ing rotat­ing num­bers by numer­i­cal val­ue, updat­ing, and mem­o­ry for items. Most of the train­ing ses­sions were 90 sec each; the rotat­ing num­bers tasks was 3 min. These activ­i­ties are sim­i­lar to those used in “edu­tain­ment” pro­grams that can be played online or with a hand­held device. The oth­er exper­i­men­tal group was trained on rea­son­ing tasks that involved iden­ti­fy­ing rel­a­tive weights of objects based on a visu­al “see­saw”, select­ing the “odd” item in a con­cept for­ma­tion type task, a task involv­ing think­ing through the effects of one action on cur­rent and future states, and three plan­ning tasks includ­ing draw­ing a con­tin­u­ous line around a grid while ascer­tain­ing that the line will not hin­der lat­er moves, a ver­sion of the Tow­er of Hanoi task, and a tile slid­ing game. The con­trol group spent time answer­ing ques­tions about obscure facts and orga­niz­ing them chrono­log­i­cal­ly based on any avail­able online resource. Results indi­cat­ed that the two exper­i­men­tal groups per­formed bet­ter than the con­trol group on only one out­come test of gram­mat­i­cal rea­son­ing; there were no dif­fer­ences between either exper­i­men­tal group and the con­trols on the remain­ing test. The exper­i­men­tal groups had improved on the trained tasks but not on the trans­fer tasks.

Sci­en­tif­ic concerns

Although some news reports sug­gest that these find­ings are defin­i­tive, there are a num­ber of con­cerns, many of which have to do with whether the find­ings have been over­gen­er­al­ized to all forms of brain train­ing because only a few tests were used. Sec­ond, there have been ques­tions raised about the amount of time allo­cat­ed to train­ing and the issue of test­ing in the home envi­ron­ment. The study report­ed no rela­tion­ship between expo­sure to train­ing and out­come, sug­gest­ing that the amount of time was not crit­i­cal. How­ev­er, there may not have been suf­fi­cient train­ing in gen­er­al, and there may be a thresh­old of expo­sure to train­ing that may be need­ed before trans­fer is observ­able. Third, there are ques­tions about whether recruit­ment of par­tic­i­pants from a show web­site that is bla­tant­ly skep­ti­cal of var­i­ous claims pro­duced a sam­ple biased against find­ing pos­i­tive effects of train­ing on gen­er­al­ized outcomes.

1. Sub­stan­tial, selec­tive and unex­plained dropout rates

There was a sub­stan­tial dropout rate for the study, as best as we can tell, with 52,617 par­tic­i­pants reg­is­ter­ing for the tri­al and 11,430 com­plet­ing pre and post train­ing bench­mark­ing assess­ments and at least two 10-minute train­ing ses­sions. If there are no oth­er inclu­sion cri­te­ria, this indi­cates a sub­stan­tial dropout rate. There was equal prob­a­bil­i­ty of assign­ment to the exper­i­men­tal and con­trol groups, and it is not indi­cat­ed how many peo­ple actu­al­ly dropped out from the reasoning/planning group with its study sam­ple size of 4,678. There were high­er dropout rates for the “brain train­ing” group of 13% and the con­trol group of 32%. In a clin­i­cal tri­al, such selec­tive and high dropout rates would be con­sid­ered very prob­lem­at­ic. How­ev­er, on bal­ance, the con­trol group did not score as supe­ri­or to the exper­i­men­tal groups on the base­line tests.

2- Ques­tion­able out­come mea­sure­ment and interpretation

I add to these ques­tions a seri­ous method­olog­i­cal con­cern about the mea­sure­ment of out­come data. Of the four trans­fer tests, only one (rea­son­ing) was scored as total cor­rect. The oth­er three tests (ver­bal short term mem­o­ry), spa­tial work­ing mem­o­ry, and paired asso­ciates) were scored as span tests (max­i­mum items cor­rect with­in a tri­al). Span tests are famous­ly insen­si­tive to change, because the max­i­mum of work­ing mem­o­ry span is 4 “chunks” of infor­ma­tion when rehearsal process­es are pre­vent­ed (see Cow­an, 2001). Ver­haeghen, Cerel­la, and Basak (2004) found that approx­i­mate­ly 10 hours of train­ing on one work­ing mem­o­ry span task pro­duced a broad­en­ing of span from one to four chunks when rehearsal was pre­vent­ed in young adults. Thus the use of span as the type of mea­sure­ment is ques­tion­able. Even when span mea­sures poten­tial­ly allow for rehearsal, as in the Owen study with a stan­dard dig­it span task, the range of per­for­mance is quite nar­row. For exam­ple, dig­it span is approx­i­mate­ly 7 items +/- 2. Total cor­rect scor­ing pro­duces a con­sid­er­ably wider range of per­for­mance, which allows for “growth”. This can be seen in the mean scores and the error bars of Fig­ure 1 in the Owens et al., (2010) arti­cle, by com­par­ing rea­son­ing to the each of the oth­er three out­come measures.

The arti­cle indi­cates that the out­come mea­sures have been used to show effects of drugs on cog­ni­tive per­for­mance and this is indeed the case, but the cit­ed arti­cles use total cor­rect but not span mea­sures for paired asso­ciates and spa­tial work­ing mem­o­ry out­comes (e.g. Turn­er, et al., 2003). Notably, if we eval­u­ate the rea­son­ing mea­sure in the Owen et al. study, there was sig­nif­i­cant improve­ment in the 2 train­ing groups com­pared to the con­trols, with effect sizes of .17 and .22. These are con­sid­ered small effect sizes but are not much dif­fer­ent than those for the effect size of modafinil on back­wards dig­it span, stop-sig­nal response time, and visu­al mem­o­ry in a much small­er sam­ple of 60 adults (Turn­er et al., 2003). The authors of the modafinil study wrote that the results of their study “sug­gest that modafinil offers sig­nif­i­cant poten­tial as a cog­ni­tive enhancer, par­tic­u­lar­ly with respect to its effects on plan­ning, accu­ra­cy and inhi­bi­tion” (p. 268). We note that modafinil has a mod­er­ate effect size of .52 on spa­tial planning.

In con­clu­sion

In con­clu­sion, in my opin­ion, the Owen et al. (2010) study con­tributes to the lit­er­a­ture on com­put­er­ized brain train­ing, by show­ing that a sub­stan­tial num­ber of indi­vid­u­als can be recruit­ed to par­tic­i­pate, with a wide range of actu­al amount of prac­tice, and that trans­fer as mea­sured did not occur in tasks mea­sured as spans, but did show small effects sim­i­lar to that of drug effects on the one test mea­sured as num­ber cor­rect. Trans­fer effects have been observed in stud­ies with old­er adults as well as younger ones in more con­trolled research envi­ron­ments; it remains to be seen whether the data col­lect­ed by the Nature study authors on old­er adults, which were not includ­ed in the pub­lished arti­cle, will show dif­fer­ent results. Obvi­ous­ly, few stud­ies in gen­er­al have been con­duct­ed on the role of auto­mat­ed cog­ni­tive train­ing in healthy adults, and more are need­ed before we can draw final con­clu­sions about its val­ue in tests of trans­fer from brain train­ing activ­i­ties. We also note that trans­fer is assumed to occur in edu­ca­tion­al envi­ron­ments; enor­mous sums of mon­ey are spent on train­ing young peo­ple not just so that they can do well in school, but so that they can lead pro­duc­tive lives afterwards.

References:

• Cow­an, N. (2001). The mag­i­cal num­ber 4 in short-term mem­o­ry: A recon­sid­er­a­tion of men­tal stor­age capac­i­ty. Behav­ioral and Brain Sci­ences, 24, 87–185.
• Owen, A. M., Hamp­shire, A., Grahn, J. A., Sten­ton, R., Dajani, S., Burns, A. S., et al., (2010). Putting brain train­ing to the test. Nature, online advance pub­li­ca­tion www.nature.com/doifinder/10.1038/nature09042
• Turn­er, D. C., Rob­bins, T. W., Clark, L., Aron, A. R., Dow­son, J., & Sahakian, B. (2003). Cog­ni­tive enhanc­ing effects of modafinil in healthy vol­un­teers. Psy­chophar­ma­col­o­gy, 165, 260–269.
• Ver­haeghen, P., Cerel­la, J., & Basak, C. (2004). A work­ing mem­o­ry work­out: How to expand the focus of ser­i­al atten­tion from one to four items in 10 hours or less. Jour­nal of Exper­i­men­tal P sychol­o­gy: Learn­ing, Mem­o­ry, and Cog­ni­tion, 30, 1322–1337.

Relat­ed articles:

BBC “Brain Train­ing” Exper­i­ment: the Good, the Bad, the Ugly

Brain Train­ing: No Mag­ic Bul­let, yet Use­ful Tool

– Eliz­a­beth Zelin­s­ki, Ph.D., is a Pro­fes­sor of Geron­tol­ogy and Psy­chol­o­gy at the Leonard Davis School of Geron­tol­ogy. Dr. Zelin­s­ki has joint appoint­ments in the Psy­chol­o­gy Depart­ment, Neu­ro­sciences and the Study of Women and Men in Soci­ety (SWMS) Pro­grams. Dr. Zelin­s­ki grad­u­at­ed sum­ma cum laude from Pace Uni­ver­si­ty and received her grad­u­ate degrees in psy­chol­o­gy, with a spe­cial­iza­tion in aging, from the Uni­ver­si­ty of South­ern Cal­i­for­nia. Dr. Zelin­s­ki is the prin­ci­pal inves­ti­ga­tor of the Long Beach Lon­gi­tu­di­nal Study, and was Co-PI in the IMPACT study.