There has been quite a bit of com­ment about the Owen et al study in Nature avail­able online on April 20, 2010. A quick syn­op­sis of the study is that the BBC show Bang Goes the The­ory worked with the study authors to pro­vide a test of the hypoth­e­sis that com­mer­cially avail­able brain train­ing pro­grams trans­fer to gen­eral cog­ni­tive abil­i­ties. The con­clu­sion was that, despite improve­ments on the trained tasks, “no evi­dence was found for trans­fer effects to untrained tasks, even when those tasks were cog­ni­tively closely related.”

The exper­i­ment

The study was con­ducted through the show’s web site. Of 52,617 par­tic­i­pants who reg­is­tered, approx­i­mately 20% (11,430) com­pleted full par­tic­i­pa­tion in the study, which con­sisted of two bench­mark­ing assess­ments 6 weeks apart with vari­ants of neu­ropsy­cho­log­i­cal tests and at least two train­ing ses­sions. Peo­ple were ran­domly assigned to one of three groups that were asked to train for about 10 min a day three times a week for the 6-week period, though they could train either more or less fre­quently. One of the two exper­i­men­tal groups was a “brain train­ing” group that com­pleted tasks includ­ing sim­ple arith­metic, find­ing miss­ing pieces, match­ing sym­bols to a tar­get, order­ing rotat­ing num­bers by numer­i­cal value, updat­ing, and mem­ory for items. Most of the train­ing ses­sions were 90 sec each; the rotat­ing num­bers tasks was 3 min. These activ­i­ties are sim­i­lar to those used in “edu­tain­ment” pro­grams that can be played online or with a hand­held device. The other exper­i­men­tal group was trained on rea­son­ing tasks that involved iden­ti­fy­ing rel­a­tive weights of objects based on a visual “see­saw”, select­ing the “odd” item in a con­cept for­ma­tion type task, a task involv­ing think­ing through the effects of one action on cur­rent and future states, and three plan­ning tasks includ­ing draw­ing a con­tin­u­ous line around a grid while ascer­tain­ing that the line will not hin­der later moves, a ver­sion of the Tower of Hanoi task, and a tile slid­ing game. The con­trol group spent time answer­ing ques­tions about obscure facts and orga­niz­ing them chrono­log­i­cally based on any avail­able online resource. Results indi­cated that the two exper­i­men­tal groups per­formed bet­ter than the con­trol group on only one out­come test of gram­mat­i­cal rea­son­ing; there were no dif­fer­ences between either exper­i­men­tal group and the con­trols on the remain­ing test. The exper­i­men­tal groups had improved on the trained tasks but not on the trans­fer tasks.

Sci­en­tific concerns

Although some news reports sug­gest that these find­ings are defin­i­tive, there are a num­ber of con­cerns, many of which have to do with whether the find­ings have been over­gen­er­al­ized to all forms of brain train­ing because only a few tests were used. Sec­ond, there have been ques­tions raised about the amount of time allo­cated to train­ing and the issue of test­ing in the home envi­ron­ment. The study reported no rela­tion­ship between expo­sure to train­ing and out­come, sug­gest­ing that the amount of time was not crit­i­cal. How­ever, there may not have been suf­fi­cient train­ing in gen­eral, and there may be a thresh­old of expo­sure to train­ing that may be needed before trans­fer is observ­able. Third, there are ques­tions about whether recruit­ment of par­tic­i­pants from a show web­site that is bla­tantly skep­ti­cal of var­i­ous claims pro­duced a sam­ple biased against find­ing pos­i­tive effects of train­ing on gen­er­al­ized outcomes.

1. Sub­stan­tial, selec­tive and unex­plained dropout rates

There was a sub­stan­tial dropout rate for the study, as best as we can tell, with 52,617 par­tic­i­pants reg­is­ter­ing for the trial and 11,430 com­plet­ing pre and post train­ing bench­mark­ing assess­ments and at least two 10-minute train­ing ses­sions. If there are no other inclu­sion cri­te­ria, this indi­cates a sub­stan­tial dropout rate. There was equal prob­a­bil­ity of assign­ment to the exper­i­men­tal and con­trol groups, and it is not indi­cated how many peo­ple actu­ally dropped out from the reasoning/planning group with its study sam­ple size of 4,678. There were higher dropout rates for the “brain train­ing” group of 13% and the con­trol group of 32%. In a clin­i­cal trial, such selec­tive and high dropout rates would be con­sid­ered very prob­lem­atic. How­ever, on bal­ance, the con­trol group did not score as supe­rior to the exper­i­men­tal groups on the base­line tests.

2– Ques­tion­able out­come mea­sure­ment and interpretation

I add to these ques­tions a seri­ous method­olog­i­cal con­cern about the mea­sure­ment of out­come data. Of the four trans­fer tests, only one (rea­son­ing) was scored as total cor­rect. The other three tests (ver­bal short term mem­ory), spa­tial work­ing mem­ory, and paired asso­ciates) were scored as span tests (max­i­mum items cor­rect within a trial). Span tests are famously insen­si­tive to change, because the max­i­mum of work­ing mem­ory span is 4 “chunks” of infor­ma­tion when rehearsal processes are pre­vented (see Cowan, 2001). Ver­haeghen, Cerella, and Basak (2004) found that approx­i­mately 10 hours of train­ing on one work­ing mem­ory span task pro­duced a broad­en­ing of span from one to four chunks when rehearsal was pre­vented in young adults. Thus the use of span as the type of mea­sure­ment is ques­tion­able. Even when span mea­sures poten­tially allow for rehearsal, as in the Owen study with a stan­dard digit span task, the range of per­for­mance is quite nar­row. For exam­ple, digit span is approx­i­mately 7 items +/- 2. Total cor­rect scor­ing pro­duces a con­sid­er­ably wider range of per­for­mance, which allows for “growth”. This can be seen in the mean scores and the error bars of Fig­ure 1 in the Owens et al., (2010) arti­cle, by com­par­ing rea­son­ing to the each of the other three out­come measures.

The arti­cle indi­cates that the out­come mea­sures have been used to show effects of drugs on cog­ni­tive per­for­mance and this is indeed the case, but the cited arti­cles use total cor­rect but not span mea­sures for paired asso­ciates and spa­tial work­ing mem­ory out­comes (e.g. Turner, et al., 2003). Notably, if we eval­u­ate the rea­son­ing mea­sure in the Owen et al. study, there was sig­nif­i­cant improve­ment in the 2 train­ing groups com­pared to the con­trols, with effect sizes of .17 and .22. These are con­sid­ered small effect sizes but are not much dif­fer­ent than those for the effect size of modafinil on back­wards digit span, stop-signal response time, and visual mem­ory in a much smaller sam­ple of 60 adults (Turner et al., 2003). The authors of the modafinil study wrote that the results of their study “sug­gest that modafinil offers sig­nif­i­cant poten­tial as a cog­ni­tive enhancer, par­tic­u­larly with respect to its effects on plan­ning, accu­racy and inhi­bi­tion” (p. 268). We note that modafinil has a mod­er­ate effect size of .52 on spa­tial planning.

In con­clu­sion

In con­clu­sion, in my opin­ion, the Owen et al. (2010) study con­tributes to the lit­er­a­ture on com­put­er­ized brain train­ing, by show­ing that a sub­stan­tial num­ber of indi­vid­u­als can be recruited to par­tic­i­pate, with a wide range of actual amount of prac­tice, and that trans­fer as mea­sured did not occur in tasks mea­sured as spans, but did show small effects sim­i­lar to that of drug effects on the one test mea­sured as num­ber cor­rect. Trans­fer effects have been observed in stud­ies with older adults as well as younger ones in more con­trolled research envi­ron­ments; it remains to be seen whether the data col­lected by the Nature study authors on older adults, which were not included in the pub­lished arti­cle, will show dif­fer­ent results. Obvi­ously, few stud­ies in gen­eral have been con­ducted on the role of auto­mated cog­ni­tive train­ing in healthy adults, and more are needed before we can draw final con­clu­sions about its value in tests of trans­fer from brain train­ing activ­i­ties. We also note that trans­fer is assumed to occur in edu­ca­tional envi­ron­ments; enor­mous sums of money are spent on train­ing young peo­ple not just so that they can do well in school, but so that they can lead pro­duc­tive lives afterwards.

References:

• Cowan, N. (2001). The mag­i­cal num­ber 4 in short-term mem­ory: A recon­sid­er­a­tion of men­tal stor­age capac­ity. Behav­ioral and Brain Sci­ences, 24, 87–185.
• Owen, A. M., Hamp­shire, A., Grahn, J. A., Sten­ton, R., Dajani, S., Burns, A. S., et al., (2010). Putting brain train­ing to the test. Nature, online advance pub­li­ca­tion www.nature.com/doifinder/10.1038/nature09042
• Turner, D. C., Rob­bins, T. W., Clark, L., Aron, A. R., Dow­son, J., & Sahakian, B. (2003). Cog­ni­tive enhanc­ing effects of modafinil in healthy vol­un­teers. Psy­chophar­ma­col­ogy, 165, 260–269.
• Ver­haeghen, P., Cerella, J., & Basak, C. (2004). A work­ing mem­ory work­out: How to expand the focus of ser­ial atten­tion from one to four items in 10 hours or less. Jour­nal of Exper­i­men­tal P sychol­ogy: Learn­ing, Mem­ory, and Cog­ni­tion, 30, 1322–1337.

Related arti­cles:

BBC “Brain Train­ing” Exper­i­ment: the Good, the Bad, the Ugly

Brain Train­ing: No Magic Bul­let, yet Use­ful Tool

– Eliz­a­beth Zelin­ski, Ph.D., is a Pro­fes­sor of Geron­tol­ogy and Psy­chol­ogy at the Leonard Davis School of Geron­tol­ogy. Dr. Zelin­ski has joint appoint­ments in the Psy­chol­ogy Depart­ment, Neu­ro­sciences and the Study of Women and Men in Soci­ety (SWMS) Pro­grams. Dr. Zelin­ski grad­u­ated summa cum laude from Pace Uni­ver­sity and received her grad­u­ate degrees in psy­chol­ogy, with a spe­cial­iza­tion in aging, from the Uni­ver­sity of South­ern Cal­i­for­nia. Dr. Zelin­ski is the prin­ci­pal inves­ti­ga­tor of the Long Beach Lon­gi­tu­di­nal Study, and was Co-PI in the IMPACT study.

Be Socia­ble, Share!

•