Flexibility is good except when it isn’t: Study finds how scientists can reach different conclusions analyzing the same brain scans

Neu­roimag­ing: Many Ana­lysts, Dif­fer­ing Results (Dana Foundation):

For decades, both the research and med­ical com­mu­ni­ties have relied on neu­roimag­ing tools like func­tion­al mag­net­ic res­o­nance imag­ing (fMRI) to give them a win­dow into the liv­ing human brain. Such scans have pro­vid­ed unprece­dent­ed insights into the brain’s struc­ture and func­tion – and the field, as a whole, has used this tech­nique to bet­ter under­stand how the brain gives rise to thoughts, emo­tions, and actions. But as neu­roimag­ing tech­nol­o­gy has advanced, so have the dif­fer­ent analy­sis tools and the num­ber of ways one can eval­u­ate the result­ing data. Now, the results of unique research project, the Neu­roimag­ing Analy­sis, Repli­ca­tion, and Pre­dic­tion Study (NARPS), sug­gest that dif­fer­ent analy­ses can lead to strik­ing­ly dif­fer­ent results from the same data set.

…

“There are so many dif­fer­ent soft­ware pack­ages now, and dif­fer­ent labs use dif­fer­ent ones for all man­ner of dif­fer­ent rea­sons. There are also dif­fer­ent philoso­phies about how analy­ses should be done. All those lit­tle dif­fer­ences can add up,” he (Edi­tor’s note: Rus­sell Pol­drack, a cog­ni­tive neu­ro­sci­en­tist at Stan­ford Uni­ver­si­ty) said. “But, when we looked close­ly at what peo­ple pro­vid­ed, we could see that the results under the hood were sub­stan­tial­ly more sim­i­lar than what they con­clud­ed. There was some­thing about going from the inter­me­di­ate steps of analy­sis work­flow to deter­min­ing the right thresh­old to denote a final yes/no answer that changed things. It was real­ly striking.”

The Study:

Vari­abil­i­ty in the analy­sis of a sin­gle neu­roimag­ing dataset by many teams (Nature). From the Abstract:

• Data analy­sis work­flows in many sci­en­tif­ic domains have become increas­ing­ly com­plex and flex­i­ble. Here we assess the effect of this flex­i­bil­i­ty on the results of func­tion­al mag­net­ic res­o­nance imag­ing by ask­ing 70 inde­pen­dent teams to analyse the same dataset, test­ing the same 9 ex-ante hypothe­ses. The flex­i­bil­i­ty of ana­lyt­i­cal approach­es is exem­pli­fied by the fact that no two teams chose iden­ti­cal work­flows to analyse the data … Our find­ings show that ana­lyt­i­cal flex­i­bil­i­ty can have sub­stan­tial effects on sci­en­tif­ic con­clu­sions, and iden­ti­fy fac­tors that may be relat­ed to vari­abil­i­ty in the analy­sis of func­tion­al mag­net­ic res­o­nance imag­ing. The results empha­size the impor­tance of val­i­dat­ing and shar­ing com­plex analy­sis work­flows, and demon­strate the need for per­form­ing and report­ing mul­ti­ple analy­ses of the same data. Poten­tial approach­es that could be used to mit­i­gate issues relat­ed to ana­lyt­i­cal vari­abil­i­ty are discussed.

The Study in Context:

• #narps: Neu­roimag­ing Analy­sis Repli­ca­tion and Pre­dic­tion Study
• Machine-learn­ing study finds EEG brain sig­na­tures that pre­dict response to anti­de­pres­sant treatments
• Study com­bines neu­roimag­ing with machine learn­ing to pre­dict, with 96% accu­ra­cy, whether high-risk 6?month-old babies will devel­op autism spec­trum dis­or­der (ASD) by age 2
• Brain scans show low­er accu­mu­la­tion of tau and amy­loid pathol­o­gy among cog­ni­tive “super-agers”
• 10 high­lights from the 2019 Sharp­Brains Vir­tu­al Summit