Brain–behavior associations depend heavily on user-defined criteria

Thanks to a recent massive study involving analysis of nearly 50,000 MRI datasets from the Adolescent Brain Cognitive Development (ABCD), Human Connectome Project (HCP), and UK Biobank, we have learned that thousands of subjects are needed to arrive at the reproducible brain–behavioral phenotype associations using univariate analytic approaches (1). The study by Marek and colleagues examined widely used brain features (e.g., cortical thickness, resting state functional connectivity metrics) to estimate relationships with multiple behavioral phenotypes (cognition and mental health). The authors argue that small brain-wide association effects and population sampling variability can result in inflated and irreproducible brain–phenotype associations in sample sizes not in the thousands. These results should be concerning to cognitive neuroscientists hoping to relate individual differences in brain structure and function to psychological phenotypes.

First let us start off by acknowledging that this is an impressive, very important study that was carefully conducted; indeed, the authors “performed billions of univariate and multivariate analyses” and should be congratulated on this tremendous feat. In addition to highlighting issues of small sample sizes and sampling variability that have contributed to the replication crisis in neuroimaging (2), the recent study by Marek and colleagues provides an opportunity for neuroscientists to revisit another source of variability that is less often discussed: assumptions regarding which brain features could meaningfully relate to behavioral phenotypes. There is a great deal of dependence on user-defined criteria inherent to all analyses of this type. For example, one user could decide that a specific brain region of interest (ROI) is the appropriate brain feature to include in a given analysis, whereas another user could choose to derive brain features at the whole-brain network level. We used HCP data from three different tasks (working memory, relational, and arithmetic) to look at both ROI- and network-level associations with task-related behavioral performance. Using Pearson correlation between a single ROI and task performance does not account for dependencies among ROIs of the same network, but a structural equation modeling (SEM) framework (Figure 1) can be used to model network activity as a latent variable with the constituent ROIs of a network as indicators, as well as model the unique association of both the latent network and ROIs with behavior. Importantly, these analyses revealed that depending on user-defined criteria (e.g., the choice to use either an ROI-level or network-level brain feature in a brain–behavior analysis), very different results can be obtained with the same dataset (3). This study is just one example of the importance of accounting for both ROI- and network levels of analysis in studies of brain–behavior relationships, and draws attention to the fact that the reproducibility crisis in neuroimaging cannot be tackled by greater sample sizes alone. The field has clearly not yet converged on a principled approach (or set of approaches) for deriving brain–behavior associations. At present, these analyses tend to follow idiosyncratic individual user-defined criteria that are highly subjective, further contributing to the heterogeneity of brain–behavior association findings in the field.

Acknowledgment LQU is supported by the National Institute on Drug Abuse (U01DA050987).

REFERENCES

1.Marek, S., Tervo-Clemmens, B., Calabro, F.J., Montez, D.F., Kay, B.P., Hatoum, A.S., et al. Reproducible Brain-Wide Association Studies Require Thousands of Individuals. Nature 2022;603(7902):654–60.

2.Genon, S., Eickhoff, S.B., Kharabian, S. Linking Interindividual Variability in Brain Structure to Behaviour. Nature Reviews Neuroscience 2022;23(5):307–18.

3.Bolt, T., Prince, E.B., Nomi, J.S., Messinger, D., Llabre, M.M., Uddin, L.Q. Combining Region- and Network-Level Brain-Behavior Relationships in a Structural Equation Model. NeuroImage 2018;165:158–69.

Fig. 1. Combining region- and network-level brain–behavior relationships. Path diagrams with standardized parameter estimates for two paths: (1) the path between the latent network variable and task accuracy, representing the unique association between overall activation estimates in the FPN or CO/S network and task accuracy, and (2) the path between the hypothesized ROI (right DLPFC/left RLPFC/right TPJ) and task accuracy, representing the association between residual activation estimates in the ROI and task accuracy (with activity due to association with the network removed). Estimates using the Power parcellation are in blue, estimates from the Gordon parcellation are in red. FPN = Frontoparietal Network; CO/S = Cingulo-opercular/Salience Network (reproduced with permission from Bolt et al. [3]).

This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits authors to copy and redistribute the material in any medium or format, remix, transform and build upon material, for any purpose, even commercially.