Abstract
The increasing popularity of naturalistic paradigms in fMRI (such as movie watching) demands novel strategies for multi-subject data analysis, such as use of neural encoding models. In the present study, we propose a shared convolutional neural encoding method that accounts for individual-level differences. Our method leverages multi-subject data to improve the prediction of subject-specific responses evoked by visual or auditory stimuli. We showcase our approach on high-resolution 7T fMRI data from the Human Connectome Project movie-watching protocol and demonstrate significant improvement over single-subject encoding models. We further demonstrate the ability of the shared encoding model to successfully capture meaningful individual differences in response to traditional task-based facial and scenes stimuli. Taken together, our findings suggest that inter-subject knowledge transfer can be beneficial to subject-specific predictive models. (Our code is available at https://github.com/mk2299/SharedEncoding_MICCAI.)
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., Malach, R.: Intersubject synchronization of cortical activity during natural vision. Science 303(5664), 1634–1640 (2004)
Sonkusare, S., Breakspear, M., Guo, C.: Naturalistic stimuli in neuroscience: critically acclaimed. Trends Cogn. Sci. (Regul. Ed.) 23(8), 699–714 (2019)
Schultz, J., Pilz, K.S.: Natural facial motion enhances cortical responses to faces. Exp. Brain Res. 194(3), 465–475 (2009)
Glasser, M.F., et al.: The minimal preprocessing pipelines for the human connectome project. Neuroimage 80, 105–124 (2013)
Hasson, U., Malach, R., Heeger, D.J.: Reliability of cortical activity during natural stimulation. Trends Cogn. Sci. (Regul. Ed.) 14(1), 40–48 (2010)
Chen, P.-H.C., Chen, J., Yeshurun, Y., Hasson, U., Haxby, J.V., Ramadge, P. J.. A reduced-dimension fMRI shared response model. In: NIPS (2015)
Varoquaux, G., Poldrack, R.A.: Predictive models avoid excessive reductionism in cognitive neuroimaging. Curr. Opin. Neurobiol. 55, 1–6 (2019)
Kell, A.J.E., Yamins, D.L., Shook, E.N., Norman-Haignere, S.V., McDermott, J.H.: A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98(3), 630–644 (2018)
Guclu, U., van Gerven, M.A.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35(27), 10005–10014 (2015)
Yamins, D.L., Hong, H., Cadieu, C.F., Solomon, E.A., Seibert, D., DiCarlo, J.J.: Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl. Acad. Sci. U.S.A. 111(23), 8619–8624 (2014)
Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex 28(12), 4136–4160 (2018)
Dubois, J., Adolphs, R.: Building a science of individual differences from fMRI. Trends Cogn. Sci. (Regul. Ed.) 20(6), 425–443 (2016)
Wen, H., Shi, J., Chen, W., Liu, Z.: Transferring and generalizing deep-learning-based neural encoding models across subjects. NeuroImage 176, 152–163 (2018)
Güçlü, U., van Gerven, M.A.J.: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage 145, 329–336 (2017)
Lin, T.-Y., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 936–944 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2015)
Hershey, S., et al.: CNN architectures for large-scale audio classification. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131–135 (2016)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Abu-El-Haija, S., et al.: Youtube-8m: a large-scale video classification benchmark. ArXiv arxiv:1609.08675 (2016)
Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P., He, K.: Detectron (2018). https://github.com/facebookresearch/detectron
Hershley, S., et al.: Models for audioset: a large scale dataset of audio events (2016). https://github.com/tensorflow/models/tree/master/research/audioset/vggish
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR arxiv:1412.6980 (2014)
Van Essen, D.C., et al.: The human connectome project: a data acquisition perspective. Neuroimage 62(4), 2222–2231 (2012)
Vu, A.T., et al.: Tradeoffs in pushing the spatial resolution of fMRI for the 7T human connectome project. Neuroimage 154, 23–32 (2017)
Tavor, I., Jones, O.P., Mars, R.B., Smith, S.M., Behrens, T.E., Jbabdi, S.: Task-free MRI predicts individual differences in brain activity during task performance. Science 352(6282), 216–220 (2016)
Kanwisher, N., McDermott, J., Chun, M.M.: The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17(11), 4302–4311 (1997)
Nasr, S., et al.: Scene-selective cortical regions in human and nonhuman primates. J. Neurosci. 31(39), 13771–13785 (2011)
Glasser, M.F., et al.: A multi-modal parcellation of human cerebral cortex. Nature 536(7615), 171–178 (2016)
Acknowledgements
This work was supported by NIH grants R01LM012719 (MS), R01AG053949 (MS), R21NS10463401 (AK), R01NS10264601A1 (AK), the NSF NeuroNex grant 1707312 (MS), the NSF CAREER 1748377 grant (MS) and Anna-Maria and Stephen Kellen Foundation Junior Faculty Fellowship (AK).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Khosla, M., Ngo, G.H., Jamison, K., Kuceyeski, A., Sabuncu, M.R. (2020). A Shared Neural Encoding Model for the Prediction of Subject-Specific fMRI Response. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12267. Springer, Cham. https://doi.org/10.1007/978-3-030-59728-3_53
Download citation
DOI: https://doi.org/10.1007/978-3-030-59728-3_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59727-6
Online ISBN: 978-3-030-59728-3
eBook Packages: Computer ScienceComputer Science (R0)
