Academia Sinica Logo

Decoding information from neural activity is important for many neuroscience applications, including basic research into sensorimotor processing, clinical diagnosis, and brain-computer interfaces. Brain data is too complex for traditional statistical techniques to provide much traction in decoding, though, and there is general agreement that deep learning approaches are necessary. However, deep learning requires large-scale datasets, which presents a challenge for neural data, because neural data is highly heterogeneous, with each recording involving different subjects, modalities, species, brain regions, stimuli, behaviour, and health states. How can we train a single model across such diverse data? Here, I will present new techniques that we have developed for training neural decoding models across diverse data. Using novel tokenization approaches, we show that we can build large-scale, transformer-based neural decoding models that can take advantage of diverse data coming from different subjects, stimuli/behaviour, brain regions, cell types, and even species. We show that our model improves as we add more diverse data, indicating that training a neural decoding model at scale is possible, as long as the system is designed to take advantage of the diversity in neural datasets.