Scaling PEFT Towards Brain Foundation Models

TL;DR: Brain encoding models are typically either readouts over frozen backbones or trained from scratch on limited neural data. We show scalable parameter-efficient finetuning (PEFT) is a better middle ground: adapt a large shared backbone with small neural adapters, then scale the adapted models out. Exploiting the backbone's own causal mask cuts feature extraction time by ~350x, which makes finetuning affordable. With ~5% of the trainable parameters and ~100x faster training, LoRA plus a lightweight readout sets a new single-model SoTA, beating even the best published ensemble. Scaling to an ensemble of 100+ adapted models then takes first place on the CNeuroMod/Algonauts 2025 challenge (both ID and OOD).

1. PEFT as a path toward brain foundation models

A brain encoding model predicts neural activity from the stimulus a subject experiences, for example, by predicting someone's brain responses while they watch a movie. Most current encoding models are built for a particular task, subjects, or set of brain regions. A brain foundation model would aim to capture more general structures that generalize across tasks, stimuli, subjects, and brain regions while remaining easy to specialize to a particular setting.

Existing approaches tend to follow one of two paths. The first starts with a model pretrained on some behavioral tasks (object recognition for vision, next-token prediction for language, etc.), freezes it, and trains a readout from its hidden states to predict neural responses. This imports a rich visual and semantic prior, but the supervision of neural signals is confined to the readout. The second approach trains a model from scratch on neural data. This allows the representation to specialize fully for predicting neural responses, but neural datasets are generally too small and expensive to support the scaling that makes modern foundation models powerful.

We argue that parameter-efficient fine-tuning (PEFT) offers a strong middle ground. It suggests a three-part scaling strategy: 1) Scale up: Use increasingly capable pretrained models to provide broad visual, semantic, and temporal representations that are hard to learn from limited neural data alone. 2) Scale down: Adapt a strong backbone with a small number of parameters, matching the scale of the update to the limited amount of neural data rather than finetuning the entire model. 3) Scale out: Attach different local updates to a strong backbone for different tasks, subjects, or even brain regions, allowing specialization without duplicating the shared model.

In the rest of this post, we show how we use this approach to take first place on the CNeuroMod/Algonauts 2025 challenge, while staying methodologically simple and computationally efficient.

2. Making causal backbone training affordable

The CNeuroMod/Algonauts 2025 benchmark [1] uses fMRI recordings from subjects watching all 7 seasons of Friends, with synchronized video, audio, and dialogue as stimuli. Predicting whole-brain responses to this kind of naturalistic, multimodal input is exactly the setting where a large pretrained backbone should help. We thus use Qwen3-Omni-30B here, the same backbone as the best published method [2] (MIRAGE), so comparisons isolate how the backbone is used rather than which one it is.

MIRAGE [2] extracts causal features from a sliding window over overlapping inputs, reprocessing the preceding stimulus at each timestep for roughly 3,500 forward passes per clip. Most of that is redundant: the backbone is already causally masked, so a single pass over a training window gives causal representations at every timestep. We also prepend a short run-up of stimulus so that the first outputs also have enough context.

This cuts feature extraction from roughly 700 GPU-hours to about two hours on A800, a ~350x reduction, across the full training set. This makes backbone computation cheap enough to finetune, so the pretrained model no longer has to stay a fixed feature extractor. We show below what happens when supervision from neural data is allowed to adapt the backbone.

3. A small adapter beats a big readout

With backpropagation through the backbone now affordable, instead of an expressive readout on frozen features, we let LoRA adapters in the backbone carry the neural supervision, combined with simple lightweight readouts.

The published MIRAGE baseline uses a 1,123M-parameter attention readout over a frozen Qwen3-Omni backbone. Throughout, the metric is the Pearson correlation between model predictions and parcel activities, averaged over subjects and parcels; the baseline reaches 0.310 in-distribution (ID) and 0.217 out-of-distribution (OOD). We use LoRA + linear readout with just 61M trainable parameters total, about 5.4% of the trainable parameters of the baseline, and reach 0.322 / 0.222, already ahead of the previous best single model on both splits.

Once the backbone can adapt, a small readout matches or beats a much larger frozen-feature readout. End-to-end, the SoTA baseline costs roughly 700 GPU-hours; ours costs about six, roughly 100x faster, while being ~20x more parameter-efficient and more accurate.

Keeping the same parameter budget, we swap the linear readout for a cross-attention readout [4]. This configuration reaches 0.327 / 0.228, surpassing not just the single-model baseline but MIRAGE's full 15-model ensemble (0.323 / 0.227), making it the best-performing published method on the benchmark on both splits, single model or ensemble [2,3].

Finally, scaling this recipe out to an ensemble of 100+ LoRA-adapted models pushes performance to 0.341 / 0.245, taking first place on the entire leaderboard, ahead of unpublished entries as well. A selected 15-model ensemble retains most of the performance of the full ensemble while being significantly more efficient. The next section shows how we scale adapters out across many specializations and combine them into a pool.

4. Scaling out: a kernel for training a population of adapted models

A 61M-parameter adapter on a 30B-parameter model changes the natural unit of specialization. Training a separate 30B model per task, subject, or brain region is unrealistic; training a small adapter for each is not. PEFT separates a large shared prior from compact local updates, making specialization cheap enough to scale out. This fits brain encoding well: subjects share representational structure without being identical, regions respond to different stimulus properties, and datasets vary in distribution, protocol, and quality. A future brain foundation model needs to capture what's shared while still modeling these differences. We think small adapters over large powerful backbones are one practical mechanism for that.

Training adapters one at a time makes the training time scale linearly with the number of adapters. But adapters are tiny and structurally identical, so the routed low-rank update for a whole pool can be expressed as a single batched gather-matmul. We develop a Triton kernel for multi-lora training so one pass over a batch updates every adapter in the pool at once.

The speedup from our multi-lora training kernel grows with pool size: gains are modest at first, but with 100+ adapters, the batched kernel already runs more than 10x faster than training them one at a time, and the advantage keeps climbing as the pool grows further.

PEFT is useful not just because one adapter is cheap, but because when it's cheap, it enables many specialized instances to coexist around one shared backbone.

With the multi-LoRA kernel accelerating training, we created an ensemble of 174 models that vary along different axes and combined their predictions.

To measure the value of ensemble pool size, we sample random subsets at every size. The result is a clear scaling effect: performance versus pool size traces a straight line on a log-log plot (R^2=0.998).

Beyond model count, diversity matters too. Different adapted models in an ensemble must capture meaningfully different solutions to be effective. Ensemble performance decreases as the mean residual correlation among models increases (R=-0.79). More diversity, better ensemble performance.

Our resulting ensemble ranks first on the CNeuroMod/Algonauts 2025 challenge (both ID and OOD) at the time of writing this post, reaching 0.341 / 0.245. A selected 15-model ensemble reaches 0.338 / 0.243, offering strong performance while remaining computationally efficient.

5. Takeaways

Most brain encoding works freeze the backbone or train smaller models from scratch on neural data. Adapting a large backbone with a small number of parameters is another option, and one that is underexplored. It suggests three axes to scale along:

Scale up the shared backbone. Large pretrained models bring increasingly rich representations of the world. Rather than retraining them from neural data or treating them as fixed feature banks, we can use small updates to specialize them. Our experiments use a 30B multimodal backbone and show that 61M-parameter LoRA plus linear readouts are sufficient to improve over a billion-parameter readout on frozen features.

Scale down the adapter update. A compact adapter can modify the representation while a simple readout maps it to neural activity. This is a better match to the limited amount of neural supervision available.

Scale out the adapted models. Small local updates can be replicated across subjects, regions, datasets, or modeling choices while sharing a strong backbone. Our multi-adapter kernel makes large pools computationally practical, and our ensemble results show that multiple adapted instances can produce substantial gains.

Our experiments do not yet constitute a brain foundation model, but they suggest a promising direction: a large shared model that captures rich priors, paired with many small neural adapters that specialize it where data are scarce. Rather than training every brain model from scratch or freezing an increasingly capable backbone, we can scale the prior up and the adaptations out. We think this is a powerful path toward brain foundation models.

Reference

[1] The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies. (Gifford et al., 2025)

[2] MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding. (Gokce et al., 2026)

[3] TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction. (d'Ascoli et al., 2026)

[4] Transformer brain encoders explain human high-level visual responses. (Adeli et al., 2026)

Authors

Mind Lab

Core Contributors

Josh Ying, Guanglong Sun, Hossein Adeli, Kieran Liu, Nolan Ho, Andrew Chen, Pony Ma

Team

Asher Cai, Song Cao, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Jun Gao, Pyke Han, Nolan Ho, Mutian Hong, Charles Huang, Fancy Kong, Andrew Lei, Lucian Li, Ray Li, Fan Lin, Kieran Liu, Logan Liu, Neo Liu, Xiang Liu, Yuxin Lu, Pony Ma, Cole Qiao, Vince Qu, Bo Wu, Chengdong Xu, Rio Yang, Regis Ye, Yihang Zeng, Di Zhang, Jiahao Zheng, Adrian Zhou, Yuhua Zhou, Murphy Zhuang and Mindverse Team

Names are listed alphabetically within team.

Citation

@misc{ying2026peft_brain,
  author       = {Ying, Josh and Sun, Guanglong and Adeli, Hossein and Liu, Kieran and Chen, Andrew and Ma, Pony and {Mind Lab}},
  title        = {Scaling {PEFT} Towards Brain Foundation Models},
  year         = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note         = {https://macaron.im/mindlab/research/scaling-peft-towards-brain-foundation-models},
}