Beyond Brain Decoding: Visual-Semantic Reconstructions to Mental Creation Extension Based on fMRI

Decoding visual information from fMRI signals is an important pathway to understand how the brain represents the world, and is a cutting-edge field of artificial general intelligence. Decoding fMRI should not be limited to reconstructing visual stimuli, but also further transforming them into descriptions, creating actions, and even generating unseen content. We purposefully propose a novel and efficient brain multimodal architecture, NeuroCreat, which combines the powerful visual and textual abilities of LLM to capture fine-grained semantic information from fMRI, transformed it into an embodied implementation of different neural representations. Specifically, we innovatively designed a brain expert adaption (BEA) module, effectively capturing commonalities and individual differences among subjects through the collaborative learning of shared/routed experts. Inspired by human visual working memory, we extracted `creation'' information from higher visual cortex for idea generation. We further constructed a prompt variant alignment module, seamlessly integrates fMRI-visual-semantic-creation into LLM to achieve flexible incorporation of different semantics in the decoding of neural representations. Experiments on different fMRI datasets show that NeuroCreat achieves SOTA performance on multiple brain decoding tasks. More importantly, we have innovatively achieved few-shot brain video creation, which opens up a new direction for demonstrating the brain'simaginative’ ability.