Abstract

Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models—through increased size, instruction-tuning, and multimodality—has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate the brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions (like image captioning, visual question answering, etc.) show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results indicate that enhancing MLLMs' ability to capture more task-specific information could allow for better differentiation between various types of instructions, and hence improve their precision in predicting brain responses.

Abstract

No Results Found

Abstract

The coronavirus disease (COVID-19) caused over 170 million illnesses and over 3 million deaths worldwide. Researchers from all over the world have been using a variety of machine-learning techniques to classify the DNA sequences of the SARS-CoV-2 virus. To classify SARS-CoV-2 (Covid-19) whole genome sequences with other viruses including Dengue, Ebola, Influenza, and also Coronaviruses that may infect humans and are members of the family of Coronaviridae like SARS-CoV, MERS-CoV, we have proposed a multi-dimensional frequency encoding scheme. In the proposed method each genome is converted into four frequency-encoded sequences for each nucleotide. Then each encoded sequence is partitioned into quartiles and fed to the CNN. The findings of the binary classification of SARS-CoV-2 with other viruses and multi-class classification of six different viruses using five well-known machine learning algorithms and the proposed approach using CNN were reported in this study. Furthermore, we applied machine learning methods for the multi-class classification of eight different SARS-CoV-2 variants. The proposed approach performed well among all machine-learning techniques for classifying genome sequences of viruses and SARS-CoV-2 variants with accuracy of 99% and 98% respectively. In addition, we have also compared the performance of our proposed approach with the existing methods from the literature.

Abstract

Despite participants engaging in single modality stimuli, such as watching images or silent videos, recent work has demonstrated that multi-modal Transformer models can predict visual brain activity impressively well, even with incongruent modality representations. This raises the question of how accurately these multi-modal models can predict brain activity when participants are engaged in multi-modal stimuli. As these models grow increasingly popular, their use in studying neural activity provides insights into how our brains respond to such multi-modal naturalistic stimuli, i.e., where it separates and integrates information from different sensory modalities. We investigate this question by using multiple unimodal and two types of multi-modal models—cross-modal and jointly pretrained—to determine which type of models is more relevant to fMRI brain activity when participants were engaged in watching movies (videos with audio). We observe that both types of multi-modal models show improved alignment in several language and visual regions. This study also helps in identifying which brain regions process unimodal versus multi-modal information. We further investigate the impact of removal of unimodal features from multi-modal representations and find that there is additional information beyond the unimodal embeddings that is processed in the visual and language regions. Based on this investigation, we find that while for cross-modal models, their brain alignment is partially attributed to the video modality; for jointly pretrained models, it is partially attributed to both the video and audio modalities.

Abstract

The intricate link between brain textbf{Functional Connectivity (FC)} and textbf{Structural Connectivity (SC)} is explored through models performing diffusion on SC to derive FC, using varied methodologies from single to multiple textbf{Graph Diffusion Kernels}. However, existing studies have not correlated diffusion scales with specific brain Regions of Interest (RoIs), limiting the applicability of graph diffusion. We propose a novel approach using textbf{Graph Diffusion Wavelets} to learn the appropriate diffusion scale for each RoI to accurately estimate the SC-FC mapping. Using the open Human Connectome Project (HCP) dataset, we achieve an average textbf{Pearson's correlation} value of 0.833, surpassing the state-of-the-art methods for prediction of FC. It is important to note that the proposed architecture is entirely linear, computationally efficient, and notably demonstrates the textbf{power-law} distribution of diffusion scales. Our results show that the bilateral frontal pole, by virtue of it having large diffusion scale, forms a large community structure. The finding is in line with the current literature on the role of the frontal pole in resting-state networks. Overall, the results underscore the potential of graph diffusion wavelet framework for understanding how the brain structure leads to functional connectivity.

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental condition characterized by varied social cognitive challanges and repetitive behaviour patterns. Identifying reliable brain imaging-based biomarkers for ASD has been a persistent challenge due to the spectrum's diverse symptomatology. Existing baselines in the field have made significant strides in this direction, yet there remains room for improvement in both performance and interpretability. We propose HyperGALE, which builds on the hypergraph by incorporating learned hyper graphs and gated attention mechanisms. This approach has led to substantial improvements in the model's ability to interpret complex brain graph data, offering deeper insights into ASD biomarker characterization. Evaluated on the exptensive ABIDE II dataset, HyperGALE not only improves interpretability but also demonstrates statistically significant enchancements in key performance metrics compared to both previous baselines and the foundation hypergraph model. The advancement HyperGALE brings to ASD research highlights the potential of sophisticated graph-based techniques in neurodevelopmental studies. The source code and implementation instructions are available at Github.

Abstract

Over the last decade, there has been growing interest in learning the mapping from structural connectivity (SC) to functional connectivity (FC) of the brain. The spontaneous fuctuations of the brain activity during the restingstate as captured by functional MRI (rsfMRI) contain rich non stationary dynamics over a relatively fxed structural connectome. Among the modeling approaches, graph difusionbased methods with single and multiple difusion kernels approximating static or dynamic functional connectivity have shown promise in predicting the FC given the SC. However, these methods are computationally expensive, not scalable, and fail to capture the complex dynamics underlying the whole process. Recently, deep learning methods such as GraphHeat networks and graph difusion have been shown to handle complex relational structures while preserving global information. In this paper, we propose a novel attentionbased fusion of multiple GraphHeat networks (AGHN) for mapping SCFC. AGHN enables us to model multiple heat kernel difusion over the brain graph for approximating the complex Reaction Difusion phenomenon. We argue that the proposed deep learning method overcomes the scalability and computational inefciency issues but can still learn the SCFC mapping successfully. Training and testing were done using the rsfMRI data of 1058 participants from the human connectome project (HCP), and the results establish the viability of the proposed model. On HCP data, we achieve a high Pearson correlation of 0.788 (DesikanKilliany atlas with 87 regions) and 0.773 (AAL atlas with 86 regions). Furthermore, experiments demonstrate that AGHN outperforms the existing methods in learning the complex nature of the structurefunction relation of the human brain.

Abstract

Automated Sleep stage classification using raw single channel EEG is a critical tool for sleep quality assessment and disorder diagnosis. However, modelling the complexity and variability inherent in this signal is a challenging task, limiting their practicality and effectiveness in clinical settings. To mitigate these challenges, this study presents an end-to-end deep learning (DL) model which integrates squeeze and excitation blocks within the residual network to extract features and stacked Bi-LSTM to understand complex temporal dependencies. A distinctive aspect of this study is the adaptation of GradCam for sleep staging, marking the first instance of an explainable DL model in this domain with alignment of its decision-making with sleep expert's insights. We evaluated our model on the publically available datasets (SleepEDF-20, SleepEDF-78, and SHHS), achieving Macro-F1 scores of 82.5, 78.9, and 81.9, respectively. Additionally, a novel training efficiency enhancement strategy was implemented by increasing stride size, leading to 8x faster training times with minimal impact on performance. Comparative analyses underscore our model outperforms all existing baselines, indicating its potential for clinical usage.

Abstract

Brain stroke has become a significant burden on global health and thus we need remedies and prevention strategies to overcome this challenge. For this, the immediate identification of stroke and risk stratification is the primary task for clinicians. To aid expert clinicians, automated segmentation models are crucial. In this work, we consider the publicly available dataset ATLAS v2.0 to benchmark various end-to-end supervised U-Net style models. Specifically, we have benchmarked models on both 2D and 3D brain images and evaluated them using standard metrics. We have achieved the highest Dice score of 0.583 on the 2D transformer-based model and 0.504 on the 3D residual U-Net respectively. We have conducted the Wilcoxon test for 3D models to correlate the relationship between predicted and actual stroke volume.