Abstract

This work presents several contributions to video representation learning and related multimodal tasks, addressing key challenges in datasets, efficient model adaptation using less data, and compositional and fine-grained visual understanding. Despite the rapid growth of online lecture videos in the past several years, video-language research has primarily focused on instructional videos/movies, resulting in a scarcity of specialized datasets for educational lecture videos. To address this, we first introduce AVLectures, a large-scale dataset of STEM lecture videos. It consists of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR for lecture frames, and optionally lecture notes, slides, assignments, and related educational content that can inspire a variety of tasks. Next, we propose a novel unsupervised temporal segmentation task to segment lecture videos into bite-sized topics. We show that multimodal cues can be effectively utilized to learn lecture-aware representations for this task, facilitating a richer analysis of educational content. Next, we address the inefficiency of adapting pre-trained models like CLIP to videos. Existing methods typically rely on large-scale, sparsely annotated video caption datasets, resulting in slow and data-intensive adaptation. We propose SRL-CLIP, a novel approach that leverages the rich, structured semantic information within Semantic Role Labels (SRLs) for highly efficient adaptation. We use VidSitu for adaptation as it provides dense SRL annotations that holistically represent the entire video. SRL-CLIP achieves comparable or superior performance on various video understanding benchmarks (zero-shot retrieval, situation recognition, dense video captioning, and localization) compared to stateof-the-art models that possess 4−8× more parameters and are post-pretrained on up to 4000× times more data. To further explore the models’ understanding of visual content, we introduce three novel benchmarks. First, VELOCITI evaluates the compositional reasoning abilities of video-language models, focusing on their ability to bind semantic concepts through time. Second, we introduce NDLB, a frame work aimed at improving fine-grained image captioning, which uses self-retrieval as a key component along with a new benchmark to check if the model can capture subtle visual distinctions. Finally, we introduce D3, a benchmark specifically designed to evaluate the fine-grained visual discrimination capabilities of MLLMs using self-retrieval, further pushing the boundaries of fine-grained visual understanding. These contributions, which include novel datasets, efficient training recipes, and insightful benchmarks, collectively advance the state of the art in multimodal and video representation learning.

Abstract

Air pollution, whether indoors or outdoors, presents substantial threats to human health. These threats arise from the presence of pollutants in the air, which can have adverse effects on respiratory and cardiovascular systems, among other health impacts. Indoor air pollution often results from activities such as cooking, heating, and the use of certain household products, while outdoor air pollution stems from industrial emissions, vehicle exhaust, and other external sources. This paper tackles these pressing concerns through a comprehensive approach involving two interlinked studies. The first study delves into the evaluation of four low-cost carbon di-oxide (CO2) sensors, specifically designed for monitoring indoor air pollution. These sensors are tested against a high quality reference device in a controlled indoor environment at IIIT-H, India. The findings emphasize the importance of local calibration, even for factory-calibrated sensors, to ensure accurate and reliable measurements. Various machine learning algorithms are employed for calibration, enhancing the sensors’ performance and usability. The second study extends the investigation to outdoor air pollution, focusing on a construction area situated approximately 300 meters from a residential apartment in Hyderabad, India. Utilizing Internet of Things (IoT)-based air pollution nodes, the research assesses pollution patterns across both indoor and outdoor environments in four adjacent blocks near the construction site. Parameters such as particulate matter 2.5 and 10 (PM2.5, PM10), CO2, volatile organic compound (VOC), temperature, and humidity are measured to provide a comprehensive understanding of the air quality. The study also explores the correlation between different pollutants and employs box plots for detailed analysis. The combined insights from these studies offer a holistic perspective on the impact of air pollution on both indoor and outdoor environments. This research contributes valuable data to the ongoing global efforts aimed at addressing air quality issues. It not only enhances our understanding of the performance of low-cost sensors for indoor air monitoring but also sheds light on the far-reaching effects of construction-related air pollution. Ultimately, the findings aim to inform policymakers, construction companies, and the public, facilitating the development of effective strategies to mitigate the health risks associated with air pollution

Abstract

Precise recognition of previously encountered locations is crucial for embodied agents to effectively localize and traverse their environment. This necessitates visual representations that remain distinctive despite substantial changes in camera perspective and environmental conditions. Current visual place recognition systems typically encode entire images for comparison. This conventional approach presents a significant hurdle when attempting to match two images of an identical location taken from disparate viewpoints: the similarity of what overlaps can be dominated by the dissimilarity of what does not overlap. To address this issue, we propose encoding and querying image segments rather than complete images. We leverage open-set image segmentation to decompose an image into ‘meaningful’ entities (encompassing things and stuff). This allows us to construct an novel image representation compris- ing multiple overlapping subgraphs that connect a segment to its neighbouring segments, which we term SuperSegment. Additionally, to efficiently encode these SuperSegments into condensed vector representations, we introduce a novel factorized formulation of feature aggregation. We demonstrate that retrieving these partial representations yields improved recognition accu- racy compared to conventional whole-image retrieval methods. Our segment-based technique, named SegVLAD, establishes new state-of-the-art in place recognition across a wide array of diverse datasets, while remaining compatible with both general-purpose and task-specific image encoders. Lastly, we showcase our method’s versatility in “revisiting anything” by assessing its performance on an object instance retrieval task. This application bridges two distinct research domains: visual place recognition and object-goal navigation, through their shared objective of identifying specific objects within a given location.

Abstract

This thesis explores the transformative potential of artificial intelligence (AI) in healthcare, particularly through enhancing diagnostic accuracy in medical imaging and streamlining data annotation processes. Central to this thesis is the development of an AI system using Unsupervised Domain Adaptation (UDA) to accurately grade retinal development in pediatric patients across different optical coherence tomography (OCT) devices. This innovative approach overcomes device-specific limitations, offering a robust and generalisable classification model without the need for device-specific data, matching the diagnostic accuracy of experienced clinicians. Additionally, the thesis addresses the variability in fundus image interpretation among medical professionals by employing a novel annotation tool. This tool facilitates a variability study among medical graders, underscoring the necessity for AI systems to account for such variability to enhance diagnostic reliability. This thesis also delves into the application of Batch Active Learning (BAL) for optimizing seed grading mechanisms in agriculture. By systematically selecting and labeling the most informative samples, the research demonstrates a significant improvement in the model’s accuracy while reducing the time and cost associated with manual data annotation. The integration of BAL with advanced machine vision techniques highlights its potential to revolutionize agricultural practices, particularly in enhancing the efficiency and accuracy of seed quality classification. This contribution underscores the versatility and scalability of AI-driven approaches in both healthcare and agriculture. Overall, the thesis provides a comprehensive evaluation of AI’s capability to handle domain shift—a significant challenge when training data differs from deployment data. It offers a framework for future models to be device-agnostic, enhancing applicability across different devices and settings. The findings not only contribute to the field of medical diagnostics but also offer insights into the broader application of AI in real-world clinical and agricultural settings.

Abstract

Machine Translation (MT), the automated process of translating text or speech between languages using computational linguistic algorithms, has drastically transformed global communication. By breaking down language barriers, MT has enabled seamless collaboration across different social, cultural, and geographical backgrounds. Although several decades of research have produced state-of-the-art models like Google Translate, these systems often falter when translating domain-specific content, such as specialized academic texts or technical manuals. These models, trained primarily on large, general-purpose datasets, struggle to accurately capture domain-specific vocabulary and linguistic nuances. For example, terms like ”chromosome” in a microbiology textbook are often poorly translated by these systems. In this thesis, we propose DomainTrans, a MT model designed to enhance translation accuracy for domain-specific texts while maintaining performance on general content. The core idea here is to form a bijection between multi-domain translation and multi-lingual translation. By treating different domain datasets as analogous to distinct languages, we leverage multi-lingual translation techniques to develop a more versatile and adaptable MT model. This thesis also studies and experiments with other key peripherals around this topic. First being domain classification, where we identify the domain to which a given text belongs as a precursor to translation. We particularly lay special emphasis on fine grained domain classification. Next we look at the effect of data availability while fine tuning models for machine translation. Towards end end of the thesis we also explore Knowledge Distillation - the process of condensing the learning done by a large or a set of large models onto a much smaller efficient to deploy model while trying to maintain similar accuracy.

Abstract

The neural mechanisms underlying the comprehension of meaningful sounds are yet to be fully understood. While previous research has shown that the auditory cortex can classify auditory stimuli into distinct semantic categories, the specific contributions of the primary (A1) and the secondary auditory cortex (A2) to this process are not well understood. We used songbirds as a model species, and analyzed their neural responses as they listened to their entire vocal repertoire (∼ 10 types of vocalizations). We first demonstrate that the distances between the call types in the neural representation spaces of A1 and A2 are correlated with their respective distances in the acoustic feature space. Then, we show that while the neural activity in both A1 and A2 is equally informative of the acoustic category of the vocalizations, A2 is significantly more informative of the semantic category of those vocalizations. Additionally, we show that the semantic categories are more separated in A2. These findings suggest that as the incoming signal moves downstream within the auditory cortex, its acoustic information is preserved, whereas its semantic information is enhanced.

Abstract

The development of computational methods for the efficient extraction of relevant legal information to aid legal practitioners is an active research area. In this regard, research efforts are being made to develop efficient methods by leveraging different kinds of information in a legal document, such as meta-data, citations, keywords, sentences, paragraphs and so on. Similar to other text documents, legal documents are composed of paragraphs. In this paper, we have analyzed how paragraph-level information could capture the similarity among judgments and improve the performance of precedence retrieval. By considering paragraph-level similarity methods with document level as the baseline, the statistical analysis on the Supreme Court of India judgment dataset shows that the paragraph-level methods with only a few paragraph interactions could capture the similarity among the judgments and exhibit more discriminating power over the baseline document-level method. Moreover, the comparison results on two benchmark datasets for the precedence retrieval on the Indian supreme court judgments task show that the paragraph-level methods exhibit comparable performance with the state-of-the-art methods by utilizing only a few paragraph interactions. Overall, the results show that paragraph-level granularity can better represent the legal document, and there is a scope to exploit them to improve the retrieval systems in the legal domain.

Abstract

Reconfigurable intelligent surfaces (RIS) are emerging as a promising technology for 6G networks due to their ability to shape the radio propagation environment. This ability helps to overcome propagation challenges, such as high path loss, absorption loss, etc., that are particularly useful at millimetre wave (mmWave) bands. Thus, RIS is envisioned to play a critical role in next-generation mmWave communication networks to enable a variety of applications, including communication, localization, and sensing. However, one critical aspect in the design of RIS-aided systems is channel estimation, as it involves estimating two channels, i.e., base station (BS) to RIS and RIS to user terminal (UT), separately based on the compound channel observed by the receiver. In this paper, we address this problem by proposing an algorithm that estimates both channels separately by leveraging the advantages of sparsity of the mmWave channel. Specifically, the proposed algorithm uses the parallel factor (PARAFAC) decomposition of the tensor, which is constructed using the received signal matrices observed under different RIS phase shift configurations. Our numerical analysis shows that the proposed algorithm provides significantly smaller normalized mean square error (NMSE) than the widely used alternating least squares (ALS) algorithm, particularly when the number of RIS phase shift configurations is smaller than the number of RIS elements. In addition, we also derived Cramer Rao Lower Bound (CRLB) for the joint parameter vector of RIS-aided mmWave channels, assuming a uniform planar array (UPA) for RIS antenna elements and uniform linear arrays (ULAs) for BS and UT antennas. Next, we also derived the CRLB for the transform parameter vector to obtain the CRLBs of BS-RIS and RIS-UT mmWave channels. Our numerical analysis demonstrates that the proposed algorithm for estimating both channels achieves their respective CRLBs for a wide range of SNRs.

Abstract

This thesis explores the intricate relationship among color saturation, salience, and visual perception, with a focus on defining color salience and investigating the potential for universal color salience. The research delves into how color saturation—defined as the intensity and purity of color—affects visual perception and contributes to the noticeability of objects in their surroundings. Special attention is given to addressing the limitations of existing theories concerning individual differences in color perception, particularly in scenarios involving color ambiguous stimuli like the internet phenomenon #thedress, where perceptions varied widely. The study draws inspiration from diverse sources, including a saree featuring 50,000 colors, prompting an examination of perceptual limits in color vision and the impact of psychophysical properties on color dominance within ensembles. Through experiments involving tasks to select dominant colors and eye-tracking analyses, the research investigates factors influencing perceived color dominance, such as hue, saturation, and value. The role of individual differences in color perception is explored using clustering techniques to group observers based on their color choices, revealing distinct perceptual strategies and trends within these clusters. Key findings challenge previous assumptions, particularly regarding the salience of hues like blue, which are found to exhibit significant dominance effects contrary to previous research. The study introduces the Relative Perceived Hue Difference (RPHD) metric to quantify color dominance, showing instances where perceived dominance does not align with actual color distribution in stimuli. Eye-tracking data further suggests that sustained attention, reflected in largest fixation points, may be influenced by dominant hues in binary stimuli, highlighting complexities in dominant color perception beyond initial visual capture. This research contributes to advancing our understanding of color perception, salience, and the underlying factors driving visual attention in individuals, offering insights into the intricate dynamics of how we perceive and prioritize colors in our visual environment.

Abstract

Interactive visualization and analysis of the class boundaries is important because it tells us how and why the classes differ. However, the problem of modeling the boundary of classes of arbitrary size, shape and density is challenging. The boundary of a class should not be limited to the points/shape which encloses the points within the class but it should be, the points/shape which encloses the region of influence of a class. The “region of influence” refers to the space around the class where any point lying within the region is likely to be classified to the class based on a nearest neighbor classifier. We have developed interactive boundary visualization toolkit for classified datasets which provides insights about the nature of the dataset and the classifier model used on the dataset. Our algorithm first generates a candidate boundary set for each class based on reverse k-nearest neighbors approach and extends this boundary iteratively through the region of influence of the class. Further, we present these boundary points enclosing the region of influence as a linear approximated shape using triangulation techniques. We show experimental results on 2D and 3D datasets. We also present a distributed map-reduce based algorithm to speed up generation of regions of influence.