Abstract

We embark on a hitherto unreported problem of an autonomous robot (self-driving car) navigating in dynamic scenes in a manner that reduces its localization error and eventual cumulative drift or Absolute Trajectory Error. Modern autonomous vehicles (AVs) often rely on vision, LiDAR, and even radar-based simultaneous localization and mapping (SLAM) frameworks for precise localization and navigation. However, modern SLAM frameworks often lead to unacceptably high levels of drift (i.e., localization error) when AVs observe few visually distinct features or encounter occlusions due to dynamic obstacles. This work argues that minimizing drift must be a key desiderata in AV motion planning, which requires an AV to take active control decisions to move towards feature-rich regions while also minimizing conventional control cost. To do so, we have two distinct formulations of this problem. In the first approach, we learn actions that lead to drift-minimized navigation through a suitable set of reward and penalty functions. We use Proximal Policy Optimization, a class of Deep Reinforcement Learning methods, to learn the actions that result in drift-minimized trajectories. We show, by extensive comparisons on a variety of synthetic, yet photo-realistic scenes made available through the CARLA Simulator, the superior performance of the proposed framework vis-a-vis methods that do not adopt ` such policies. In the second approach, we first introduce a novel data-driven perception module that observes LiDAR point clouds and estimates which features/regions an AV must navigate towards for drift minimization. Then, we introduce an interpretable model predictive controller (MPC) that moves an AV toward such feature-rich regions while avoiding visual occlusions and gracefully trading off drift and control cost. Our experiments on challenging, dynamic scenarios in the state-of-the-art CARLA simulator indicate our method reduces drift up to 76.76% compared to benchmark approaches. We further proposed a novel approach that combines differentiablity with classical odometry algorithms (ICP) to identify drift minimizing features in a point cloud. We also developed a modular and scalable platform for testing and deploying self-driving systems on a real car. This works sets up the groundwork for scalable active navigation in real world

Abstract

The task of entity extraction has been thoroughly explored in the NLP community over the last two decades. There is a myriad of downstream tasks for Natural Language Understanding that utilizes entity extraction - ranging from Information Retrieval, Question Answering, Fact Extraction and Verification, Knowledge Graph Completion, etc. However, the datasets and domains that have been used for evaluation and benchmarking such entity extraction models have mostly been straightforward, structurally simple, semantically non-complex, and well-present across the breadth of training data. Most entity extraction tasks comprise of a significant overlap in entities across train and test sets. This does not mimic the real-world scenario, where rare emergent entities show a larger presence, and the entities themselves are often complex and semantically ambiguous. Our work investigates entity extraction in 2 settings - in the domain of scientific research papers, and in the area of linguistically low-resource structurally complex settings. In the first scenario, we establish an end-to-end pipeline that extracts certain task-specific entities from research documents, that help in a meaningful mapping of the research landscape. We show that the domain of scientific documents is non-trivial for entity extraction tasks because scientific entities follow a long-tail distribution. We incorporate multiple strategies like Distant Supervision, Graph Ranking, and Sequence Labelling to solve this task. Furthermore, we introduce multiple human-annotated gold standard datasets for both modular and complete evaluation of this entire pipeline. In the second setting, we investigate the task of low-resource semantically ambiguous complex entities. We experiment with multiple transformer-based architectures and pre-training strategies to solve this task. Our work significantly outperforms the baseline and outperforms multiple ensemble and gazetteer-based systems. We hope that our work will help undertake further research in this critical area of Natural Language Processing.

Abstract

Video is a complex modality consisting of multiple events, complex action, humans, objects and their interactions densely entangled over time. Understanding videos has been the core and one of the most challenging problem in computer vision and machine learning. What makes it even harder is the lack of structured formulation of the task specially when long videos are considered consisting of multiple events and diverse scenes. Prior works in video understanding have tried to address the problem only in a sparse and a uni-dimensional way, for example action recognition, spatio-temporal grounding, question answering and, free form captioning. However it requires holistic understanding to fully capture all the events, actions, and relations between all the entities, and represent any natural scene with the highest detail in the most faithful way. It requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation Recognition (VidSitu) through semantic role labeling is framed as a task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities. This is one of the most dense video understanding task posing several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation due to the free form captions for representing the roles. In this work, we propose the addition of spatio-temporal grounding as an essential component of the structured prediction task in a weakly supervised setting, without requiring ground truth bounding boxes. Since evaluating free-form captions can be difficult and imprecise this not only improves the current formulation and the evaluation setup, but also improves the interpretability of the models decision, because grounding allows us to visualise where the model is looking while generating a caption. To this end we present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time

Abstract

Ancient paper documents and palm leaf manuscripts from the Indian subcontinent have made a significant contribution to the world literary and culture. These documents often have complex, uneven, and irregular layouts. The process of digitization and deciphering the content from these documents without human intervention pose difficulties in a broad range of areas, including language, script, layout, elements, position, and number of manuscripts per image. Large-scale annotated Indic manuscript image datasets are needed for this kind of research. In order to meet this objective, we present Indiscapes, the first dataset containing multi-regional layout annotations for ancient Indian manuscripts. We also adapt a fully convolutional deep neural network architecture for fully automatic, instance-level spatial layout parsing of manuscript images in order to deal with the challenges such as presence of dense, irregular layout elements, pictures, multiple documents per image and the wide variety of scripts. Eventually, We demonstrate the effectiveness of proposed architecture on images from the Indiscapes dataset. Despite advancements, the segmentation of semantic layout using typical deep network methods is not resistant to the complex deformations that are observed across semantic regions. This problem is particularly evident in the domain of Indian palm-leaf manuscripts, which has limited resources. Therefore, we present Indiscapes2, a new expansive dataset of various Indic manuscripts with semantic layout annotations, to help address the issue. Indiscapes2 is 150% larger than Indiscapes and contains materials from four different historical collections. In addition, we propose a novel deep network called Palmira for reliable, deformation-aware region segmentation in handwritten manuscripts. As a performance metric, we additionally report a boundary-centric measure called Hausdorff distance and its variations. Our tests show that Palmira offers reliable layouts and outperforms both strong baseline methods and ablative versions. We also highlight our results on Arabic, South-East Asian and Hebrew historical manuscripts to showcase the generalization capability of PALMIRA. Even though we have reliable deep-network based approaches for comprehending manuscript layout, these models implicitly assume one or two manuscripts per image during the process, whereas in a real-world scenario there are often cases where multiple manuscripts are typically scanned together into a scanned image to maximise scanner surface area and reduce manual labour. Now, making sure that each individual manuscript within a scanned image can be isolated (segmented) on a per-instance basis became the first essential step in understanding the content of a manuscript. Hence, there is a need for a precursor system which extracts individual manuscripts before downstream processing. The highly curved and deformed boundaries of manuscripts, which frequently cause them to overlap with each other, introduce another complexity when confronting issue. We introduce another new document image dataset named IMMI (Indic Multi Manuscript Images) to address these issues. We also present a method that generates synthetic images to augment sourced non-synthetic images in order to boost the efficiency of the dataset and facilitate deep network training. Adapted versions of current document instance segmentation frameworks are used in our experiments. The results demonstrate the efficacy of the new frameworks for the task. Overall, our contributions enable robust extraction of individual historical manuscript pages. This in turn, could potentially enable better performance on downstream tasks such as region-level instance segmentation, optical character recognition and word-spotting in historical Indic manuscripts at scale.

Abstract

Brain decoding involves the reconstruction of stimuli from brain recordings. These recordings can be obtained by presenting stimuli to a subject in various forms, such as text, image, and speech. Despite extensive research on brain decoding, important questions remain unanswered. Can we develop multi-view decoders capable of decoding concepts from brain recordings of any view, including picture, sentence, or word cloud? Can we build a system that can use brain recordings to automatically generate descriptions of what a subject is viewing using keywords or sentences? How about a system that can automatically extract important keywords from sentences that a subject is reading? Answering these questions requires innovative approaches to brain decoding, as traditional methods have not yet been proven adequate. Previous brain decoding efforts have focused only on single-view analysis and hence cannot help us build such systems. As a first step toward building such systems, inspired by Natural Language Processing literature on multi-lingual and cross-lingual modelling, this thesis proposes novel brain decoding setups: (1) Multi-view Decoding (MVD), (2) Cross-view Decoding (CVD), and (3) Abstract v/s Concrete Decoding. In MVD, the goal is to build an MV decoder that can take brain recordings for any view as input and predict the concept. In CVD, the goal is to train a model which takes brain recordings for one view as input and decodes a semantic vector representation of another view. Specifically, this thesis studies practically useful CVD tasks like image captioning, image tagging, keyword extraction, and sentence formation. In Abstract v/s Concrete Decoding, the goal is to build a decoder trained on concrete concepts and test it on both abstract and concrete concepts and similarly build a decoder trained on abstract concepts and test it in both types of concepts. Extensive experiments lead to MVD models with ∼0.68 average pairwise accuracy across view pairs and CVD models with ∼0.8 average pairwise accuracy across tasks. It was found that the decoder trained on concrete concepts can decode both abstract and concrete objects with great and better accuracy than the model trained on abstract objects. Analysis of the contribution of different brain networks reveals exciting cognitive insights: (1) Models trained on picture or sentence view of stimuli are better MV decoders than a model trained on word cloud view. (2) Our extensive analysis across 9 broad brain regions, 11 language sub-regions, and 16 visual sub-regions of the brain helped us localize, for the first time, the parts of the brain involved in cross-view tasks like image captioning, image tagging, sentence formation, and keyword extraction. (3) The visual brain network is very important for processing Word+Picture stimuli for concrete concepts. Surprisingly, this is not the case for abstract concepts where voxels from the language and DMN brain network are more activated.

Abstract

The task of saliency prediction focuses on understanding and modeling human visual attention (HVA), i.e., where and what people pay attention to given visual stimuli. Audio has ideal properties to aid gaze fixation while viewing a scene. There exists substantial evidence of audio-visual interplay in human perception, and it is agreed that they jointly guide our visual attention. Learning computational models for saliency estimation is an effort to inch machines/robots closer to human cognitive abilities. The task of saliency prediction is helpful in many digital content-based applications like automated editing, perceptual video coding, human-robot interactions,etc. The field has progressed from using hand-crafted features to deep learning-based solutions. Efforts on static image saliency prediction methods are led by convolutional architectures. The ideas were extended to videos by integrating temporal information using 3D convolutions or LSTM’s. Many sophisticated multimodal, multi-stream architectures have been proposed to process multimodal information for saliency prediction. Despite existing works of Audio-Visual Saliency Prediction (AVSP) models claiming to achieve promising results by fusing audio modality over visual-only models, most of these models only consider visual cues and fail to leverage auditory information that is ubiquitous in dynamic scenes. In this thesis, we investigate the relevance of audio cues in conjunction with the visual ones and conduct extensive analysis to analyse the cause of AVSP models being superior by employing well-established audio modules and fusion techniques from diverse correlated audio-visual tasks. Our analysis on ten diverse saliency datasets suggests that none of the methods worked for incorporating audio. Our endeavour suggests that augmenting audio features ends up learning a predictive model agnostic to audio . Furthermore, we bring to light, why AVSP models show a gain in performance over visual-only models, though the audio branch is agnostic at inference. Our experiments clearly indicate that visual modality dominates the learning; the current models largely ignore the audio information. The observation is consistent while using three different audio backbones and four different fusion techniques and contrasts with the previous methods, which claim audio as a significant contributing factor. The performance gains are a byproduct of improved training and the additional audio branch seems to have a regularizing effect. We show that similar gains are achieved while sending random audio during training. Overall our work questions the role of audio in current deep AVSP models and motivates the community to a clear avenue for reconsideration of the complex architectures by demonstrating that simpler alternatives work equally well.

Abstract

Detecting humans in images and videos has emerged as an essential aspect of intelligent video systems that solve pedestrian detection, tracking, crowd counting, etc. It has many real-life applications varying from visual surveillance and sports to autonomous driving. Despite achieving high performance, the single camera-based detection methods are susceptible to occlusions caused by humans, which drastically degrades the performance where crowd density is very high. Therefore multi-camera setup becomes necessary, which incorporates multiple camera views for detections by computing precise 3D locations that can be visualized and transformed to Top View also termed as Bird’s Eye View (BEV) representation and thus permits better occlusion reasoning in crowded scenes. The thesis, therefore, presents a multi-camera approach that globally aggregates the multi-view cues for detection and alleviates the impact of occlusions in a crowded environment. But it was still primarily unknown how satisfactorily the multi-view detectors generalize to unseen data. In different camera setups, this becomes critical because a practical multi-view detector should be usable in scenarios such as i) when the model trained with few camera views is deployed, and one of the cameras fails during testing/inference or when we add more camera views to the existing setup, ii) when we change the camera positions in the same environment and finally iii) when deploying the system on the unseen environment; an ideal multi-camera setup system should be adaptable to such changing conditions. While recent works using deep learning have made significant advances in the field, they have overlooked the generalization aspect, which makes them impractical for real-world deployment. We formalized three critical forms of generalization and outlined the experiments to evaluate them: generalization with i) a varying number of cameras, ii) varying camera positions, and finally, iii) to new scenes. We discover that existing state-of-the-art models show poor generalization by overfitting to a single scene and camera configuration. To address the concerns: (a) we generated a novel Generalized MVD (GMVD) dataset, assimilating diverse scenes with changing daytime, camera configurations, varying number of cameras, and (b) we discuss the properties essential to bring generalization to MVD and developed a barebones model to incorporate them. We performed a series of experiments on the WildTrack, MultiViewX, and the GMVD datasets to motivate the necessity to evaluate the generalization abilities of MVD methods and to demonstrate the efficacy of the developed approach.

Abstract

Space, time, and number are fundamental to human cognition. The mental representations on these entities are crucial for planning and decision-making. In our everyday lives, we are always thinking in terms of quantities — how long would we take to reach the workplace, what would be a shorter route to get to a specific store from where we are, how many cupcakes should we prepare for the people we have invited, how do we throw a stone that will dislodge a shuttlecock stuck in the tree, and so on. Even for simple tasks like grasping, reaching, or catching a ball, subtle calculations involving distance, speed, and time are essential. To successfully execute our actions, we need to synchronize these entities efficiently. For example, to grab an object kept on the table, one needs to integrate information from time, space, and number dimensions to evaluate the obstacles present in that environment and the distance between the object and our body. Further, spatiotemporal integration of information is needed for successful reaching and grasping. Over the last two decades, numerous studies have advanced our knowledge of how humans utilize perceptual information to estimate magnitudes such as space, time, and number. One of the most popular theories of magnitude processing, A Theory of Magnitude (ATOM), suggests that a generalized magnitude system in the brain processes information related to space, time, and numbers. Since these magnitudes are processed by a common magnitude system, they interact with one another. Earlier studies investigating the number-time interaction have provided support to ATOM’s predictions. On the contrary, more recent studies have argued against ATOM and suggested that the cross-dimensional magnitude interactions may emerge from cognitive factors like attention and memory. Such contradicting findings raise a fundamental question as to whether a common magnitude system indeed exists, or such crossdimensional magnitude interactions result from cognitive factors. This is still an unsettled question. In the present thesis, we examine the influence of numerical magnitude on temporal processing in a different experimental setup. More specifically, we investigated whether numerical magnitude affects our temporal experience or simply biases judgment of time. Further, we examined whether large numerical magnitude is always perceived to be longer in time (as predicted by ATOM) or attentional modulation can cause such crossdimensional magnitude interactions. We also tested the generality of the ATOM framework in a cross-modal setting wherein numerical magnitude and temporal information were presented in two different modalities and evaluated ATOM’s prediction in a cross-domain setting. The overall results from the five empirical investigations suggest that the processing of numbers and time may not require to invoke a common magnitude processing system. The cross-dimensional magnitude interactions (in this case, number and time) may emerge from the modulation of attentional mechanisms.

Abstract

Retail stores have become a part of our daily lives and play a vital economic role in society. It has been reported that shelf-space allocation decisions in a retail store significantly impact the retailer’s revenue. So, developing approaches for efficient product placement in available shelf space in retail stores is one of the key research issues. Several research efforts have been made to propose approaches for improving product placement in retail stores based on the knowledge of customer purchase history. Traditionally, several dynamic programming model based approaches were proposed to solve this problem. More recently, data mining approaches, like frequent pattern mining and utility pattern mining, have been employed to improve product placement based on the patterns extracted from customer purchase transactions. In this thesis, we address the issue of product placement in a retail store by considering the diversity of products. Providing diverse options to customers can be useful in increasing the long term sustainability of retail stores. Other recommendation systems have also used increasing diversity to improve user interest. In the context of retail stores, a store that can cater to the needs of a diverse customer base (by providing diverse recommendations) has a higher chance to stay relevant for customers in the long term, thus, facilitating long term sustainability of retail businesses. We have proposed an approach for facilitating the placement of items in a diversified manner in a given retail store based on the concept hierarchy that exists among the items without compromising the revenue. In the proposed approach, a concept hierarchy based approach is employed to determine the diversity value of the itemset quantitatively. The diversity of the given itemset captures the extent of its items belonging to multiple categories. By combining the notion of utility and diversity, we propose a framework to compute diverse net revenue of the itemset. Given a set of transactions, price values of items, and concept hierarchy, we propose a methodology to build the Concept Hierarchy Utility Itemset Index (CHUI), which contains potential itemsets with high revenue and diversity. Next, we propose the approach to perform product placement based on the knowledge of itemsets in the CHUI. We conduct experiments on a real dataset, namely, Instacart retail dataset (containing 49,688 items and 1,31,208 transactions), to demonstrate the proposed approach’s overall effectiveness

Abstract

Rice is a fascinating and complex plant. Consumed by more than half of the world’s population, it is important to have a comprehensive understanding of the organism to advance crop engineering and breeding strategies. Abiotic stresses like drought, high temperature, salinity and flood have affected its growth and productivity. Furthermore, global climate change has added to the severity of these stresses, suggesting the need for varieties with improved stress tolerance for sustainable crop production. Improving stress tolerance requires an in-depth understanding of the biological processes, transcriptional pathways and hormone signaling involved in stress response. With the surge in omics data, it has paved the way for deciphering the biological information underlying complex traits. However, dealing with such large datasets calls for the development of powerful bioinformatics methods for a thorough transcriptome analysis. A popular approach is the construction and analysis of co-expression networks representing transcriptionally coordinated genes that are often part of the same biological process. Using prior knowledge and data integration further enhances the elucidation of gene regulatory relationships in this network. With this objective we have developed NetREx, a Network based Rice Expression Analysis Server, that hosts ranked co-expression networks of Oryza sativa using publicly available mRNAseq data across uniform experimental conditions. It provides a range of interactable data viewers and modules for analysing user queried genes across different stress conditions (drought, flood, cold and osmosis) and hormonal treatments (abscisic and jasmonic acid) and tissues (root and shoot). Subnetworks of user-defined genes can be queried in preconstructed tissue-specific networks, allowing users to view the fold-change, module memberships, gene annotations and analysis of their neighborhood genes and associated pathways. The webserver also allows querying of orthologous from Arabidopsis, wheat, maize, barley, and sorghum. Here we demonstrate that NetREx can be used to identify novel candidate genes and tissue-specific interactions under stress conditions and can aid in the analysis and understanding of complex phenotypes linked to stress response in rice. Available at: https://bioinf.iiit.ac.in/netrex/. In the second part of the thesis, we present a meta-analytic study using co-expressed modules to understand the biological functions associated with different abiotic stresses in the root tissue. The osmotic stress condition is an extremely severe stress condition involving the effects of multiple stresses like drought, salinity and ionic stress and is discussed in detail. The early responsive modules are analyzed and a causal flow of mechanisms and signaling pathways is established.