Abstract

The growing scope of virtual reality (VR) applications demonstrates the potential to address burgeoning problems in today’s and future world. The power of VR comes through the sense of embodiment, allowing people to experience the virtual environment from a first-person perspective [71]. It empowers the experiencer to perceive a real-world inside the simulated world, breaking the limits of the physical dimension, further enabling the capturing of natural psychological, behavioural and physiological responses as one would manifest in real-life situations. These measures act as a magnifying glass to understand the underlying perception and cognition processes of every interaction made with the environment. This thesis studies the affective-state (psychological), psycho-motor (behavioural) and skin conductance (physiological) response to the affective stimuli projected through VR. The stimuli consist of pleasant and unpleasant videos varied across valence and arousal dimensions, selectively chosen from the VR Stanford affective database [76]. In this research, emotional affective state was studied by asking participants to rate their feeling after exploring each video by using the self-assessment manikin (SAM) scale. We chose Skin-Conductance Response (SCR) to measure physiological response corresponding to the affective state while exploring the affective videos and used head-tracking data to measure the psycho-motor scanning behaviour in correspondence with the affective state. We conducted two studies, in which we used HMD to display the 360° affective videos to the participants. Participants were asked to explore the video, as they would do in real world, by making their head movement in all the three directions, i.e. yaw (left-right headmovement), pitch (up-down head-movement) and roll (shoulder-tilt head-movement). The first study focuses on the physiological and psychological relationship with VR emotional stimuli with an objective to find differences in SCR data in response to varying emotional stimuli. This was a pilot study carried out with 10 volunteers, and all were shown 10 videos (5 pleasant and 5 unpleasant). We observed no significant difference in physiological activity while exploring pleasant as compared to unpleasant emotional videos. The second study focuses on the behavioural and psychological relationship with VR emotional stimuli to analyse the differences in head-orientation in all the three directions while exploring pleasant and unpleasant videos. Here, 30 volunteers participated, and all were shown 4 videos (2 pleasant and 2 unpleasant), pseudo-randomly selected from 10 videos (5 pleasant and 5 unpleasant). We observed a significant positive correlation between the standard deviation of head-movement and valence and a significant positive correlation between scanning speed and valence. Participants showed natural exploring behaviour in the region-wise analysis, where they primarily explored the front view and rarely explored the rear-view. In the second study, we also assessed the individual predispositional emotional state by using Beck Depression Inventory-II (BDI) which groups the participants into minimal, mild, moderate and severe categories. The assessment was performed, keeping in view the increasing worldwide prevalence of depression and the current state of the art, demanding the need for the alternative assessment measures and diagnostic solutions where one-to-one supervision is not required. This is because the doctor-patient ratio is too low [118, 90] leading to huge gap, and the traditional methods involve subjective measures and interviews, making the whole process gradual and inaccessible. VR has the potential to bridge this gap by providing objective assessment measures and remotely accessible solutions. For this, the current research also compares the affective state and scanning behaviour across different depression groups. The relation between virtual affective experience and 360° exploration behaviour is discussed in light of the BDI categories.

Abstract

Upon nitrogen starvation, Schizosaccharomyces pombe exits the mitotic cell cycle and becomes irreversibly committed to the completion of the meiosis program. In meiosis, DNA replication (S-phase) is followed by two successive rounds of cell divisions (Meiosis I and Meiosis II) without intermediate interphase. Meiotic cell divisions are coordinated with sporulation events to produce haploid spores. In the last few decades, experiments on fission yeast have revealed different molecular players involved in two meiotic cell divisions, MI and MII. How the MI entry, MI to MII transition, and MII exit occur because of the dynamics of the regulatory network is not well understood. In this thesis, a comprehensive mathematical model of meiotic regulatory network was developed. This network was assembled from several experimental observations in the literature for meiotic cell divisions and exit. The core of the regulatory network is the regulation of cyclin-dependent kinase (Cdk) 1 and anaphase-promoting complex or cyclosome (APC/C) by meiosis-specific factors. The network was translated into a set of ordinary differential equations to simulate the dynamics of meiotic progression. We also performed one and two-parameter bifurcations to study the role of different feedback loops in meiosis. The model accounts for the phenotypes of several experimental data including single and multiple mutations and demonstrate the control strategy involving multiple feedback loops to yield two successive division cycles. The G2 to MI transition (MI entry) involves activation of meiosis-specific transcription factor Mei4, which controls the bistable activation of Cdk1. The logic of meiotic progression involves a transition from an oscillatory regime during MI-to-MII transition to a low Cdk1 state at the MII exit with an increase in meiosis-specific APC/C coactivators. The differential regulation of APC/C coactivators and its inhibitors is crucial for the dynamics of both MI-to-MII transition and MII exit. Further, we developed an integrated model of entire meiotic cell cycle by connecting the regulation of G1 to S phase transition and irreversible commitment with the meiotic progression. S-phase entry and commitment depends on the feedback regulation of an RNA-binding protein Mei2, which regulates inhibitor of meiotic mRNAs including Mei4. This integrated model captures the dynamics of wild type and 60 different mutations. The model serves as a key tool for experimentalist to perform in silico mutations and test the hypothesis. Overall, this thesis work provides systems-level interpretation of experimental data, deduces the molecular logic of meiosis and provides insights that help in further experiments and modeling.

Abstract

Motion planning deals with the computation of a sequence of robot states and control actions to navigate between a start and a goal robot state. As part of this planning, a robot must detect environmental objects, monitor their changes, reason about them, and finally compute kinodynamically feasible realtime trajectories for task-specific navigation. The work in this thesis is motivated by real-time cooperative motion planning applications. Such applications require multiple robots to work together and complete a common task through communication and negotiation. Furthermore, robots must leverage cooperation to improve sensing, thereby, allowing them to develop a shared understanding of the environment and plan their motion effectively. In our work, we primarily focus on the cooperative task of target following and its applications. The algorithms developed aim to be applicable towards aerial outdoor motion capture, search and rescue (SAR) applications for victim and rescuer tracking, and human target guided cooperative load transport applications. For the tasks mentioned above, computing real-time coordinated plans is challenging because of the associated high-dimensional multi-agent configuration space, non-linear inter-agent dependencies like collision-avoidance, limited communication bandwidth and range, non-convex and discontinuous perceptual constraints of common target perception and tracking, and, kinodynamically constrained common payload manipulation and transportation. The key contributions of this thesis that overcome the challenges mentioned above are as follows. We first develop a decentralized robot motion planning pipeline for target following, which mitigates the large planning configuration space problem. Additionally, we embed non-convex collision avoidance and inter-robot constraints into locally convex approximations, promoting the use of convex optimization methodologies for real-time planning. Model-predictive control facilitates high-frequency replanning, which handles imperfect system models, noisy observations, and communication limitations. Model-based optimal control algorithms and model-free sequential decision-making algorithms are leveraged to plan motion in perceptually driven tasks. Finally, a hierarchical decomposition of the complex kinodynamically constrained planning problem is proposed to compute feasible and safe trajectories in real-time. In the first part of the thesis, we introduce a novel decentralized multi-agent algorithm for real-time target following and collision avoidance using model-predictive control (MPC). The algorithm is the first of its kind convex-optimization-MPC approach and is a comprehensive solution to multi-agent collision avoidance for cooperative target following scenarios. The method is validated extensively in synthetic environments. In the second part, we empirically develop stochastic target perception models and embed it into the previously designed decentralized MPC to compute formation motion plans for cooperative target following. The controller actively computes robot control actions that minimize the fused uncertainty in target perception, thereby, binding perception and planning into a single MPC. Subsequently, we leverage model-free deep reinforcement learning algorithms to determine robot motion plans, which are optimized for target pose and shape estimation. Furthermore, we experimentally compare the performance of model-free and model-based approaches for the target following task. Both the approaches are evaluated and validated extensively in simulation and using real-robot experiments for the task of outdoor motion capture. In the third part, we analyze the challenges in kinematic planning for multirobot spatial payload manipulation and target following, and introduce a hierarchical motion planning solution that builds on the previously designed decentralized MPC. The performance is corroborated in challenging simulation environments. In the last part, we present a hierarchical energy-conscious control allocation algorithm which is constrained kinodynamically by multiple agents and electrically constrained by the actuators and battery limitations. The approach is the first of its kind control allocation algorithm, which models a multi-robot system as an over-actuated multi-agent system to plan its motion. The controller is verified in simulation, and a preliminary experimental testbed design is showcased for physical validation. We conclude this thesis by summarizing our novel model-predictive control formulation for the perceptually driven task of outdoor motion capture. Subsequently, we outline the key observations from the model-free approach for designing controllers for the same perceptually driven task. We then discuss the extension of the model-based target following controller towards planning motion of a high degree of freedom cooperative payload manipulation systems. Finally, we detai

Abstract

Majority of speech and natural language processing applications rely on word and document representations (or embeddings). The document embeddings encode semantic information which makes them suitable for tasks such as topic identification (document classification), topic discovery (document clustering), language model adaptation, and query-based document retrieval. These embeddings are usually learned from widely available un-labelled data; hence generative or probabilistic topic models which aim to capture the distribution of data are suitable. Although there exist several probabilistic and neural network-based topic models to learn these embeddings, they often ignore to capture the uncertainty in the estimated embeddings. Thus, any error in the estimation of these embeddings affects the performance in downstream tasks. The uncertainty in the embeddings is usually due to shorter, ambiguous or noisy sentences/documents. This thesis presents model(s) for learning to represent document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in their covariances. Further, these learned uncertainties in embeddings are exploited by the proposed generative Gaussian linear classifier for topic identification. This thesis proposes to use subspace multinomial model (SMM), a simple log-linear model for learning document embeddings. Experiments on 20Newsgroups text corpus show that the embeddings extracted from SMM are superior when compared to popular topic models such as latent Dirichlet allocation, sparse topical coding in topic identification and document clustering tasks. Using the variational Bayes framework on SMM, the model is able to infer the uncertainty in document embeddings, represented by (posterior) Gaussian distributions. Additionally, the common problem of intractability which appears while performing variational inference in mixed-logit models is addressed using Monte Carlo sampling via the re-parametrization trick. The resulting Bayesian SMM achieves state-of-the-art perplexity results on 20Newsgroups text and Fisher speech corpora. The proposed generative classifier exploits the learned uncertainty in the document embeddings; and achieves state-of-the-art classification results on the aforementioned corpora as compared to other unsupervised topic and document models. Furthermore, this thesis presents a multilingual extension of the Bayesian SMM for zero-shot cross-lingual topic identification. The proposed model achieves superior classification results when compared to the systems based on multilingual word embeddings and neural machine translation inspired sequence-to-sequence bidirectional long-short term memory models.

Abstract

With the increase in number and size of the problems in the computer vision domain, a lot of research has focused on speeding up the algorithms in these domains. Most of these algorithms require the solution of a linear system at some point. This thesis explores the application of numerical linear algebra(NLA) techniques for increasing the efficiently of algorithms related to the bundle adjustment[62] problem. Since, the dimensions of the linear systems associated with these problems is large, iterative methods such as conjugate gradients(CG) and generalized minimal residual(GMRES), are used. However, the efficiency of these methods depends on the condition number(defined as the ratio of the largest eigenvalue to the smallest eigenvalue) of the coefficient matrix. In general, the matrices associated with these problems are very badly conditioned. This motivates the use of preconditioning matrices, or preconditioners, which are close to the original matrix in some sense, and simplifies the solution of the linear systems. In this thesis, some novel preconditioners are developed and their effects on the efficiency of the algorithms are presented.Two NLA techniques, namely the domain decomposition method and the multigrid method, are used to develop preconditioners for the bundle adjustment problem. This is the first time these techniques have been applied to this problem. Another method is explored in this thesis that is based on deflation of unfavourable eigenvectors. These preconditioners are integrated with popular solvers like SSBA1 and Ceres[2] and their improvements are demonstrated experimentally

Abstract

The internet has caused a revolutionary information explosion and we have more resources available than ever before. The size of just the English language Wikipedia articles is about 159.69 GB. As the resources we consume keep getting bigger and more complex, we need systems to query these datasets. Initially, to query large databases, simple Information Retrieval techniques like fetching keywords and combining them with Boolean logic results were used. But now, we have search engines that can take queries in the form of multiple keywords and even questions. The retrieval side of these search engines also started with simple Document Retrieval methods like document frequencies and inverse document frequencies and has evolved to complex ranking methods based on neural networks that can fetch entire questions. Question Answering is a major field in NLP. It combines query formulation and document retrieval and in some cases, summarization of the retrieved documents. We now have complete end-to-end based systems that can fetch a suitable answer from a corpus. These systems can be trained on large datasets - both closed domain or open domain. However, these systems rely heavily on curated datasets which require a lot of manual labour and resources. For Indian languages where reliable and substantial datasets are scarce, neural networks that rely purely on data cannot be used. Hence, providing more data and introducing data-independent techniques becomes important. We shall explore both these aspects. First, we present a Reading Comprehension Task in Hindi - HindiRC. The dataset has been divided into different grades based on reading proficiency and we perform the baseline experiment on each grade separately which shows the increasing level of difficulty. Using various linguistic cues and metrics we further prove that the grades are reflective of linguistic complexity. In addition to addressing the data scarcity, we propose a Hindi Question Generation methodology. The rule-based method is based on Computational Paninian Grammar Framework relations. No additional resources are required and it can be used to increase the number of questions for Hindi datasets. We also prove that the generation method tends to overgenerate questions; further inflating the number. Along with the dataset and question generation method, we aim to provide more resources for Question Answering in Hindi.

Abstract

Real-world images often suffer from variation in weather conditions such as rain, fog, snow, and temperature. These variations in the atmosphere adversely affect the performance of computer vision models in real-world scenarios. This problem can be bypassed by collecting and annotating images for each weather. However, collecting and annotating images in such conditions is an extremely tedious task, which is time-consuming as well as expensive. So, in this work, we address the forementioned problems. Among all the weather conditions, we focus on the distortions in the image caused by high temperature, also known as atmospheric turbulence. These distortions introduce geometrical deformation around the boundaries of an object in an image which causes a vision algorithm to perform poorly and pose a major concern. Hence, in this thesis, we address the problem of artificially generating atmospheric turbulence and restoring the images from it. In the first part of our work, we attempt to model atmospheric turbulence. Since such models are critical to extending computer vision solutions developed in the laboratory to real-world use cases. And, simulating atmospheric turbulence by using statistical models or by computer graphics is often computationally expensive. To overcome this problem, we train a generative adversarial network(GAN) which outputs an atmospheric turbulent image by utilizing less computational resources than traditional methods. We propose a novel loss function to efficiently learn the atmospheric turbulence at the finer level. Experiments show that by using the proposed loss function, our network outperforms the existing state-of-the-art image to image translation network in turbulent image generation. In the second part of the thesis, we address the ill-posed problem of restoring images degraded due to atmospheric turbulence. We propose a deep adversarial network to recover the images which are distorted due to atmospheric turbulence and show the applicability of restored images in several tasks. Unlike previous methods, our approach neither uses any prior knowledge about atmospheric turbulence conditions at inference time nor requires the fusion of multiple images to get a single restored image. To train our models, we synthesized turbulent images by following a series of efficient 2D operations. Thereafter, using our trained models we run inference on real and synthesized turbulent images. Our final restoration models DT-GAN+ and DTD-GAN+ qualitatively and quantitatively outperforms the general state-of-the-art image-to-image translation models. The improved performance of our model is due to the use of optimized residual structures along with channel attention and sub-pixel mechanism which exploits the information between the channels and removes atmospheric turbulence at the finer level. We also perform extensive experiments on restored images by utilizing them for downstream tasks such as classification, pose estimation, semantic keypoint estimation, and depth estimation. In the third part of our work, we study the problem of the semantic segmentation model in adapting to hot climate cities. This issue can be circumvented by collecting and annotating images in such weather conditions and training segmentation models on those images. But, the task of semantically annotating images for every environment is painstaking and expensive. Hence, we propose a framework that improves the performance of semantic segmentation models without explicitly creating an annotated dataset for such adverse weather variations. Our framework consists of two parts, a restoration network to remove the geometrical distortions caused by hot weather and an adaptive segmentation network that is trained on an additional loss to adapt to the statistics of the ground-truth segmentation map. We train our framework on the Cityscapes dataset, which showed a total IoU gain of 12.707 over standard segmentation models. In the last part of our work, we improve the performance of our joint restoration and segmentation network via a feedback mechanism. In, the previous approach the restoration network does not learn directly from the errors of the segmentation network. In other words, the restoration network is not task aware. Hence, we propose a semantic feedback learning approach, which improves the task of semantic segmentation giving a feedback response into the restoration network. This response works as an attend and fix mechanism by focusing on those areas of an image where restoration needs improvement. Also, we proposed loss functions: Iterative Focal Loss (iFL) and Class-Balanced Iterative Focal Loss (CBiFL), which are specifically designed to improve the performance of the feedback network. These losses focus more on those samples that are continuously miss-classified over successive iterations. Our approach gives a gain of 17.41 mIoU over the standard segmentation model, in

Abstract

Advancements in 3D sensing and reconstruction has made a huge leap for modeling large-scale environments from monocular images using structure from motion (SfM) and simultaneous localization and mapping (SLAM) algorithms. SfM and SLAM based 3D reconstruction has applications for digital archival and modeling of real-world objects and environments, visual localization for geo-tagging and information retrieval, and mapping and navigation for robotic and autonomous driving applications. In this thesis, we address problems in the area of large-scale structure from motion (SfM) for 3D reconstruction and localization. We introduce new methods for improving efficiency and accuracy of state-of-the-art pipeline for structure from motion. Large-scale SfM pipeline deals with large unorganized collections of images pertaining to a particular geographical site. These image collections are formed by either retrieving relevant images using textual queries from the Internet, or can be captured for the specific purpose of 3D modeling, mapping, and navigation. Internet image collections tend to be more noisy and present more challenges for reconstruction as compared to datasets captured with specific intention to reconstruct. In this thesis, we propose methods that help with organizing these large, unstructured, and noisy images into a structure that is useful for SfM methods, a match-graph (or a view-graph). We first propose a geometry-aware two stage approach for pairwise image matching that is both more efficient and superior in quality of correspondences. We then extend this idea to SfM pipeline and present an iterative multistage framework for coarse to fine 3D reconstruction. Finally, we suggest that a key to solving many of the reconstruction problems is to address the problem of filtering and improving the view-graph in a way that is specific to the underlying problem. To this effect, we propose a unified framework for view-graph selection and show its application to achieve multiple reconstruction objectives.

Abstract

Natural Answer Generation(NAG) is the task of generating full-length sentential answers to factbased questions. This task is a desired attribute of conversational and task-oriented dialogue systems which aim to interact with the user in natural responses. Moreover, multi-modal interfaces enabling speech based input in addition to providing natural responses enhance the ease with which the user interacts with such systems. NAG systems have been widely studied in knowledge-base (KB) question answering tasks where the system generates natural answers to fact-based questions utilizing the structured information in the KB. Even though the task of generating natural answers to textual questions has been explored in recent years, there is little research on natural sentence generation over spoken content. To address this challenge, we propose the task of full-length natural answer generation from spoken question. We approach the task iteratively with sequential sub-tasks. First we study a pointergenerator based full-length natural answer generator system which generates natural answer from a textual question and factoid answer. The pointer-generator system effectively copies facts from the input addressing the challenges of out-of-vocabulary words and named-entities present in datasets. To train and evaluate our system, we develop a dataset of 315,000 samples comprising of question, factoid answer and full-length answer triples to train the system. We test our system over cross-domain samples extracted from a knowledge-base dataset such as Freebase and a machine comprehension dataset such as NewsQA. The system achieves a BLEU score of 74.05 and a Rogue-L score of 86.25. As a second sub-task, we study the effect of spoken input on dialogue systems. Spoken dialogue systems typically use one or several (top-N) ASR sequence(s) for inferring the semantic meaning and tracking the state of the dialogue, either with an end-to-end system or with a Spoken Language Understanding (SLU) module associated to a dialogue state tracker (DST). However ASR graphs, such as confusion networks (confnets), provide a compact representation of a richer hypothesis space than a top-N ASR list. In this sub-task, we study the benefits of using confusion networks with a state-of-the-art neural dialogue state tracker (DST). We encode the 2-dimensional confnet into a 1-dimensional sequence of embeddings using an attentional confusion network encoder which can be used with any DST system. Our confnet encoder is plugged into the state-of-the-art ’Global-locally Self-Attentive Dialogue State Tacker’ (GLAD) model for DST and obtains significant improvements in both accuracy and inference time compared to using top-N ASR hypotheses. Subsequently, we address the next sub-task of generating full length natural answer from spoken question. To the best of our knowledge, this is the first attempt towards generating full-length natural answers from spoken content. We approach this task by representing the spoken sequence compactly as a confusion network extracted from a pre-trained Automatic Speech Recognizer. We develop a pointer-generator system over the confusion network using the pre-assigned ASR scores and global attention to copy words from the confusion network and textual factoid answer. We release a large-scale dataset of 259,788 samples of spoken questions, their factoid answers and corresponding full-length textual answers called the ConfNet2Seq dataset. Following our proposed approach, we achieve comparable performance with best ASR hypothesis. Using ASR scores to copy words from the confusion network has the limitation that only the best ASR hypothesis is always chosen to be copied from the confusion network. The final sub-task addresses this issue with a novel hierarchical pointer-generator network which copies words from alternative hypothesis of the confusion network instead of being limited to the best ASR hypothesis. We analyse the benefits of using an ASR graph over the hierarchical pointer-generator network for the task of full-length natural answer generation from spoken questions. We experiment over increasingly noisy data to evaluate the performance of our system in noisy setting and show that our system outperforms the ConfNet2Seq system on the ConfNet2Seq dataset and performs similar to the ConfNet2Seq system over very noisy data. We also perform cross-dataset evaluation to evaluate the efficacy of our method over new domains.

Abstract

The internet has heralded an age of information explosion. A simple search query yields thousands of results. Each of these, in turn, is linked to a lengthy and verbose document. There is an immediate need to find faster and efficient ways to consume this ever-growing ocean of information. The purpose of summarization is to generate a shorter representation of the input that captures its gist and preserves its intent. In this work, we explore extractive summarization techniques, which involve identifying and extracting the most informative parts of a text. Most extractive summarization systems have been developed for the domains of newswire articles and scientific texts. These systems usually operate by ranking all the source sentences, according to some heuristic or metric, and then select the top sentences as the summary. They work well in domains where the discourse revolves around a central theme and information is often enforced by reiteration across several sentences. However, documents such as fictional narratives do not talk about a single topic. They describe a sequence of events and often contain dialogue. They do not contain repetitive information and each sentence contributes to developing the plot further. Hence, selecting a subset of sentences does not accurately capture the story. In the first part of this thesis, we discuss telegraphic summarization. Just like a telegram, a telegraphic summary does not contain complete sentences. Instead, shorter phrases are extracted across multiple sentences to capture the crux of the document. This summarization technique is better suited for the domain of fiction. We describe a set of guidelines to create such summaries. These guidelines were used to annotate a gold corpus of 200 English short stories. In the later part of this thesis, we model the task of telegraphic summarization as the well-known sentence compression task. Sentence compression is the task of shortening a sentence while retaining its meaning. Most methods proposed for this task rely on labeled or paired corpora (containing pairs of verbose and compressed sentences), which are often expensive to collect. To overcome this limitation, we present a novel unsupervised deep learning framework (SCAR) for deletion-based sentence compression. SCAR is primarily composed of two encoder-decoder pairs: a compressor and a reconstructor. The compressor generates a sequence of zeroes and ones, representing token deletion decisions, used to mask the input sentence. The reconstructor tries to regenerate the input from the masked sentence. The model is trained completely on unlabeled data and does not require additional inputs such as explicit syntactic information or optimal compression length. SCAR’s merit lies in the novel Linkage loss function, which correlates the compressor and its effect on reconstruction, guiding it to drop inferable tokens. SCAR achieves higher ROUGE scores than the existing state-ofthe-art methods and baselines on benchmark datasets. We also conduct a user study to demonstrate the application of our model as a text highlighting system. Using our model to underscore salient information facilitates speed-reading and reduces the time required to skim a document.