Abstract

Navigation of Unmanned Aerial Vehicles (UAVs) amongst high-rises in urban environments be-comes inevitable due to increasing demand for various applications in Urban Air Mobility (UAM).Moreover, these city scale urban environments are populated with tall buildings and skyscrapers withinherent piecewise planar structures. Leveraging this inherent structural information for navigating aUAV equipped with a single camera and an IMU is the primary focus of this thesis. We make threecontributions that leverage these geometric cues to detect and map these planar structures for obstacleavoidance and path planning.Our first contribution,Multi-View Planarity Constraints for skyline estimation from UAV im-ages in city scale urban environments, is a three-stage pipeline that brings together several modulescombining a data-driven monocular plane segmentation network and geometric constraints from 3D vi-sion to showcase a quick reconstruction of planar facades within a few views. We evaluate the efficacyof our pipeline with various constraints and errors from multi-view geometry using ablation studies. Wethen retrieve the skyline of the buildings in synthetic as well as real-world scenes.In our second contribution,A new geometric approach for three view line reconstruction andmotion estimation in Manhattan Scenes, we propose a novel method of pose estimation using linefeatures from three views of a Manhattan Scene. We leverage the vanishing point directions to estimatethe relative rotations as well as to fix the 3D line direction. In consequence we build a constraints matrix,which has the relative translations and 3D line depth as its null space. We then perform 1-parameter lineBA using factor graph based cost function. We compare the efficacy of our method with standard linetriangulation in synthetic as well as real-world scenes.Finally, we proposeUrbanFly: Uncertainty-Aware Planning for Navigation Amongst High-Rises with Monocular Visual-Inertial SLAM Maps, a sequential convex program based novel trajec-tory planner tailored for outdoor urban scenes. It uses the sparse point clouds generated by a MonocularVisual Inertial SLAM (VINS) backend to build a cuboid representation of the environment through adata-driven monocular plane segmentation network. Our chosen world model provides faster distancequeries than the more common voxel-grid representation. The trajectory optimizer is initialized based onsafety estimates on a set of randomly drawn trajectories. We perform extensive simulations in a tightlyintegrated perception-control loop to validate its efficacy. UrbanFly outperforms competing baselinesin metrics such as collision rate, trajectory length, etc., on a high fidelity AirSim simulator augmentedwith synthetic and real-world dataset scenes.

Abstract

The current research presents an experimental investigation on coal ash samples (fly ash, bottom ash, and pond ash) collected from the NALCO thermal power plant and Talcher Thermal Power station, Odisha, India, and a numerical investigation on the dynamic response of shallow foundation on soil and pond ash deposit. Morphological, mineralogical, and geotechnical properties of coal ash (fly ash, bottom ash, and pond ash) samples were extensively investigated. The pond ash sample was chosen for further study because of its unique gradation of all collected samples. It was intended to determine the dynamic characteristics of pond ash subjected to dynamic excitation through laboratory tests and numerical analysis. A set of multiple cyclic triaxial tests were planned and performed on pond ash to examine the influence of test parameters like relative compaction, effective confining pressure, frequency, and shear strain. The properties obtained through the experimental tests were applied in the numerical analysis on pond ash subjected to different seismic excitations, i.e., the Nepal earthquake (Mw: 7.8) and Northeast India earthquake (Mw: 7.5). Liquefaction-induced deformation of pond ash was evaluated from the numerical analysis. Initially, morphological, mineralogical, and geotechnical properties of coal ash (fly ash, bottom ash, and pond ash) were investigated. Then pond ash was subjected to dynamic loads using cyclic triaxial testing equipment. The factors influencing the liquefaction potential of pond ash were evaluated. At low frequency (0.3 Hz), the pond ash specimen takes 87 cycles to liquefy at its most densified state. Also, the maximum dynamic shear modulus was noticed at this stage, i.e., 6534.80 kPa, and the decay was observed at 87 cycles. In addition, the decay in dynamic shear modulus was rapid with the enhancement of cyclic shear strain. The presence of 50% fines and the highly compacted state of the pond ash specimen caused close packing of particles which offered more resistance to shear strain; this resulted in possession of the high shear modulus as observed in this study. The maximum value of the damping ratio was obtained for the highly compacted specimen confined at high pressure subjected to the high frequency at medium cyclic shear strain considered in this study. The amplification of shear strain from medium to large shear strain caused a decrement in the damping ratio of pond ash. The presence of 50% fines in the present pond ash specimen made the specimen exhibit less dynamic shear modulus (Gdyn) (i.e., 33% less) than that of past studies by Mohanty and Patra (2014). The presence of predominant silt range particles (51%) made the specimen offer less resistance against the applied high strains despite better interlocking of silt and sand range particles of compacted pond ash specimens in the present study. 50% degradation of the dynamic shear modulus occurred within 4-28 cycles of loading. The dissipation of energy over the first few loading cycles was noticed between 12% and 94%, and it continued with a decreasing trend afterward. The observed values of the damping ratio were in contrast with that of the past studies on pond ash tested for similar adopted parameters (Mohanty and Patra, 2014) and contrary to the studies on different kinds of sandy soils (Kokusho, 1980; Govindarajulu, 2005). Studies by Chattaraj and Sengupta (2016) on fly ash with a predominant range of silt particles (~80%) revealed that the damping ratio was increased with an increase in shear strain; this observation contrasts with the present study. The trend of a damping ratio beyond 0.5% cyclic shear strain is in good agreement with the past studies of Kumar et al. (2017). A regression analysis has been carried out for the energy dissipated during cyclic loading under a varying state of compaction, effective confining pressure, and frequency. Few relationships between energy dissipated during cyclic loading under varying parameters of the state of compaction, effective confining pressure, and frequency were generated. The percentage variation of observed and predicted results is 4%. The 1D, 2D, and 3D ground response analysis of soil and pond ash and seismic response analysis of soil and pond ash with the foundation were subjected to Nepal and Northeast India earthquake excitations. The size of strip footing and shallow foundation (square) for 2D and 3D response analysis has been considered 1.5m x 0.8m and 1.5m x 1.5m x 0.8m, respectively. The PGA value in the case of 1D ground response analysis of pond ash has been noticed as 1.24 times and 1.63 times over 2D and 3D ground response analysis, respectively, when excited under the Northeast India earthquake. The PGA value in the case of 1D ground response analysis of pond ash has been noticed as 0.73 times and nearly two times over 2D and 3D ground response analysis, respectively, when excited under the Nepal earthquake. Soil and pond ash were found to be liq

Abstract

Geometric approach to solving a problem seems very natural or sometimes elegant compared to an analytical solution. Several machine learning problems have hidden geometric structures that can be exploited in proposing new solutions. In this work, we consider a specific structure called Riemannian structure of the search space of the problem. Often such spaces are nonlinear spaces where the standard techniques of optimization cannot work for the lack of linearity. But these spaces have a very useful property of being locally linear, which in turn helps us to consider extensions of classical optimization to such spaces. In particular, we consider two machine learning problems - the extreme classification problem and the tensor completion problem, where we show the presence of Riemannian structure. Once we identify this structure, we show how to use it to come up with efficient algorithms. For the extreme classification problem, a proof the convergence is given for the proposed algorithm. We experimentally verify the efficiency of the method in terms of train time and test accuracy comparing with several state-of-the-art methods on extremely large datasets. In the case of tensor completion problem, we propose a general problem that considers several works in the literature as its special case. For this general problem we propose a Riemannian optimization algorithm that solves the problem efficiently. We verify the correctness of the proposed algorithm both theoretically and experimentally through the special cases.

Abstract

Natural Language Generation is the task of producing understandable human text from variety of input sources. With the advent of pretrained language models (PLMs), text generation capabilities of current systems have achieved unprecedented heights. Pretrained language models have become the new normal and naturally serve as a backbone architecture for numerous tasks. It has been observed that these models learn various intricacies involved in language understanding and generation during pretraining step. These models are pretrained over large corpus; they also discover the world knowledge through text, some of which are absorbed in model parameters. Although, during finetuning on certain knowledge-intensive tasks (like text coherence, data-to-text, summarization, translation, etc.), it fails to utilize the intrinsic knowledge stored in it’s parameters effectively. In this thesis, we will tackle this problem by incorporating external facts to improve results on two downstream tasks: multilingual facts-to-text generation and text coherence modeling. We observe close association of facts in improving text generation by directly focusing on fact-to-text generation task. The fact-to-text generation is a variant of data-to-text where structured input data is knowledge graph triples. Data-to-text generative system consumes structured input data like tables, databases, knowledge bases, time-series data etc., and produces human-readable text summaries. The first part of the thesis addresses the multilingual fact-to-text generation, where facts are used to generate sentences in multiple languages. The fact-to-text generation requires a dataset where knowledge graph triples are well-aligned with semantically equivalent textual information. Manual creation of such a highquality fact-to-text dataset requires human supervision and is quite challenging to scale. Unsupervised alignment has recently emerged as an active area of research to overcome lack of labelled data and difficulty in domain adaptation. However, not much work has been done for low-resource languages that provide two significant challenges: (1) unavailability of pair of triples and native text with same content distribution and (2) limited Natural language Processing resources in low-resource languages. Hence, we rigorously investigate cross-lingual fact-to-text problem of aligning English structured data with sentences in multiple low-resource languages and develop a new dataset called XALIGN consisting of 0.45M pairs across seven low-resource languages. We propose two different methods of cross-lingual fact-to-text alignment: (a) Non-parametric approaches and (b) Parametric approaches. Additionally, we experimented with strong baseline results by adapting popular natural language generation methods for the cross-lingual fact-to-text task An essential requirement for any system that generates text is the coherence. In the second part of thesis, we address detection of text coherence. A large body of previous work have leveraged entity-based methods, syntactic patterns, discourse relations, and traditional deep learning architectures for text coherence assessment. However, these approaches do not consider factual information present in the documents. Transitions of facts associated with entities across sentences could help better capture the essence of textual coherence. We hypothesise that coherence assessment is a cognitively complex task that requires deeper fact-aware models and can benefit from other related tasks. To demonstrate this, we develop a novel deep learning model that fuses document-level information with factual information. We further enhance the model efficacy by training it simultaneously with Natural Language Inference tasks in multi-task learning setting, taking advantage of inductive transfer between the two tasks. Our experiments with popular benchmark datasets across multiple domains demonstrate that the proposed model consistently outperforms existing methods on synthetic coherence evaluation tasks and two real-world tasks involving predicting varying degrees of coherence.

Abstract

In times where autonomous navigation has the center stage, it is important for the embodied agents to learn to navigate through the environment with the degree of flexibility, that resembles the human-like sensibility and decision-making capacity. To do so, it is important to understand the spatial and semantic correlation between the regions and the objects and encapsulate this knowledge into the agent. In this thesis, we contribute to the same by proposing frameworks which utilize the relational co-occurrence of different regions and the objects within them in the problem of ”Object Goal Navigation”. The Object Goal Navigation (ObjectNav) task requires a robot to move to an instance of an out of view target object class, when initialised at a random position, in an unseen environment. The first framework proposed, uses a history of robot trajectories to learn a Spatial Relational Graph (SRG) and Graph Convolutional Network (GCN)-based embeddings for the likelihood of proximity of different semantically-labeled regions and the occurrence of different object classes in these regions. When the robot has to locate a specific target object instance during evaluation, a Bayesian inference approach and the SRG are used to estimate the visible regions, and the learned GCN embeddings are used to rank different visible regions and select the region to explore next. The approach is tested using the Matterport3D (MP3D) benchmark dataset of indoor scenes in AI Habitat, a visually realistic simulation environment. We report a improvement in performance over relevant baselines. The second framework described, first builds a semantic map of the environment at gradually over time, and then repeatedly selects a long-term goal based on the semantic map to locate the target object instance. The decision about ”where to go” (long-term goal) is formulated as a vision-based deep reinforcement learning problem. Specifically, an Encoder Network is trained to process a semantic map, extract high-level features, and select a long-term goal. In addition, we incorporate data augmentation and Q-function regularization to make the long-term goal selection process more effective. We report experimental results using the photo-realistic Gibson benchmark dataset in the AI Habitat 3D simulation environment to demonstrate that our framework substantially improves performance on standard measures in comparison with a state of the art baseline

Abstract

Speech is the most convinient and expressive form of communication among humans. Since the past few decades there have been continous efforts from the technology community around the world to establish speech based human-computer interaction. In this respect, several speech based technologies including (but not limited to) automatic speech recognition, text to speech synthesis, speaker recognition, speaker verification and emotion recognition have evolved. Among the various natural signals that can be recorded using sensors (such as seismic signal, radio signals etc.), we have physiological understanding of the corresponding production process of only a few signals such as speech and cardiac signal. In this thesis, we explore the possibility of extracting sound information from speech using it’s corresponding production characteristics. In particular, certain acoustic features are proposed for the identification of stops and nasals in continous speech. These acoustic features extract the corresponding sound’s production characteristics at two levels, namely, excitation source level and vocal tract system level. The acoustic features are extracted only at glottal closure instants using recently proposed signal processing techniques. The proposed algorithms do not rely on training on a database, and hence can be used for analysis on under-resourced languages. Further another study is presented to assess the individual contributions of magnitude and phase spectra of a given speech signal in preserving it’s intelligibility under different noise conditions. The results in this study can be used to develop algorithms for modifying the speech signal such that it’s intelligibility is improved in noise.

Abstract

Immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) are among the fasted growing and fascinated technology today. As the name suggests, these technologies promise to provide users with a much better experience using immersive head-mounted displays. Medicine, culture, education, and architecture are some areas that have already taken advantage of this technology. Popular video conferencing platforms such as Microsoft Teams, Zoom, and Google Meet is working to improve user experience by allowing users to use their digital avatars. Nevertheless, they lack immersiveness and realism. What if we extend these applications to virtual reality platforms so people can feel lively talking to each other? Integrating virtual reality platforms in collaborative spaces such as virtual telepresence systems have become quite popular after globalization since it enables multiple users to share the same virtual environment, thus mimicking real-life face-to-face interactions. For a better immersive experience in virtual telepresence/communication systems, it is essential to recover the entire face, including the portion masked by the headsets (e.g., Head-Mounted Displays abbreviated as HMDs). Several methods have been proposed in the literature that deal with this problem in various forms, such as HMD removal and face inpainting. Despite some remarkable explorations, none of these methods promises to provide usable results as expected in virtual reality platforms. Addressing these challenges in the real-world deployment of AR/VR-based applications draws emerging attention. Considering the existing limitations and usability of previous solutions, we explore various research challenges and propose a practical approach to facial de-occlusion/HMD removal for virtual telepresence systems. This thesis is well-documented to motivate and introduce the audience to various research challenges in facial de-occlusion, familiarizing them with existing solutions and their inapplicability in our problem domain, followed by the idea and formulation of our proposed solution to tackle the problem. With this view, the first chapter lays the outline of this thesis. In the second chapter, we propose a method for facial de-occlusion and discuss the importance of personalized facial de-occlusion methods in enhancing the sense of realism in virtual environments. The third chapter talks about the refinement of to previously proposed network in improving the reconstruction of the eye region. Last but not least, the final chapter briefly discusses the existing face datasets in face reconstruction and inpainting, followed by an overview of the dataset we collected for this work, from acquisition to making it usable for training deep learning models. In addition, we also attempt to extend image-based facial de-occlusion to video frames using off-the-shelf approaches, briefly explained in Appendix A.

Abstract

Air quality monitoring and analysis of a given location can be challenging without being able to determine the factors influencing its behavior. Stationary air pollution monitoring systems mostly ac- count for temporal variations in few locations due to the traditional choice of sparse deployment. In addition, the data from these monitoring stations may not be easily accessible. Internet of things (IoT) has allowed deployment of cost-efficient networks of smart devices with an ease in connectivity to the internet enabling easier air pollution monitoring. This thesis mainly proposes two test cases that highlight prominent factors that deeply influence the pollution pattern in a location. In the first test case, a temporal variation analysis is performed on two networks of pollution monitoring systems to understand the effects of implementing a nationwide lock- down during the COVID-19 pandemic. The first network comprises of the costly and bulky air quality monitoring stations of Central Pollution Control Board (CPCB) sparsely deployed in Hyderabad city and the second network comprises of the smaller and low-cost IoT nodes densely deployed inside the IIIT-H campus. Differential analyses were conducted on the data to understand the effects of lockdown on Particulate Matter (PM) levels by factoring in the yearly and seasonal trends, followed by Welch’s t- test to check whether the PM values have changed w.r.t. values in pre-lockdown period. Lastly, Pearson pair-wise correlation coefficient is estimated between PM and temperature values to show the effect of temperature changes on the PM values irrespective of lockdown. This test case established the effects of human-centric factors such as vehicular and industrial emissions, commercial activities and commo- tion on ambient PM variations as a sudden drop in the anthropogenic activities during the nationwide lockdown has caused a decline in PM levels. In the second test case, a novel methodology to analyze a moving object database is proposed by performing a case study on mobile IoT data collecting Particulate Matter across a road stretch of India, and look into the neighboring spatial and anthropogenic factors such as human activities, settlement patterns and vegetation profile corresponding to each geo-location of the PM data. The thematic inter- actions of spatial and anthropogenic factors of each location with the corresponding ambient PM levels resulted in a factor-based data structure that highlights the PM distribution mapped to each factor. This case study not only enabled to showcase the influence of spatial and anthropogenic factors of a location in influencing its ambient PM levels, but also provided a use case for handling mobile object databases, a challenging issue prevalent amidst the GIS community. Both the test cases effectively demonstrated the contribution of different surrounding factors in affecting the PM values, thus providing a gateway for accurately modeling the ambient air quality data based on key parameters.

Abstract

Supervised methods in deep learning approaches hinge on the assumption that the training and testing data are sampled from the same distribution. However, this is not the case in most realistic scenarios and thus leads to poor performance when these models are deployed in domains with different data distribution than the training set. Unsupervised Domain Adaptation (UDA) tackles the problem of adapting the data distributions between a labelled source domain and an unlabeled target domain. In contrast, Semi-supervised Domain Adaptation (SSDA) assumes a partially labelled target domain, a more realistic scenario in many computer vision tasks. Domain randomization is a popular approach wherein models are trained on synthetically generated data. With complete control over the synthetic data generation process, domain randomization introduces randomness in various properties of the object as well as the scene. Similar to data augmentation techniques in deep learning, the hope here is to introduce invariance for different non-causal features of the data and even nudge the model towards learning the causal correlations for the task at hand. In this thesis, we explore and study various approaches to domain adaptation. First, we present the image level domain adaptation methods, which use image-level manipulation or transformations to achieve domain invariance. We first analyze the domain randomization approach used in an object detection setting. For this, we use synthetically generated data and train a FasterRCNN model aimed at the object detection task. Domain randomization helps boost the performance of object detection models, and a model trained entirely on synthetic data outperforms the one trained on real data. With fine-tuning, the performance of the model trained on synthetically generated data increases drastically. Next, we extend the work on domain adaptation in the frequency domain, wherein the image level adaptation occurs in the frequency domain. To this end, we propose new combination strategies to combine the frequency components. We propose masking techniques which consider the frequency of the components in the combination process. Fourier domain adaptation techniques have seen some success in the image segmentation task from synthetic like GTA5 [1] and SYNTHIA [2] to realistic domains like Cityscapes [3]; however, these domains contain syntactically similar images. For the synthetic dataset that we use, we find that these frequency domain bases stylization methods do not improve performance over the domain randomization method. Finally, we present two novel methods for domain adaptation using feature level alignment. One of the primary challenges in SSDA is the skewed ratio between the number of labelled source and target samples, causing the model to be biased towards the source domain. Recent works in SSDA show that aligning only the labelled target samples with the source samples potentially leads to incomplete domain alignment of the target domain to the source domain. In our first approach, we train the source and target feature spaces separately. To ensure that the feature space of the target domain is generalized well, we employ semi-supervised methods to leverage the labelled and unlabeled samples. The Domain Adapters, which are parametric functions, are then trained to learn the feature level transformation from the target domain to the source domain. During inference, we use the target domain’s feature extractor and then pass the features to the respective Domain Adapter for that target-source pair. The transformed feature representation in the source domain is then fed to the source classifier. We show that keeping the feature extractors separate is advantageous if the domain gap between the source and the target domain is insignificant. Finally, we present SPI, which leverages contrastive losses to learn a semantically meaningful and a domain agnostic feature space using the supervised samples from both domains. To mitigate challenges caused by the skewed label ratio, we pseudo-label the unlabeled target samples by comparing their feature representation to those of the labelled samples from both the source and target domains. Furthermore, to increase the support of the target domain, these potentially noisy pseudo-labels are gradually injected into the labelled target dataset over the course of training. Specifically, we use a temperature scaled cosine similarity measure to assign a soft pseudo-label to the unlabeled target samples. Additionally, we compute an exponential moving average of the soft pseudo-labels for each unlabeled sample. These pseudo-labels are progressively injected (or removed) into (from) the labelled target dataset based on a confidence threshold to supplement the alignment of the source and target distributions. Finally, we use a supervised contrastive loss on the labelled and pseudo-labelled datasets to align the s

Abstract

Text Classification is a technique for assigning a set of categories to give text. A significant type of text classification is Sentiment Analysis. Sentiment Analysis is a technique in natural language processing to identify and quantify subjective states and information systematically. With the exponential increase in traffic over the internet, we have a lot of textual content available in different settings. This data provides an opportunity for efficient processing and application in solving problems across multiple domains. Few such critical applications include analysing public behaviour online and how they interact. Other is to analyse news content by different media houses that can cause public divide and polarisation due to its subjective nature. Firstly, we aim to create a novel dataset for bias detection in Hindi. We also explore classification tasks in a multi-modal setting to analyse and understand how information in multiple modalities affects classification outcomes. We further experiment with deep learning models to classify text where the language is more subtle and often unrealised for humans making its classification a tricky task. Recently, we have seen a rise in content from highly subjective news media outlets created to polarise and create a divide between people. This reporting can have far-reaching consequences, from orchestrating riots and public agitations to influencing the results of an election. This biased reporting makes its analysis and detection crucial. We aim to do it for the Hindi Language. As no data were available, we decided to develop our dataset. Our dataset consists of 1388 Hindi news articles and their headlines from various Hindi news media outlets, which were annotated for being biased towards, against, or neutral towards the BJP (Bhartiya Janata Party). Further, we explore a multi-modal classification task of identifying misogynous memes posted online. In this work, we also explore how the visual modality affects the classification task. We build an ensemble of deep learning text-transformers and vision-language transformers like UNITER and OSCAR and compare them to better understand the role of vision in multi-modal classification tasks. We also experiment with task-specific pretraining of our vision-language(VL) transformers. Lastly, we experiment with employing transformers trained on similar tasks on other similar datasets to explore transfer learning. Lastly, We employ deep-learning transformers to detect patronising and condescending language targeting vulnerable communities (e.g. refugees, homeless people, and low-income families). We experiment with techniques like focal-loss and weighted random sampling to deal with class imbalances and to improve our results. We also take up the task of classifying more fine-grained categories of condescending language.