Abstract

The Himalayan region, spanning 2,500 kilometers in northern India, is highly prone to seismic activity. Situated in seismic zones IV and V, this region experiences frequent and devastating earthquakes, responsible for 70% of the world's fatal landslides. Factors such as steep slopes, heavy rainfall, uneven topography, geological conditions, climate, and unplanned urbanization exacerbate the susceptibility of the Himalayan landscape to landslides during earthquakes. The ongoing collision between the Indian and Eurasian plates generates faults and stress, making the region a high-risk area for future earthquakes. Khattri's 1999 research suggests a 56% probability of a magnitude 8.5 or greater earthquake occurring in the Himalayan seismic gap within the next century. Therefore, earthquake- induced landslides are a significant concern, necessitating enhanced preparedness to mitigate social and economic setbacks. To assess slope stability under seismic conditions, researchers employ deterministic, probabilistic, and statistical techniques. While predicting earthquakes with absolute certainty is impossible, seismic hazard studies estimate expected ground motion levels. These studies play a vital role in identifying ground shaking intensities that can trigger slope failures, quantifying hazards associated with specific locations. Integrated seismic hazard assessments, considering slope properties, enable the evaluation of likely ground motion scenarios. Therefore, conducting comprehensive and up-to-date seismic hazard studies is crucial to assess future landslide hazards in the earthquake-prone Himalayan region. Several seismic hazard analyses have been conducted in the Himalayan region, resulting in the development of peak ground acceleration for specific locations. However, challenges exist in using these ground intensities to accurately predict seismic vulnerability. Outdated or macro-level hazard maps, infrequent updates to earthquake databases, lack of expanded prediction equations, generalized GIS databases, data uncertainties, and stochastic-based earthquake catalogs contribute to these challenges. Depth ranges and maximum magnitude evaluations are crucial in seismic hazard assessments. Previous studies used standard or average depth ranges, but this study incorporated appropriate focal depths for point and linear sources. It also utilized a probabilistic approach called Regional Rupture Character (RRC) to estimate maximum magnitudes, setting it apart from studies using different ground motion prediction equations (GMPEs). Furthermore, the study introduced a fully probabilistic technique, called fully probabilistic seismic hazard assessment (FPSHA), to assess ground motion triggering landslides, a novel approach for the region. This study conducted seismic hazard analyses for the Darjeeling Sikkim Himalayan region using three frameworks: deterministic seismic hazard analysis (DSHA), probabilistic seismic hazard analysis (PSHA), and fully probabilistic seismic hazard analysis (FPSHA). DSHA emphasized seismic sources as the primary threat but did not consider uncertainties in the earthquake database and GMPE, potentially impacting results. While deterministic hazard maps evaluate intolerable failure consequences, they lack probabilistic information. PSHA, considering uncertainties in the earthquake database, provides estimates of ground motion exceedance over a specific time period. PSHA ground motions were significantly lower than those from DSHA, possibly due to inclusion of uncertainties. However, PSHA ground motions varied compared to FPSHA, which integrated PSHA with a dynamic slope stability model considering slope properties. FPSHA assessed the most probable ground motion scenarios for landslide triggering over the next 50 years for all slope models. Significant differences in ground motion levels were observed between FPSHA and both DSHA and PSHA, attributed to uncertainties in slope models, GMPE, and seismic source models. It is apparent from this research that seismic landslide hazards can be overestimated or underestimated when relying solely on DSHA and PSHA approaches. While PSHA provides probabilistic ground motions based on historical earthquakes, it is recommended for general seismic infrastructure design. FPSHA, on the other hand, estimates ground motions based on earthquake statistics and soil properties, offering suitable design ground motions for landslide triggering conditions. By considering all possible earthquake scenarios leading to slope instability, FPSHA accounts for specific conditions that can trigger landslides. The updated hazard maps and design charts developed in this study have various applications. They can be utilized in seismic infrastructure design, hazard zoning mapping, landslide monitoring, seismic slope stability analysis, land use planning, code requirements, and implementation of mitigation measures. These outc

Abstract

Autonomous systems perform various tasks across different industries ranging from finance to healthcare to space applications. However, these systems are often deployed in the open world, where it is hard to obtain complete specifications of the objectives and constraints. Operating based on an incomplete model can produce undesired effects, i.e., Negative Side Effects (NSEs). Negative side effects affect the system’s safety and reliability and can be of two types: Markovian and non-Markovian. In this thesis, we try to mitigate negative side effects in environments modeled as Markov decision processes (MDPs). Unlike previous works in this area that associate negative side effects with stateaction pairs, our framework associates them with entire trajectories, which is more general and captures non-Markovian dependence on states and actions. Non-Markovian negative side effects are produced when the agent executes a certain sequence of actions in the deployed environment. Prior works mitigate Markovian negative side effects and can not be easily extended to non-Markovian negative side effects. We build a framework, Controller-Assisted Safe Planning (CASP), for mitigating the non-Markovian negative side effects. Our primary contributions are: 1. We design a model based on Finite State Controllers (FSCs) that can predict the severity of negative side effects for a given trajectory. 2. We learn the model parameters using observed data containing state-action trajectories and the severity of the associated negative side effects. The model is learned such that it generalizes well to unseen data. Information about negative side effects is gathered through Oracle feedback and compactly represented as a finite state controller. 3. We develop a constrained MDP model that uses information from both the underlying MDP and the learned model for planning while avoiding negative side effects. Our empirical evaluation demonstrates the effectiveness of our approach in learning and mitigating Markovian and non-Markovian negative side effects.

Abstract

Document summarization aims to create a precise and coherent summary of a text document. There exist a plethora of deep learning summarization models that are developed mainly for English, which often requires (i) a large training corpus and (ii) efficient pre-trained language models and tools. However, English summarization models for low-resource languages like Indian languages are often limited by rich morphological variation and syntactic and semantic differences. Also, the restricted form of supervision limits the generality and usability of low-resource languages due to the lack of annotated corpora. The Graph Autoencoder (GAE) model has recently shown superior performance on several NLP tasks, even with limited resources. In this work, we propose GAE-ISUMM, an unsupervised Indic Summarization model that extracts summaries. In particular, our proposed model uses GAE and leverages: (i) learning document representations and (ii) jointly learning sentence representations and summary of the document. For evaluation purposes, we introduce TELSUM, a manually annotated summarization dataset comprised of 501 document-summary pairs. Extensive experiments on existing low-resource datasets (XL-Sum) and TELSUM provide the following insights: (i) our proposed model displays state-of-the-art results on XL-Sum and report benchmark results on TELSUM, (ii) Surprisingly, the inclusion of positional and cluster information in the proposed model further improved the performance of summaries. We open-source our dataset and code 1 . On the other hand, with the advancement of various deep-learning methodologies and transformerbased models, summarization has advanced to a new level. However, consistent and standard datasets must be produced to benefit from these deep learning algorithms fully. The creation of dedicated resources is rarely seen for low-resource Indian languages, unlike English which hinders the progress of summarization. To this end, we create summarization resources for Indian languages by introducing ISummCorp (Indic Summarization Corpora) and IndicSumm (Indic Language Summarization Models). IsummCorp is a highly abstractive summarization dataset sourced from the Times Of India (TOI). It is manually annotated by experts across eight Indian languages. Human and intrinsic evaluations demonstrate the high quality, abstraction, and compactness of ISummCorp. IndicSumm is a set of diverse monolingual and multilingual models based on ISummCorp. We refined IndicSumm, by finetuning the sophisticated, multilingual pre-trained mT5 model. With ISummCorp, we show that a model can perform better in a monolingual environment when trained with enough monolingual data than in a multilingual finetuning scenario. To investigate the potential of monolingual models, we finetuned mT5 using ISummCorp in both monolingual and multilingual situations and achieved better performance in a monolingual setting. Furthermore, we compare IndicSumm to other multilingual summarization models (XL-Sum and IndicBART) and achieve the state-of-the-art results.

Abstract

The goal of this thesis is to design a flow synthesis based visual servoing framework enabling longrange obstacle avoidance for Micro Air Vehicles (MAV) flying amongst tall skyscrapers. Recent deep learning based frameworks use optical flow to do high-precision visual servoing. This work explores the question: can a surrogate flow be designed for these high-precision visual-servoing methods which leads to obstacle avoidance? The concept of saliency is explored for identifying high-rise structures in/close to the line of attack amongst other competing skyscrapers and buildings as a collision obstacle. A synthesised flow is used to displace the salient object segmentation mask. This flow is computed in a way that ensures that the visual servoing controller maneuvers the MAV safely around the obstacle. In this approach, a multi-step Cross-Entropy Method (CEM) based servo control is employed to achieve flow convergence, resulting in obstacle avoidance. This novel pipeline is deployed on an MAV to successfully and persistently maneuver amongst high-rises and reach the goal in simulated and photo-realistic realworld scenes. Extensive experimentation is conducted and results are presented to compare the proposed approach with optical flow and short-range depth-based obstacle avoidance methods hence conclusively demonstrating the proposed framework’s merit. Additional visualisations can be found here.

Abstract

This paper proposes a buffer-assisted amplify-and-forward (AF) unmanned aerial vehicle (UAV) relay for communication between two ground nodes without a direct link. These communications are essential in disaster rescue areas where fairness plays a significant role. To achieve fairness, we aim to maximize the minimum information rate. The traditional way of prefixed scheduling the time slots to transfer and receive at the UAV do not guarantee that the communication system uses high signal-to-noise ratio (SNR) communication links instead of low SNR links. Therefore, we choose to employ a buffer at the UAV to store the information and transfer it to the destination nodes in the high SNR links. The pairing of time slots is necessary here since we are using a buffer, and we do not know in which time slot data is transmitted to the destination node after it is received at the UAV. Consequently, we formulate a fairness maximization problem by jointly optimizing the trajectory and power control. Unfortunately, this formulation results in a non-convex problem. We propose a solution based on the principles of the minorize-maximize (MM) algorithm and linear programming relaxation techniques to solve the fairness problem and pairing of time slots. Numerical results demonstrate that the trajectory, power control and paired slots favor the UAV and ground nodes to communicate in the high SNR channel links, thus maintaining fairness.

Abstract

Text-to-speech systems convert any given text to speech. They play a vital role in making Humancomputer interaction (HCI) possible. As humans, we don’t just rely on text (language) to communicate; we use many other mechanisms like voice, gestures, expressions, etc., to communicate efficiently. In natural language processing, vocabulary and grammar tend to take center stage, but those elements of speech only tell half the story. Affective prosody of speech provides larger context and gives meaning to words, and keeps listeners engaged. Current HCI systems largely communicate in text, and they lack a lot of prosodic information, which is crucial in a conversation. To make the HCI systems communicate in speech, text to speech systems should be able to synthesize speech that is expressive and controllable. But the existing text to speech systems learn the average variation in the dataset it’s trained on, which synthesizes samples in a neutral way without many prosodic variations. To this end, we develop a textto-speech system that can synthesize the given emotion where the emotion is represented as a tuple of Arousal, Valance and Dominance (AVD) values. Text to speech systems have a lot of complexities. Training such a system requires the data to be very clear, noiseless, and collecting such data is difficult. If the data is noisy, it will reflect unnecessary artifacts in the synthesized samples. Training emotion based text to speech models is considerably more difficult and not strait forward. The fact that obtaining emotion annotated data for the desired speaker is costly and very subjective makes it a cumbersome task. Current emotion based systems can synthesize emotions with some limitations. (1) Emotion controllability comes at the cost of loss in quality, (2) Have discreet emotions which lack the finer control, and (3) cannot be generalized to new speakers without the annotated emotion data. We propose a system that overcomes the above-mentioned problems by leveraging the largely available corpus of noisy speech annotated with emotions. Even though the data is noisy, our technique trains an emotion based text to speech system that can synthesize desired emotion without any loss of quality in the output. We present a method to control the emotional prosody of Text to Speech (TTS) systems by using phoneme-level intermediate variances/features (pitch, energy, and duration) as levers. We learn how the variances change with respect to emotion. We bring the finer control in the synthesized speech by using AVD values, which can represent emotions in a 3D space. Our proposed method also doesn’t require emotion annotated data for the target speaker. Once trained on the emotion annotated data, it can be applied to any system which has the prediction of the variances as an intermediate step. vi vii With thorough experimental studies, we show that the proposed method improves over the prior art in accurately emulating the desired emotions while retaining the naturalness of speech. We extend the traditional evaluation of using individual sentences for a complete evaluation of HCI systems. We present a novel experimental setup by replacing an actor with a TTS system in offline and live conversations. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly preferred over the state-of-the-art TTS system and adds the much-coveted ”human touch” in machine dialogue.

Abstract

We embark on a hitherto unreported problem of an autonomous robot (self-driving car) navigating in dynamic scenes in a manner that reduces its localization error and eventual cumulative drift or Absolute Trajectory Error. Modern autonomous vehicles (AVs) often rely on vision, LiDAR, and even radar-based simultaneous localization and mapping (SLAM) frameworks for precise localization and navigation. However, modern SLAM frameworks often lead to unacceptably high levels of drift (i.e., localization error) when AVs observe few visually distinct features or encounter occlusions due to dynamic obstacles. This work argues that minimizing drift must be a key desiderata in AV motion planning, which requires an AV to take active control decisions to move towards feature-rich regions while also minimizing conventional control cost. To do so, we have two distinct formulations of this problem. In the first approach, we learn actions that lead to drift-minimized navigation through a suitable set of reward and penalty functions. We use Proximal Policy Optimization, a class of Deep Reinforcement Learning methods, to learn the actions that result in drift-minimized trajectories. We show, by extensive comparisons on a variety of synthetic, yet photo-realistic scenes made available through the CARLA Simulator, the superior performance of the proposed framework vis-a-vis methods that do not adopt ` such policies. In the second approach, we first introduce a novel data-driven perception module that observes LiDAR point clouds and estimates which features/regions an AV must navigate towards for drift minimization. Then, we introduce an interpretable model predictive controller (MPC) that moves an AV toward such feature-rich regions while avoiding visual occlusions and gracefully trading off drift and control cost. Our experiments on challenging, dynamic scenarios in the state-of-the-art CARLA simulator indicate our method reduces drift up to 76.76% compared to benchmark approaches. We further proposed a novel approach that combines differentiablity with classical odometry algorithms (ICP) to identify drift minimizing features in a point cloud. We also developed a modular and scalable platform for testing and deploying self-driving systems on a real car. This works sets up the groundwork for scalable active navigation in real world

Abstract

The task of entity extraction has been thoroughly explored in the NLP community over the last two decades. There is a myriad of downstream tasks for Natural Language Understanding that utilizes entity extraction - ranging from Information Retrieval, Question Answering, Fact Extraction and Verification, Knowledge Graph Completion, etc. However, the datasets and domains that have been used for evaluation and benchmarking such entity extraction models have mostly been straightforward, structurally simple, semantically non-complex, and well-present across the breadth of training data. Most entity extraction tasks comprise of a significant overlap in entities across train and test sets. This does not mimic the real-world scenario, where rare emergent entities show a larger presence, and the entities themselves are often complex and semantically ambiguous. Our work investigates entity extraction in 2 settings - in the domain of scientific research papers, and in the area of linguistically low-resource structurally complex settings. In the first scenario, we establish an end-to-end pipeline that extracts certain task-specific entities from research documents, that help in a meaningful mapping of the research landscape. We show that the domain of scientific documents is non-trivial for entity extraction tasks because scientific entities follow a long-tail distribution. We incorporate multiple strategies like Distant Supervision, Graph Ranking, and Sequence Labelling to solve this task. Furthermore, we introduce multiple human-annotated gold standard datasets for both modular and complete evaluation of this entire pipeline. In the second setting, we investigate the task of low-resource semantically ambiguous complex entities. We experiment with multiple transformer-based architectures and pre-training strategies to solve this task. Our work significantly outperforms the baseline and outperforms multiple ensemble and gazetteer-based systems. We hope that our work will help undertake further research in this critical area of Natural Language Processing.

Abstract

Video is a complex modality consisting of multiple events, complex action, humans, objects and their interactions densely entangled over time. Understanding videos has been the core and one of the most challenging problem in computer vision and machine learning. What makes it even harder is the lack of structured formulation of the task specially when long videos are considered consisting of multiple events and diverse scenes. Prior works in video understanding have tried to address the problem only in a sparse and a uni-dimensional way, for example action recognition, spatio-temporal grounding, question answering and, free form captioning. However it requires holistic understanding to fully capture all the events, actions, and relations between all the entities, and represent any natural scene with the highest detail in the most faithful way. It requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation Recognition (VidSitu) through semantic role labeling is framed as a task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities. This is one of the most dense video understanding task posing several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation due to the free form captions for representing the roles. In this work, we propose the addition of spatio-temporal grounding as an essential component of the structured prediction task in a weakly supervised setting, without requiring ground truth bounding boxes. Since evaluating free-form captions can be difficult and imprecise this not only improves the current formulation and the evaluation setup, but also improves the interpretability of the models decision, because grounding allows us to visualise where the model is looking while generating a caption. To this end we present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time

Abstract

Ancient paper documents and palm leaf manuscripts from the Indian subcontinent have made a significant contribution to the world literary and culture. These documents often have complex, uneven, and irregular layouts. The process of digitization and deciphering the content from these documents without human intervention pose difficulties in a broad range of areas, including language, script, layout, elements, position, and number of manuscripts per image. Large-scale annotated Indic manuscript image datasets are needed for this kind of research. In order to meet this objective, we present Indiscapes, the first dataset containing multi-regional layout annotations for ancient Indian manuscripts. We also adapt a fully convolutional deep neural network architecture for fully automatic, instance-level spatial layout parsing of manuscript images in order to deal with the challenges such as presence of dense, irregular layout elements, pictures, multiple documents per image and the wide variety of scripts. Eventually, We demonstrate the effectiveness of proposed architecture on images from the Indiscapes dataset. Despite advancements, the segmentation of semantic layout using typical deep network methods is not resistant to the complex deformations that are observed across semantic regions. This problem is particularly evident in the domain of Indian palm-leaf manuscripts, which has limited resources. Therefore, we present Indiscapes2, a new expansive dataset of various Indic manuscripts with semantic layout annotations, to help address the issue. Indiscapes2 is 150% larger than Indiscapes and contains materials from four different historical collections. In addition, we propose a novel deep network called Palmira for reliable, deformation-aware region segmentation in handwritten manuscripts. As a performance metric, we additionally report a boundary-centric measure called Hausdorff distance and its variations. Our tests show that Palmira offers reliable layouts and outperforms both strong baseline methods and ablative versions. We also highlight our results on Arabic, South-East Asian and Hebrew historical manuscripts to showcase the generalization capability of PALMIRA. Even though we have reliable deep-network based approaches for comprehending manuscript layout, these models implicitly assume one or two manuscripts per image during the process, whereas in a real-world scenario there are often cases where multiple manuscripts are typically scanned together into a scanned image to maximise scanner surface area and reduce manual labour. Now, making sure that each individual manuscript within a scanned image can be isolated (segmented) on a per-instance basis became the first essential step in understanding the content of a manuscript. Hence, there is a need for a precursor system which extracts individual manuscripts before downstream processing. The highly curved and deformed boundaries of manuscripts, which frequently cause them to overlap with each other, introduce another complexity when confronting issue. We introduce another new document image dataset named IMMI (Indic Multi Manuscript Images) to address these issues. We also present a method that generates synthetic images to augment sourced non-synthetic images in order to boost the efficiency of the dataset and facilitate deep network training. Adapted versions of current document instance segmentation frameworks are used in our experiments. The results demonstrate the efficacy of the new frameworks for the task. Overall, our contributions enable robust extraction of individual historical manuscript pages. This in turn, could potentially enable better performance on downstream tasks such as region-level instance segmentation, optical character recognition and word-spotting in historical Indic manuscripts at scale.