Abstract

The launch of level 5 autonomous driving is imminent which would result in removal of controls like steering wheel, acceleration and brakes. This warrants the need for a better system to direct the self-driving agent to perform maneuvers if the passenger wishes to do so. Hence, this thesis proposes a set of solutions tackle the problem of vision and language navigation in an autonomous driving setting which has been broken down into multiple sub problems. The first problem we attempt to solve, we present a simple baseline for visual grounding for autonomous driving which outperforms the state of the art methods, while retaining minimal design choices. The framework minimizes the cross-entropy loss over the cosine distance between multiple image ROI features with a text embedding (representing the given sentence/phrase). We use pre-trained networks for obtaining the initial embeddings and learn a transformation layer on top of the text embedding.We perform experiments on the Talk2Car dataset and achieve 68.7% AP50 accuracy, improving upon the previous state of the art by 8.6%. Humans have a natural ability to effortlessly comprehend linguistic commands such as “park next to the yellow sedan” and instinctively know which region of the road the vehicle should navigate. Extending this ability to autonomous vehicles is the next step towards creating fully autonomous agents that respond and act according to human commands where knowing only the location of the object of interest in the scene is not enough. To this end, a novel task of Referring Navigable Regions (RNR), i.e., grounding regions of interest for navigation based on the linguistic command is proposed. RNR is different from Referring Image Segmentation (RIS), which focuses on grounding an object referred to by the natural language expression instead of grounding a navigable region. For example, for a command “park next to the yellow sedan,” RIS will aim to segment the referred sedan, and RNR aims to segment the suggested parking region on the road. We introduce a new dataset, Talk2Car-RegSeg, which extends the existing Talk2car dataset with segmentation masks for the regions described by the linguistic commands. We benchmark the proposed dataset using a novel transformer-based architecture. We present extensive ablations and show superior performance over baselines on multiple evaluation metrics. A downstream path planner generating trajectories based on RNR outputs confirms the efficacy of the proposed framework.

Abstract

Dialogue systems help us to access information and services more effectively. This thesis presents the work done on developing Interactive Question Answering Systems, a type of dialogue systems, for English and Telugu. The domains chosen for English was Hyderabad MMTS enquiry and for Telugu it was Hyderabad Tourism. Both the systems take input in the form of text and the output is also text. To be able to study a wider range of user behaviour both the systems are developed for a different domain and language. The MMTS Question Answering system follow the frame based approach. This model can handle spelling errors and can also deal with insufficient information in the user query. In this system, the data is stored using SQL databases. We convert the user query to a proper SQL query and extract required information from the database. We tried to port the above model into Telugu language for Tourism domain but the process failed because of a lack of proper language processing tools for Telugu language. So we had to design a new approach. The approach used for Telugu can be extended to other resource poor Indian languages as well. The dialogue model consists of two parts namely Data Management and Query Processing. Data Management deals with storing the data in a particular format which helps in easy and quick retrieval of requested information. Query Processing deals with producing a relevant system response for a user query. Our model can handle code-mixed queries which are very common in Indian languages and also handles context which is a major challenge in dialogue systems. It also handles spelling mistakes and a few grammatical errors. The model is domain and language independent. Using this model, we developed a system for Telugu language for ’Tourist places of Hyderabad’ domain. As there is no automated evaluation tool available for dialogue systems, we went for human evaluation of our system. Five people evaluated our system and the results are reported in the thesis. In this thesis, we have also made an attempt to extract adjacency pairs from conversational units with multiple utterances from each user using dialogue acts. Adjacency pairs help in automatic extraction of dialogue structure. In dealing with minimal information in utterances, this becomes useful. In the machine learning algorithms that we were using for Natural language Generation, we relied on conversational units where the speakers have uttered only one sentence in their turn as training data, This restricts the data, and a good amount of manual work goes into converting these conversations units to adjacency pairs. To ease this process, we have tried to automate the process of extracting adjacency pairs from the data using dialogue acts.

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, domestic service robots, and more, the need for privacy of individuals interacting with such systems has become evident. We aim to investigate the quality of solutions that claim to protect the reward functions that contain sensitive personal information, from being exploited. Optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. Differential Privacy (DP) - a popular technique used for computational privacy, empowers us to provide mathematically quantifiable privacy guarantees for queries made to “privatised” algorithms. Our work introduces the reward reconstruction attack that tries to reverse engineer the reward from a private policy using Inverse RL algorithms. This attack is the key part of the novel Privacy-Aware Inverse RL Analysis (PRIL) framework. We assume that the entire training process and the optimal policy is released publicly. We study the set of existing differentially-private RL policies derived from various RL algorithms including Value Iteration, Deep Q Networks (DQNs), and Vanilla Proximal Policy Optimization (PPO). We propose the new PRIL analysis framework, that performs reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the novel reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using an Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using the PRIL framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on multiple instances of OpenAI Gym environments of varying complexities, specifically the discrete state-space Frozen Lake domain and the continuous state-space Mountain Car domain, employing a different Linear Programming based Inverse RL method for each domain. From our experiments, we observe that there is no indication of the policy improving at reconstructing the reward with a relaxation in the budget - thus rendering all strategies ineffective at providing true reward privacy. Based on the analysis performed, we infer that there exists a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We quantify the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards. We calculate the distance between the original reward and the reconstructed reward via various distance metrics. The larger the reward distance, the better the private agent’s ability to protect the sensitive reward function using which it was trained. We present the trends in test-time performance utility of private agents upon varying the level of privacy guaranteed (by altering the privacy budget during training). We also formally discover the differentially-private Bellman Update algorithm along with its proof of sensitivity. This is one of the first few works in this sub-field, and we hope that it is pursued with more rigour in the future.

Abstract

To execute meaningful real-world tasks, robots must perceive and understand the objects that surround them. Manipulation, autonomous driving, and human-robot interaction are just a few examples where robots must reason about previously unseen instances of objects. We present DRACO, a method for Dense Reconstruction And Canonicalization of Object shape from one or more RGB images. Canonical shape reconstruction—estimating 3D object shape in a coordinate space canonicalized for scale, rotation, and translation parameters—is an emerging paradigm that holds promise for a multitude of robotic applications. Prior approaches either rely on painstakingly gathered dense 3D supervision, or produce only sparse canonical representations, limiting real-world applicability. DRACO performs dense canonicalization using only weak supervision in the form of camera poses and semantic keypoints at train time. During inference, DRACO predicts dense object-centric depth maps in a canonical coordinatespace, solely using one or more RGB images of an object. Extensive experiments on canonical shape reconstruction and pose estimation show that DRACO is competitive or superior to fullysupervised methods. This thesis delves into understanding objects to capture their pose and shape from a single image. Canonicalization is the operation of mapping an input representation to an invariant representation with respect to an operation. This input representation can be a point cloud of an object. For the scope of this thesis, we fix this operation to be a 3D transformation. Lets take an example of an object, say a car. By finding a canonical frame for this car, we have its 3D reconstruction in a frame that is invariant to any 3D transformation. This is important for tasks such as registration, feature propagation and annotation. Using this invariant 3D pose we can register different poses of the car to a single consistent frame. This single consistent frame across multiple cars is called the canonical frame. Humans have an inherent understanding of this canonical frame. For instance, we know that the headlights of the car are towards its forward direction. This forward direction is with respect to our mental canonical frame and this allows us to estimate the pose of the car with respect to our view. canonicalization – the process of mapping an object instances to a category-level container – has emerged as a useful tool for category-level understanding. We can extend datasets,automatically annotate for parts, register and manipulate, if we can map every object to the canonical orientation. Our method, DRACO, estimates a canonical frame by predicting NOCS maps and reconstructs the object of interest in using a monocular image. NOCS maps are 2D images that store the location of every pixel with respect to a canonical frame that is invariant to transformation. Previous methods use pain-stakingly collected datasets to learn a dense canonical frame or weakly-supervise canonicalization for sparse keypoints. Our method uses weak-supervision in the form of semantic keypoints and camera relative motion to learn an object’s reconstruction and canonical frame which performs on par or better against fullysupervised methods. This enables estimating canonical frame for in-the-wild objects that enables canonicalizing them. Extensive experiments on our new large-scale mixed reality dataset (20K images/category) demonstrate on par shape reconstruction and canonicalization quality compared to state-of-the-art fully-supervised approaches, and weakly-supervised baselines that strictly use more information. This thesis introduces concepts of single-view depth estimation for reconstructing objects using a monocular image and delves into canonicalization in a weakly-supervised manner.

Abstract

Developers perform a wide variety of tasks during software development and maintenance. The most common tasks are documentation, bug fixing, enhancing capabilities, removing obsolete capabilities, and optimizations. Each task varies in complexity and effort. Developers spend significant time acquiring the software system’s knowledge. Specifically, they find comprehension of collected information laborious and often repetitive. This difficulty is due to the dispersed information across multiple artifacts such as source code, unit tests, bug reports, API documentation, commit history, and developer discussions. The comprehension effort gets confounded as the number of artifacts increases. Also, complete comprehension is difficult as software systems grow in size and get more distributed. Due to this, developers have a fragmented or incomplete understanding of the software system. To overcome this, they adopt tactical strategies to familiarize themselves with the program. They either read code, add trace statements, use debuggers, establish and test hypotheses, or observe the behavior. Due to this, the changes are tedious and are of lower quality. So, supporting developers to acquire task-specific knowledge rapidly is essential to develop and maintain a software system effectively. In this context, we analyze the developers’ information needs, propose a model and developed a technique to help faster code comprehension. Developers choose to read software code. They do so due to a lack of time and face minimum barriers. Developers read code on a need-basis without any plan – opportunistic. They choose either case-based or scavenging approaches [86]. Large-scale studies have observed that developers read code despite the availability of modern tools and Integrated Development Environment (IDE) [91, 125]. While code reading is prevalent among developers, the effectiveness of approaches followed by them is unknown. So, we study the effectiveness of comprehension by reading code in the context of reusing an external module, which is a common developer task. Reuse is a complex task, but developers are still required to integrate an external module rapidly. Observing developers undertake the reuse task is difficult due to the unplanned nature of the task. So, analyzing the historical bug reports and commit history helps us reason about the effectiveness of code reading practice. We observed reuse of the JMapViewer module in two independently managed opensource software applications – Mobile Atlas Creator and JTileDownloader. Based on the bug reports and large commits, we conjecture that comprehension was limited. It was to the extent required to accomplish the integration. We observed many inconsistencies, emergent behavior, conflicting scenarios, and fragility in the software structure. From this study, we infer that developers limit knowledge acquisition in an opportunistic task. Furthermore, we also conclude that the issue is due to the information overload they face when reading code. There is a growing interest in understanding developers’ tasks and how they comprehend code. Nevertheless, little is known about developers’ information needs when reading code. For example, How do developers locate concepts in a new program? What information do they seek to assess the impact of a change? What knowledge about the system do they need to triage bugs? How do they decide where to start for root cause analysis? When working with code, developers are constantly engaged in reasoning about the choices, implications, purpose, relevance, and expectations. In doing so, they assess a code fragment’s relevancy to the task. Typically, a comprehension task beings with an information-seeking activity. We find the current developer tool-sets fall short in assisting developers in reducing the information overhead they face during comprehension. When developers read a code, they create rich mental models of the program structure. However, these models are volatile and have no formal methods to record and share among developers. As there are no records, developers repeatedly read the same code and rebuild the mental model again. Having to repeat this across multiple source code is tedious, more so if the code is unfamiliar. Recently, researchers are examining large software projects using Mining Software Repositories (MSR) techniques to address a variety of Software Engineering questions. Overwhelmingly, recent research studies have focused on what can happen or why it happens. However, when reading code, questions related to what is happening are essential. So, we explored how to assist a developer in reading code by identifying and extracting task-specific information from software artifacts. We base our approach on the program comprehension model and the program navigation models [23, 111, 114]. These models suggested that developers use beacons or cues to establish the rele

Abstract

Text recognition has been an active field in computer vision even before the beginning of the deep learning era. Due to the varied applications of recognition models, the research area has been classified into diverse categories based on the domain of the data used. Optical character recognition (OCR) is focused on scanned documents, whereas images with natural scenes and much complex backgrounds fall into the category of scene text recognition. Scene text recognition has become an exciting area of research due to the complexities and difficulties such as complex backgrounds, improper illumination, distorted images with noise, inconsistent usage of fonts and font sizes that are not usually horizontally aligned. Such cases make the task of scene text recognition more complicated and challenging. In recent years, we have observed the rise of deep learning. Subsequently, there has been an incremental growth in the recognition algorithms and datasets available for training and testing purposes. This surge has caused the performance of recognizing text in natural scenes to rise above the baseline models that were previously trained using hand-crafted features. Latin texts were the center of attention in most of these works and did not profoundly investigate the field of scene text recognition for non-Latin languages. Upon scrutiny, we observe that the performance of the current best recognition models has reached above 90% over scene text benchmark datasets. However, these recognition models do not perform as well on non-Latin languages as they did on Latin (or English) datasets. This striking difference in the performances over different languages is a rising concern among the researchers focusing on lowresource languages, and it is indeed the motivation behind our work. Scene text recognition in low-resource non-Latin languages is difficult and challenging due to the inherent complex scripts, multiple writing systems, various fonts and orientations. Despite such differences, we can also achieve Latin (English) text-like performance for low-resource non-Latin languages. In this thesis, we look at all the parameters involved in the process of text recognition and determine the importance of those parameters through thorough experiments. We use synthetic data for controlled experiments where we test the parameters as mentioned earlier in an isolated fashion to effectively identify the catalysts of text recognition. We analyse the complexity of the scripts via these synthetic data experiments. We present the results of our experiments on two baseline models, CRNN and STAR-Net models, on available datasets to ensure generalisability. In addition to this, we also propose an error correction module for correcting the labels by utilizing the training data of real test datasets. To further improve the results on real test datasets, we propose transfer learning from English to exploit the abundant data that is available for learning. We show that the transfer from English is not feasible, and it actually lowers the performance of the individual language models. Due to the failure of English transfer experiments, we shift our focus onto just the Indian languages and examine the characteristics of each language via character n-gram plots, visual features like vowel signs, conjunct characters and other word statistics. They also share a resemblance to each other concerning certain other factors. We then propose to apply transfer learning across languages to enhance the performance of the language models. We depict the improvement on real datasets because of the transfers among Indian languages that are visually closer or sometimes better than the individual models. The transfers among languages prove to be much more profitable than transfers from English. We comprehend the significance of the variety and number of fonts during data generation via synthetic data experiments on English test datasets. Synthetic data embodies various fonts to ensure diversity of data to create robust recognition systems. In order to strengthen data diversity, we incorporate over 500 Hindi fonts (including Unicode and non-Unicode fonts) into the synthetic data for improved performance on Hindi real test datasets. We also manifest the process to utilize and incorporate nonUnicode fonts of Indian languages into the training process error-free. In addition to these fonts, we make specific changes to encompass an augmentation pipeline that adds to the diversity of data. We utilize more than nine augmentation techniques to boost the performance of Hindi STR systems. We achieved significant improvements over previous works with our evaluations over natural settings. Through our experiments, we set new benchmark accuracies for STR on Hindi, Telugu, and Malayalam languages from the IIIT-ILST dataset by gaining 6%, 5%, and 2% gains in Word Recognition Rates (WRRs) compared to previous works. Similarly, we also ach

Abstract

Many populous countries including India are burdened with a considerable backlog of legal cases. Development of automated systems that could process legal documents and augment and help legal practitioners can mitigate this. However, there is a dearth of high-quality corpora that is needed to develop such data-driven systems. The problem gets even more pronounced in the case of low resource languages such as Hindi. Additionally one of the most common and time sensitive cases handled by the courts are bail cases. In this Thesis, we first introduce the Hindi Legal Documents Corpus (HLDC), a corpus of more than 900K legal documents in Hindi. Documents are cleaned and structured to enable the development of downstream applications. We then introduce and tackle the task of bail prediction. We select the bail cases from our HLDC corpus and further extract the facts and arguments and judge’s summary. We experiment with a battery of models and propose a Multi-Task Learning (MTL) based model for the same. Our MTL model uses summarization as an auxiliary task along with bail prediction as the main task. The intermediate summarisation step is a novel introduction which serves dual purposes. First, it reduces the document size without compromising on the information. Since many transformer models have constraint on the input length, sending a summarised version of the documents allows us to overcome this barrier. Second, it builds towards explainable legal NLP systems as it allows us to identify salient sentences. This Thesis lays the foundation for research in Legal NLP for Hindi court documents. The multitude of legal NLP tasks and challenges are indicative of the need for further research in this area.

Abstract

Natural Language Processing (NLP) has evolved drastically in many areas due to the popularity and success of a variety of different Machine Learning and specifically Deep Learning techniques and their applications commercially as well as academcially in the last decade or so. Typically most of the work done in these fields globally has centred around solving tasks with the use of resource rich languages such as English as a base upon which models have been built. The large chasm this technological and academic revolution brings about, to a world that is divided in a multitude of regional cultures and languages that haven’t evolved at the same pace, creates a very real obstacle. We aim to play a part in bridging this gap that has been brought about with the creation of a rich resource for one such low resource language in Telugu, a Dravidian language. We aim to focus on curating and annotating a dialogue corpus as it allows us to approach a variety of tasks and holds various different real world applications including the rather pertinent current phenomenon of improving Human Computer Interactions especially in the context of low resource languages such as Telugu. It is important to also take note of the evolution of language and different means of communication as a function of time and one product of this observation is the increased usage of code-mixing in colloquial discourse. In order to develop robust models and accurately recreate Human-Human interactions with the use of computers, we need to be able work with and model on code-mixed data. Therefore another focus of our study is in the creation of a dialogue corpus that is heavily code-mixed with Telugu and English and the subsequent modelling of the dialogue system using deep learning techniques. In order to determine the resourcefulness of the Telugu-English code-mixed dialogue corpus that we have carefully curated, we put it to the test by performing two different widely studied NLP tasks in : Text-based Speaker Identification and Automatic Humour Recognition. We provide a comprehensive analysis and survey of modern deep learning techniques for the task of text-based speaker identification and present a detailed explanation of all the choices made while undertaking this study. The deep learning techniques implemented in this body of work include the likes of Convolutional Neural Networks, LSTMs, and Transformer based models.

Abstract

Social media has enabled individuals to easily communicate with others while freely sharing their ideas and thoughts by providing anonymity. Due to the substantial growth of user-generated web content in general and social media networks in particular, the amount of discriminatory content is also steadily increasing, leading to significant toxicity in abusive behaviours and online discourse. One type of such harmful content is hate speech, which directly attacks or promotes hate towards an individual member or a group based on their actual or perceived aspects of identity, such as ethnicity, religion, and discrimination. Developing models for this type of data can be difficult due to the fact that the data available on online media may be inadequate, unstructured, and highly imbalanced. In this thesis, we focus on discrimination, especially the form of discrimination perpetrated based on sex. Sexism, a form of oppression based on one’s sex, manifests itself in numerous ways and causes enormous suffering. During the last years, the role of women within online platforms has gained attention, unfortunately, because of the growing hatred and abuse against them. In view of the growing number of experiences of sexism reported online (accounts of sexism) and the increase in sexist posts in online conversations, automatically categorizing both sorts of posts can assist the fight against sexism, as it can facilitate effective analyses by gender studies researchers and government officials involved in policy making. In this thesis, we investigate the detection and categorization of both types of sexist posts. The existing work on sexism classification has certain limitations in terms of the categories of sexism used and/or whether they can co-occur. To the best of our knowledge, we work with considerably more categories of sexism than any prior work through our 23-class problem formulation. We present the first semi-supervised work for the multi-label classification of accounts describing any type(s) of sexism. We devise self-training based techniques tailor-made for the multi-label nature of the problem to utilize unlabeled samples for augmenting the labeled set. We identify high textual diversity with respect to the existing labeled set as a desirable quality for candidate unlabeled instances and develop methods for incorporating it into our approach. We also explore ways of infusing class imbalance alleviation for multi-label classification into our semi-supervised learning, independently and in conjunction with the method involving diversity. In addition to data augmentation methods, we develop a neural model which combines biLSTM and attention with a domain-adapted BERT model in an end-to-end trainable manner. Further, we formulate a multi-level training approach in which models are sequentially trained using categories of sexism of different levels of granularity. Moreover, we devise a loss function that exploits any label confidence scores associated with the data. We also propose a knowledge-based cascaded multi-task framework for fine-grained multi-label sexism classification that allows for shared learning across multiple tasks through common layers/weights and a combined loss function. We leverage several supporting tasks, including homogeneous and heterogeneous auxiliary tasks. Homogeneous tasks are set up without incurring any manual labeling cost and heterogeneous tasks are set up which have a high correlation with the accounts of sexism. Unlabeled accounts of sexism are utilized through unsupervised learning to help construct our multi-task setup. In addition, we incorporate a knowledge module within the framework to infuse domain-specific external knowledge features into the learning process. Further, we investigate transfer learning that employs weakly labeled accounts of sexism and transfers the learning to the multi-label sexism classification. We also devise objective functions that exploit label correlations in the training data explicitly. In order to evaluate the versatility of our approaches, we experiment with adapting them for the toxic comment classification dataset. Multiple proposed multi-task methods outperform the state-of-the-art multi-label sexism classification and toxic comment classification across five standard metrics. In addition to analysing the textual accounts describing sexism suffered or observed in a multi-label multi-class setup, we also analyze how sexism is expressed in online conversations, especially in the social networks, wherein the text perpetrates sexism. Moreover, we also aimed to address the problem of detecting sexism rather than only the categorization. Specifically, we explore the problem of detecting whether a tweet/gab post is sexist or not. We further discriminate the detected sexist post into one of the fine-grained sexism categories in a single-label multi-class manner since labeled data (text perpetrates sexism) is not available wherein

Abstract

Air pollution poses a threat to the lives of all living beings. Government authorities generally monitor pollution levels using a high-grade setup. The high-grade instruments are expensive and also require space for setup. A few low-cost sensors have been introduced for monitoring air quality. But, those sensors also have a few shortcomings. This thesis mainly focuses on the performance evaluation of low-cost PM sensors and an image processing-based technique to estimate air quality. Firstly, the performance of three new and popular low-cost particulate matter (PM) sensors, namely SDS011, Prana Air, and SPS30, for measuring PM2.5 and PM10 levels is evaluated against a standard reference Aeroqual Series-500. The test setup was exposed to PM concentrations ranging from 30 µg/cm3 to 600 µg/cm3 . The results were based on 1 min, 15 min, 30 min, and 1 hr average readings. The experiments were carried out in indoor as well as outdoor environments. The comparative evaluation was performed before and after calibration. The performance of these sensors is evaluated in terms of coefficient of determination (R2 ), coefficient of variation (Cv), and root mean square error (RMSE). A real-time Air Quality Index (AQI) estimation technique using images and weather sensors on Indian roads is also presented. A mixture of image features, i.e., traffic density, visibility, and sensor features, i.e., temperature and humidity, were used to predict the AQI. Object detection and localizationbased Deep Learning (DL) method and image processing techniques were used to extract image features. At the same time, an ML model was trained on those features to estimate the AQI. For this experiment, a dataset containing 5048 images was collected over four months from September-December 2021 using AQSense device that was developed in International Institute of Information Technology (IIIT), Hyderabad, and co-located AQI values across different seasons were collected by driving on the roads of Hyderabad city in India. The experimental results report an overall accuracy of 82% for AQI prediction. A few challenges faced during the measurement campaign are also discussed.