Abstract

The last few decades have witnessed tremendous growth in the sensors industry, spurred by advancements in electronics, nanoscience, biotechnology and computer science. They are employed for a wide variety of applications and they possess the potential to play a pivotal role in biochemical applications. Among the various sensing modalities being explored, we have Radio Frequency based sensing. It studies the interaction between EM waves and matter and uses this interaction for qualitative and quantitative analysis. RF sensors offer a reduction in process complexity, speed, cost, portability and ease of integration with existing technology. In this work, Radio Frequency (RF) based sensing technique is applied for different biochemical applications. In particular, emphasis is given to combating two significant challenges of RF sensing, namely, specificity and reusability. In addition to that, the use of RF sensing in new sub-domains is explored, which would enable the development of a portable biochemical sensing platform. The sensors put into use in this work are Interdigitated Capacitor based sensor and Microstrip Patch based sensor. This sensor is reusable and can be put into use for diverse applications. Reusability of the sensor implies that it can not only be used several times, but the same sensor can detect different sets of analytes with equal efficiency. We have performed both computational and experimental studies on surfactants, nanoparticles and biomolecules using RF sensing. Surfactants are essentially surface-active agents that are amphiphilic. Above a specific concentration called the Critical Micelle Concentration (CMC), they self assemble to form micelles. The estimation of CMC is vital for all the varied applications of surfactants. The first application develops a fast, facile, efficient method for CMC detection that works well for all classes of surfactants. With their plethora of inherent advantages, Nanoparticles play vital roles in the biochemical and biomedical domains. Their study is essential to utilize them for diverse applications efficiently. There is a growing need to develop a methodology that performs preliminary characterization of nanoparticles in the absence of sophisticated laboratory facilities. In the second application, we study nanoparticles of various morphology and arrangements and characterize them. Our third application deals with developing a biomolecule detection platform that is rapid, portable, efficient and reusable. Gold nanoparticles are functionalized to improve the reusability and specificity aspects of conventional RF sensors. Further, the Biotin-Streptavidin binding is chosen to demonstrate the methodology

Abstract

The popularity of egocentric cameras and their always-on nature has lead to the abundance of daylong first-person videos. Because of the extreme shake and highly redundant nature, these videos are difficult to watch from beginning to end and often require summarization tools for their efficient consumption. However, traditional summarization techniques developed for static surveillance videos, or highly curated sports videos and movies are, either, not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. In this work, we present a novel unsupervised reinforcement learning technique to generate video summaries from day long egocentric videos. Our approach can be adapted to generate summaries of various lengths making it possible to view even one minute summaries of one’s entire day. The technique can also be adapted to various rewards, such as distinctiveness, indicativeness of the summary. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state of the art. Quantitative comparison on the benchmark Disney dataset shows that our method achieves significant improvement in Relaxed F-Score (RFS) (32.56 vs. 19.21) and BLEU score (12.12 vs. 10.64). Finally, we show that our technique can be applied for summarizing traditional, short, hand-held videos as well, where we improve the state of the art F-score on benchmark SumMe and TVSum datasets from 41.4 to 45.6 and 57.6 to 59.1 respectively

Abstract

Random Number Generators (RNGs) are the solution for cryptographic applications to enhance hardware security. These RNGs should have three specific properties unpredictability, aperiodic, and good statistical criteria. This brief presents a True Random Number Generator (TRNG) based on Ring oscillators' jitter with MLFSR. The MLFSR is augmented with a set of prime primitive polynomials, Boolean, and nonlinear functions to attain a non-linear, unpredictable, and extended sequence period. Paper mainly focused on achieving high randomness by integrating the TRNG with a new promising crypto engine Keccak as a post-processing block, leading to more extensive security in data transfer, encryption keys, data authenticity of ICs, and IoT based applications. The TRNG design is coded in Verilog HDL and implemented on the FPGA Zed board. Hashing is performed with a throughput of 2.4Gbps at 100MHz either with RNG data or SHA data. The HRNG (Hardware Random Number Generator) is an integrated block of TRNG and hashing along with the IP cores of memory, clock Generator, and Integrated Logic Analyzer, implemented on the FPGA hardware. Evaluated the randomness of the generated non-deterministic bit streams (10Mb) using the NIST 800-22 & Diehard test suite and successfully passed.

Abstract

The launch of level 5 autonomous driving is imminent which would result in removal of controls like steering wheel, acceleration and brakes. This warrants the need for a better system to direct the self-driving agent to perform maneuvers if the passenger wishes to do so. Hence, this thesis proposes a set of solutions tackle the problem of vision and language navigation in an autonomous driving setting which has been broken down into multiple sub problems. The first problem we attempt to solve, we present a simple baseline for visual grounding for autonomous driving which outperforms the state of the art methods, while retaining minimal design choices. The framework minimizes the cross-entropy loss over the cosine distance between multiple image ROI features with a text embedding (representing the given sentence/phrase). We use pre-trained networks for obtaining the initial embeddings and learn a transformation layer on top of the text embedding.We perform experiments on the Talk2Car dataset and achieve 68.7% AP50 accuracy, improving upon the previous state of the art by 8.6%. Humans have a natural ability to effortlessly comprehend linguistic commands such as “park next to the yellow sedan” and instinctively know which region of the road the vehicle should navigate. Extending this ability to autonomous vehicles is the next step towards creating fully autonomous agents that respond and act according to human commands where knowing only the location of the object of interest in the scene is not enough. To this end, a novel task of Referring Navigable Regions (RNR), i.e., grounding regions of interest for navigation based on the linguistic command is proposed. RNR is different from Referring Image Segmentation (RIS), which focuses on grounding an object referred to by the natural language expression instead of grounding a navigable region. For example, for a command “park next to the yellow sedan,” RIS will aim to segment the referred sedan, and RNR aims to segment the suggested parking region on the road. We introduce a new dataset, Talk2Car-RegSeg, which extends the existing Talk2car dataset with segmentation masks for the regions described by the linguistic commands. We benchmark the proposed dataset using a novel transformer-based architecture. We present extensive ablations and show superior performance over baselines on multiple evaluation metrics. A downstream path planner generating trajectories based on RNR outputs confirms the efficacy of the proposed framework.

Abstract

Dialogue systems help us to access information and services more effectively. This thesis presents the work done on developing Interactive Question Answering Systems, a type of dialogue systems, for English and Telugu. The domains chosen for English was Hyderabad MMTS enquiry and for Telugu it was Hyderabad Tourism. Both the systems take input in the form of text and the output is also text. To be able to study a wider range of user behaviour both the systems are developed for a different domain and language. The MMTS Question Answering system follow the frame based approach. This model can handle spelling errors and can also deal with insufficient information in the user query. In this system, the data is stored using SQL databases. We convert the user query to a proper SQL query and extract required information from the database. We tried to port the above model into Telugu language for Tourism domain but the process failed because of a lack of proper language processing tools for Telugu language. So we had to design a new approach. The approach used for Telugu can be extended to other resource poor Indian languages as well. The dialogue model consists of two parts namely Data Management and Query Processing. Data Management deals with storing the data in a particular format which helps in easy and quick retrieval of requested information. Query Processing deals with producing a relevant system response for a user query. Our model can handle code-mixed queries which are very common in Indian languages and also handles context which is a major challenge in dialogue systems. It also handles spelling mistakes and a few grammatical errors. The model is domain and language independent. Using this model, we developed a system for Telugu language for ’Tourist places of Hyderabad’ domain. As there is no automated evaluation tool available for dialogue systems, we went for human evaluation of our system. Five people evaluated our system and the results are reported in the thesis. In this thesis, we have also made an attempt to extract adjacency pairs from conversational units with multiple utterances from each user using dialogue acts. Adjacency pairs help in automatic extraction of dialogue structure. In dealing with minimal information in utterances, this becomes useful. In the machine learning algorithms that we were using for Natural language Generation, we relied on conversational units where the speakers have uttered only one sentence in their turn as training data, This restricts the data, and a good amount of manual work goes into converting these conversations units to adjacency pairs. To ease this process, we have tried to automate the process of extracting adjacency pairs from the data using dialogue acts.

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, domestic service robots, and more, the need for privacy of individuals interacting with such systems has become evident. We aim to investigate the quality of solutions that claim to protect the reward functions that contain sensitive personal information, from being exploited. Optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. Differential Privacy (DP) - a popular technique used for computational privacy, empowers us to provide mathematically quantifiable privacy guarantees for queries made to “privatised” algorithms. Our work introduces the reward reconstruction attack that tries to reverse engineer the reward from a private policy using Inverse RL algorithms. This attack is the key part of the novel Privacy-Aware Inverse RL Analysis (PRIL) framework. We assume that the entire training process and the optimal policy is released publicly. We study the set of existing differentially-private RL policies derived from various RL algorithms including Value Iteration, Deep Q Networks (DQNs), and Vanilla Proximal Policy Optimization (PPO). We propose the new PRIL analysis framework, that performs reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the novel reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using an Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using the PRIL framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on multiple instances of OpenAI Gym environments of varying complexities, specifically the discrete state-space Frozen Lake domain and the continuous state-space Mountain Car domain, employing a different Linear Programming based Inverse RL method for each domain. From our experiments, we observe that there is no indication of the policy improving at reconstructing the reward with a relaxation in the budget - thus rendering all strategies ineffective at providing true reward privacy. Based on the analysis performed, we infer that there exists a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We quantify the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards. We calculate the distance between the original reward and the reconstructed reward via various distance metrics. The larger the reward distance, the better the private agent’s ability to protect the sensitive reward function using which it was trained. We present the trends in test-time performance utility of private agents upon varying the level of privacy guaranteed (by altering the privacy budget during training). We also formally discover the differentially-private Bellman Update algorithm along with its proof of sensitivity. This is one of the first few works in this sub-field, and we hope that it is pursued with more rigour in the future.

Abstract

To execute meaningful real-world tasks, robots must perceive and understand the objects that surround them. Manipulation, autonomous driving, and human-robot interaction are just a few examples where robots must reason about previously unseen instances of objects. We present DRACO, a method for Dense Reconstruction And Canonicalization of Object shape from one or more RGB images. Canonical shape reconstruction—estimating 3D object shape in a coordinate space canonicalized for scale, rotation, and translation parameters—is an emerging paradigm that holds promise for a multitude of robotic applications. Prior approaches either rely on painstakingly gathered dense 3D supervision, or produce only sparse canonical representations, limiting real-world applicability. DRACO performs dense canonicalization using only weak supervision in the form of camera poses and semantic keypoints at train time. During inference, DRACO predicts dense object-centric depth maps in a canonical coordinatespace, solely using one or more RGB images of an object. Extensive experiments on canonical shape reconstruction and pose estimation show that DRACO is competitive or superior to fullysupervised methods. This thesis delves into understanding objects to capture their pose and shape from a single image. Canonicalization is the operation of mapping an input representation to an invariant representation with respect to an operation. This input representation can be a point cloud of an object. For the scope of this thesis, we fix this operation to be a 3D transformation. Lets take an example of an object, say a car. By finding a canonical frame for this car, we have its 3D reconstruction in a frame that is invariant to any 3D transformation. This is important for tasks such as registration, feature propagation and annotation. Using this invariant 3D pose we can register different poses of the car to a single consistent frame. This single consistent frame across multiple cars is called the canonical frame. Humans have an inherent understanding of this canonical frame. For instance, we know that the headlights of the car are towards its forward direction. This forward direction is with respect to our mental canonical frame and this allows us to estimate the pose of the car with respect to our view. canonicalization – the process of mapping an object instances to a category-level container – has emerged as a useful tool for category-level understanding. We can extend datasets,automatically annotate for parts, register and manipulate, if we can map every object to the canonical orientation. Our method, DRACO, estimates a canonical frame by predicting NOCS maps and reconstructs the object of interest in using a monocular image. NOCS maps are 2D images that store the location of every pixel with respect to a canonical frame that is invariant to transformation. Previous methods use pain-stakingly collected datasets to learn a dense canonical frame or weakly-supervise canonicalization for sparse keypoints. Our method uses weak-supervision in the form of semantic keypoints and camera relative motion to learn an object’s reconstruction and canonical frame which performs on par or better against fullysupervised methods. This enables estimating canonical frame for in-the-wild objects that enables canonicalizing them. Extensive experiments on our new large-scale mixed reality dataset (20K images/category) demonstrate on par shape reconstruction and canonicalization quality compared to state-of-the-art fully-supervised approaches, and weakly-supervised baselines that strictly use more information. This thesis introduces concepts of single-view depth estimation for reconstructing objects using a monocular image and delves into canonicalization in a weakly-supervised manner.

Abstract

Developers perform a wide variety of tasks during software development and maintenance. The most common tasks are documentation, bug fixing, enhancing capabilities, removing obsolete capabilities, and optimizations. Each task varies in complexity and effort. Developers spend significant time acquiring the software system’s knowledge. Specifically, they find comprehension of collected information laborious and often repetitive. This difficulty is due to the dispersed information across multiple artifacts such as source code, unit tests, bug reports, API documentation, commit history, and developer discussions. The comprehension effort gets confounded as the number of artifacts increases. Also, complete comprehension is difficult as software systems grow in size and get more distributed. Due to this, developers have a fragmented or incomplete understanding of the software system. To overcome this, they adopt tactical strategies to familiarize themselves with the program. They either read code, add trace statements, use debuggers, establish and test hypotheses, or observe the behavior. Due to this, the changes are tedious and are of lower quality. So, supporting developers to acquire task-specific knowledge rapidly is essential to develop and maintain a software system effectively. In this context, we analyze the developers’ information needs, propose a model and developed a technique to help faster code comprehension. Developers choose to read software code. They do so due to a lack of time and face minimum barriers. Developers read code on a need-basis without any plan – opportunistic. They choose either case-based or scavenging approaches [86]. Large-scale studies have observed that developers read code despite the availability of modern tools and Integrated Development Environment (IDE) [91, 125]. While code reading is prevalent among developers, the effectiveness of approaches followed by them is unknown. So, we study the effectiveness of comprehension by reading code in the context of reusing an external module, which is a common developer task. Reuse is a complex task, but developers are still required to integrate an external module rapidly. Observing developers undertake the reuse task is difficult due to the unplanned nature of the task. So, analyzing the historical bug reports and commit history helps us reason about the effectiveness of code reading practice. We observed reuse of the JMapViewer module in two independently managed opensource software applications – Mobile Atlas Creator and JTileDownloader. Based on the bug reports and large commits, we conjecture that comprehension was limited. It was to the extent required to accomplish the integration. We observed many inconsistencies, emergent behavior, conflicting scenarios, and fragility in the software structure. From this study, we infer that developers limit knowledge acquisition in an opportunistic task. Furthermore, we also conclude that the issue is due to the information overload they face when reading code. There is a growing interest in understanding developers’ tasks and how they comprehend code. Nevertheless, little is known about developers’ information needs when reading code. For example, How do developers locate concepts in a new program? What information do they seek to assess the impact of a change? What knowledge about the system do they need to triage bugs? How do they decide where to start for root cause analysis? When working with code, developers are constantly engaged in reasoning about the choices, implications, purpose, relevance, and expectations. In doing so, they assess a code fragment’s relevancy to the task. Typically, a comprehension task beings with an information-seeking activity. We find the current developer tool-sets fall short in assisting developers in reducing the information overhead they face during comprehension. When developers read a code, they create rich mental models of the program structure. However, these models are volatile and have no formal methods to record and share among developers. As there are no records, developers repeatedly read the same code and rebuild the mental model again. Having to repeat this across multiple source code is tedious, more so if the code is unfamiliar. Recently, researchers are examining large software projects using Mining Software Repositories (MSR) techniques to address a variety of Software Engineering questions. Overwhelmingly, recent research studies have focused on what can happen or why it happens. However, when reading code, questions related to what is happening are essential. So, we explored how to assist a developer in reading code by identifying and extracting task-specific information from software artifacts. We base our approach on the program comprehension model and the program navigation models [23, 111, 114]. These models suggested that developers use beacons or cues to establish the rele

Abstract

Text recognition has been an active field in computer vision even before the beginning of the deep learning era. Due to the varied applications of recognition models, the research area has been classified into diverse categories based on the domain of the data used. Optical character recognition (OCR) is focused on scanned documents, whereas images with natural scenes and much complex backgrounds fall into the category of scene text recognition. Scene text recognition has become an exciting area of research due to the complexities and difficulties such as complex backgrounds, improper illumination, distorted images with noise, inconsistent usage of fonts and font sizes that are not usually horizontally aligned. Such cases make the task of scene text recognition more complicated and challenging. In recent years, we have observed the rise of deep learning. Subsequently, there has been an incremental growth in the recognition algorithms and datasets available for training and testing purposes. This surge has caused the performance of recognizing text in natural scenes to rise above the baseline models that were previously trained using hand-crafted features. Latin texts were the center of attention in most of these works and did not profoundly investigate the field of scene text recognition for non-Latin languages. Upon scrutiny, we observe that the performance of the current best recognition models has reached above 90% over scene text benchmark datasets. However, these recognition models do not perform as well on non-Latin languages as they did on Latin (or English) datasets. This striking difference in the performances over different languages is a rising concern among the researchers focusing on lowresource languages, and it is indeed the motivation behind our work. Scene text recognition in low-resource non-Latin languages is difficult and challenging due to the inherent complex scripts, multiple writing systems, various fonts and orientations. Despite such differences, we can also achieve Latin (English) text-like performance for low-resource non-Latin languages. In this thesis, we look at all the parameters involved in the process of text recognition and determine the importance of those parameters through thorough experiments. We use synthetic data for controlled experiments where we test the parameters as mentioned earlier in an isolated fashion to effectively identify the catalysts of text recognition. We analyse the complexity of the scripts via these synthetic data experiments. We present the results of our experiments on two baseline models, CRNN and STAR-Net models, on available datasets to ensure generalisability. In addition to this, we also propose an error correction module for correcting the labels by utilizing the training data of real test datasets. To further improve the results on real test datasets, we propose transfer learning from English to exploit the abundant data that is available for learning. We show that the transfer from English is not feasible, and it actually lowers the performance of the individual language models. Due to the failure of English transfer experiments, we shift our focus onto just the Indian languages and examine the characteristics of each language via character n-gram plots, visual features like vowel signs, conjunct characters and other word statistics. They also share a resemblance to each other concerning certain other factors. We then propose to apply transfer learning across languages to enhance the performance of the language models. We depict the improvement on real datasets because of the transfers among Indian languages that are visually closer or sometimes better than the individual models. The transfers among languages prove to be much more profitable than transfers from English. We comprehend the significance of the variety and number of fonts during data generation via synthetic data experiments on English test datasets. Synthetic data embodies various fonts to ensure diversity of data to create robust recognition systems. In order to strengthen data diversity, we incorporate over 500 Hindi fonts (including Unicode and non-Unicode fonts) into the synthetic data for improved performance on Hindi real test datasets. We also manifest the process to utilize and incorporate nonUnicode fonts of Indian languages into the training process error-free. In addition to these fonts, we make specific changes to encompass an augmentation pipeline that adds to the diversity of data. We utilize more than nine augmentation techniques to boost the performance of Hindi STR systems. We achieved significant improvements over previous works with our evaluations over natural settings. Through our experiments, we set new benchmark accuracies for STR on Hindi, Telugu, and Malayalam languages from the IIIT-ILST dataset by gaining 6%, 5%, and 2% gains in Word Recognition Rates (WRRs) compared to previous works. Similarly, we also ach

Abstract

Many populous countries including India are burdened with a considerable backlog of legal cases. Development of automated systems that could process legal documents and augment and help legal practitioners can mitigate this. However, there is a dearth of high-quality corpora that is needed to develop such data-driven systems. The problem gets even more pronounced in the case of low resource languages such as Hindi. Additionally one of the most common and time sensitive cases handled by the courts are bail cases. In this Thesis, we first introduce the Hindi Legal Documents Corpus (HLDC), a corpus of more than 900K legal documents in Hindi. Documents are cleaned and structured to enable the development of downstream applications. We then introduce and tackle the task of bail prediction. We select the bail cases from our HLDC corpus and further extract the facts and arguments and judge’s summary. We experiment with a battery of models and propose a Multi-Task Learning (MTL) based model for the same. Our MTL model uses summarization as an auxiliary task along with bail prediction as the main task. The intermediate summarisation step is a novel introduction which serves dual purposes. First, it reduces the document size without compromising on the information. Since many transformer models have constraint on the input length, sending a summarised version of the documents allows us to overcome this barrier. Second, it builds towards explainable legal NLP systems as it allows us to identify salient sentences. This Thesis lays the foundation for research in Legal NLP for Hindi court documents. The multitude of legal NLP tasks and challenges are indicative of the need for further research in this area.