Abstract

Immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) are among the fasted growing and fascinated technology today. As the name suggests, these technologies promise to provide users with a much better experience using immersive head-mounted displays. Medicine, culture, education, and architecture are some areas that have already taken advantage of this technology. Popular video conferencing platforms such as Microsoft Teams, Zoom, and Google Meet is working to improve user experience by allowing users to use their digital avatars. Nevertheless, they lack immersiveness and realism. What if we extend these applications to virtual reality platforms so people can feel lively talking to each other? Integrating virtual reality platforms in collaborative spaces such as virtual telepresence systems have become quite popular after globalization since it enables multiple users to share the same virtual environment, thus mimicking real-life face-to-face interactions. For a better immersive experience in virtual telepresence/communication systems, it is essential to recover the entire face, including the portion masked by the headsets (e.g., Head-Mounted Displays abbreviated as HMDs). Several methods have been proposed in the literature that deal with this problem in various forms, such as HMD removal and face inpainting. Despite some remarkable explorations, none of these methods promises to provide usable results as expected in virtual reality platforms. Addressing these challenges in the real-world deployment of AR/VR-based applications draws emerging attention. Considering the existing limitations and usability of previous solutions, we explore various research challenges and propose a practical approach to facial de-occlusion/HMD removal for virtual telepresence systems. This thesis is well-documented to motivate and introduce the audience to various research challenges in facial de-occlusion, familiarizing them with existing solutions and their inapplicability in our problem domain, followed by the idea and formulation of our proposed solution to tackle the problem. With this view, the first chapter lays the outline of this thesis. In the second chapter, we propose a method for facial de-occlusion and discuss the importance of personalized facial de-occlusion methods in enhancing the sense of realism in virtual environments. The third chapter talks about the refinement of to previously proposed network in improving the reconstruction of the eye region. Last but not least, the final chapter briefly discusses the existing face datasets in face reconstruction and inpainting, followed by an overview of the dataset we collected for this work, from acquisition to making it usable for training deep learning models. In addition, we also attempt to extend image-based facial de-occlusion to video frames using off-the-shelf approaches, briefly explained in Appendix A.

Abstract

Air quality monitoring and analysis of a given location can be challenging without being able to determine the factors influencing its behavior. Stationary air pollution monitoring systems mostly ac- count for temporal variations in few locations due to the traditional choice of sparse deployment. In addition, the data from these monitoring stations may not be easily accessible. Internet of things (IoT) has allowed deployment of cost-efficient networks of smart devices with an ease in connectivity to the internet enabling easier air pollution monitoring. This thesis mainly proposes two test cases that highlight prominent factors that deeply influence the pollution pattern in a location. In the first test case, a temporal variation analysis is performed on two networks of pollution monitoring systems to understand the effects of implementing a nationwide lock- down during the COVID-19 pandemic. The first network comprises of the costly and bulky air quality monitoring stations of Central Pollution Control Board (CPCB) sparsely deployed in Hyderabad city and the second network comprises of the smaller and low-cost IoT nodes densely deployed inside the IIIT-H campus. Differential analyses were conducted on the data to understand the effects of lockdown on Particulate Matter (PM) levels by factoring in the yearly and seasonal trends, followed by Welch’s t- test to check whether the PM values have changed w.r.t. values in pre-lockdown period. Lastly, Pearson pair-wise correlation coefficient is estimated between PM and temperature values to show the effect of temperature changes on the PM values irrespective of lockdown. This test case established the effects of human-centric factors such as vehicular and industrial emissions, commercial activities and commo- tion on ambient PM variations as a sudden drop in the anthropogenic activities during the nationwide lockdown has caused a decline in PM levels. In the second test case, a novel methodology to analyze a moving object database is proposed by performing a case study on mobile IoT data collecting Particulate Matter across a road stretch of India, and look into the neighboring spatial and anthropogenic factors such as human activities, settlement patterns and vegetation profile corresponding to each geo-location of the PM data. The thematic inter- actions of spatial and anthropogenic factors of each location with the corresponding ambient PM levels resulted in a factor-based data structure that highlights the PM distribution mapped to each factor. This case study not only enabled to showcase the influence of spatial and anthropogenic factors of a location in influencing its ambient PM levels, but also provided a use case for handling mobile object databases, a challenging issue prevalent amidst the GIS community. Both the test cases effectively demonstrated the contribution of different surrounding factors in affecting the PM values, thus providing a gateway for accurately modeling the ambient air quality data based on key parameters.

Abstract

Supervised methods in deep learning approaches hinge on the assumption that the training and testing data are sampled from the same distribution. However, this is not the case in most realistic scenarios and thus leads to poor performance when these models are deployed in domains with different data distribution than the training set. Unsupervised Domain Adaptation (UDA) tackles the problem of adapting the data distributions between a labelled source domain and an unlabeled target domain. In contrast, Semi-supervised Domain Adaptation (SSDA) assumes a partially labelled target domain, a more realistic scenario in many computer vision tasks. Domain randomization is a popular approach wherein models are trained on synthetically generated data. With complete control over the synthetic data generation process, domain randomization introduces randomness in various properties of the object as well as the scene. Similar to data augmentation techniques in deep learning, the hope here is to introduce invariance for different non-causal features of the data and even nudge the model towards learning the causal correlations for the task at hand. In this thesis, we explore and study various approaches to domain adaptation. First, we present the image level domain adaptation methods, which use image-level manipulation or transformations to achieve domain invariance. We first analyze the domain randomization approach used in an object detection setting. For this, we use synthetically generated data and train a FasterRCNN model aimed at the object detection task. Domain randomization helps boost the performance of object detection models, and a model trained entirely on synthetic data outperforms the one trained on real data. With fine-tuning, the performance of the model trained on synthetically generated data increases drastically. Next, we extend the work on domain adaptation in the frequency domain, wherein the image level adaptation occurs in the frequency domain. To this end, we propose new combination strategies to combine the frequency components. We propose masking techniques which consider the frequency of the components in the combination process. Fourier domain adaptation techniques have seen some success in the image segmentation task from synthetic like GTA5 [1] and SYNTHIA [2] to realistic domains like Cityscapes [3]; however, these domains contain syntactically similar images. For the synthetic dataset that we use, we find that these frequency domain bases stylization methods do not improve performance over the domain randomization method. Finally, we present two novel methods for domain adaptation using feature level alignment. One of the primary challenges in SSDA is the skewed ratio between the number of labelled source and target samples, causing the model to be biased towards the source domain. Recent works in SSDA show that aligning only the labelled target samples with the source samples potentially leads to incomplete domain alignment of the target domain to the source domain. In our first approach, we train the source and target feature spaces separately. To ensure that the feature space of the target domain is generalized well, we employ semi-supervised methods to leverage the labelled and unlabeled samples. The Domain Adapters, which are parametric functions, are then trained to learn the feature level transformation from the target domain to the source domain. During inference, we use the target domain’s feature extractor and then pass the features to the respective Domain Adapter for that target-source pair. The transformed feature representation in the source domain is then fed to the source classifier. We show that keeping the feature extractors separate is advantageous if the domain gap between the source and the target domain is insignificant. Finally, we present SPI, which leverages contrastive losses to learn a semantically meaningful and a domain agnostic feature space using the supervised samples from both domains. To mitigate challenges caused by the skewed label ratio, we pseudo-label the unlabeled target samples by comparing their feature representation to those of the labelled samples from both the source and target domains. Furthermore, to increase the support of the target domain, these potentially noisy pseudo-labels are gradually injected into the labelled target dataset over the course of training. Specifically, we use a temperature scaled cosine similarity measure to assign a soft pseudo-label to the unlabeled target samples. Additionally, we compute an exponential moving average of the soft pseudo-labels for each unlabeled sample. These pseudo-labels are progressively injected (or removed) into (from) the labelled target dataset based on a confidence threshold to supplement the alignment of the source and target distributions. Finally, we use a supervised contrastive loss on the labelled and pseudo-labelled datasets to align the s

Abstract

Text Classification is a technique for assigning a set of categories to give text. A significant type of text classification is Sentiment Analysis. Sentiment Analysis is a technique in natural language processing to identify and quantify subjective states and information systematically. With the exponential increase in traffic over the internet, we have a lot of textual content available in different settings. This data provides an opportunity for efficient processing and application in solving problems across multiple domains. Few such critical applications include analysing public behaviour online and how they interact. Other is to analyse news content by different media houses that can cause public divide and polarisation due to its subjective nature. Firstly, we aim to create a novel dataset for bias detection in Hindi. We also explore classification tasks in a multi-modal setting to analyse and understand how information in multiple modalities affects classification outcomes. We further experiment with deep learning models to classify text where the language is more subtle and often unrealised for humans making its classification a tricky task. Recently, we have seen a rise in content from highly subjective news media outlets created to polarise and create a divide between people. This reporting can have far-reaching consequences, from orchestrating riots and public agitations to influencing the results of an election. This biased reporting makes its analysis and detection crucial. We aim to do it for the Hindi Language. As no data were available, we decided to develop our dataset. Our dataset consists of 1388 Hindi news articles and their headlines from various Hindi news media outlets, which were annotated for being biased towards, against, or neutral towards the BJP (Bhartiya Janata Party). Further, we explore a multi-modal classification task of identifying misogynous memes posted online. In this work, we also explore how the visual modality affects the classification task. We build an ensemble of deep learning text-transformers and vision-language transformers like UNITER and OSCAR and compare them to better understand the role of vision in multi-modal classification tasks. We also experiment with task-specific pretraining of our vision-language(VL) transformers. Lastly, we experiment with employing transformers trained on similar tasks on other similar datasets to explore transfer learning. Lastly, We employ deep-learning transformers to detect patronising and condescending language targeting vulnerable communities (e.g. refugees, homeless people, and low-income families). We experiment with techniques like focal-loss and weighted random sampling to deal with class imbalances and to improve our results. We also take up the task of classifying more fine-grained categories of condescending language.

Abstract

ML-powered document image analysis approaches can enable intelligent solutions in bringing handwritten information into the digital world. Two major components of handwriting understanding include handwritten text recognition(HTR) and handwritten search. The former task enables the conversion of handwritten text to digital format, whereas the latter task provides easy access to the handwritten information scattered across books, archives, manuscripts and so on. We primarily focus on both these problem statements in this thesis. Handwritten document image analysis for Indic scripts is still in its nascent stage compared to Latin scripts. For example, many commercial applications and open-source demonstrations are available for Latin scripts and there are hardly such known applications for Indic scripts. It is challenging to develop solutions for Indic scripts due to (i) variety of scripts within the Indic subcontinent, (ii) lack of huge annotated datasets and challenges in collecting data for multiple scripts, and (iii) inherent challenges in Indic scripts like inflections, joining multiple glyphs to form an akshara(equivalent to a character in Latin scripts). While challenging, it is also crucial to develop HTR and handwritten search approaches for Indic scripts. Therefore, this thesis is majorly focused on discussing approaches for Indic HTR and Indic handwritten search. In the last two decades, large digitization projects converted paper documents and ancient historical manuscripts into digital forms. However, they remain often inaccessible due to the unavailability of robust HTR solutions. Recognizing handwritten text is fundamental to any modern document analysis system. In recent years, efforts toward developing text recognition systems have advanced due to the success of deep neural networks and the availability of annotated datasets. This is especially true for Latin scripts. In this thesis, we discuss the standard text recognition pipeline that comprises various neural network modules. Then, we present a simple and effective way to improve the text recognition pipeline and training approach. We report the improvement from our approach on four benchmark datasets in Latin and Indic scripts. The existing state-of-the-art approaches for Latin HTR and Latin handwritten search are highly datadriven. Due to the lack of availability of large-scale data, developing Indic HTR and Indic handwritten search is challenging. Therefore, we release a collective Indic handwritten dataset with text images from majorly spoken 10 Indic scripts. We establish a high baseline for text recognition in prominent Indic scripts. Our recognition scheme follows the contemporary design principles from other recognition literature, and yields competitive results on English. We also explore the utility of pre-training for Indic HTRs. We hope our efforts will catalyze research and fuel applications related to handwritten document understanding in Indic scripts. Finally, we investigate the problem of handwritten search and retrieval for unlabeled collections. Handwritten search pipelines are needed in online platforms like E-libraries and digital archives. Such pipelines can efficiently search through handwritten collections and present relevant results, much like Google Search. With its ease of access and time-saving capability, the handwritten search application can prove to be valuable to many communities that study such historical documents. In this thesis, we present one such pipeline for handwritten search that performs retrieval on new and unseen collections. The proposed retrieval is not fine-tuned for specific writing styles or unknown vocabulary in the new collection. Therefore, it can be applied to new unlabeled collections.

Abstract

Linguistic diversity is one of the defining traits of the Indian subcontinent. With thousands of official languages and countless mother tongues, multilingualism is a way of life for the people. Today, as speech technology gains popularity due to the rising number of smart assistants, this abundance of natural languages is an obvious target for developing new speech systems. A majority of Indian languages happen to be low resource due to any of the following reasons that arise from the Indian context: 1) A lack of native speakers of the desired language; 2) A lack of recorded spontaneous speech for modelling natural speaker behaviour; 3) Poor quality of recordings due to external noise or minimal proper documentation; 4) Use of prolific code-switching in natural speech. Such limitations to the data that can be collected encourages us to look for alternatives that can both compensate for the lack of target data as well as properly utilise the feature of the languages available. Models to handle multilingualism are becoming increasingly complex, but they fail to be robust to the Indian context. Without any precise understanding of the way the models operate, it is impossible to adapt them to Indian language requirements. In addition, there is a lack of linguistic studies which have been conducted on Indian multilingualism in the context of developing speech systems. Due to this, even the requirements for making a system robust to the Indian context are largely unclear. The first step to adapting existing speech systems is understanding the models currently in use. Then, it is important to understand the context – in this case, to become familiar with the Indian context and the way Indian languages interact. This provides a theoretical basis which helps properly utilise existing resources with lower data reliance to manage Indian speech systems. This also forms the foundation for developing new frameworks which can either leverage the unique context of Indian languages or benefit from their shared features. In this thesis we have produced tools and results which will form the basis for future work on both multilingual and acquired language studies. We analyse the working of the joint acoustic model in multilingual automatic speech recognition systems, and determine which phonological properties of languages most affect model performance. We also demonstrate how the output patterns and accuracy are affected by the choice of language for each of the architectures that was used. Such results can be used to make appropriate choices for architectures based on the available data as well as the desired outputs for different applications, avoiding any unnecessary complications. We also study acquired language pronunciations, providing a dictionary for converting American English lexicons to Indian English as well as phonetic transformation rules which characterise Indian English relative to British Received Pronunciation. In our analysis, we also create a linguistic profile for 18 major Indian languages which describes the phonemes and phonological behaviours of Indian native languages which may be considered "universal" or regional. These tools can be adapted for handling variations in pronunciation or adapting accents to make existing systems more robust. Thus, the work in this thesis is a foundation for further studies of Indian language speech systems.

Abstract

The configurational sampling is central to characterize the equilibrium properties of complex molecular systems, but it remains a significant computational challenge. The conventional molecular dynamics simulations of limited duration often result in inadequate sampling and thus inaccurate equilibrium estimates. Replica exchange with non-equilibrium switches (RENS) is a collective variable-free computational technique to achieve extensive sampling from a sequence of equilibrium and non-equilibrium MD simulations without modifying the underlying potential energy surface of the system. Unlike the conventional replica exchange molecular dynamics (REMD) simulation, which demands a significant number of replicas for better accuracy, RENS employs non-equilibrium heating and cooling work simulations prior to configurational swaps to improve the acceptance probability for replica exchange. Here, we have implemented the RENS algorithm on four model systems and examined its performance against the conventional MD and REMD simulations. The desired equilibrium distributions were generated by RENS for all the model systems, whereas REMD and MD simulations could not do so due to inadequate sampling on the same timescales. The calculated work distributions from RENS obeyed the expected non-equilibrium fluctuation theorem. The results also demonstrated that the switching time of the nonequilibrium work simulations can be adaptively altered to fine-tune the acceptance probability and the reduced work of switching. The modular implementation of RENS algorithm not only enables us to readily extend it to multiple replicas, but also paves the way for extension to larger molecular systems in the future

Abstract

INTERNET is the most familiar word from everyone’s mouth nowadays. Children or adults of all ages are utilizing it in some way. Major change has been seen from communicating with pigeons and then moving on with telephone to voice-over IPs. Internet connects smaller businesses to larger ones and makes communication faster, reliable and accessible. Every second, the world creates an incredible amount of data, out of which the majority is unstructured. It has become pivotal to understand this vast amount of unstructured, raw data to get innovations and data-driven decisions. But the question that remains is what gives coherence to this raw data? The answer to this is through Natural Language Processing (NLP). In simple terms, NLP refers to the capability of computers to decipher human speech or, as it is spoken or written to text. NLP is a part of Artificial Intelligence that causes the systems to understand, grasp, and interpret how these systems communicate with human languages. It includes teaching computers how to make sense of the natural language humans use and to derive meaning from it. Many algorithms work on this aspect and train machines to understand natural language. Most of these algorithms are based on Machine learning, in which the computer program is trained using a large amount of data about the language to understand the language with some accuracy. This work will mainly focus on improving two NLP applications for Indian languages, viz Machine Translation and Text Classification. India is a multi-lingual country with thousands of languages and regional dialects spoken by its citizens. Some of them are lost, while others are still in use with significant speakers. Most of the Indian languages are related to each other due to shared ancestry or prolonged contact, and they share a lot of linguistic and structural features due to this relatedness. So, it is essential to utilize this languagerelatedness to build efficient NLP systems. We make use of relatedness between Indian languages and attempt to quantify their similarity. In the process of similarity score calculation, we examine different factors (script, string metric, size and type of parallel corpus, etc.) that can affect that similarity value. We also try to exploit language relatedness to train efficient multilingual NMT systems (one-to-many and many-to-one) and multilingual text classification systems for languages of the Indian subcontinent. We also focused on contributing to resource creation and annotation strategy for low-resource domains for Indian languages, further motivating researchers to curate new datasets for low-resource fields with little or no annotated data.

Abstract

Virtual assistants are trained to understand and process natural language, which in general are used to follow simple commands like making calls, controlling devices, setting alarms, etc. Customer chat supports in multiple applications also use virtual assistants for resolving simple issues or raising tickets. Human assistants come into action when there is a problem with understanding the issue or understanding the language. The aim here is to have the issues resolved quickly and efficiently by AI instead of a human. The virtual assistance in the past decade has improved vastly over the past decade. Widely used assistants like SIRI, Alexa, and Google Assistant were developed in the 2000s and had significant improvement in understanding natural language over time. Multiple virtual assistants have extensive uses in various fields like entertainment, essential tasks, voice search, etc. Virtual assistance in education is equally necessary as today’s world is much more dependent on online schooling. Learning is a crucial task in a student’s life, which needs assistance in many stages. We started by working on generating questions that will help in cross-checking key points from a passage of text. We worked on the grammatical part of the language Telugu which could give us better results than the state-of-art systems. We tried to create rules that capture the grammatical patterns for potential POS(Parts of Speech) tags that carry important information (instead of tags like auxiliaries) and suffixes of words in a Telugu sentence. In this work we present a learning assistant that tests one’s knowledge and gives feedback that helps a person learn at a faster pace. A learning assistant (based on an automated question generation) has extensive uses in education, information websites, self-assessment, FAQs, testing ML agents, research, etc. Multiple researchers and companies have worked on Virtual Assistance, but majorly in English. We built our learning assistant for the Telugu language to help with teaching in the mother tongue, which is the most efficient way of learning. Our system is built primarily based on Question Generation in Telugu. Many experiments have been conducted on Question Generation in English using multiple methods like rule/template based, supervised, semi supervised [39, 81, 31, 26, 1]. We have built the first hybrid machine learning and rule-based solution in Telugu, which proves efficient for short stories and short passages in children’s books. Our work covers the fundamental question forms with question types such as adjective, yes/no, adverb, verb, when, where, whose, quotative, and quantitative (how many/how much). We constructed rules for question generation using Part of Speech (POS) tags and Universal Dependency (UD) tags along with linguistic information of the surrounding relevant context of the word. Our system is primarily built on question generation in Telugu and is also capable of evaluating the user’s answers to the generated questions

Abstract

Semantic segmentation is an essential primitive in real-time systems such as autonomous navigation, which require processing at high frames per second. Hence for models to be practically applicable, it is essential that they have to be compact, fast as well as achieve high prediction accuracies. Previous research into semantic segmentation has focused on creating high-performance deep learning architectures. Most of the time, these best-performing models are complex, deep, have large processing times, and demand a significantly higher amount of processing capacity. Another relevant area of research is model compression, by which we can obtain lightweight models. Considering that there also have been works that produced mainstream light-weight semantic segmentation models at the expense of performance, we design models that bring a desirable balance between performance and latency. Specifically, methods and architectures that give a high performance while being real-time and working on resourceconstrained settings. We identify the redundancies in the existing state-of-the-art approaches and propose compact architecture family called ESSNet with accuracy comparable to the state-of-the-art while utilizing only a fraction of the space and computational power of those networks. We propose convolutional module designs with sparse coding theory as a premise and we also present two real-time encoder backbones employing our proposed modules. We empirically evaluate the efficacy of our proposed layers and compare them with existing approaches. Secondly, we explore the need for optimization during the training phase in the proposed models and present a novel training method called Gradual Grouping that results in models with improved implementation efficiency vs accuracy trade-offs. Additionally, we conduct extensive experiments by varying architecture hyper-parameters such as network depth, kernel sizes, dilation rates, split branching and additional context extraction modules. We also present a compact architecture using multi-branch separable convolutional layers with different dilation rates.