Abstract

ML-powered document image analysis approaches can enable intelligent solutions in bringing handwritten information into the digital world. Two major components of handwriting understanding include handwritten text recognition(HTR) and handwritten search. The former task enables the conversion of handwritten text to digital format, whereas the latter task provides easy access to the handwritten information scattered across books, archives, manuscripts and so on. We primarily focus on both these problem statements in this thesis. Handwritten document image analysis for Indic scripts is still in its nascent stage compared to Latin scripts. For example, many commercial applications and open-source demonstrations are available for Latin scripts and there are hardly such known applications for Indic scripts. It is challenging to develop solutions for Indic scripts due to (i) variety of scripts within the Indic subcontinent, (ii) lack of huge annotated datasets and challenges in collecting data for multiple scripts, and (iii) inherent challenges in Indic scripts like inflections, joining multiple glyphs to form an akshara(equivalent to a character in Latin scripts). While challenging, it is also crucial to develop HTR and handwritten search approaches for Indic scripts. Therefore, this thesis is majorly focused on discussing approaches for Indic HTR and Indic handwritten search. In the last two decades, large digitization projects converted paper documents and ancient historical manuscripts into digital forms. However, they remain often inaccessible due to the unavailability of robust HTR solutions. Recognizing handwritten text is fundamental to any modern document analysis system. In recent years, efforts toward developing text recognition systems have advanced due to the success of deep neural networks and the availability of annotated datasets. This is especially true for Latin scripts. In this thesis, we discuss the standard text recognition pipeline that comprises various neural network modules. Then, we present a simple and effective way to improve the text recognition pipeline and training approach. We report the improvement from our approach on four benchmark datasets in Latin and Indic scripts. The existing state-of-the-art approaches for Latin HTR and Latin handwritten search are highly datadriven. Due to the lack of availability of large-scale data, developing Indic HTR and Indic handwritten search is challenging. Therefore, we release a collective Indic handwritten dataset with text images from majorly spoken 10 Indic scripts. We establish a high baseline for text recognition in prominent Indic scripts. Our recognition scheme follows the contemporary design principles from other recognition literature, and yields competitive results on English. We also explore the utility of pre-training for Indic HTRs. We hope our efforts will catalyze research and fuel applications related to handwritten document understanding in Indic scripts. Finally, we investigate the problem of handwritten search and retrieval for unlabeled collections. Handwritten search pipelines are needed in online platforms like E-libraries and digital archives. Such pipelines can efficiently search through handwritten collections and present relevant results, much like Google Search. With its ease of access and time-saving capability, the handwritten search application can prove to be valuable to many communities that study such historical documents. In this thesis, we present one such pipeline for handwritten search that performs retrieval on new and unseen collections. The proposed retrieval is not fine-tuned for specific writing styles or unknown vocabulary in the new collection. Therefore, it can be applied to new unlabeled collections.

Abstract

Linguistic diversity is one of the defining traits of the Indian subcontinent. With thousands of official languages and countless mother tongues, multilingualism is a way of life for the people. Today, as speech technology gains popularity due to the rising number of smart assistants, this abundance of natural languages is an obvious target for developing new speech systems. A majority of Indian languages happen to be low resource due to any of the following reasons that arise from the Indian context: 1) A lack of native speakers of the desired language; 2) A lack of recorded spontaneous speech for modelling natural speaker behaviour; 3) Poor quality of recordings due to external noise or minimal proper documentation; 4) Use of prolific code-switching in natural speech. Such limitations to the data that can be collected encourages us to look for alternatives that can both compensate for the lack of target data as well as properly utilise the feature of the languages available. Models to handle multilingualism are becoming increasingly complex, but they fail to be robust to the Indian context. Without any precise understanding of the way the models operate, it is impossible to adapt them to Indian language requirements. In addition, there is a lack of linguistic studies which have been conducted on Indian multilingualism in the context of developing speech systems. Due to this, even the requirements for making a system robust to the Indian context are largely unclear. The first step to adapting existing speech systems is understanding the models currently in use. Then, it is important to understand the context – in this case, to become familiar with the Indian context and the way Indian languages interact. This provides a theoretical basis which helps properly utilise existing resources with lower data reliance to manage Indian speech systems. This also forms the foundation for developing new frameworks which can either leverage the unique context of Indian languages or benefit from their shared features. In this thesis we have produced tools and results which will form the basis for future work on both multilingual and acquired language studies. We analyse the working of the joint acoustic model in multilingual automatic speech recognition systems, and determine which phonological properties of languages most affect model performance. We also demonstrate how the output patterns and accuracy are affected by the choice of language for each of the architectures that was used. Such results can be used to make appropriate choices for architectures based on the available data as well as the desired outputs for different applications, avoiding any unnecessary complications. We also study acquired language pronunciations, providing a dictionary for converting American English lexicons to Indian English as well as phonetic transformation rules which characterise Indian English relative to British Received Pronunciation. In our analysis, we also create a linguistic profile for 18 major Indian languages which describes the phonemes and phonological behaviours of Indian native languages which may be considered "universal" or regional. These tools can be adapted for handling variations in pronunciation or adapting accents to make existing systems more robust. Thus, the work in this thesis is a foundation for further studies of Indian language speech systems.

Abstract

The configurational sampling is central to characterize the equilibrium properties of complex molecular systems, but it remains a significant computational challenge. The conventional molecular dynamics simulations of limited duration often result in inadequate sampling and thus inaccurate equilibrium estimates. Replica exchange with non-equilibrium switches (RENS) is a collective variable-free computational technique to achieve extensive sampling from a sequence of equilibrium and non-equilibrium MD simulations without modifying the underlying potential energy surface of the system. Unlike the conventional replica exchange molecular dynamics (REMD) simulation, which demands a significant number of replicas for better accuracy, RENS employs non-equilibrium heating and cooling work simulations prior to configurational swaps to improve the acceptance probability for replica exchange. Here, we have implemented the RENS algorithm on four model systems and examined its performance against the conventional MD and REMD simulations. The desired equilibrium distributions were generated by RENS for all the model systems, whereas REMD and MD simulations could not do so due to inadequate sampling on the same timescales. The calculated work distributions from RENS obeyed the expected non-equilibrium fluctuation theorem. The results also demonstrated that the switching time of the nonequilibrium work simulations can be adaptively altered to fine-tune the acceptance probability and the reduced work of switching. The modular implementation of RENS algorithm not only enables us to readily extend it to multiple replicas, but also paves the way for extension to larger molecular systems in the future

Abstract

INTERNET is the most familiar word from everyone’s mouth nowadays. Children or adults of all ages are utilizing it in some way. Major change has been seen from communicating with pigeons and then moving on with telephone to voice-over IPs. Internet connects smaller businesses to larger ones and makes communication faster, reliable and accessible. Every second, the world creates an incredible amount of data, out of which the majority is unstructured. It has become pivotal to understand this vast amount of unstructured, raw data to get innovations and data-driven decisions. But the question that remains is what gives coherence to this raw data? The answer to this is through Natural Language Processing (NLP). In simple terms, NLP refers to the capability of computers to decipher human speech or, as it is spoken or written to text. NLP is a part of Artificial Intelligence that causes the systems to understand, grasp, and interpret how these systems communicate with human languages. It includes teaching computers how to make sense of the natural language humans use and to derive meaning from it. Many algorithms work on this aspect and train machines to understand natural language. Most of these algorithms are based on Machine learning, in which the computer program is trained using a large amount of data about the language to understand the language with some accuracy. This work will mainly focus on improving two NLP applications for Indian languages, viz Machine Translation and Text Classification. India is a multi-lingual country with thousands of languages and regional dialects spoken by its citizens. Some of them are lost, while others are still in use with significant speakers. Most of the Indian languages are related to each other due to shared ancestry or prolonged contact, and they share a lot of linguistic and structural features due to this relatedness. So, it is essential to utilize this languagerelatedness to build efficient NLP systems. We make use of relatedness between Indian languages and attempt to quantify their similarity. In the process of similarity score calculation, we examine different factors (script, string metric, size and type of parallel corpus, etc.) that can affect that similarity value. We also try to exploit language relatedness to train efficient multilingual NMT systems (one-to-many and many-to-one) and multilingual text classification systems for languages of the Indian subcontinent. We also focused on contributing to resource creation and annotation strategy for low-resource domains for Indian languages, further motivating researchers to curate new datasets for low-resource fields with little or no annotated data.

Abstract

Virtual assistants are trained to understand and process natural language, which in general are used to follow simple commands like making calls, controlling devices, setting alarms, etc. Customer chat supports in multiple applications also use virtual assistants for resolving simple issues or raising tickets. Human assistants come into action when there is a problem with understanding the issue or understanding the language. The aim here is to have the issues resolved quickly and efficiently by AI instead of a human. The virtual assistance in the past decade has improved vastly over the past decade. Widely used assistants like SIRI, Alexa, and Google Assistant were developed in the 2000s and had significant improvement in understanding natural language over time. Multiple virtual assistants have extensive uses in various fields like entertainment, essential tasks, voice search, etc. Virtual assistance in education is equally necessary as today’s world is much more dependent on online schooling. Learning is a crucial task in a student’s life, which needs assistance in many stages. We started by working on generating questions that will help in cross-checking key points from a passage of text. We worked on the grammatical part of the language Telugu which could give us better results than the state-of-art systems. We tried to create rules that capture the grammatical patterns for potential POS(Parts of Speech) tags that carry important information (instead of tags like auxiliaries) and suffixes of words in a Telugu sentence. In this work we present a learning assistant that tests one’s knowledge and gives feedback that helps a person learn at a faster pace. A learning assistant (based on an automated question generation) has extensive uses in education, information websites, self-assessment, FAQs, testing ML agents, research, etc. Multiple researchers and companies have worked on Virtual Assistance, but majorly in English. We built our learning assistant for the Telugu language to help with teaching in the mother tongue, which is the most efficient way of learning. Our system is built primarily based on Question Generation in Telugu. Many experiments have been conducted on Question Generation in English using multiple methods like rule/template based, supervised, semi supervised [39, 81, 31, 26, 1]. We have built the first hybrid machine learning and rule-based solution in Telugu, which proves efficient for short stories and short passages in children’s books. Our work covers the fundamental question forms with question types such as adjective, yes/no, adverb, verb, when, where, whose, quotative, and quantitative (how many/how much). We constructed rules for question generation using Part of Speech (POS) tags and Universal Dependency (UD) tags along with linguistic information of the surrounding relevant context of the word. Our system is primarily built on question generation in Telugu and is also capable of evaluating the user’s answers to the generated questions

Abstract

Semantic segmentation is an essential primitive in real-time systems such as autonomous navigation, which require processing at high frames per second. Hence for models to be practically applicable, it is essential that they have to be compact, fast as well as achieve high prediction accuracies. Previous research into semantic segmentation has focused on creating high-performance deep learning architectures. Most of the time, these best-performing models are complex, deep, have large processing times, and demand a significantly higher amount of processing capacity. Another relevant area of research is model compression, by which we can obtain lightweight models. Considering that there also have been works that produced mainstream light-weight semantic segmentation models at the expense of performance, we design models that bring a desirable balance between performance and latency. Specifically, methods and architectures that give a high performance while being real-time and working on resourceconstrained settings. We identify the redundancies in the existing state-of-the-art approaches and propose compact architecture family called ESSNet with accuracy comparable to the state-of-the-art while utilizing only a fraction of the space and computational power of those networks. We propose convolutional module designs with sparse coding theory as a premise and we also present two real-time encoder backbones employing our proposed modules. We empirically evaluate the efficacy of our proposed layers and compare them with existing approaches. Secondly, we explore the need for optimization during the training phase in the proposed models and present a novel training method called Gradual Grouping that results in models with improved implementation efficiency vs accuracy trade-offs. Additionally, we conduct extensive experiments by varying architecture hyper-parameters such as network depth, kernel sizes, dilation rates, split branching and additional context extraction modules. We also present a compact architecture using multi-branch separable convolutional layers with different dilation rates.

Abstract

Dance movement is not just an external body movement but internal too. It is both a physical and mental phenomena. As per Vaiśeṣika theory both these phenomena are different in nature. Mental phenomena, like thoughts, are sequential in nature, as it is in pure time. Physical phenomena, like body movement, are in time as well as space. If thoughts are to be put in front of the world, then they need body which utilizes space. How does the thought that is in mind finds expression in space using body? The link between the two, mind (mental) and body (physical), is manas (mind). Manas connects mental and physical phenomena. It helps us to understand the internal body which plays a major role in dance. It is through body that a dancer dances. Manas resides and moves in the internal body. Hence tracing the movement of manas is essential for the construction of accurate grammar of dance. We take a model of human as per Vaiśeṣika theory to interpret dance movement. According to this theory human being is made up of three basic elements, ātman (self), manas (mind) and śarīra (body). Ātman with the help of manas makes an effort (prayatna, ceṣṭā) on the muscles and joints of body to create dance. Manas is the link between ātman and śarīra. When we do a simple movement, manas actually goes to millions of places in the body. Even if we take gross [aggregate] movements of the manas moving in the body, we can model dance movement. Thus, understanding the movement of manas will help us to understand and derive the grammar of movement. We plan to specify syntax of movement, which integrates physical as well as mental aspect of the movement. Thus when the movement is executed as a conscious experience then it is noticed that certain structures emerge. These structure are the phenomenal concepts also known as movement words. These movement words are further classified giving the detail. These details are then used to reconstruct the movement. Keywords: Dance, Philosophy of mind, Phenomenology, Ontology, Bharatanāṭyam, Vaiśeṣika Darśana, Kinaesthetic Experience, Embodied Dance

Abstract

In India, energy consumption has increased by 50% during the last decade, and with rapid electrification, ever-growing population and increasing incomes, it is expected to grow by 2.3 times in the next two decades. Residential sector alone contributes about 24% of India’s total energy use and is set to go upwards in the coming years. Factors such as appliance ownership, occupant behaviour, thermal comfort levels and the climate conditions are the major contributors to household electricity consumption in India. In the Indian dwellings about 45% of total energy consumed is used to meet the requirement of thermal comfort. Air-Conditioners (AC) usage patterns influence the annual load curve significantly even though it is used only for a few months, for few hours every year. Due to the availability of the limited energy resources, meeting these increasing energy demands is a major challenge. In order to reduce these demands, a deeper understanding of energy consumption patterns and demand profiles along with the factors that are affecting the usage levels will help. This thesis proposes a well-organised data driven approach that is helpful towards future solutions to address this challenge. Based on the recent studies, a thorough understanding of residential energy consumption is required for designing and execution of energy saving measures. Whereas most of the studies in India focused on survey-based approach in identifying electricity usage patterns of residential sector, a very few studies were done using the captured real-time data of electricity and indoor temperature patterns. Therefore, there is a need to accurately collect and understand data on energy consumption at household level, which is of interest to end-customers, utilities, appliance manufacturers and most importantly policy makers. In this thesis, a total of 25 homes from Hyderabad, covering various built types (standalone houses/apartments), income levels (low/medium/high income), occupancy levels and ownership of AC units were monitored during three different seasons (summer, monsoon and winter) by installing electricity and indoor temperature loggers. A detailed survey questionnaire was also conducted to understand dwelling and occupancy characteristics. Using the monthly energy bills and the survey questionnaire data, a correlation matrix was developed to identify the factors that affect the household electricity consumption. It was found that, factors such as ‘connected load’ of home appliances, ‘number of ACs’, ‘number of rooms’ and ‘income levels’ had significant impact on the ‘Annual Energy Consumption (AEC)’ of a dwelling. Seasonal consumption trends were identified using the data from the loggers installed in the dwellings and it was observed that AC home consume nearly three times more than a non-AC home during summer, but the gap between them was minimal during monsoon and in winter. Further study was done on AC homes to determine the usage patterns of the primary-AC and the true contribution of it towards the total home energy. On an average, primary-AC was used for 6.2 hours a day with mean daily consumption of 6 kWh/day during summer. It was found that primary-AC was contributing about 44% towards the total home energy.

Abstract

Open-Vocabulary Keyword spotting solves the problem of spotting audio keywords in an utterance. The keyword set can incorporate keywords the system has seen and not seen during training, functioning as zero-shot keyword spotting. The traditional method involves using ASR to transcribe audio to text and search in the text space to spot keywords. Other methods include obtaining the posteriors from a Deep Neural Network (DNN) and using template matching algorithms to find similarities. Keyword spotting does not require transcribing the entire audio and focuses on detecting the specific words of interest. In this thesis, we aim to explore the usage of the Automatic Speech Recognition (ASR) system for Keyword Spotting. We demonstrate that the intermediate representation of ASR can be used for open vocabulary keyword spotting. With this, we show the effectiveness of using Connectionist Temporal Classification (CTC) loss for learning word embeddings for keyword spotting. We propose a novel method of using the CTC loss function with the traditional triplet loss function to learn word embeddings for keyword spotting on the TIMIT English language audio dataset. We show this method achieves an Average Precision (AP) of 0.843 over 344 words unseen by the model trained on the TIMIT dataset. In contrast, the Multi-View recurrent method that learns jointly on the text and acoustic embeddings achieves only 0.218 for out-of-vocabulary words. We propose a novel method to generalize our approach to Tamil, Vallader, and Hausa low-resource languages. Here we use transliteration to convert the Tamil language script to English such that the Tamil words sound similar written with English alphabets. The model predicts the transliterated text for input Tamil audio with CTC and triplet loss functions. We show that this method helps transfer the knowledge learned from high resource language English to low resource language Tamil. We further reduce the model size to make it work in a small footprint scenario like mobile phones. To this extent, we explore various knowledge distillation loss functions such as MSE, KL Divergence, and CosEmbedding loss functions. We observe that small-footprint ASR representation is competitive with knowledge distillation methods for small-footprint keyword spotting. This methodology makes use of existing ASR networks trained with massive datasets. It converts them into open vocabulary keyword spotting systems that can also be generalized to low-resource language.

Abstract

Automatic Speech Recognition (ASR) refers to the task of transcribing a given audio sample into its corresponding words. In the recent years, interest in improving the ASR systems has grown tremendously as the commercial viability of the devices such as the Google Assistant, Alexa, and Siri that use these systems has grown exponentially. As is true for most tasks that use Machine Learning-based systems, building robust ASR systems requires large amounts of annotated data. However, publicly available resources to build ASR systems for Indian languages is scarce. Therefore, in order to counter this scarcity, we explore how language relatedness can improve the performance of Language Models (LM) and thereby the ASR systems that use them. We begin by investigating how pooling the datasets of closely related Indian languages such as Marathi and Hindi and building a HMM-based Hybrid multilingual ASR system can help improve the quality of transcriptions as measured by Character Error Rate (CER) and Word Error Rate (WER). Having found little evidence that such models outperform baseline monolingual ASR systems, we look to increase the lexical overlap between the two languages by using subword-based language models instead of the default choice, i.e., word-based languages. We investigate the performance of such models under various conditions and show that they outperform word-based counterparts as measured by upto 26.35% CER. With the increased interest in End-to-End ASR systems, we shift our focus to how LMs play a role in them. Previous research has found that Acoustic Models (AM) of an Automatic Speech Recognition (ASR) system are susceptible to dialect variations within a language, thereby adversely affecting the ASR. To counter this, researchers have proposed to build a dialect-specific AM while keeping the LM constant for all the dialects. In this thesis, we start by studying how dialect mismatched LMs adversely affect the performance of the ASR by considering three regional Telugu dialects: Telangana, Rayalaseema, and Coastal Andhra. We show that dialect variations that surface in the form of a different lexicon, grammar, and occasionally semantics can significantly degrade the performance of the LM under mismatched conditions. Therefore, this degradation has an adverse effect on the ASR even when dialect-specific AM is used so much so that it actively hinders the performance of the ASR, i.e., performs worse than an ASR system with no LM. Next, we look to remove the need for building dialectspecific model by proposing a multi-dialect ASR system that outperforms the dialect-specific ones. We find that simple pooling of the data and training a multi-dialect ASR benefits the low-resource dialect the most (Rayalaseema). Subsequently, we incorporate dialect-specific information by adding a Dialect ID into the LM of an End-to-End ASR. We train these End-to-End ASR systems under a Multi-Task Learning (MTL) framework, where the the primary task is to transcribe the audio and the secondary task is to predict the dialect. Such a model outperforms naive multi-dialect ASRs by up to 8.24% in relative WER.