Abstract

The widespread popularity of over-parameterized deep neural networks (NNs) is backed by its ‘unreasonable’ performance on unseen data that is Independent, Identically Distributed with respect to the train data. The generalization of NNs cannot be reasoned with the traditional machine learning wisdom that the increase in the number of parameters leads to overfitting on train samples and subsequently reduced generalization. Various generalization measures have been proposed in recent times to explain the generalization property of deep learning networks. Despite some promising investigations, there is little consensus on how we can explain the generalization of NNs. Furthermore, the ability of neural networks to fit any train data for any random configuration of labels makes it more challenging to explain their generalization performance. Despite this ability to completely fit any given data, the neural network seems to be able to ‘cleverly’ learn a generalizing solution. We hypothesize that the ‘simple’ solution lies in a constrained subspace of the hypothesis space. We propose a constraint formulation of neural networks to close the generalization gap. We show that, through a principled constraint, we can achieve comparable train test performance for neural networks. We propose a way to constrain each output of the neural network as the convex combination of its inputs to ensure certain desirable geometry of the decision boundaries. This document covers two major aspects. Firstly, we show the improved generalization of neural networks using convex constraints. The second section goes beyond the IID setting and investigates the generalization of neural networks on the Out Of Distribution test sets.In the first section of the document, we investigate the constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and significantly reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi-layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOCNNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on six commonly used image classification datasets with three different neural network architectures.In the second section, we revisit the ability of networks to completely fit any given data and yet ‘cleverly’ learn the generalizing hypothesis from the large number of variances that can explain the train data. In accordance with concurrent findings, our explorations show that neural networks learn the most ‘low-lying’ variance in the data. They learn the features that easily correlate with the label and do no further exploration after finding such a solution. With this insight, we reinvent the need to understand the generalization of neural networks and improve them. We go beyond traditional IID and OOD evaluation benchmarks to further our understanding of learning in deep networks. Through our explorations, we give a possible explanation as to why neural networks can do well in certain benchmarks and why other inventive methods fail to give any consistent improvement over a simple neural network. Domain Generalization (DG) requires a model to learn a hypothesis from multiple distributions that generalizes to an unseen distribution. DG has been perceived as a front face of OOD generalization. We present empirical evidence to show that the primary reason for generalization in DG is the presence of multiple domains while training. Furthermore, we show that methods for generalization in IID are equally important for generalization in DG. Tailored methods fail to add performance gains in the Traditional DG (TDG) evaluation. Our experiments prompt if TDG has outlived its usefulness in evaluating OOD generalization? To further strengthen our investigation, we propose a novel evaluation strategy, ClassWise DG (CWDG), where for each class, we randomly select one of the domains and keep it aside for testing. We argue that this benchmarking is closer to human learning and relevant in real-world scenarios. Counter-intuitively, despite being exposed to all domains during training, CWDG is more challenging than TDG evaluation. While explaining the observations, our work makes a case for more fundamental analysis around the DG problem before exploring new ideas to tackle it K

Abstract

Agriculture Ontology for Telugu language (AOT) is a new ontology that we have created for plants and their diseases that models semantic relations between the entities in this domain. The ontology provides a formal way for the representation of entities and their mutual relations. Several methodologies that exploit numerous techniques of various fields (machine learning, text mining, knowledge representation and reasoning, information retrieval, and natural language processing) proposed to bring some level of automation in ontology acquisition from unstructured text. Question Answering Systems, Anaphora Resolution, Dialogue Systems are some of the realworld applications of ontology. This thesis presents a dataset of Ontology for the Telugu language and a Question Answering system which generates its responses by accessing the ontology as its database. We create an architecture for a Telugu agriculture-based Question Answering system using Ontology as the data resource in this thesis. We used Machine Learning techniques for Question Classification, Named Entity Recognition for Keyword Extraction, to generate a query for ontology used for answer generation. A Question Answering system’s question analysis is a critical component. We attempted to categorise Telugu queries both semantically and architecturally. Structural classification aids us in accurately parsing questions and, as a result, correctly identifying question arguments. Semantic classification helps in the mapping of question arguments to Ontologies and, as a result, the extraction of replies from the data resource. We give an architecture for an Interactive QA system as an extension to our QA system. For this we are extending our Ontology to accommodate a vector, which contains the symptoms of the disease in order to identify the disease when described by the farmer. We introduced a dialog manager to accommodate the new changes.

Abstract

We look at Quantum query complexity, a model which has been prominent in evaluating quantum algorithms, and other related complexity measures. There exist two popular lower bounding techniques on quantum query complexity: polynomial method and adversary method. They have been used to determine the quantum query complexity of a function in the case of most quantum algorithms. These, with the recent addition of spectral sensitivity act as concrete lower bounding techniques for quantum query complexity. We look at the quantum query complexity of Gap Majority, a partially symmetric Boolean function. We characterize ts query complexity using Adversary method. We use this result and show a lower bound on noisy randomised query complexity in terms of quantum query complexity. This also forms a base to explore the quantum query complexity of partially symmetric functions. We also give an explicit solution to the positive adversary method for total symmetric Boolean functions. This acts as a stronger evidence for the fact that all the aforementioned concrete lower bounding techniques give the same value for every total symmetric Boolean function [Nai21]. Lastly, we also see how big the separation could be between the block sensitivity and the sensitivity for any total symmetric Boolean function. Block sensitivity and sensitivity have been studied extensively in relation to query complexity. We show the biggest possible separation, even up to constants, and also show a function that achieves this thereby making the bound tight.

Abstract

With the ever-increasing size of documents on the web, tasks such as relevant information extraction with a query and similar document retrieval have become big challenges. Typical information retrieval and classification systems have been posed with questions like – Which documents match the given short text query? Which documents are similar to the given query document? Are the given two query documents possible duplicates? Researchers have been trying to represent documents and queries to answer these questions. Building frameworks for extracting such representations have been crucial to information retrieval research. Handcrafted textual features have been traditionally utilized to represent documents. Recently, advanced learning models such as transformers relying on attention mechanism have formed the basis of learning techniques proven to push the state of the art in language processing. These models have been built to sentence and document representations, however, their applications in extracting domainspecific inter-document information have been limited. Documents are found in various domains – medical, scientific journals, software engineering etc. Short documents are more prevalent in empirical medicine in the form of evidence-based systematic reviews and large scale software repository in the form of bug reports. The data entry design of both these systems has been primarily free-text with weak structure guidance. The large and growing number of published studies makes the task of identifying relevant studies in an unbiased way becomes complex and challenging. Similarly in the context of bug repository, with the rapid increase in features and codebase in large scale repositories, the task of triaging and extracting potential duplicates becomes time-consuming. Thus, this thesis aims at providing document retrieval and representation methods by enhancing query formulation and generating effective document embeddings. Firstly we devise a method of assisting Cochrane experts with writing systematic reviews of a study. In order to write a systematic review, researchers have to conduct a thorough search over the published articles relevant to a study. We conduct experiments that test the effectiveness of information retrieval methods to overcome the limitations of boolean search and title/abstract screening of documents. We provide a method using expansion techniques according to Relevance Feedback to retrieve relevant documents. The feedback mechanism uses term boosting over multiple iterations constructs grading feedback for ranking documents. Secondly, we focus on representing documents over large databases to improve retrieval performance while capturing semantic concept described in natural language. We present a novel method of representing a bug report by utilizing inter-document context. We use pre-trained transformers with a unique choice of triplet formulation to learn rich document embeddings. Experimentation with downstream tasks like duplicate bug detection and classification performs better than the existing methods in terms of recall rates.

Abstract

The question “what is art” is a forever open inquiry. The study proposes a formal ontological approach to theorizing art. This leads us to question the fascinating density of the realms of imagination and freedom of creation, which can only be accessed through chasing formal structures of the mind and self. Therefore, this inquiry deals with the folding of various categories of existence, in Art and in the specific act of painting. The research explores the Indian and Greco-European traditions of metaphysical thought to arrive at a framework for describing the artistic phenomenon of painting. While we finally commit to the frameworks given in the Kashmiri Saiva darshana and Spanda-shastra therein, the inquiry journeys into a brief history of formal thought that does not limit itself to regional interpretative silos. Therefore, in describing the universal Art phenomenon, we enable conversation across cultures around the same theme. This is a departure from the art-historical mode of theorizing art, while not dismissing the value of the sense of past in guiding such an exploration. We endeavor into observing the workings of the private mind in the work of art, rather than only the existential fact of its material presence. Traditions of the practice of observing mental functions, and strains from diverse formal thought traditions, enable opening the mysteries of the phenomena of art, especially painting. We attempt to clarify the formal states involved in the phenomenon of Painting.

Abstract

Disfluencies in speech are the abrupt breaks or hesitations that disrupt the forward flow of speech. The presence of disfluencies can adversely affect the performance of many speech based applications like Automatic Speech Recognition (ASR) systems. So, in order to make the systems perform effectively, disfluencies need to be identified in the speech signal. This task is referred to as disfluency detection. The works in this thesis focus on detecting multiple disfluencies in spontaneous lecture-mode speech. The IIITH-Indian English Disfluency (IIITH-IED) dataset is used for this purpose. The main focus of this thesis is to incorporate feature-level improvements in the disfluency detection pipeline to effectively model and capture the characteristics of disfluencies directly from speech. In order to improve frame-level disfluency detection, Shifted Delta Cepstral (SDC) coefficients, which have the ability to capture temporal variations in the speech signal effectively, are explored. Further different configurations of SDC features are experimented with for each type of disfluency to check the effect of varying the SDC parameters. Since, disfluencies also depend on the speaking style and can vary from speaker to speaker, speaker characteristics are also included in the disfluency detection systems by giving the d-vectors (speaker embeddings) as input along with cepstral and SDC features. An improvement in the detection accuracy is obtained for all the disfluencies, showing the importance of considering temporal variations and speaker characteristics in detecting disfluencies. In the literature, most works use frame-level or utterance-level analysis for detecting disfluencies. To analyse speech disfluencies at a supra-segmental level, syllable-level disfluency is performed here. Pre-linguistic Automatic syllabification is used to segment the input speech into perceptually distinct syllable-like units. Then, statistical prosody features are used to detect the presence of speech disfluencies in each syllable-like region. In order to add complementary acoustic information into the disfluency detection systems developed, a BiLSTM based feature extractor is used to get an acoustic representation of the syllable-like units. This acoustic representation is concatenated with the prosody features and it is found that the combined features provide an accuracy of 91.24% while distinguishing disfluent and non-disfluent speech syllables.

Abstract

Ancient historical handwritten documents were one of the earliest forms of written media and contributed to the most valuable cultural and natural heritage of many countries globally. They hold early written knowledge about subjects like science, medicine, Buddhist doctrines and astrology. India has the most extensive collection of manuscripts and studies have been conducted on their digitization for passing on their wealth of wisdom to future generations. Targeted annotation systems for automatic multi-region instance segmentation of their document images exist but with relatively inferior quality layout prediction compared to their human-annotated counterparts. Precise boundary annotations of image regions in historical document images are crucial for downstream applications like OCR, which rely on region-class semantics. Some document collections contain densely laid out, highly irregular, and overlapping multi-class region instances with a large range in aspect ratio. Addressing this, a web-based layout annotation and analytics system is proposed in this thesis. The system, called HInDoLA, features an intuitive annotation GUI, a graphical analytics dashboard, and interfaces with machine-learning-based intelligent modules on the backend. HInDoLA has successfully helped us create the first-ever large-scale dataset for layout parsing of Indic palm-leaf manuscripts named Indiscapes. Keeping the non-technical nature of domain experts in mind, the tool offers an interactive and relatively fast annotation process with the help of two modes, namely Fully Automatic mode and Semi-Supervised Intelligent Mode. We then discuss the semi-supervised approach superiority over fully automatic approaches for Historical document annotation. Fully automatic boundary estimation approaches tend to be data-intensive, cannot handle variable-sized images, and produce sub-optimal results for images mentioned above. BoundaryNet, a novel resizing-free approach for high-precision semi-automatic layout annotation, is another main contribution from this thesis. An attention-guided skip network first processes the variablesized user-selected region of interest. The network optimization is guided via Fast Marching distance maps to obtain a good quality initial-boundary estimate and an associated feature representation. These outputs are processed by a Residual Graph Convolution Network optimized using Hausdorff loss to get the final region boundary. A challenging image manuscript dataset demonstrates that BoundaryNet outperforms solid baselines and produces high-quality semantic region boundaries. Qualitatively, our approach generalizes across multiple document image datasets containing different script systems and layouts, all without additional fine-tuning.

Abstract

Tunnels are the most important components of infrastructure, as they play an important role in public transport, water transport, and hydroelectric power generation. The tunnel industry has long believed that tunnels are safe from any seismic activity. Recent rigorous damage to tunnels has proven the, unlike scenario. Several case histories raised the concern for the safety of underground structures. Examples include Chi-Chi Earthquake in Taiwan (1999), Wenchuan Earthquake in Sichuan (2008), Nepal Earthquake (2011, 2015) and Sikkim earthquake (2011). At this point, it will be prudent to understand the seismic performance of rock support systems while designing tunnels. The need for more power plants has led to significant amount of hydroelectric tunnelling work in the Himalayan region. The challenges for tunnelling in Himalayan region are enormous owing to the heterogeneous geology and the high seismic activity. The analysis and understanding of the prevalent rock mass in such areas are a challenge and necessary for the successful execution of tunnelling project. A tunnel construction typically consists of the ground support system and the tunnel lining construction. The effect of earthquake activity is measured only after there is a static design available. The main aim of this thesis is to provide a simplistic design approach as an effective mechanism for engineers to design tunnel support. This study explores the effects of earthquakes on the tunnel lining. The analyses are carried out to understand the effect of rockmass quality ‘Q’, size of tunnels, effect of discontinuities on the seismic performance of tunnel lining. In this thesis an attempt is made to improve the current design approach for the support design of tunnel using Q system and expected PGA values at the tunnel site. For this, case study of three hydropower tunnels from India has been selected: 1) (4111 MW) Vishnugarh Pipalkoti hydropower project (Uttarakhand), 2) 450 MW Shongtong Karcham hydroelectric project (Himachal Pradesh), and 3) 37.5 MW Parnai hydroelectric project (Jammu and Kashmir). Seismic parameters have been evaluated by carrying out probabilistic seismic hazard assessment (PSHA) for these specific sites. This study uses pseudo-static to understand tunnel behavior in static and seismic cases. It is found that axial force increases with rock mass quality Q and tunnel size. Three sedimentary rock mass classes are modelled with different geo-mechanical parameters determined empirically. The study used Phase2 version 8.0 numerical programme, which is frequently used to understand tunnel seismic response. In this study, a tunnel model with diameters of 6, 12, 18, and 24 m was assessed at a depth of 100 m surrounded by rock masses with Q values ranging from 1 to 30 . As a result of such an analysis, researchers and field engineers can account for a continuum's seismic response. A parametric study was conducted to assess the seismic activity of tunnels with discontinuities in a fragmented geological setting with high overburden. Determining the influence of joint orientation, spacing, friction, and stiffness on the dynamic forces developed in a liner requires numerous pseudo-static finite element analyses. The role of joint stiffness and shear strength in the tunnel response was found to be critical. Reduced joint dip angles improved tunnel stability, as observed in the field. For competent rocks with joints, neither seismicity nor joint orientation had an effect on tunneling. However, the weak rock mass deforms and is affected regardless of joint position. The results agreed with the Norwegian Rock Index System (Q) and Bhasin et al. (2006) earlier tests. It is necessary to modify the current Q-system tunnel design chart to use Qseismic instead of Qstatic to achieve the new support system. Barton (1984) advised halving the Q value. However, here it is suggested that Qseismic = Qstatic, where ‘’ is a coefficient mostly reliant on the PGA and the rock mass quality Q, respectively. It is hypothesized that ‘’ depends on other qualities of the rock mass, the size of the tunnel, and the parameters of the shotcrete liner. There is a strong correlation between tunnel deterioration and the Peak ground Acceleration (PGA). Increases in PGA put more strain on the tunnel's liner. There is a correlation developed between ‘’ and PGA for the three case study locations with varying seismic stress. For the optimistic design of the tunnel support system, the proposed research observation is also used to design a key tool for evaluating the support system for Himalayan tunnel projects.

Abstract

Action recognition is indispensable not only to the umbrella field of computer vision but in multitudes of allied fields such as video surveillance, human computer interaction, robotics and human robot interaction. Typically action recognition is performed over RGB videos, however, in recent years skeleton action recognition has also gained a lot of traction. Much of it is owed to the development of frugal motion capture systems, which enabled the curation of large scale skeleton action datasets. In this thesis, we focus on skeleton-based human action recognition. We begin our explorations with skeleton-action recognition in the wild by introducing Skeletics-152, a curated and 3-D pose-annotated subset of RGB videos sourced from Kinetics-700, a large-scale action dataset. We extend our study to include out-of-context actions by introducing Skeleton-Mimetics, a dataset derived from the recently introduced Mimetics dataset. We also introduce Metaphorics, a dataset with caption-style annotated YouTube videos of the popular social game Dumb Charades and interpretative dance performances. We benchmark state-of-the-art models on the NTU-120 dataset and provide multi-layered assessment of the results. The results from benchmarking the top performers of NTU-120 on the newly introduced datasets reveal the challenges and domain gap induced by actions in the wild. Overall, our work characterizes the strengths and limitations of existing approaches and datasets. Via the introduced datasets, our work enables new frontiers for human action recognition. When moving ahead from the traditional supervised recognition to the more challenging zero-shot recognition, the language component of the action name becomes important. Focusing on this, we introduce SynSE, a novel syntactically guided generative approach for Zero-Shot Learning (ZSL). Our end-to-end approach learns progressively refined generative embedding spaces constrained within and across the involved modalities (visual, language). The inter-modal constraints are defined between action sequence embedding and embeddings of Parts of Speech (PoS) tagged words in the corresponding action description. We deploy SynSE for the task of skeleton-based action sequence recognition. Our design choices enable SynSE to generalize compositionally, i.e., recognize sequences whose action descriptions contain words not encountered during training. We also extend our approach to the more challenging Generalized Zero-Shot Learning (GZSL) problem via a confidence-based gating mechanism. We are the first to present zero-shot skeleton action recognition results on the large scale NTU-60 and NTU-120 skeleton action datasets with multiple splits. Our results demonstrate SynSE’s state of the art performance in both ZSL and GZSL settings compared to strong baselines on the NTU-60 and NTU-120 datasets. 3-D virtual avatars are extensively used in gaming, educational animation and physical exercise coaching applications. The development of these visually interactive systems relies heavily of how humans perceive the actions performed by the 3-D avatars. To this end, we perform a short user-study to gain insights into the recognizability of human actions performed virtually. Our results reveal that actions performed by 3-D avatars are significantly easier to recognize as compared to those performed 3-D skeletons. Concrete actions, i.e actions which can only be performed using a fixed set of movements are recognized more quickly and accurately as compared to abstract actions. Overall, in this thesis we study various new-frontiers in skeleton action recognition by means of novel datasets and tasks. We drift from unimodal approaches to a multimodal setup by incorporating language in a unique syntactically aware fashion with hopes of utilizing similar ideas in more challenging problems like skeleton action generation.

Abstract

With a tremendous increase in the user population of the world wide web across the globe, the diversity amongst its users has also increased unpredictably. It is now more than ever, essential to provide accessibility and enhance universal inclusivity. Web accessibility enables equal access to information on the world wide web regardless of disability. It is imperative that we address the problem of web accessibility by leveraging the current trends in the state-of-the-art. In this dissertation, we present a systematic mapping study on web accessibility with an emphasis on the web annotation intervention. The aim of this study is to provide a structure to the body of knowledge and identify key knowledge gaps. We performed a systematic investigation of the publications in the areas of web accessibility in the context of web annotation. As a part of this study, we framed research questions, defined inclusion exclusion criteria and formulated a classification scheme. We then mapped and analysed the obtained set of research publications. From a preliminary set of 2018 articles, we selected a set of 55 for our study. The insights gained from this mapping study can be used to develop tools and formulate strategies for the enhancement of accessibility and overcome existing barriers. In this dissertation we focus on the human users and aid them in better knowledge formation. We suggest that users face accessibility challenges in gathering information, especially when it is in a different representation system than they are used to. We identify this as a semantic gap. To overcome this we propose a client-side, browser based information modification tool called Renarration UI. We focus on both the design and implementation aspects of this tool and validate it by conducting an empirical study (n=10). Results are encouraging and suggest that renarration of web pages has potential to address these information accessibility issues. We also propose another client-side, browser-based information augmentation tool called SeRena. This semantic renarration tool enhances the accessibility of the enduser by facilitating the end-user with relevant additional information. To accomplish this we develop an ontology and populate a Knowledge Base(KB) with instances of the ontology. We then semantically annotate the webpage to provide the end-user with additional information from the KB.