Abstract

In this work, we propose Spatio-Temporal Laban Features: a novel motion descriptor based on Laban Theory and provide a platform to document, analyse and annotate performed dance. Dance is an indispensable part of our life. It has accompanied social and religious events, ceremonies, rituals, celebrations, etc throughout our history. However, dance evolves, it is very susceptible to creative changes by practitioners. Being a part of our cultural heritage, especially classical and folk dance, it needs to be preserved to track it’s evolution through time. With the boom in popularity of media platforms, multimedia content has permeated virtually every aspect of our lives. The volume of usergenerated content on the web has skyrocketed, and unlike before, dance is also leaving behind more trail than ever. However, the majority of such content is undocumented primarily due to high costs associated with the time and the resources required annotate such data manually or through sensors. We leverage the advancements in Machine Learning to reduce these costs and automate motion indexing. We create a semantically searchable dance database with support for Labanotation generation, automatic annotation, and retrieval. Our approach is built on a thorough review of Laban Theory which is a very popular annotation system based on movement analysis. We apply a pose estimation module to retrieve pose and generate Labanotation over recorded videos. Labanotation provides us with a comprehensive annotation system that we use to generate our novel motion descriptor and it can be further exploited to build an ontology as well. It is also very relevant for the preservation and digitization of online resources, especially for dance cultural heritage, which has been our focus in this work. We built a semi-automatic annotation model that generates semantic annotations over a big video database. We combine two publicly available ballet datasets and test our model on it. High-level concepts such as ballet pose and steps are used to make the semantic library. These also act as descriptive meta-tags for any ballet video making the videos retrievable using a semantic text or even a video query. The final platform thus allows a user to generate and study the form of any dance sequence using Labanotation as well as search for particular steps to learn from a video database.

Abstract

The increasing popularity of semantic web has led to the creation and enrichment of ontologies and other linked datasets using automated algorithms. Alongside these algorithms, quality evaluation of the enriched ontologies is necessary to ensure reliability and reuse. Current quality evaluation approaches seek to evaluate ontologies in either syntactic (degree of following ontology development guidelines) or semantic (degree of semantic validity of enriched concepts/relations) aspects. Current syntactic quality evaluation approaches often use pre-defined rules and priorities for evaluation, which provide limited flexibility and customisability to the end user. Any extension to such works in an attempt to adapt it for a specific task or a different goal also requires significant programming, further posing hindrances in extensibility, especially for non-programming ontology engineers. to create or update rules at run time. Semantic approaches, on the other hand, which often involve crowdsourcing, suffer in output quality with increase in size and diversity of crowd validators. Most current approaches use naive majority voting, with some using spam-filtering techniques. While these methods are useful for eliminating spammers, the accuracy of these works are debatable in more general scenarios where only all validators have some degree of knowledge but only few are absolute experts regarding the ontology domain. In these scenarios, it would be useful to have an algorithm that grades validators according to their expertise and weights their decisions proportionally to their experience. This work thus seeks to propose a semi-automated framework that seeks to achieve both syntactic and semantic quality evaluation in an integrated framework. The proposed framework consists of: a) SynEvaluator and b) SemValidator for evaluating the syntactic and semantic aspects of ontology quality respectively. SynEvaluator uses a novel theoretical rule-creation framework that aims to improve customisability and flexibility for the end user. It does so by allowing for dynamic, task-specific creation and updation of syntactic rules at runtime, all without any need for programming. SemValidator uses crowdsourcing for semantic validation. It also uses a novel algorithm, TweetExpert, to estimate the Twitter-based expertise of validators for weighing their decisions as a means of quality control. The efficacy and validity of the framework is shown empirically on multiple ontologies.

Abstract

In machine learning, tasks like making predictions using a model and learning model parameters can often be formulated as optimization problems. The feasibility of using a machine learning model depends on the efficiency with which the corresponding optimization problems can be solved. As such, the area of machine learning throws up many challenges and interesting problems for research in the field of optimization. While in some cases, it is possible to directly apply off-the-shelf optimization methods for problems in machine learning, in many other cases, it becomes necessary to develop optimization algorithms that are tailor-made for specific problems. On the other hand, developing optimization algorithms for specific problem domains can itself be helped by machine learning techniques. Learning optimization algorithms from data can help relieve tedious effort required to develop optimization methods for new problem domains. The challenge here is to appropriately parameterize the space of algorithms for different optimization problems. In this context, we explore the interplay between the areas of optimization and machine learning and make contributions in specific problems of interest that lie in the overlap of these fields. We look into the optimization challenge presented by multi-class classification with a large number of output classes and propose a partial-linearization based approach that intuitively generalizes over several popular optimization algorithms applicable to this task. It is popular to use measures like average precision (AP) and normalized discounted cumulative gain (NDCG) to evaluate a model for a classification task or a ranked retrieval task, however, they are not as popularly used for learning the parameters of these models. This is because of the difficulty in optimizing these non-decomposable loss functions. We propose a useful characterization of a large class of such loss functions that we show are amenable to efficient optimization and present an algorithm that can be used to efficiently optimize this class of loss functions. On the other hand, we explore the possible application of machine learning techniques for developing optimization algorithms, specifically, for combinatorial optimization problems. A wide class of approximate algorithms for NP-hard combinatorial optimization problems solve a continuousrelaxation of the original problem and then round the fractional solution to obtain an approximate discrete solution. The rounding procedures involved in such methods are non-trivial and generally have to be ingeniously designed for each task. We propose a framework that allows the learning of such rounding procedures from unsupervised data. Such a learning based approach permits rapid development of algorithms for novel optimization problems.

Abstract

The present-day smart homes are proliferated with innumerable smart home devices. The community is accepting smart home devices. More than 70% of homes in North America have an IoT device. The growing number of smart home devices has led to its variegated usage patterns. Unlike mobile phones, IoT devices are usually static, have less computation power and are shared among the members and visitors of a house. Several studies have shown different usage patterns of smart devices based on the relationships among the people in the house. However, devices in a house can be used to cause dangerous consequences to the property or its members. Smart homes do not usually appear to be places where security and privacy could be compromised. We challenge this premise by proving that when individuals in a smart home have malicious intentions, they can cause maximum damage to the people and devices of the house. Though the issue is not commonly addressed, one can use smart home devices to hamper the security or privacy of the house members. Since those who get access to these devices are trusted, it is imperative to manage the rights of these users. A strict access control policy is required to grant only required privileges to the users of the devices and monitor their activities to prevent abuse. Our work aims to regulate the usage of devices in a shared multi-user, multi-device setup by laying out an access control system for the home. Designing such a system has been previously attempted. However, those works failed to acknowledge the practicality of every device being used differently by each user. We assign each user a different role for every device based on the user’s suitability with that of the device. Existing methods to manage privileges in smart home systems have not considered allocating privileges to users based on (i) the relationship of the user with the device, (ii) the location and risk of the device and (iii) the current environment. In this work, we take a multi-perspective view on the problem of sharing fine-grained privileges of IoT devices among multiple users in a smart home. We propose the concepts of user role (subset of privileges specific to each device), tasks and security levels (level for each privilege) to allot the proper privileges to users. We limit the exploitation of privileges assigned to legitimate insiders of the house. Thus, our work matches the aspirations of previous surveys on building a comprehensive access control system to manage privileges in a shared smart home. We deployed our solution as a tool on a python-based Flask server that receives requests from users in the home through a REST API.Administrators in a smart home are often overwhelmed by the number of options, controls and environments given to them to manage. Since devices are varied, managing a smart home becomes a non-trivial task. To aid administrators to manage the environment in a hassle-free manner, we have also designed Vigile. This system verifies whether the aspirations of privilege distribution to users in the smart home matches with the actual allocation of privileges.

Abstract

The use of local detectors and descriptors in typical computer vision pipelines work well until variations in viewpoint and appearance change become extreme. Past research in this area has typically focused on one of two approaches to this challenge: the use of projections into spaces more suitable for feature matching under extreme viewpoint changes, and attempting to learn features that are inherently more robust to viewpoint change. In this paper we present a novel framework that combines learning of invariant descriptors through data augmentation and orthographic viewpoint projection. We propose rotation-robust local descriptors, learnt through training data augmentation based on rotation homographies, and a correspondence ensemble technique that combines vanilla feature correspondences with those obtained through rotation-robust features. Using a range of benchmark datasets as well as contributing a new bespoke dataset for this research domain, we evaluate the effectiveness of the proposed approach on key tasks including pose estimation and visual place recognition. Our system outperforms a range of baseline and state-of-the-art techniques, including enabling higher levels of place recognition precision across opposing place viewpoints, and achieves practically-useful performance levels even under extreme viewpoint changes. We reduce pose estimation error by 86.72% relative to state of the art. In particular, our integration of Visual Place Recognition (VPR) with SLAM by leveraging the robustness of deep-learnt features and our homography-based extreme viewpoint invariance significantly boosts the performance of VPR, feature correspondence and pose graph sub-modules of the SLAM pipeline. We demonstrate a localization system capable of state-of-the-art performance despite perceptual aliasing and extreme 180-degree-rotated viewpoint change in a range of real-world and simulated experiments. Our system is able to achieve early loop closures that prevent significant drifts in SLAM trajectories.

Abstract

With the number of protein sequences increasing rapidly, it becomes imperative to have a basic idea of the function and structure of a protein before the three-dimensional structure becomes available. Protein-drug interactions play essential roles in many biological processes and therapeutics. Prediction of the active binding site of a protein helps discover and optimise these interactions leading to the design of better ligand molecules. The secondary structure provides clues to the shape that the protein can be expected to take. It tells us whether an amino acid belongs to a coil turn, alpha-helix or beta-sheet structure. Deep Learning is a class of machine learning algorithms that progressively uses multiple layers to extract higher-level features from raw input. Deep learning methods eliminate feature engineering for supervised learning tasks by translating the raw inputs into intermediate representations that capture the more abstract and composite information, removing redundancies in the original input. The rapid adoption and success of deep learning algorithms in various sections of structural biology beckon deep learning algorithms for accurate binding site detection and secondary structure prediction. Protein-drug interactions play essential roles in many biological processes and therapeutics. Prediction of the active binding site of a protein helps discover and optimise these interactions leading to the design of better ligand molecules. The tertiary structure of a protein determines the binding sites available to the drug molecule. To quickly and accurately predict the binding site from sequence alone without utilising the three-dimensional structure is challenging. In the first study, a Residual Neural Network (leveraging skip connections) [1] is implemented to predict a protein’s most active binding site. An Annotated Database of Druggable Binding Sites from the Protein DataBank, sc-PDB [2], is used for training the network. Features extracted from the Multiple Sequence Alignments (MSAs) of the protein generated using DeepMSA, such as Position-Specific Scoring Matrix (PSSM), Secondary Structure (SS3), and Relative Solvent Accessibility (RSA), are provided as input to the network. A weighted binary cross-entropy loss function is used to counter the substantial imbalance in the two classes of binding and non-binding residues. The network performs very well on single-chain proteins, providing a pocket that has good interactions with a ligand. Secondary structure predictions predict three classes: C (Coil Turn), H (Alpha Helix) and E(Beta Sheet). In the second study, a Transformer (based on the multi-attention mechanism) network is used to train on the TR4590 dataset, which contains 4590 proteins with a sequence similarity cut off 25% and Xray resolution better than 2.0A. Ten models are trained across 10-fold cross-validation, and a weighted ˚ cross-entropy loss function is used. The ten trained models are run on the test set containing 1199 sequences. The mean of the probabilities of each class is taken, and then the class with the maximum probability is considered the class to which the amino acid belongs. The model achieves an accuracy of 82.51%.

Abstract

Transmission Control Protocol(TCP) is one of the most widely deployed transport layer protocol around the world. Since TCP is designed for reliable packet transfer, whenever the packet transfer is deemed unsuccessful, TCP takes various steps to resolve it such as re-transmitting the packet, changing the packet sending rate, etc. Congestion in the network is one such reason for unsuccessful packet transfer. The subsequent packet retransmission reduces the goodput in the network. There have been many variations of TCP for appropriate congestion control. These variations manage the sending rate of the users based upon the feedback they get from the network. In like manner, there are congestion control methods that leverage the facilities of the Software Defined Network(SDN) framework and manage congestion from the controller. The global image of the network in the controller enables the network administrators to deploy rules to manage the occurrences of congestion. In this work, we aim to couple both of these approaches. We make use of the reactive approach of the TCP congestion control at the user’s TCP stack. This helps us to manage congestion as it is detected at the user. To proactively reduce the possibility of the occurrence of congestion, we use a SDN controller to detect a congestion event even before it happens. We provide a utility based framework for this joint proactive and reactive mechanisms for congestion control. We propose a system optimization function based upon [15] and show that our user’s and network controller’s congestion control methods optimize the system utility. We simulate our model and show that our model significantly improves the system utility when compared with other congestion control methods

Abstract

Humans have used different characteristics to identify each-other since early times. This practice of identification based on person-specific features called biometric traits has developed over time to use more sophisticated techniques and characteristics like fingerprints, irises and gait in order to improve the identification performance. Fingerprints due to their distinctiveness, persistence over time and ease of capture, have become of the most widely use biometric traits for identification. However, with this ever increasing dependence on fingerprint biometrics, it is very important to ensure the safety of these recognition system against potential attackers. One of the most common and successful ways to circumvent these systems is through the use of fake or artificial fingers synthesized using commonly available materials like silicon and clay to match the real fingerprint of any particular person. Most fingerprint recognition systems employ a spoof detection module to filter out these fake fingerprints. While they seem to work well in general, it is a well-established fact that spoof detectors are not able to identify spoof fingerprints synthesized using ”unseen” or ”novel” spoof materials, i.e, the materials which were not available during the training phase of the detector. While it is possible to synthesize a few fingers using the various available materials, present-day spoof detectors require a large amount of samples for their training, which is practically not feasible due to the high cost and high complexity of fabrication of spoof fingers. In this thesis, we propose a method for creating artificial fingerprint images using only a very limited number of artificial fingers created from a specific material. We train a style-transfer network using available spoof fingerprint images which learns to extract material properties from the image, and then for each material, uses the limited set of spoof fingerprint images to generate a huge dataset of artificial fingerprint images without actually fabricating spoof fingers. These artificial fingerprint images can then be utilised by the spoof detector for training. Through our experiments, we show that the use of these artificially generated spoof images for training can improve the performance of existing spoof detectors over unseen spoof materials. Another major limitation of present-day recognition systems is their high resource requirements. Most fingerprint recognition systems use a spoof detector as a separate system either in series or in parallel with a fingerprint matcher leading to very high memory and time requirements during inference. To overcome this limitation, we explore the relationship between these two tasks in order to develop a common module capable of performing both spoof detection and matching. Our experiments show a high level of correlation between the features extracted for spoof detection and matching. We propose a new joint model which achieves similar fingerprint spoof detection and matching performance on various datasets as current state-of-the-art methods while using 50% less time and 40% less memory, thus providing a significant advantage for recognition systems deployed on resource-constrained devices like mobile phones.

Abstract

A wide variety of information pertaining to human behaviour, ranging from connections on social networks to accounts of discrimination and abuse, is being produced on the internet. Predictive models built using this data can be valuable for various domains including computational social science, personalization, recommendation, and marketing. Creating such predictive models can be challenging owing to the fact that the data available on online media relevant to human behaviour may be unstructured, inadequate and/or highly imbalanced. In this thesis, we consider two contrasting instances of human behaviour: trust and discrimination, especially gender-based discrimination, i.e., sexism. We formulate a novel approach based on matrix factorization for predicting user-user trust by capitalizing on community memberships and devising a new way to model the homophily theory. With regard to sexism, we introduce the problem of multi-label categorization of accounts describing sexism of any kind(s) and propose neural network based solutions for it. The prediction of trust relations between users of social networks is critical for finding credible information. Inferring trust is challenging, since user-specified trust relations are highly sparse and power-law distributed. In this thesis, we explore utilizing community memberships for trust prediction in a principled manner. We also propose a novel method to model homophily that complements existing work and a way of capitalizing on helpfulness scores of users’ item ratings. To the best of our knowledge, this is the first work that mathematically formulates a hypothesis pertaining to community memberships for unsupervised trust prediction. We propose and model the hypothesis that a user is more (or as) likely to develop a trust relation within the user’s community than (or as) outside it. Unlike existing work, our approach for encoding homophily directly links user-user similarities with the pair-wise trust model. Our formulation for tapping rating helpfulness scores relates to the notion that users receiving low rating helpfulness scores should not receive high levels of trust on average. We derive mathematical factors that model our hypothesis relating community memberships to trust relations, the homophily effect, and the premise involving helpfulness scores. Along with an existing method, they are combined into chTrust, the proposed multi-faceted optimization framework. Our experiments on the standard Ciao and Epinions datasets show that in the majority of the settings involving the fraction of the trust relations used for training, whether all user pairs or only the low-degree ones are considered in the evaluation, and the dataset, the proposed framework outperforms all but one of the unsupervised trust prediction baselines considered; no proposed or baseline method performs the best across all settings.We also develop a supervised method for trust prediction that avoids any dependence on auxiliary information such as ratings or reviews. We propose combining representation learning with the inference of trust relations. Using only a small amount of binary user-user trust relations, our approach simultaneously learns embeddings for the users and a trust prediction model. Another human behaviour that we endeavour to analyze through predictive models is discrimination, especially the form of discrimination that is perpetrated on the basis of sex. Sexism, an injustice that gives rise to enormous suffering, manifests in blatant as well as subtle ways. In the wake of growing documentation of experiences of sexism on the web, the automatic categorization of accounts of sexism has the potential to assist social scientists and policy makers in studying and thereby countering sexism. The existing work on sexism classification has certain limitations in terms of the categories of sexism used and/or whether they can co-occur. To the best of our knowledge, this is the first work on the multi-label classification of sexism of any kind(s), and we contribute the largest dataset for sexism categorization. We also consider the related task of the classification of misogyny. While sexism classification is performed on textual accounts describing sexism suffered or observed, misogyny classification is carried out on tweets perpetrating misogyny. We devise a novel neural framework for classifying sexism and misogyny that can combine text representations obtained using models such as BERT with distributional and linguistic word embeddings using a flexible architecture involving recurrent components and optional convolutional ones. Further, we leverage unlabeled accounts of sexism to infuse domain-specific elements into our framework. In order to evaluate the versatility of our neural approach for tasks pertaining to sexism and misogyny, we experiment with adapting it for misogyny identification. For categorizing sexism, we investigate m

Abstract

Computer Aided Diagnosis (CAD) systems are developed to aid doctors and clinicians in diagnosis after interpreting and examining a medical image. CAD systems aids in performing the task more consistently. With the arrival of data-driven deep learning paradigm and availability of large amount of data in the medical domain, CAD systems are being developed to diagnose a large variety of diseases ranging from different types of cancers, heart and brain diseases, Alzheimer’s disease, and diabetic retinopathy, etc. These systems are highly competent in performing the task on which they are trained. Although such systems perform at par with the trained clinicians, they suffer from a limitation in that they are completely black box in nature and are trained only on image-level class labels. This poses a problem in deploying CAD systems as stand alone solutions for disease diagnosis. This is because decisions in the medical domain are about health of a patient and are well reasoned and backed by evidence, sometimes from multiple modalities. Hence, there is a critical need for CAD system’s decisions to be explainable. Restricting our focus to only image modality, a solution to design explainable CAD systems could be to train the system using both class labels and local annotations and derive the explanation in a fully supervised manner. However, getting these local annotations is very expensive, time-consuming, and infeasible in most circumstances. In this thesis we address this explainability and data scarcity problem and propose two different approaches towards the development of weakly supervised explainable CAD systems. Firstly, we aim to explain the classification decision by providing heatmaps denoting important regions of interest in the image, which helped the model make the prediction. In order to generate anatomically accurate heatmaps, we provide a mixed set of annotations to our model - class labels for the entire training set of images and rough localization of suspect regions for a smaller subset of images in the training set. The proposed approach is illustrated on two different disease classification tasks based on disparate image modalities - Diabetic macular edema (DME) classification from OCT slices and Breast Cancer detection from mammographic images. Good classification results are shown on public datasets, supplemented by explanations in the form of suspect regions; these are derived using just a third of images with local annotations, emphasizing the potential for generalisability of the proposed solution.