Abstract

Face Reenactment and Synthetic Talking Head works have been widely popular for creating realistic face animations by using a single image of a person. In light of the recent developments in processing facial features in images and videos, as well as the ability to create realistic talking heads, We are focusing on two promising applications. These applications include utilizing face reenactment for movie dubbing and compressing video calls where the primary object is a talking face. We propose a novel method to generate realistic talking head videos using audio and visual streams. We animate a source image by transferring head motion from a driving video using a dense motion field generated using learnable keypoints. We use audio as an additional input for high-quality lip sync, by helping the network to attend to the mouth region. We use additional priors using face segmentation and face mesh to preserve the structure of the reconstructed faces. Finally, we incorporate a carefully designed identity- aware generator module to get realistic quality of talking heads. The identity-aware generator takes the source image and the warped motion features as input to generate a high-quality output with fine-grained details. Our method produces state-of-the-art results and generalizes well to unseen faces, languages, and voices. We comprehensively evaluate our approach using multiple metrics and outperforming the current techniques both qualitative and quantitatively. Our work opens up several applications, including enabling low-bandwidth video calls and movie dubbing. We leverage the advancements in talking head generation to propose an end-to-end system for video call compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them. We use a state-of-the-art face reenactment network to detect keypoints in the non-pivot frames and transmit them to the receiver. A dense flow is then calculated to warp a pivot frame to reconstruct the non-pivot ones. Transmitting keypoints instead of full frames leads to significant compression. We propose a novel algorithm to adaptively select the best-suited pivot frames at regular intervals to provide a smooth experience. We also propose a frame-interpolater at the receiver’s end to improve the compression levels further. Finally, a face enhancement network improves reconstruction quality, significantly improving several aspects, like the sharpness of the generations. We evaluate our method both qualitatively and quantitatively on benchmark datasets and compare it with multiple compression techniques

Abstract

Multi-robot formation control has various applications in domains such as vehicle troops, platoons, payload transportation, and surveillance. Maintaining formation in a vehicle platoon requires designing a suitable control scheme that can tackle external disturbances and uncertain system parameters while maintaining a predefined safe distance between the robots. A crucial challenge in this context is dealing with the unknown/uncertain friction forces between wheels and the ground, which vary with changes in road surface, wear in tires, and speed of the vehicle. Although state-of-the-art adaptive controllers can handle a priori bounded uncertainties, they struggle with accurately modeling and identifying frictional forces, which are often state-dependent and cannot be a priori bounded. This thesis proposes a new adaptive sliding mode controller for wheeled mobile robot-based vehicle platoons that can handle the unknown and complex behavior of frictional forces without prior knowledge of their parameters and structures. The controller uses the adaptive sliding mode control techniques to regulate the platoon’s speed and maintain a predefined inter-robot distance, even in the presence of external disturbances and uncertain system parameters. This approach involves a two-stage process: first, the kinematic controller calculates the desired velocities based on the desired trajectory; and second, the dynamics model generates the commands to achieve the desired motion. By separating the kinematics and dynamics of the robot, this approach can simplify the control problem and allow for more efficient and robust control of the wheeled mobile robot. The stability of the closed-loop system employing both the proposed controllers are studied analytically via Lyapunov theory. The effectiveness of the proposed controller is demonstrated through simulations using Gazebo, a popular robot simulation tool. The simulations show that the proposed controller outperforms existing state-of-the-art controllers in terms of stability, convergence, and robustness to changes in frictional forces. The simulations also demonstrate the controller’s ability to maintain formation under various road conditions, including slopes, curves, and rough terrain.

Abstract

Natural Language Generation (NLG) focuses on the automatic generation of natural language text, which should ideally be coherent, fluent, and stylistically appropriate for a given communicative goal and target audience. The tasks in NLG are varied, whether it be summarization, headline generation, dialogue generation etc., and are also heavily dependent on the domain being considered. Recent research has focused on creating domain-specific datasets and developing domain-specific models to make NLP systems more suited to real-world applications. Training models on data specific to a domain has been observed to yield significantly better results across different domains, whether it be legal, financial or biomedical. However, we observe that there has not been much work done on problems in the tourism domain. The tourism industry is important for the benefits it brings and due to its role as a commercial activity that creates demand and growth for many more industries. Currently, there does not exist any standard benchmark for the evaluation of travel and tourism-specific data science tasks and models. To address this gap, we propose a benchmark, TOURISMNLG, of five natural language generation (NLG) tasks for the tourism domain and release corresponding datasets with standard train, validation and test splits. Moreover, as NLG systems are diversifying across languages, the datasets we create and the models we contribute are also multilingual in nature, which is beneficial for the tourism industry globally. Further, previously proposed data science solutions for tourism problems do not leverage the recent benefits of transfer learning. Thus, in this thesis, we also contribute the first rigorously pretrained mT5 and mBART model checkpoints for the tourism domain. The models have been pretrained on four tourismspecific datasets covering different aspects of tourism. Using these models, we present initial baseline results on the benchmark tasks, that indicate an improvement in performance as compared to the respective models without domain-specific pretraining. Additionally, we consider the problem of summarization for Indian languages, as described in the ILSUM (Indian Language SUMmarization) shared task, which focuses on summarising content from the news domain in three important Indian languages: Indian English, Hindi, and Gujarati. We evaluate the performance of existing pretrained models for the task and present our results and findings. We also talk about steps that must be taken to create high-quality summarization datasets for Indian languages. We hope that the contributions of this thesis will promote active research for natural language generation for travel and tourism, as well as other domain-specific and language-specific tasks and models

Abstract

Climate change is one of the most pressing issues of our time, and understanding the discourse surrounding it is crucial for effective communication and action. The discourse encircling climate change can circumscribe a wide range of perspectives, attitudes and opinions. It is essential to analyze this discourse to identify current challenges, road-maps, and systematic changes governments, organizations, and institutions require to combat the effects of climate change. Social media is an important platform for climate change discourse due to its widespread use and real-time nature. This makes it possible to analyze the discourse in near real-time, providing valuable insights into public opinions, attitudes, assess topic framing, event dependent attention to the issue and concerns surrounding climate change. We evaluate the contextual and social features that play key role in the coverage on different platforms. In this thesis, we focus on the fine-grained classification and stance detection of climate changerelated social media text surrounding the United Nations Climate Change Conference. We establish two corpora, ClimateStance and ClimateEng with the help of tweets posted during the 2019 United Nations Framework Convention on Climate Change with Intergovernmental Panel in Geneva. We comprehensively outline the dataset collection, pre-processing, annotation methodology, and dataset composition. We have put together a set of guidelines and specifications for creating expandable corpora ClimateEng, ClimateStance which is a collection of 3777 tweets that have been manually labeled with information about events, states, the categories they belong to, and their corresponding stance. We benchmark both datasets for climate change prevention stance detection and fine-grained classification using state-ofthe-art methods in text classification and experiments along with results are discussed in detail. In addition, we create a dataset called ClimateReddit, which is based on Reddit and includes 6262 comments from climate-change related subreddits. We perform semi-supervised learning on the corpus with pseudo-labelling and manually annotate 329 comments for the tasks of fine-grained classification and stance detection of climate-change data. We compare the results with the best-performing models for both tasks from the supervised experiments. Finally, we provide linguistic analysis of ClimateEng, ClimateStance and ClimateReddit using techniques such as part-of-speech tagging and named-entity recognition. Further, we extend our work in code-mixed setting. We collect Hindi and English code-mixed data from twitter during 2020 and construct a corpus of code-mixed Twitter data. We define the task of finegrained classification for the same and outline data-collection and annotation methodology for code-mix data.

Abstract

The Large Hadron Collider (LHC) experiment is searching for exotic beyond the Standard Model signal processes. These processes occur at extremely low rates compared to the Standard Model ones making it difficult to extract these signals effectively. Classical cut-based methods that rely heavily on hand-crafted strategies fail to perform well on these exotic signals. For the next stage of runs at the LHC, there is an urgent need to develop newer algorithms more capable of identifying these rare signals. Deep learning methods have recently gained traction to assist these searches due to their exceptional ability to analyse complex patterns and show immense potential to improve the sensitivity of experiments. In this thesis, I describe a few approaches to adapt and incorporate deep learning techniques into collider experiments. First, I present a case study of a phenomenological search for a hypothetical heavy quark, where I augment the standard search strategy with a simple deep-learning model to yield better sensitivity and reach. In the case study, I also discuss the caveats of such methods and point to a potential solution. Secondly, I present a simple adaptation of the deep learning training strategy to align it with physics goals. Specifically, I study modifications to the standard Cross Entropy loss and propose a new loss function, based on the Lovasz extension, to directly optimise the sensitivity metric (Z-score). Finally, I present a novel generative model based on the GAN (Generative Adversarial Framework) to generate particle jets. This model can quickly generate jets from different processes and assist jet studies.

Abstract

Metaverse is predicted to be the next game changer after artificial intelligence. In this work, we operationally define metaverse applications as virtual reality (VR) applications with possibly multi-modal, multi-user, and multi-sensory inputs. The idea of creating a VR space with a hardware-software co-design system can enhance user interaction. Additionally, the VR based systems interacting in the 3-dimensional environment can be beneficial in addressing the socio-economic reforms currently faced in society. The potential extent that VR systems can be used in medical, educational and defence capabilities is substantial. Such scale can be recorded only if components of VR based systems like head-mounted devices are compliant with standards, and are available to common masses. The road to widespread usage of VR systems is a challenge. Cost, technological capabilities, hardware-software co-design, learning curve, mass adaptation and usability analysis of VR systems are significant issues that need to be addressed. In this thesis, we focus on designing and implementing low-cost, customizable head-mounted devices for Virtual Reality applications. The cost aspect of the head-mounted device is addressed by making it application-specific (using degrees of freedom) and customizable (hardware and software development). As part of this work we studied currently available solutions and compared their sensing capabilities. The outcome resulted in choosing sensors based on the type of locomotion involved in the application and finalizing the hardware schematic. We further designed and fabricated the prototype to address VR applications with 3 degrees of freedom (roll, pitch and yaw). The application used to validate the prototype lies in Optometry domain. We propose a head-mounted device named CHORD (Customizable Head-mounted device for Occular disoRder Detection) to detect visual acuity disorders using VR applications. CHORD uses a standalone system with customizable VR scenes to detect myopic vision, astigmatism, colour vision deficiency and age-related macular degeneration. The product is validated using an empirical study with results showing an average accuracy of 85% in the tests conducted.

Abstract

Natural Language Processing (NLP) is a rapidly growing field focusing on the interaction between computers and human languages. It involves utilizing computational techniques to understand and generate natural language text. Several NLP tasks, such as question answering, text summarization, and machine translation, are widely researched for many languages, including Indian languages. Indian languages are resource-scarce and have distinctive characteristics posing different challenges for NLP. Recent advancements in NLP have helped in the development of models and techniques that are specific to Indian languages. However, for Telugu, a south Indian language, a lot of research is still needed to improve the performance of several NLP systems. Progress of such systems will benefit the huge Telugu-speaking community worldwide to communicate and access information in Telugu through various NLP applications. This motivates us to develop essential NLP resources and systems for Telugu. Firstly, the thesis provides an overview of the fundamental concepts and techniques used in NLP. Then we approach three specific NLP tasks: paraphrasing, question answering, and spelling correction. For these tasks, we address the problem of resource scarcity and then present the techniques to create the data for such low-resource languages. In this thesis, we have presented paraphrasing, question-answering, and spelling correction resources for the Telugu language. For paraphrasing, we presented two manually created and annotated corpora of size 1544 and 10000+ samples, respectively. We have also discussed the necessity for manual intervention while creating such resources. We have introduced a Telugu Question Answering Dataset - TeQuAD, with a size of 82k parallel triples. We also proposed the guidelines and methodologies that can be followed to create a Question Answering dataset for low-resource languages. We presented a Spell correction system for the Telugu language with the help of a synthetically created dataset.

Abstract

The applications of shared autonomy or human-robot interaction are growing rapidly in the field of autonomous robotics. Assisting human beings in dynamically changing environments in urban areas is still an active area of research. In crowded scenarios, in a structured environment such as public places with occlusions and dynamic obstacles, moving vehicles, people and so on - is the most critical part of this challenge. And, our work focuses on developing effective control strategies using model predictive control (MPC) because it is best known for handling such uncertainty and complex system dynamics relatively easily. While the extensive use of data-driven techniques using machine learning has become the de facto solution today, the underlying physics, the model of a system, and its behaviour are neces- sary to develop control laws. We first design an innovative MPC controller for a social person follower that can move safely around humans. We further incorporate motion-planning, target-tracking, and so- cial norms into a single holistic framework, being the first of its kind on a differential drive-wheeled mobile robot. To develop this robust person following behavior, we also employ path prediction using LSTM (Long-Short Term Memory) a type of recurrent neural network for supervised learning. This allows us for out-of-sight tracking and natural anticipation of a person’s future state. We also develop a local-map-based early relocation (ER) strategy that can reduce oscillations in the path, maintain the field of view (FOV) for long-term indoor navigation. Thus, we move beyond trivial person following to anticipating future visit locations and following them in the present. Overall, a non-linear MPC-based control law is designed using an online optimization problem with constraints on both kinematics and dynamics, as well as social norms of safety around humans. We implement these using 2D simulations in Matlab, and in Python to test the controller performance, runtime analysis, and error analysis. We show that the MPC framework can run in real-time with an adequate margin for adapting to changing human movement patterns, and agile enough for its changing movement speeds.

Abstract

In the research field of Artificial Neural Networks, there are ongoing efforts to address the issue of noise corruption in patterns. Various approaches and techniques have been explored to suppress noise and improve the accuracy of pattern classification. One of the research themes in this direction is the theory of Support Vector Machines [52] (SVM). SVMs aim to find an optimal hyperplane that maximizes the margin between classes, effectively reducing the impact of noise on classification. This approach formulates the problem of noise suppression as a quadratic programming problem. Hopfield Associative Memory [5] is another research area that has been investigated for noise suppression. The concept of null vectors of perceptron’s, which refers to vectors in the null space of the perceptron's transformation matrix, is introduced and utilized to suppress noise. By studying the null vectors of Extreme Machine Learning [49], researchers have explored their potential for noise suppression in pattern classification tasks. In addition to these approaches, the idea of stacking associative memories has been utilized to develop Hopfield neural networks [5] capable of suppressing noise in different dimensions. This includes 1-D, 2-D, and 3-D Hopfield neural networks that are designed to effectively handle noise corruption in patterns of various dimensions. Furthermore, to further enhance classification accuracy by suppressing noise, a real-valued Hopfield neural network based on a novel model of artificial neuron called the ceiling neuron has been proposed and studied. This innovation aims to improve noise suppression capabilities and overall performance in pattern classification tasks. Overall, this thesis explores and innovates different approaches to suppress noise in pattern classification using various neural network models, such as Support Vector Machines, Hopfield Associative Memory, and real-valued Hopfield neural networks based on ceiling neurons. These efforts contribute to the ongoing research in noise suppression techniques and aim to improve the accuracy and reliability of pattern classification systems.

Abstract

Distributed data analytics engines are designed to process and analyze large data sets in a distributed environment. The architecture of these engines is based on two key components: a distributed file system and a distributed computing framework. In this thesis, we consider the following two problems, coded data rebalancing in distributed file systems and coded distributed computing. For the data rebalancing problem, we consider replication-based distributed storage systems in which each node stores the same quantum of data and each data bit stored has the same replication factor across the nodes. Such systems are referred to as balanced distributed databases. When existing nodes leave or new nodes are added to this system, the balanced nature of the database is lost, either due to the reduction in the replication factor, or the non-uniformity of the storage at the nodes. This triggers a rebalancing algorithm, that exchanges data between the nodes so that the balance of the database is reinstated. The goal is then to design rebalancing schemes with minimal communication load. In a recent work [1] by Krishnan et al., coded transmissions were used to rebalance a carefully designed distributed database from a node removal or addition. These coded rebalancing schemes have optimal communication load, however, require the file-size to be at least exponential in the system parameters. In this work, we consider a cyclic balanced database (where data is cyclically placed in the system nodes) and present coded rebalancing schemes for node removal and addition in such a database. These databases (and the associated rebalancing schemes) require the file-size to be only cubic in the number of nodes in the system. We bound the advantage of our node removal rebalancing scheme over the uncoded scheme, and show that our scheme has a smaller communication load. In the node addition scenario, the rebalancing scheme presented is a simple uncoded scheme, which we show has optimal load. For the distributed computing problem, we consider the widely popular MapReduce distributed computing framework, which consists of three phases, namely Map, Shuffle, and Reduce. In previous works by Li et al. [2, 3], an optimal coded distributed computing scheme has been presented. This optimal scheme however requires very high file complexity, i.e., exponential in the number of servers K. To address this issue, low complexity distributed computing schemes using binary matrices derived from combinatorial designs have been presented in [4, 5]. In our work, we use similar principles to construct a distributed computing scheme via subspace designs, the q-analogs of combinatorial designs. While the scheme requires low file complexity, it has a higher communication load, when compared to the optimal scheme with equivalent parameters. Also, it is primarily useful for large local storage scenarios. Moreover, we also provide numerical comparisons with some existing baseline schemes.