Abstract

Urea assisted denaturation of protein and RNA has been shown to be a valuable tool for studying the stabilities and the folding phenomena of these biomolecules. It has been shown that stacking interactions between nucleobases and urea are one of the driving forces of denaturation. In this study, the ability of urea to form unconventional stacking interactions with RNA bases is investigated by performing high level quantum calculations (RI-MP2/aug-cc-pVDZ level) on few thousands of model systems. Four systems were considered based on the RNA nucleobases (GUA, ADE, CYT, and URA) for the investigation. For each system, a set of models were designed to study the role of hetero-atoms/groups of the nucleobases on stacking interactions with urea moiety with respect to every possible pair. Furthermore, for each model, several plane parallel complexes were generated with urea on top of aromatic systems to exhaustively study all possible factors for urea-nucleobases stacking interactions. Energy decomposition analysis (EDA) and natural bond orbital (NBO) analysis were performed on stacked complexes. AIM analysis was done for the most stable system of each model for all systems to gain better insights on non-covalent stacking interactions. Dispersion component was found to be heavily stabilizing, while the EHF was found to be repulsive for all the four systems indicating lack of hydrogen bonding (HB) type interactions and presence of dispersion type interactions. Interactions of most stable complex for each nucleobase with Urea moeities were also studied to identify the importance of each part of Urea in these stacking interactions. Amide and Carbonyl group of urea molecule was found to play a major role in stacking interactions. We demonstrate that along with functional groups present on the nucleobases, the orientation of urea molecules plays a vital role in stabilizing the urea-nucleobase non-covalent interactions. The proposed study quantifies and provides a comprehensive theoretical description of urea nucleobase unconventional stacking interactions which helps to unravel urea driven RNA unfolding mechanism.

Abstract

Understanding speech in the absence of audio, from the visual perception of lip-motion can aid a variety of computer vision applications. System comprehending ‘silent speech presents a promising potential for low bandwidth video-calling, speech transmission in auditory noisy environment to aid for hearing impaired. While presenting numerous opportunities, it is highly difficult to model lips in silent speech video by observing lip-motion of speaker. Albeit developments in automatic-speech recognition (ASR) has yielded better audio-speech recognition systems in last two decades, in the presence of noise their performance drastically deteriorates. This calls for a computer vision solution to the speech under- standing problem. In this thesis, we present two solutions for modelling lips in silent speech videos. In the first part of the thesis, we propose a word-spotting solution for searching spoken keywords in silent lip-videos. In this work on visual speech recognition our contributions are twofold: we develop a pipeline for recognition-free retrieval, and show its performance against recognition-based retrieval on a large-scale dataset and another set of out-of-vocabulary words. 2) We introduce a query expansion technique using pseudo-relevant feedback and propose a novel re-ranking method based on maximizing the correlation between spatio-temporal landmarks of the query and the top retrieval candidates. The proposed pipeline improves baseline performance by over 35% for word-spotting task on one of the largest lipreading corpus. We demonstrate the robustness of our method through a series of experiments, by investigating domain invariance, out-of-vocabulary prediction and careful analysis of results on dataset. We also present the qualitative results showing success and failure cases. We finally show the application of our method by spotting words in an archaic speech video. In the second part of our work, we propose a lip-synchronization solution for ‘visually redubbing speech videos in a target language. Current methods of adapting a native speech video in foreign lan- guage either through placement of subtitle in the video, which distracts the viewer or through audio redubbing the video in the target language. This causes unsynchronized lip-motion of the speaker with respect to the redubbed audio, resulting in video appearing unnatural. In this work, we propose two lip synchronization methods: 1) cross-accent lip-synchronization for change in accent of the same language audio dubbing, and 2) cross-language lip-synchronization for speech videos dubbed in a differentlanguage. Since viseme remains the same in cross-accent dubbing, we propose a dynamic programing algorithm to align the visual speech from the original video with the accented speech in the target audio. In cross-language dubbing overall linguistics changes, hence we propose a lip-synthesis model conditioned upon on the redubbed audio. Finally, a user-based study is conducted, which validates our claim of better viewing experience in comparison to baseline methods. We present the application of both these methods by ‘visually redubbing Andrew Ngs machine learning tutorial video clips in Indian accented English and Hindi language respectively. In the final part of this thesis, we propose an improved method of 2D lip-landmark localization method. We investigated the current landmark localization techniques in facial domain and human- pose estimation to discover the shortcoming in adapting these methods for the task of lip-landmark localization. Present state-of-the-art methods in the domain considers lip-landmarks as a subset of facial landmarks and hence doesn’t explicitly optimizes for it. In this work we propose a new lip-centric loss formulation on the existing stacked-hourglass architecture which improves the baseline performance. Finally we use 300W and 300VW faces dataset to show the performance of our methods and compare them with the baselines. Overall, in this thesis we examined the current methods of lip modelling, investigated them for their shortcomings and proposed solutions to overcome those challenges. We perform detailed studies, and ablation studies to study our proposed methods and reported both success and failure cases for the same. We compare our solutions with the current baseline on challenging datasets, reporting quantitative results and demonstrating qualitative performances. Our proposed solutions improves the baseline performances in their individual domains.

Abstract

Future graphics and imaging applications from self-driving cars, to pervasive sensing, demand orders of magnitude more computation than we currently have. Real graphics and imaging applications appear embarrassingly parallel, but have complex dependencies, and are limited by locality and synchronization. Increasingly, the cost of communication, both within a chip and over a network dominates computation and power consumption. Driven by these trends, writing high-performance image processing code is challenging. To implement these upcoming sophisticated DIP algorithms and to process the large amount of data captured from sources such as satellites or medical instruments, intelligent high speed real-time systems have become imperative. Image processing algorithms implemented in hardware (instead of software) have recently emerged as the most viable solution for improv- ing the performance of image processing systems. The outstanding features which FPGAs offer such as optimization, high computational density, low cost etc, make them an increas- ingly preferred choice of experts in image processing field today. Despite of huge advancements in High-Level Synthesis (HLS) for FPGAs, designers are still required to have detailed knowledge about coding techniques and the targeted architecture to achieve efficient solutions. To address the challenge of productively building efficient,high- performance programs, this thesis describes an automatic approach to accelerate image processing pipelines using FPGAs. An image processing pipeline can be viewed as a graph of interconnected stages that processes images successively. Each stage typically performs a point-wise, stencil, or other more complex operations on image pixels. Recent efforts have led to the development of domain-specific languages (DSL) and optimization frameworks for image processing pipelines. We develop an approach to map image processing pipelines ex- pressed in the PolyMage DSL to efficient parallel FPGA designs. Our approach exploits reuse and available memory bandwidth (or chip resources) maximally. When compared to Dark-room, a state-of-the-art approach to compile high-level DSL to FPGAs, our approach (a) leads to designs that deliver significantly higher throughput, and (b) supports a greater variety of filters. Furthermore, the designs we generate obtain an mprovement even over pre-optimized FPGA implementations provided by vendor libraries for some of the benchmarks

Abstract

Growing infrastructure is demanding huge amounts of construction and fill materials, though the source available is limited. This unavoidable scenario in civil engineering needs alternative materials which can be substitutes to the conventional fill materials. It would be more advantageous if waste materials are made substitutes. Out of the motivation gained from the past research on waste utilization, waste materials considered for the present study are rice husk ash (RHA), quarry waste (QW) and granulated rubber (GR). Geotechnical properties of combinations of materials were investigated for the under two different studies namely, Study-I and Study-II. Under Study-I, light weight rice husk ash has been strengthened using quarry waste as additive and under Study-II, a light weight high strength material has been proposed using granulated rubber as additive to quarry waste. A series of experimental tests have been performed in Study-I and Study-II to explore their geotechnical behavior. Based on the outcomes of the experimental investigation of Study-I, the RQ50 mix (consisting equal amounts of rice husk ash and quarry waste by weight) was found to be the optimum mix. Numerical analysis under seismic loading has been performed on earth dam with RQ50 mix, QW as shell materials, in order to check their suitability to utilize in earth dam. The seismic performance of earth dam with RQ50 as shell material has been compared to that of dam with dense quarry waste as shell material, using finite element software PLAXIS 2D. The results from numerical study concluded the better seismic performance of earth dam with RQ50 than that with quarry waste, though the mix was lighter compared to quarry waste material. Thus, this combination can form an efficient material to be used as shell material in earth dam. The laboratory outcomes of Study-II have concluded the reduction in maximum dry density of quarry waste by about 0.77gm/cc upon adding 30% of granulated rubber for which the friction angle was found to improve by about 14 ̊ . The experimental results concluded QGR mixes (quarry waste mixes with granulated rubber) to be efficient backfill material with high strength and lightweight. QGR30 mix (70% QW+30% GR by weight) has been identified as best performing mix from the laboratory tests. The lateral earth pressure reduction on retaining wall by utilizing QGR30 as backfill compared to sand, quarry waste would be about 85%, 73% respectively. The advantage of lightweight and high strength fill is very economical as it would result in reduced lateral pressure, thereby the reduction in dimensions of supporting wall. The savings in the cost of backfill material using QGR30 over sand would be around 8.35%. Thus, from this study, the QGR mixes were found to be promising alternatives to the conventional fill materials, having high strength and low weight.

Abstract

DNA is the storehouse of genetic information in biological cells. It plays a vital role in transmission of the stored hereditary information from one generation to the next via the gene expression process. DNA damages caused by various damaging agents including free radicals, UV radiation, and anticancer drugs can alter genetic information and adversely affect gene expression pathways leading to various complex genetic disorders and cancers. To safeguard genetic information, biological cells have sophisticated survival mechanisms to recognize and rectify DNA damages with high fidelity. The task of recognition and repair of damaged DNA is carried out by a group of specific proteins. It is essential to understand the molecular basis of protein-mediated DNA damage repair to comprehend on the initiation and evolution of various complex genetic diseases and disorders. A critical molecular event that occurs during most protein-mediated DNA repair processes is the extrusion of orphaned bases at the damaged site facilitated by specific repairing enzymes. The molecular-level understanding of the mechanism, dynamics, and energetics of base extrusion is necessary to elucidate the molecular basis of protein-mediated DNA damage repair. The present thesis investigates the molecular mechanism and energetics of protein-induced base flipping and of DNA bending in various damaged DNA-protein complexes using molecular dynamics simulation and enhanced sampling free energy methods. In particular, we have explored the molecular mechanism of dinucleotide extrusion in a mismatched DNA (containing a stretch of three contiguous thymidine-thymidine base pairs) facilitated by Radiation sensitive 4 (RAD4), a key DNA repair protein, on an atom-by-atom basis using molecular dynamics (MD) and umbrella-sampling (US) simulations. Using atomistic models of RAD4-free and RAD4-bound mismatched DNA, the free energy profiles associated with extrusion of mismatched partner bases are determined for both systems. The mismatched bases adopted the most stable intra-helical conformation and their extrusion was unfavorable in RAD4-free mismatched DNA due to the presence of prohibitively high barriers (> 12.0 kcal/mol) along the extrusion pathways. Upon binding of RAD4 to the DNA, the global free energy minimum is shifted to the extra-helical state indicating the key role of RAD4-DNA interactions in catalyzing the dinucleotide base extrusion in the DNA-RAD4 complex. The critical residues of RAD4 contributing to the conformational stability of the mismatched bases are identified and the energetics of insertion of a -hairpin of RAD4 into the DNA duplex is examined.The conformational landscape-based mechanistic insight into RAD4-mediated base extrusion provided here may serve as a useful baseline to understand the molecular basis of xeroderma pigmentosum C (XPC)-mediated DNA damage repair in humans. In order to understand the molecular basis of why RAD4 recognizes certain mismatched DNA sequences with higher specificity, we have examined the base flipping dynamics in different RAD4-DNA complexes with different mismatched DNA sequences. The sequence dependent variations in the activation barrier for base extrusion and the most-stable conformations of mismatched partner bases are examined. The simulation results reveal that the energetics of base flipping and stability of the damaged bases are sensitive to the sequence of the damaged DNA. We have also explored the molecular basis of DNA damage recognition by enzyme 8-oxoguanine DNA glycosylase (MutM) and examined the base flipping energetics of guanine (G) and 8-oxoguanine (8OG) in the presence and absence of enzyme MutM. A detailed analysis of free energy profiles have revealed atomistic insights into enzyme-binding induced base flipping energetics, pathways and barriers for all the four systems. Similarly, to understand the sequence dependent damage (8OG)-induced distortions, the free energies of base flipping and DNA bending are calculated for 32 different DNA sequences. Our simulations revealed that base flipping is highly sequence dependent. Depending on the position of nucleotides on either (3' and 5') side of the base, significant alterations have been observed in base flipping and DNA bending energetics.

Abstract

Non-canonical base pairs are found to play a crucial role in folding of RNA molecules. Some of these non-canonical base pairs have been found to occur in nature in intrinsically unstable geometries. An understanding of how various interactions stabilize these base pairs will not only help in predicting higher order structures from secondary structures, but also provide insights into the sequence-structure function landscape of RNA molecules. One such intrinsically unstable non-canonical base pair is the G:G W:H Cis base pair, which converges to a stable G:G W:W Trans geometry when independently optimized using quantum mechanical (QM) methods. This study aims to uncover the external factors that stabilize this base pair using a combination of structural bioinformatics tools and QM methods. Such investigations were earlier used successfully to show that formation of higher order structures with external factors stabilized G:C W:W Trans base pair. Structural bioinformatics tools were used on a non-redundant set of 167 RNA crystal structures. G:G W:H Cis base pair was found to occur 123 times in this dataset. These occurrences of G:G W:H Cis base pair were categorized into 6 different environmental contexts. Environmental contexts are defined based on the external factors (other bases and metal ions) present in the vicinity of G:G W:H Cis base pairs,similar to the higher order structures identified in the study of G:C W:W Trans base pair. Quantum mechanics based geometry optimization calculations were performed on models, incorporating these environmental contexts at B3LYP level and using 6-31++(2d,2p) basis set to investigate the possible roles of these external factors towards stabilizing the geometry of this base pair. Interaction energies were calculated at MP2 level and using aug-cc-pVDZ basis set.The external factors that were found to stabilize the G:G W:H Cis base pair geometry are 1) Triplet formation by the guanine base G H , whose hoogstein edge is involved in G:G W:H Cis base pair, forming additional base pair with either adenine or cytosine through it’s watson crick edge, 2) Quartet formation of G:G W:H Cis base pair, with the G H base forming two additional base pairs, one with adenine though it’s sugar edge and the other with cytosine through it’s watson crick edge and 3) co-ordinate covalent bonds formed by Mg +2 cation with the O6 atoms of both the guanine bases. It is found that, in the absence of external factors, the G:G W:H Cis geometry is unstable even with two good hydrogen bonds and this could be due to the positioning the highly electronegative, and electron rich, O6 atoms closer to each other resulting in electrostatic repulsion. The aforementioned external factors hold the oxygen atoms together by 1) hydrogen bonds with hydrogen atoms of the exocyclic amino groups of the another partner base or 2) co-ordinate covalent bonds with metal ions which act like clamps, to maintain its observed crystal geometry.

Abstract

Users navigating the web often deviate from the intended path and find themselves in a state of being lost. Early research shows, different cognitive factors such as domain knowledge, spatial ability, working memory, and prior experience contributes to the successful navigation. Existing cognitive models such as CoLiDeS, CoLiDeS+, CoLiDeS+Pic, MESA, and SNIF-ACT provide deeper analysis to understand the user behavior on the web. At the end, the goal of these models is to help the practitioners of the web in evaluating “the effectiveness of their designs”, in terms of user navigation. These models consider the selection process of a link as the key step in navigation and do so by measuring the semantic similarity of the link with respect to the goal. However, they do not completely account for all types of cognitive factors, and also the position of the links and content on a web page in the selection process. In this work, we present the results from two eye tracking studies conducted to understand the user behaviour during the link selection process. We observed that the users select a link with maximum similarity to the goal (information scent) and the information scent is always increased in a successful navigation path. We have also observed that the position and the information scent has not affected the way users attend the links. Although existing models do not consider the content while evaluating the links, we noticed that users spent significant amount of time in attending the content during the selection process. By comparing CoLiDes+ with Information Search process, we also found that the users feel more uncertain during the link selection stage of CoLiDes+. Incorporating all these observations, we modelled a tool to assist in navigation to reduce the user uncertainty and disorientation. While the above studies consider the user behavior as goal-oriented, often users browse without any predefined goal to start with. In this context, we carried out a behavioral study to understand the role of cognitive factors spatial ability and prior knowledge, on information seeking patterns. We observed the high spatial ability resulted in surface navigation whereas the low spatial ability resulted in deep navigation pattern. Users with high domain knowledge spent more time per page and visited less number of pages. These findings lead us to design an assistive feature by presenting the information from other sections on a given page. We believe this will result in better information discovery for the low spatial ability and the domain knowledge.

Abstract

Handwriting recognition (HWR) in Indic scripts is a challenging problem due to the inherent subtleties in the scripts, cursive nature of the handwriting and similar shape of the characters. Though a lot of research has been done in the field of text recognition, the focus of the vision community has been primarily on English. Furthermore, a lack of publicly available handwriting datasets in Indic scripts has also affected the development of handwritten word recognizers and made direct comparisons across different methods an impossible task in the field. Also, due to this lack annotated data, it becomes challenging to train deep neural networks which contain millions of parameters. These facts are quite surprising considering the fact that there are over 400 million Devanagari speakers in India alone. We first tackle the problem of lack of annotated data using various approaches. We describe a framework for annotating large scale of handwritten word images without the need for manual segmentation and label re-alignment. Two new word level handwritten datasets for Telugu and Devanagari are released which were created using the above mentioned framework. We synthesize synthetic datasets containing millions of realistic images with a large vocabulary for the purpose of pre-training using publicly available Unicode fonts. Later on, pre-training using data from Latin script is also shown to be useful to overcome the shortage of data.Capitalizing on the success of the CNN-RNN Hybrid architecture, we propose various improvements in the architecture and it’s training pipeline to make it even more robust for the purposes of handwriting recognition. We now change the network to use a Resnet-18 like structure for the convolutional part along with adding a spatial transformer network layer. We also use an extensive data augmentation scheme involving multi-scale, elastic, affine and test time distortion. We outperform the previous state-of-the-art methods on existing benchmark datasets for both Latin and Indic scripts by quite some margin. We perform an ablation study to empirically show how the various changes we made to the original CNN-RNN Hybrid network have improved its performance with respect to handwriting recognition. We dive deeper into the working of our networks convolutional layers and verify the robustness of convolutional-features through layer visualizations. We hope the release of the two datasets mentioned in this work along with the architecture and training techniques that we have used instill interest among fellow researchers of the field.

Abstract

Videos engulf the media archive in every capacity, generating a rich and vast source of information from which machines can learn how to understand such data and predict useful attributes that can help technologies make human lives better. One of the most significant and instrumental part of a computer vision system is the comprehension of human actions from visual sequences – a paragon of machine intelligence that can be achieved through computer vision. The problem of human action recognition has high importance as it facilitates several applications built around recognition of human actions. Understanding actions from monocular sequences has been studied immensely, while comprehending human actions from a skeleton video has developed in recent times. We propose action recognition frameworks that use the skeletal data (viz. 3D locations of some joints) of human body to learn spatio-temporal representations necessary for recognizing actions. Information about human actions is composed of two integral dimensions, space and time, along which the variations occur. Spatial representation of a human action aims at encapsulating the config- uration of human body essential to the action, while the temporal representation aims at capturing the evolution of such configurations across time. To this end, we propose to use geometric relations betweenhuman skeleton joints to discern the body pose relative to the action and physically inspired kinematic quantities in order to understand the temporal evolution of body pose. Spatio-temporal understanding of human actions is thus conceived as a comprehension of geometric and kinematic information with the help of machine learning frameworks. Using a representation inculcating an amalgamation of geo- metric and kinematic features, we recognize human actions from skeleton videos (S-videos) using such frameworks. We first present a non-parametric approach for temporal sub-segmentation of trimmed action videos using angular momentum trajectory of the skeletal pose sequence. Meaningful summarization of the pose sequence is a product of the sub-segmentation achieved through systematic sampling of the seg- ments. Descriptors capturing geometric and kinematic statistics encoded as histograms and spread across a periodic range of orientations are computed to represent the summarized pose sequences, which are fed to a kernelized classifier for recognizing actions. This framework for understanding human ac- tions instils the effects of using geometric and kinematic properties of human pose sequences, important to spatio-temporal modeling of the actions. However, a downside of this framework is the inability to scale with availability of large amount of visual data.To mitigate the impending drawback, we next present geometric deep learning frameworks and specifically, graph convolutional networks, for the same task. Representation of human skeleton as a sparse spatial graph is intuitive and a structured form which lies on graph manifolds. A human action video hence results in the formation of a spatio-temporal graph and graph convolutions facilitate the learning of a spatio-temporal descriptor for the action. Inspired by the success of Deformable Part-based Models (DPMs) for the task of object understanding from images, we propose a part-based graph con- volutional network (PB-GCN) that operates on a human skeletal graph divided into parts. Incorporating the culmination of understandings from the success of geometric and kinematic features, we propose to use relative coordinates and temporal displacements of the 3D joints coordinates as features at the vertices in the skeletal graph. Owing to these signals, the prowess of graph convolutional networks is further boosted to attain state-of-the-art performance among several action recognition systems using skeleton videos. In this thesis, we meticulously examine the growth of the idea about using geometry and kinematics, transition to geometric deep learning frameworks and design a PB-GCN with geometric + kinematic signals at the vertices, for the task of human action recognition using skeleton videos.

Abstract

360° field of view (FOV) is becoming a vital part of various human-robot/computer applications, such as urban search and rescue task. Despite growing importance of 360° FOV, very few studies have assessed the effect of 360° visual display on users' remote spatial ability, specifically in the case of 2D 360° User Interface (UI), which is the focus of the current study. We aim to evaluate the efficacy and efficiency of desktop 360° display designs on teleoperators' egocentric spatial ability, specifically navigation and direction judgment in an unknown environment. A previous study[1] showed an advantage of the segmented interface over seamless 360° UI on angle estimation error. However, we are still not clear as to the determining factors that lead to a difference in interface interaction ability. We conducted two studies that studied the scanning effort while using a 360° interface and the ability to estimate the mental egocentric bearing of a target in an environment. We measured the effects of gaming experience, and time constriction, with metrics consisting of time to task completion, target bearing estimation, and latency of target estimation. Further data was collected by an eye-tracking device to help support and complement the data from the studies. We found that gaming experience is a significant positive factor in accuracy and time to task completion. Further, a seamless panoramic view was found to be least effective compared to a segmented interface view of the 360° video feed for target bearing estimation across the entire field of vision presented through the interfaces. Eye tracking data supports the behavioral data collected. It was found that the estimation frequencies in the first study were high for forward heading direction, which was seen in the increased eye movement activity in the same region. In the second study, it was found that angle estimation error was the lowest along mental egocentric axes. This was again observed in the eye tracker data that reported increased activity along the mental egocentric axes.