Abstract

The reconstruction of 3D objects from monocular images is an active field of research in 3D computer vision which is further boosted by advancements in deep learning. In context of human body, modeling realistic 3D virtual avatars from 2D images is a recent trend, thanks to the advent of AR/VR & metaverse. The problem is challenging, owing to non-rigid nature of human body, especially because of the garments. Various attempts have been made to solve the problem, at least for relatively tighter clothing styles, but loose clothing styles still pose a huge challenge. This problem has also sparked quite an interest in the fashion e-commerce domain, where the objective is to model the 3D garments, independent from the underlying body, in order to enable intriguing applications like virtual try-on systems. 3D garment digitization has been garnering a lot of interest in the past few years, as the demand for online window-shopping and other e-commerce activities has increased in the recent years, where the unfortunate crisis of COVID-19 plays a huge role. Though the problem of 3D digitization of garments seems intriguing, solving it is not as straightforward as it looks. There are existing works out there in the field, majority of which are deep learning based solutions. Most of these methods rely on predefined garment templates which makes the task of texture synthesis easier, but restrict the usage to a fixed number of garment styles for which templates are available. Additionally, these methods do not deal with issues like complex poses and self-occlusions which are very common under in-the-wild assumption. Template-free methods are also explored which enables modeling arbitrary clothing styles, however, they lack texture information which is essential for high-quality photorealistic appearance. The thesis aims to resolve aforementioned issues by providing novel solutions. The main objective is 3D digitization of garments from a monocular RGB image of a person wearing the garment, both in template-based and template-free settings. Initially, we address challenges in existing state-of-the-art template-based methods. We aim to handle complex human poses, occlusions etc. by proposing to use a robust keypoint regressor which estimates keypoints on input monocular image. These keypoints define thin-plate-spline (TPS) based warping of texture from input image to the UV space of a predefined template. Then, we utilize a deep inpainting network to handle missing texture information. In order to train these neural networks, we curate a synthetic dataset of garments with varying textures, draped on 3D human characters in various complex poses. This dataset helps in robust training and generalization to real images. We achieve state-of-the-art results for specific clothing styles (e.g. t-shirt and trouser). However, template-based methods cannot model any arbitrary garment style. Therefore, we next aim to handle arbitrary garment styles in a template-free setting. Existing state-of-the-art template-free methods can model geometrical details of arbitrary garment styles up to some extent, but fail to recover texture information. To model arbitrary geometry of garments, we propose to use an explicit, sparse representation introduced for modeling human body. This representation handles self-occlusion and loose clothing as well. We extend this representation by introducing semantic segmentation information for differentiating between various clothing styles (top wear /bottom wear) and human body present in the input image. Furthermore, this representation is exploited in a novel way to provide seams for texture mapping, thereby retaining high-quality textural details and providing way to lot of useful applications like texture editing, appearance manipulation, texture super-resolution etc. The proposed method is the first one to model arbitrary garment styles and recover textures as well. We evaluate our proposed solutions on various publicly available datasets, outperforming existing state-of-the-art methods. We also discuss the limitations in the proposed methods and provide potential solutions that can be explored. Finally, we discuss the future extensions of the proposed methods. We believe this thesis significantly improves the research landscape in 3D garment digitization and accelerates the progress in this direction.

Abstract

Drug discovery involves the process of designing molecules that interact well with a given target protein structure in order to modulate to its function. With large scale availability of 3D structures of proteins, structure based drug design has been developed as a rational approach towards de novo drug design. In its core, it is about developing drug molecules that bind well with the receptor at its binding site and also portray ideal drug-like properties for Absorption, Distribution, Metabolism and Excretion (ADME). A prerequisite to this is finding druggable and functionally relevant binding sites on the 3D structure of the protein. Once the binding site is identified large libraries of molecules are screened against it to identify top candidate molecules, a process more commonly known as virtual screening. However, these chemical libraries represent only a minute portion of the larger chemical space where more ideal molecules may reside. To access this larger chemical space generative model need to be designed that can learn a distribution over this space for efficient sampling. Such a situation specifically beckons for inverse molecule design wherein generative models sample molecules conditioned to have specific properties and chemical structures. The rapid adoption and success of deep learning algorithms in various sections of structural biology motivates the usage of such algorithms for important tasks in drug design. In this thesis, we present three studies, namely DeepPocket, Apobind and MolGPT, that address important tasks in drug design such as binding site detection, molecular generation, and optimization using deep learning. DeepPocket uses 3D convolutional neural networks to identify and segment out druggable binding sites on the surface of the protein. However, DeepPocket was initially trained on only annotated ligand-bound (holo) protein structures since only such datasets were available for binding site detection. Therefore, Apobind a dataset of annotated ligand-unbound (Apo) structures is mined to circumvent this limitation and also act as an useful resource for other tasks like binding affinity prediction and molecular docking. Finally, MolGPT, a novel molecular generative model based on the attention-based transformer architecture is discussed while and its efficacy in molecular optimization and inverse molecule design is demonstrated.

Abstract

Emotions are an essential part of daily communication and play a critical role in conveying a person’s mental state. Understanding, categorization, and controllable emotions generation are widely researched in NLP and social sciences. Different work in social sciences aimed to define emotions and identify different emotions. Ekman’s initial work identified six primary emotions: anger, disgust, fear, joy, sadness, and surprise. Plutchik identified eight emotions and classified them into four polar opposite pairs. With the advent of computer science, different computational approaches have been developed to identify and categorize emotions and generate emotional text. In this thesis, we develop data-driven, psychologically grounded computational approaches to solve two broad problems relating to emotions in text. The first problem is identifying emotions at a paragraph level. Existing work detects emotion at a sentence level having a limited context. We aim to detect emotions at a paragraph level consisting of multiple sentences. Thus, the model needs to consider the emotional cues present in an individual sentence and paragraph as a whole unit. We also explore the classification problem in scientific documents when the context is large but cannot be broken down into individual sentences. The second problem that we explore is changing the emotion of a sentence while controlling its intensity. In the process, the meaning of the sentence should be preserved. This task of changing specific attributes of a sentence while preserving its overall meaning is defined as style transfer. For both the tasks, the proposed system achieved state-of-the-art. Pretrained language models are current de facto in Natural Language Processing (NLP). We develop an ensemble solution consisting of three finetuned language models for the first task. The language models are RoBERTa, BART, and RoBERTa model that is first finetuned on sentence-level emotion detection tasks. RoBERTa model is used due to its remarkable performance in many natural language understanding tasks. BART has shown good performance for tasks involving multi-sentence level context like summarization, thus is used here. RoBERTa model that is first finetuned on sentence-level emotion detection tasks is used to identify strong sentence-level cues. Our proposed system was ranked one globally for the Empathy Detection and Emotion Classification shared task at the WASSA workshop at ACL 2022. We also explore the power of the pretrained language model for the classification of scientific documents when the context is large but cannot be divided into sentences. Our proposed system of using highly contextualized embeddings of science BERT achieved a new state of the art and was ranked one globally for 3C Citation Context Classification shared task at the Scholarly Document Processing workshop at NAACL 2021.

Abstract

Acetone sensors find applications in healthcare, environmental, food, agriculture, and industrial fields. Semiconducting metal oxide (SMO)-based acetone sensors are costeffective, stable, and could be user-friendly and portable. In SMO-based acetone sensors, interferents like alcohol, which is structurally and chemically similar to acetone, impair the usability of the sensor. In this experimental study, we have investigated how Ag nanoparticles loading onto nanostructured ZnO materials improves the sensor performance, including the selectivity towards acetone gas over ethanol. The sensor involves facile chemical synthesis methods for nanostructured ZnO preparation and preparation/loading Ag nanomaterials onto ZnO. We have varied the Ag nanoparticle loading to see its effect on the acetone sensing performance. Ag is the cheaper noble metal and has been widely used in optoelectronics fields. UV-visible spectrophotometry, Raman scattering spectroscopy and field emission scanning electron microscopy (FESEM) have been used to characterize the sensor materials. It is found that an optimal Ag nanoparticle loading onto nanostructured ZnO shows remarkable acetone selectivity with high sensor response, fast response and recovery at relatively low working temperature compared to pristine ZnO. The nanostructured ZnO/Ag sensor material does not exhibit any interference from water vapor, a significant constituent of the air and exhaled breath. Furthermore, the sensor response and logarithmic acetone concentration yield a good linearity relationship which suggests suitable nanostructured ZnO/Ag material has immense potential as a sensitive and selective low-cost acetone sensor.

Abstract

This research aims to study Indian folk paintings in terms of style, icons, and their interrelations. The thesis is an attempt to do the above through presenting a formal theory of paintings, built upon, and developed from pre-existing formal theories in India and the west. We have selected Indian folk paintings as a case to demonstrate our formal theory for the following reasons: first, Indian folk paintings have not been studied in terms of their formal aspects; second, their unique compositional sense of style stability and iconic similarity have remained under-researched. The thesis begins by explaining the nature of space of the painting. It gradually develops the following five core chapters: (1) Concept of style (2) Concept of icon (3) Blending of style and icons in paintings (4) Case study of Indian folk paintings and (5) Formal analysis of folk paintings of India. The idea developed in the above core chapters is to study Indian folk paintings in terms of their artistic qualities. The styles and patterns of the Indian folk paintings are as diverse as the nation. However, Indian folk paintings show diversity in style and similarity in iconic representations. For example, one can see similar iconic representations of snake, elephant, tiger and even humans being repeated across various Indian folk painting styles. The purpose of our research is to qualify these differences in style and study the artistically and contextually informed aspects influencing an artist’s craft. We will describe and analyse folk paintings in terms of their formal aspects, which helps to develop and shape our research process. Finally, to validate our research efforts in qualifying style in Indian folk paintings, with undertake an evaluation employing existing arthistorical theories and modules.From the above study, this thesis will attempt to derive and explain formal features guiding the style of an artist and her art, in this case folk paintings from India. Our thesis leads further to the following inquiry- how can we define the concept of style in painting? What are the definitions of icons? What is the nature of space of a painting which allows blending of the style and icon? What is the role of memory and imagination in creating icon and understanding a pictorial style? “If the icon is what an artist wants to expose, the style is the medium for this exposure” (Merleau-Ponty, 1993). Painting reflects the ‘way’ (style/action) and ‘what’ (icon) in a concrete form. In painting, style/action can be understood only through strokes. “These strokes are traces of the artist's moving hand. The hand movements are guided by the icon, which is etched in the artist's mind. Strokes are nothing but the parts of that icon” (MerleauPonty, 1993). For example, if the intention is to draw a tiger, one starts by drawing its parts like head, tail, body. Here the strokes are syntactic, which reveal the semantics of the icon to a spectator. Only the strokes and patches are visible in a painting. “The primary sine qua non to create or to understand icons is imagination” (Feldman, 1967). The artist using her imagination to create artwork is understandable. However, it is indeed intriguing how the viewer comprehends the artist's mind by using her imaginative prowess. How does the viewer get access to the world of the artist's imagination? The artist makes her art accessible through her artworks. The strokes and colour patches in the painting become paths through which the viewer enters the world of the artist's imagination. This research is about the role of style and icon in creating these paths and the journey that the artist undergoes while creating and, the viewer, while seeing. Thisstudy involves an analysis of five Indian folk painting styles. They belong to five different Indian states: Telangana, Odisha, West Bengal, Bihar, and Rajasthan. We tried to trace the changes and developments within a particular style and clarify the core elements in each of the region’s (mentioned above) art school style and their compositional principles. For this, we have made an exhaustive and elaborate database of thousands of images of paintings from each style. The images belong to varying time periods and are created by several artists. We adopted qualitative methodologies to interact and interview artists to know and get a clear perception of their painting styles. We have videotaped the interviews to aid in capturing the process in the making of a painting from the perspective o the artist.

Abstract

Crowd counting is an important task in security, surveillance and monitoring. There are many competitive benchmark datasets available in this domain. The data distribution in the crowd counting datasets show a heavy-tailed and discontinuous nature. This nature of the dataset is majorly ignored while building solutions to this problem. However, the skew in datasets contradicts few assumptions made by the stages of the training pipeline. As a consequence of the skew in the dataset, unacceptably large standard deviation wrt to the customarily used performance measures (MAE, MSE) is observed. To address these issues, this thesis provides modifications that incorporate the dataset skew in training and evaluation pipelines. In the training pipeline, to enable principled and balanced minibatch sampling, a novel smoothed Bayesian binning approach is presented that stratifies the entire count range. Further, these strata are sampled to construct uniform minibatches. The optimization is upgraded with a novel strata-aware cost function that can be readily incorporated into the existing crowd counting deep networks. In the evaluation pipeline, as an alternative to the customary evaluation MAE, this thesis provides three alternative evaluation measures. Firstly, a strata-level performance in terms of mean and standard deviation gives range specific insights. Secondly, relative error perspective is brought in by using a novel Thresholded Percentage Error Ratio (TPER). Lastly, a localization included counting error metric Grid Average Mean absolute Error (GAME) is used to evaluate the different networks. In this thesis, it is shown that proposed binning-based modifications retain their superiority wrt the novel strata-level performance measure. Overall, this thesis contributes a practically useful training pipeline and detail-oriented characterization of performance for crowd counting approaches.

Abstract

This thesis proposes solutions to two major problems (highlighted) faced when developing a selfdriving car using traditional modular approaches. That is to sense, percieve, localize/map, predict and plan/control. The first one is doing real-time Simultaneous Localization and Mapping SLAM at low sampling rates and/or at high linear or angular velocities. The keynote of the work showcases SROM’s ability to maintain localization at low sampling rates or at high linear or angular velocities where most popular LiDAR-based localization approaches get degraded fast. We also demonstrate SROM to be computationally efficient and capable of handling high-speed maneuvers. It also achieves low drifts without the need for any other sensors like IMU and/or GPS. Our method has a two-layer structure wherein first, an approximate estimate of the rotation angle and translation parameters are calculated using a Phase Only Correlation (POC) method. Next, we use this estimate as an initialization for a point-to-plane ICP algorithm to obtain fine matching and registration. Another key feature of the proposed algorithm is the removal of dynamic objects before matching the scans. This improves the performance of our system as the dynamic objects can corrupt the matching scheme and derail localization. Our SLAM system can build reliable maps at the same time generating high-quality odometry. We exhaustively evaluated the proposed method in many challenging highways/country/urban sequences from the KITTI dataset and the results demonstrate better accuracy in comparisons to other state-of-the-art methods with reduced computational expense aiding in real-time realizations. We have also integrated our SROM system with our in-house autonomous vehicle and compared it with the state-of-the-art methods like LOAM and LeGO-LOAM. The second one is the problem of an agent/robot with non-holonomic kinematics avoiding dynamic and static obstacles whose state and velocity noise are non-parametric. Additionally, there may be bounds/constraints on the configurational space of the robot in the form of lane/corridor boundaries. In this work we consider the problem of an agent/robot with non-holonomic kinematics avoiding dynamic and static obstacles. Additionally there may be bounds/constraints on the configurational space of the robot in the form of lane/corridor boundaries. The robot’s state and velocity noise, the lanes, the obstacles, and the robot’s control noise are modelled as non-parametric distributions as Gaussian assumptions of noise models are violated in real-world scenarios. Under these assumptions, we formulate a robust MPC that samples robotic controls effectively in a manner that aligns the robot to the goal state while avoiding obstacles and staying within the lane bounds under the duress of such non-parametric noise. In particular, the MPC incorporates a distribution matching cost that effectively aligns the distribution of the current collision cone to a certain desired distribution whose samples are collision-free. This cost is posed as a distance function in the Hilbert Space, whose minimization typically results in the collision cone samples becoming collision-free. We show tangible performance gains compared to methods that model the collision cone distribution by linearizing the Gaussian approximations of the original nonparametric state and obstacle distributions. We also show superior performance to methods that pose a chance constraint formulation of the Gaussian approximations of non-parametric noise without subjecting such approximations to further linearizations. The performance gain is shown both in terms of trajectory length and control costs that vindicates the efficacy of the proposed method. Finally, we show the proposed method being used to navigate with a non holonomic differential drive robot in real-time in a realistic setting in Gazebo with dynamic and static obstacles. To the best of our knowledge, this is the first presentation of non-holonomic collision avoidance of stationary obstacles, moving obstacles and lane constraints in the presence of non-parametric state, velocity, actuator and lane boundary noise models.

Abstract

The last few decades have witnessed tremendous growth in the sensors industry, spurred by advancements in electronics, nanoscience, biotechnology and computer science. They are employed for a wide variety of applications and they possess the potential to play a pivotal role in biochemical applications. Among the various sensing modalities being explored, we have Radio Frequency based sensing. It studies the interaction between EM waves and matter and uses this interaction for qualitative and quantitative analysis. RF sensors offer a reduction in process complexity, speed, cost, portability and ease of integration with existing technology. In this work, Radio Frequency (RF) based sensing technique is applied for different biochemical applications. In particular, emphasis is given to combating two significant challenges of RF sensing, namely, specificity and reusability. In addition to that, the use of RF sensing in new sub-domains is explored, which would enable the development of a portable biochemical sensing platform. The sensors put into use in this work are Interdigitated Capacitor based sensor and Microstrip Patch based sensor. This sensor is reusable and can be put into use for diverse applications. Reusability of the sensor implies that it can not only be used several times, but the same sensor can detect different sets of analytes with equal efficiency. We have performed both computational and experimental studies on surfactants, nanoparticles and biomolecules using RF sensing. Surfactants are essentially surface-active agents that are amphiphilic. Above a specific concentration called the Critical Micelle Concentration (CMC), they self assemble to form micelles. The estimation of CMC is vital for all the varied applications of surfactants. The first application develops a fast, facile, efficient method for CMC detection that works well for all classes of surfactants. With their plethora of inherent advantages, Nanoparticles play vital roles in the biochemical and biomedical domains. Their study is essential to utilize them for diverse applications efficiently. There is a growing need to develop a methodology that performs preliminary characterization of nanoparticles in the absence of sophisticated laboratory facilities. In the second application, we study nanoparticles of various morphology and arrangements and characterize them. Our third application deals with developing a biomolecule detection platform that is rapid, portable, efficient and reusable. Gold nanoparticles are functionalized to improve the reusability and specificity aspects of conventional RF sensors. Further, the Biotin-Streptavidin binding is chosen to demonstrate the methodology

Abstract

The popularity of egocentric cameras and their always-on nature has lead to the abundance of daylong first-person videos. Because of the extreme shake and highly redundant nature, these videos are difficult to watch from beginning to end and often require summarization tools for their efficient consumption. However, traditional summarization techniques developed for static surveillance videos, or highly curated sports videos and movies are, either, not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. In this work, we present a novel unsupervised reinforcement learning technique to generate video summaries from day long egocentric videos. Our approach can be adapted to generate summaries of various lengths making it possible to view even one minute summaries of one’s entire day. The technique can also be adapted to various rewards, such as distinctiveness, indicativeness of the summary. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state of the art. Quantitative comparison on the benchmark Disney dataset shows that our method achieves significant improvement in Relaxed F-Score (RFS) (32.56 vs. 19.21) and BLEU score (12.12 vs. 10.64). Finally, we show that our technique can be applied for summarizing traditional, short, hand-held videos as well, where we improve the state of the art F-score on benchmark SumMe and TVSum datasets from 41.4 to 45.6 and 57.6 to 59.1 respectively

Abstract

Random Number Generators (RNGs) are the solution for cryptographic applications to enhance hardware security. These RNGs should have three specific properties unpredictability, aperiodic, and good statistical criteria. This brief presents a True Random Number Generator (TRNG) based on Ring oscillators' jitter with MLFSR. The MLFSR is augmented with a set of prime primitive polynomials, Boolean, and nonlinear functions to attain a non-linear, unpredictable, and extended sequence period. Paper mainly focused on achieving high randomness by integrating the TRNG with a new promising crypto engine Keccak as a post-processing block, leading to more extensive security in data transfer, encryption keys, data authenticity of ICs, and IoT based applications. The TRNG design is coded in Verilog HDL and implemented on the FPGA Zed board. Hashing is performed with a throughput of 2.4Gbps at 100MHz either with RNG data or SHA data. The HRNG (Hardware Random Number Generator) is an integrated block of TRNG and hashing along with the IP cores of memory, clock Generator, and Integrated Logic Analyzer, implemented on the FPGA hardware. Evaluated the randomness of the generated non-deterministic bit streams (10Mb) using the NIST 800-22 & Diehard test suite and successfully passed.