Abstract

Consider an autonomous agent capable of observing multiple humans making a pizza and making one the next time! Motivated to contribute towards creating systems capable of understanding and reasoning instructions at the human level, in this thesis, we tackle procedure learning. Procedure learning involves identifying the key-steps and determining their logical order to perform a task. The first portion of this thesis focuses on the datasets curated for procedure learning. Existing datasets commonly consist of third-person videos for learning the procedure, making the manipulated object small in appearance and often occluded by the actor, leading to significant errors. In contrast, we observe that videos obtained from first-person (egocentric) wearable cameras provide an unobstructed and clear view of the action. To this end, for studying procedure learning from egocentric videos, we propose the EgoProceL dataset. However, procedure learning from egocentric videos is challenging because the camera view undergoes extreme changes due to the wearer’s head motion and introduces unrelated frames. Due to this, current state-of-the-art methods’ assumptions that the actions occur at approximately the same time and are of the same duration do not hold. Instead, we propose to use the signal provided by the temporal correspondences between key-steps across videos. To this end, we present a novel self-supervised Correspond and Cut (CnC) framework that identifies and utilizes the temporal correspondences between the key-steps across multiple videos to learn the procedure. We perform experiments on the benchmark ProceL and CrossTask datasets and achieve state-of-the-art results. In the second portion of the thesis, we look at various approaches to generate the signal for learning the embedding space. Existing approaches use only one or a couple of videos for this purpose. However, we argue that it makes key-steps discovery challenging as the algorithms lack an inter-videos perspective. To this end, we propose an unsupervised Graph-based Procedure Learning (GPL) framework. GPL consists of the novel UnityGraph that represents all the videos of a task as a graph to obtain both intra-video and inter-videos context. Further, to obtain similar embeddings for the same key-steps, the embeddings of UnityGraph are updated in an unsupervised manner using the Node2Vec algorithm. Finally, to identify the key-steps, we cluster the embeddings using KMeans. We test GPL on benchmark ProceL, CrossTask, and EgoProceL datasets and achieve an average improvement of 2% on third-person datasets and 3.6% on EgoProceL over the state-of-the-art. We hope this work motivates future research on procedure learning from egocentric videos. Furthermore, the unsupervised approaches proposed in the thesis will help create scalable systems and drive future research toward creative solutions

Abstract

Cell division refers to the process by which cells grow, replicate their genetic material, and divide to form daughter cells. Major biological processes, namely reproduction, development, wound healing and tissue regeneration, require cell division. Cells switch between quiescence and proliferation states for maintaining tissue homeostasis and regeneration. The cell division process is regulated by a wide range of extracellular and intracellular cues like growth factors, stress, and reactive oxygen species (ROS). The decision to exit or enter quiescence is dysregulated in proliferative and degenerative diseases. Hence, understanding the molecular mechanisms that control the reversible transition between quiescence and proliferation is crucial. In this thesis, we study the regulatory network involved in this decision-making in normal and disease (cancers and Alzheimer’s Disease) conditions and characterize the metabolic adaptation of cancers using systems biology approach. At the restriction point (R-point), cells become irreversibly committed to the completion of the cell cycle independent of mitogen. The mechanism involving hyperphosphorylation of retinoblastoma (Rb) and activation of transcription factor E2F is linked to the R-point passage. However, stress stimuli trigger exit from the cell cycle back to the mitogen-sensitive quiescent state after Rb hyper-phosphorylation, but only until APC/C-Cdh1 inactivation. In the work presented here, we developed a mathematical model to investigate the reversible transition between quiescence and proliferation in mammalian cells with respect to mitogen and stress signals. The model integrates the current mechanistic knowledge and accounts for the recent experimental observations with cells exiting quiescence and proliferating cells. We show that Cyclin E-Cdk2 couples Rb-E2F and APC/C-Cdh1 bistable switches and temporally segregates the Rpoint and the G1/S transition. A redox-dependent mutual antagonism between APC/CCdh1 and its inhibitor Emi1 makes the inactivation of APC/C-Cdh1 bistable. We show that the levels of Cdk inhibitor (CKI) and mitogen control the reversible transition between quiescence and proliferation. Further, we propose that shifting of the mitogeninduced transcriptional program to G2-phase in proliferating cells might result in an intermediate Cdk2 activity at the mitotic exit and the immediate inactivation of APC/CCdh1. Our study builds a coherent framework and generates hypotheses that have been confirmed by experimental findings. Proliferative diseases like cancer arise due to alterations in the regulation of the cell cycle. An emerging hallmark of cancer is metabolic reprogramming, which presents opportunities for cancer diagnosis and treatment based on metabolism. A comprehensive metabolic network analysis of renal cell carcinoma (RCC) subtypes, including clear cell, papillary, and chromophobe, was performed by integrating transcriptome data with the human genome-scale metabolic model to understand the coordination of metabolic pathways in cancer cells. We identified metabolic alterations of each subtype with respect to tumor-adjacent normal samples and compared them to understand the differences between subtypes. We found that genes of amino acid metabolism and redox homeostasis are significantly altered in RCC subtypes. Chromophobe showed metabolic divergence compared to other subtypes with upregulation of genes involved in glutamine anaplerosis and aspartate biosynthesis. A difference in transcriptional regulation involving HIF1A is observed between subtypes. We identified E2F1 and FOXM1 as other major transcriptional activators of metabolic genes in RCC. These results highlight the crosstalk between metabolism and cell division. Further, the co-expression pattern of metabolic genes in each patient showed variations in metabolism within RCC subtypes. We also found that co-expression modules of each subtype have tumor stage-specific behavior, which may have clinical implications. Intriguingly, cell cycle dysregulation triggers not only proliferative diseases such as cancers but also drives degenerative diseases like Alzheimer's disease (AD). Aberrant production and aggregation of amyloid beta oligomers (Aβ) into plaques is a frequent feature of AD. However, therapeutic approaches targeting Aβ accumulation fail to reverse or inhibit disease progression. The approved cholinesterase inhibitor drugs are also mostly symptomatic treatments. During human brain development, the progenitor cells differentiate into neurons and switch to a postmitotic, resting state. However, cell cycle re-entry often precedes the loss of neurons. In this study, we developed mathematical models of multiple routes leading to cell cycle re-entry in neurons that incorporate the crosstalk between cell cycle, neuronal and apoptotic signaling mechanisms. We show that the integration of multiple feedback loops influenc

Abstract

The detection of analytes using Radio Frequency (RF) based sensors has gained immense popularity in recent times. The change in the sensor’s resonant frequency in the presence of different analytes is used as the basis for detection. Although, there have been plethora of advancements in developing RF biosensors, yet most of them have been designed and fabricated considering air all around them. However, in a practical scenario, a solute is mostly used with solvent for testing. The solvent alone produces huge shift in resonant frequency whereas, the shift due to solute and solvent together is significantly narrower (in MHz) when compared to the solvent alone. This indicates that the shift due to solute is narrowband in nature. Most of these sensors utilize traditional Vector Network Analyzers (VNAs) for measuring the s-parameters. However, traditional VNAs are broadband devices that are expensive, large and cumbersome to operate. These VNAs also have much larger bandwidths than the observed frequency shifts as mentioned above and often prove to be overkill. Thus, the optimal solution is to develop narrowband measurement platforms that precisely monitor the frequency shifts. This article tackles the challenge and presents a solvent specific integrated solution for the detection of materials having different permittivities. An Integrated Biosensing Network Analyser (IBNA) is designed and fabricated using microstrip line technology to detect different analytes (Glucose and Sucrose). IBNA primarily consists of a sensing and a measuring unit specifically designed for solvent like water. While the Interdigitated Capacitor based RF Bio Liquid Cavity resonating at 2.3 GHz performs the sensing operation, the Dual Band Six-Port Reflectometer (DBSPR), with an operating frequency of 1.1 GHz and 2.3 GHz, participates in the measurement of the s-parameters. Furthermore, to demonstrate the working of the design, the detection of 1 M Glucose and Sucrose is performed by EM-simulations and experiments. The sensing is carried out efficiently, with Glucose and Sucrose yielding frequency shifts of 15 MHz and 20 MHz at room temperature respectively. The results help in establishing the utility of IBNA and its superiority over traditional network analyzers

Abstract

One of the significant causes of life loss during an earthquake is the collapse of structures in the near-fault regions. Broadly, two sets of parameters affect the structural response, namely: (i) characteristics of structures, like configuration, stiffness, strength, and ductility, and (ii) characteristics of ground motions. Usually, the former are dependent on the latter. But the latter is derived independently by geologists, seismo-tectonists, seismologists, geotechnical engineers and structural engineers. Often, the considered seismic hazard has uncertainties arising from the way strong ground motion is considered in the near-fault regions. To understand and quantify the hazard in the near-fault regions, the ground motions characteristics are examined of the strong motion records of 1999 Chi-Chi, Taiwan earthquake. It is observed that the ground motions in the near-fault region exhibit higher PGA values on the hanging wall side as compared to the footwall side. Further, the behaviours of two reinforced concrete buildings with moment resisting frames of five and ten storeys are studied when subjected to two types of horizontal near-fault ground motions with and without velocity pulse. The responses observed and the damage caused in the moment frame buildings owing to these nearfault ground motions are identified. It is observed that the near-fault ground motions with pulse induce higher drift demands compared to those without pulse and far-fault ground motions. Also, the response of the structure subjected to near-fault ground motion with pulse indicate the impulsive action that requires a revised mitigation strategy other than the conventional earthquake resistant design procedures. Inclusion of a structural wall is identified as a simple solution to control the damage arising from near-fault ground motion effects. The maximum drift demand on the structures with structural walls with SPD of 2.75%, 3.75% and 9% are studied and improved performance is observed as the structural plan density of structural walls is increased.

Abstract

The Internet has many tools that connect people from different ends of the globe. Geographical boundaries do not hinder the connection between two people. Communication is easily facilitated, and news is easily spread. However, this connectivity can be exploited and used negatively. Bullying, hate speech, and fake news can all be spread across the internet, and the common denominator in combating these issues is Sentiment Analysis. Sentiment Analysis is the task of finding the sentiment (positivity or negativity) of a text. This process can be an essential feature for bullying detection, hate speech detection, and fake news detection models. However, there are many libraries and functions available to use, and choosing the optimal one that has the most accuracy to use as a feature for other Natural Language Processing (NLP) tasks is essential. During our search to find functions that calculate sentiment, otherwise known as pre-trained sentiment models, we find that most of the models are trained on polarised texts and datasets that already have sentiment annotated. Hence, we use a nonpolarised text form: news articles and we use a dataset that is not meant to be used with Sentiment Analysis. This helps us test different models. Through our experiments we find that VADER, Stanza, and Transformer’s distilBERT work with a good accuracy, and are ideal to use as the basis for calculating a sentiment feature whereas lexical-based models such as TextBlob and using SentiWordNet are less accurate in its calculations. The dataset we use is the Reuter 50 50 dataset that is meant to be used for another NLP task: Author Identification/Author Profiling. [60] Working with the dataset led to our second task: to create a model that works with minimal requirements for data and computational power but still gives a comparable accuracy to other deep learning methods of Author Identification. Author Identification is that task of being given a random text that could have been written by a group of suspect authors, and finding with reasonable accuracy, which author wrote the text. This is useful to stop other people taking credit for works they have not written and giving credit to the rightful author. We find that using a mixture of word-based features, stylometric feature, and syntax/punctuation features are good features to train a model for Author Identification. We also find that the Boosting class of algorithms outperforms other classes such as Linear models, Nearest Neighbour-Based Models, and Tree Models. XGBoost performs the best out of all the algorithms we test. Linear Models such as SVMs, the Naive Bayes, and the K-Nearest Neighbour algorithm perform abysmally.

Abstract

Crowdsourcing effectively solves a large variety of tasks by employing a distributed human population. Information aggregation from multiple reports provided by potentially unreliable or malicious agents is a primary challenge in crowdsourcing systems. As a result, research in this area has focused on incentivising agents to exert efforts and report truthfully. In particular, Peer Based Mechanisms (PBMs) appropriately reward agents for reporting accurately and truthfully. However, we observe that with PBMs, crowdsourcing systems may not be fair. As PBMs evaluate agents’ reports based on their consistency with their peers, agents may not receive deserved rewards despite investing efforts and reporting truthfully. Unfair rewards for the agents may discourage participation. Motivated by this, we aim to build a general framework that assures fairness in PBMs. Towards this, we propose the idea of providing trustworthy agents with additional chances of pairing while evaluating their reports. Providing additional chances will help to reduce the penalty obtained by trustworthy agents from unfair pairings, improving their expected reward. To decide which agents to give additional chances we adopt a reputation model that quantifies agents’ trustworthiness in the system. Based on this approach, we build a general iterative framework, REFORM, which adopts the reward scheme of any existing PBM and uses a suitable reputation model. To quantify fairness in PBMs, we introduce two general notions of fairness for PBMs, namely γ-fairness and qualitative fairness. γ-fairness is based on the proximity of the expected rewards a PBM assures to a truthful agent with the optimal reward it can provide. Qualitative fairness prioritises agents who consistently report accurate over other agents. In this work, we also consider that the tasks in the setting are time-sensitive. The task’s requester expects agents to submit the task reports at the earliest. We refer to such a setting as temporal settings. In a temporal setting, the reputation model needs to consider both accuracy of reports and the time taken to report. However, no existing reputation models consider the time taken to report. Towards this, we introduce Temporal Reputation Model (TERM) to quantify an agent’s trustworthiness in a temporal setting. TERM assigns scores to the agents based on their reporting behaviour and the time taken to report. Later, we demonstrate REFORM’s significance by deploying the framework with RPTSC’s reward scheme and TERM. Specifically, we prove that REFORM considerably improves fairness; while incentivising truthful and early reports. Furthermore, we conduct synthetic simulations to validate our results.

Abstract

Decision-making is hard when presented with a large set of similar options that insignificantly tradeoff amongst a range of attributes. This phenomenon is encountered within the skyline set because no skyline point is strictly better than any other skyline point, and therefore, every skyline point can excel ever so slightly in some subspace of attributes. The objective of this work is to determine the set of all sets of skyline points that are hard to choose from and hence all points in a set compete for attention from the same type of consumer (such sets are called competitive skyline cliques) In this work, two skyline points are defined to be competitive if the differences between the two points across each attribute are bounded by specified thresholds. We introduce maximal competitive skyline cliques (MCSCs) – maximal sets of mutually competitive skyline points and provide algorithms that enumerate all MCSCs. While the problem of enumerating all MCSCs is structurally similar to the NP-Hard problem of enumerating all maximal cliques in a graph, because of properties exhibited by our competitiveness metric, we show that enumerating all MCSCs can be performed efficiently in polynomial time. Furthermore, in 2D space, we provide an optimal linear-time sweepline algorithm. We also provide a bounded approximation for MCSCs that are easier to enumerate. Our extensive experiments using both synthetic and real (UCars, FIFA22 and Pokemon) datasets demonstrate the ´ efficacy of MCSCs and the efficiency of our algorithms.

Abstract

For decades, process technology scaling has catered to ever increasing demands of high-speed integrated circuits. Many applications, such as image and multimedia data processing, artificial intelligence, machine learning etc., depend on these circuits to process huge swaths of data in real time. Be it in the data centers, the cloud networks or the mobile devices, the computing efficiency of the circuits has had to play catch-up with this ever-increasing demand. However, it has become abundantly clear in recent years that cranking the technology knob isn’t a feasible solution to future needs of high-speed computing. Across the spectrum, data processing applications are endowed with resilience to acceptable levels of errors in the underlying computations. These applications incorporate, what can only be called a “forgiving” nature, into their algorithms without compromising with the end-user experience. And this intrinsic robustness to errors can be leveraged even further by the design methodology of Approximate Computing. It has now become an independent field that explores methods to reduce computation costs by allowing minor degradation in intermediate computations. Several approximate arithmetic units have been extensively studied and implemented with significant impacts on the system costs in terms of power, area and speed. Image processing algorithms with intrinsic robustness to errors can be approximated for significant resource and energy savings while still meeting the end-user requirements. FPGA-based implementations can increase their suitability for real-time high-speed multimedia applications by leveraging Approximate Computing. With high volume of pixel level computations, algorithms such as the Harris Corner Detector (HCD) and Unsharp Making (USM), become targets for such approximation strategies. In this thesis, we propose hardware implementations of these algorithms that rely on approximating the intermediate multiplication operations using Dynamic Range Unbiased Multiplier (DRUM). With runtime configurable bit-width control of DRUM instances, their quality of outputs is shown to depend on the varying accuracy. We explore how the errors due to approximate operations propagate to the output. Further,the experimental results from implementations on Virtex-7 and Zynq-7000 FPGA devices are documented and an analytical approach based on quality metric comparisons with base implementation is presented.

Abstract

The task of Visual Grounding is at the intersection of computer vision and natural language processing tasks. The Visual Grounding (VG) task requires spatially localizing an entity in a visual scene based on its linguistic description. The capability to ground language in the visual domain is of significant importance for many real-world applications, especially for human-machine interaction. One such application is language-guided navigation, where the navigation of autonomous vehicles is modulated using a linguistic command. The VG task is intimately linked with the task of vision-language navigation (VLN), as both the tasks require reasoning about the linguistic command and the visual scene simultaneously. Existing approaches to VG can be divided into two categories based on the type of localization performed: (1) bounding-box/proposal-based localization and (2) pixel-level localization. This work focuses on pixel-level localization, where the segmentation mask corresponding to the entity/region referred to by the linguistic expression is predicted. The research in this thesis focuses on a novel modeling strategy for visual and linguistic modalities for the VG task, followed by the first-ever visual grounding based approach to the VLN task. We first present a novel architecture for the task of pixel-level localization, also known as Referring Image Segmentation (RIS). The architecture is based on the hypothesis that both intra-modal (wordword and pixel-pixel) and inter-modal (word-pixel) interactions are required to identify the referred entity successfully. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intra-modal interactions. We address this limitation by performing all three interactions synchronously in a single step. We validate our hypothesis empirically against existing methods and achieve State-Of-the-Art results on RIS benchmarks. Finally, we propose the novel task of Referring Navigable Regions (RNR), i.e., grounding regions of interest for navigation based on the linguistic command. RNR is different from RIS, which focuses on grounding an object referred to by the natural language expression instead of grounding a navigable region. We additionally introduce a new dataset, Talk2Car-RegSeg, which extends the existing Talk2car dataset with segmentation masks for the regions described by the linguistic commands. We present extensive ablations and show superior performance over baselines on multiple evaluation metrics. A downstream path planner generating trajectories based on RNR outputs confirms the efficacy of the proposed framework.

Abstract

Finding or tracking the location of an object with precision and accuracy is a crucial problem in defence applications, robotics and computer vision. Radars fall into the spectrum of high-end defence sensors or systems upon which the security and surveillance of the entire world depends. There has been a lot of focus on Multi Sensor Tracking (MST) in recent years, with radars as the sensors. The Indian Air Force (IAF) uses a MST system to detect flights pan India, developed and supported by Bharat Electronics Limited (BEL), a defence agency that we are working with. In this thesis, we describe the Machine Learning approaches, which are built on top of the existing system the Air force uses currently. We have used 3 Machine Learning approaches, the first on a smaller dataset and the second and third on a bigger dataset. Each subsequent approach does a better job of reducing the errors faced by the system. The final approach trained our models on about 11 million anonymized real Multi Sensor tracking data points provided by radars performing tracking activity across the Indian air space. This approach has shown an increase in tracking accuracy by 5.5% from 91.5% to 97%. The model and the corresponding code were transitioned to BEL, which have been tested in their simulation environment with a plan to take forward for ground testing. Our final approach comprises of 3 steps: (a) We train a Neural Network and a CatBoost model and ensemble them using a Logistic Regression model to predict one type of error, namely Splitting error, which can help to improve the accuracy of tracking. (b) We again train a Neural Network and a CatBoost model and ensemble them using a different Logistic Regression model to predict the second type of error, namely Merging error, which can further improve the tracking accuracy. (c) We use cosine similarity to find the nearest neighbour and correct the data points predicted to have Splitting/Merging errors by predicting the original global track of these data points.