Abstract

Age related macula degeneration (AMD), Cystoid Macular Edema (CME) and glaucoma are retinal diseases that affects vision and often lead to irreversible blindness. Many imaging modalities have been used for screening of retinal diseases. Optical coherence tomography (OCT) is one such imaging modality that provides structural information of retina. Optical coherence tomography angiography (OCTA) also provides vascular information of retina in addition to structural information. With advancement in OCT and OCTA technology, the number of patients being scanned using these modalities is increasing exponentially. Manual analysis of vast data often makes human experts feel fatigue. Also, this extends the time to treat any patient and creates a demand to develop a fast and accurate automated OCT image analysis system. The system proposed in this thesis aims at analysing retinal anatomy and its diseases. We approach the problem of automatically analysing the retinal anatomy by segmenting its layers using a deep learning framework. In literature these algorithms usually requires preprocessing steps, such as denoising, image flatenning and edge detection, all of which involve separate parameter tunings. We propose a deep learning technique to automate all these steps and handle the presence/absence of pathologies. This model consists of a combination of Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM). The CNN is used to extract layers of interest image and extract the edges, while the LSTM is used to trace the layer boundary. This model is trained on a mixture of normal and AMD cases using minimal data. Validation results on three public datasets show that the pixel-wise mean absolute error obtained with our system is 1.30 ± 0.48 which is lower than inter-marker error of 1.79 ± 0.76. The performance of the proposed module is also on par with the existing methods. We next propose three modules for disease analysis, first for diagnosing them at volume level, second at slice level and finally at pixel level by segmenting the abnormality. Firstly, we propose a novel method for glaucoma assessment at volume level using data from OCTA modality. The goal of this module is to gain insights about glaucoma indicators from both structural information (OCT volume) and vascular information (angioflow images). The proposed method achieves a mean sensitivity of 94% specificity of 91% and accuracy of 92% on 49 normal and 18 glaucomatous volumes. Secondly, for slice level disease(AMD and CME) classification from OCT volumes, we learn a decision boundary using a CNN in the new extremal representation space ofan image. Evaluating on four publically available datasets with training set consisting of 3500 OCT slices and test set having 1995 OCT slices, this module achieves a mean sensitivity of 96%. specificity of 97% and accuracy of 96%. Finally, we propose an automated cyst segmentation algorithm from OCT volumes. We propose a biologically inspired method based on selective enhancement of the cysts, by inducing motion to a given OCT slice. A CNN is designed to learn a mapping function that combines the result of multiple such motions to produce a probability map for cyst locations in a given slice. The final segmentation of cysts is obtained via a simple clustering of the detected cyst locations. The proposed method is trained on OPTIMA Cyst segmentation challenge (OCSC) train test and achieves a mean Dice Coefficient (DC) of 0.71, 0.69 and 0.79 on the OCSC test set, DME dataset and AEI dataset respectively. Thus, the complete system can be employed in screening scenarios to aid retinal experts in accurate and faster analysis of retinal anatomy and disease.

Abstract

With overwhelming amount of visual data on the internet, it is beyond doubt that a search capability for this data is needed. While several commercial systems have been built to retrieve images and videos using meta-data, many of the images and videos do not have a detailed descriptions. This problem has been addressed by content based image retrieval (CBIR) systems which retrieve the images by their visual content. In this thesis, we will demonstrate that images and videos can be retrieved using the pose of the humans present in them. Here pose is the 2D/3D spatial arrangement of anatomical body parts like arms and legs. Pose is an important information which conveys action, gesture and the mood of the person. Retrieving humans using pose has commercial implications in domains such as dance (query being a dance pose) and sports (query being a shot). In this thesis, we propose three pose representations that can be used for retrieval. Using one of these representations, we will build a real-time pose retrieval system over million movie frames. Our first pose representation is based on the output of human pose estimation algorithms (HPE) [5, 26, 80, 103] which estimate the pose of the person given an image. Unfortunately, these algorithms are not entirely reliable and often make mistakes. We solve this problem by proposing an evaluator that predicts if a HPE algorithm has succeeded. We describe the stages required to learn and test an evaluator, including the use of an annotated ground truth dataset for training and testing the evaluator, and the development of auxiliary features that have not been used by the (HPE) algorithm, but can be learnt by the evaluator to predict if the output is correct or not. To demonstrate our ideas, we build an evaluator for each of four recently developed HPE algorithms using their publicly available implementations: Andriluka et al. [5], Eichner and Ferrari [26], Sapp et al. [80], and Yang and Ramanan [103]. We demonstrate that in each case our evaluator is able to predict whether the algorithm has correctly estimated the pose or not. In this context we also provide a new dataset of annotated stickmen. Further, we propose innovative ways in which a pose evaluator can be used. Specifically, we show how a pose evaluator can be used to filter incorrect pose estimates, to fuse outputs from different HPE algorithms, and to improve a pose search application. Our second pose representation is inspired by poselets [13] which are body detectors. First, we introduce deep poselets for pose-sensitive detection of various body parts, that are built on convolutional neural network (CNN) features. These deep poselets significantly outperform previous instantiations of Berkeley poselets [13]. Second, using these detector responses, we construct a pose representation that is suitable for pose search, and show that pose retrieval performance is on par with the state of the artpose representations. The compared methods include Bag of visual words [85], Berkeley poselets [13] and Human pose estimation algorithms [103, 18, 68]. All the methods are quantitatively evaluated on a large dataset of images built from a number of standard benchmarks together with frames from Hollywood movies. Our third pose representations is based on embedding an image into a lower dimensional pose-sensitive manifold. Here we make the following contributions: (a) We design an optimized neural network which maps the input image to a very low dimensional space where similar poses are close by and dissimilar poses are farther away, and (b) We show that pose retrieval system using these low dimensional representation is on par with the deep poselet representation. Finally, we describe a method for real time video retrieval where the task is to match the 2D human pose of a query. A user can form a query by (i) interactively controlling a stickman on a web based GUI, (ii) uploading an image of the desired pose, or (iii) using the Kinect and acting out the query himself. The method is scalable and is applied to a dataset of 22 movies totaling more than three million frames. The real time performance is achieved by searching for approximate nearest neighbors to the query using a random forest of K-D trees. Apart from the query modalities, we introduce two other areas of novelty. First, we show that pose retrieval can proceed using a low dimensional representation. Second, we show that the precision of the results can be improved substantially by combining the outputs of independent human pose estimation algorithms. The performance of the system is assessed quantitatively over a range of pose queries.

Abstract

Diabetic Retinopathy (DR) is a condition where individuals with diabetes develop a disease in the inner wall of eye known as retina. DR is a major cause of visual impairments and early detection can prevent vision loss. Use of automatic systems for DR diagnosis is limited due to their lower accuracy. As an alternative, reading-centers are becoming popular in real-world scenarios. Reading center is a facility where retinal images coming from various sources are stored and trained personals(who might not be experts) analyze them. In this thesis we look at techniques to increase efficiency of DR image-readers working in reading centers. The first half of this thesis aims at identifying efficient image-reading technique which is both fast and accurate. Towards this end we have conducted an eye-tracking study with medical experts while they were reading images for DR diagnosis. The analysis shows that experts employ mainly two types of reading strategies: dwelling and tracing. Dwelling strategy appears to be accurate and faster than tracing strategy. Eye movements of all the experts are combined in a novel way to extract an optimal image scanning strategy, which can be recommended to image-readers for efficient diagnosis. In order to increase the efficiency further, we propose a technique where saliency of lesions can be boosted for better visibility of lesions. This is named as an Assistive Lesion Emphasis System(ALES) and demonstrated in the second half of the thesis. ALES is developed as a two stage system: saliency detection and lesion emphasis. Two biologically inspired saliency models, which mimic human visual system, are designed using unsupervised and supervised techniques. Unsupervised saliency model is inspired from human visual system and achieved 10% higher recall than other existing saliency models when compared with average gazemap of 15 retinal experts. Supervised saliency model developed as deep learning based implementation of biologically inspired saliency model proposed by Itti-Koch(Itti, L., Koch, C. and Niebur, E., 1998) and achieves 10% to 20% higher AUC compared to existing saliency model when compared with manual markings. Saliency maps generated by these models are used to boost the prominence of lesions locally. This is done using two types of selective enhancement techniques. One technique uses multiscale fusion of saliency map with original image and other uses spatially varying gamma correction; both increases CNR of lesions by 30%. One saliency model and one selective-enhancement technique are clubbed together to illustrate two complete ALESs. DR diagnosis done by analyzing ALES output using optimal strategy should presumably be faster and accurate.

Abstract

In most of the applications, wireless sensor networks (WSNs) are deployed in an unattended environment. Thus, sensor nodes can be physically captured by the adversaries, and the adversaries can then extract the stored information in those nodes using the power analysis attacks. Using the extracted information they can manufacture new sensor nodes. These newly manufactured sensor nodes can also have additional functionalities, which can be used to launch different attacks. After that these new sensor nodes can be deployed by the adversaries in the network. These nodes can launch different attacks in the network, which can cause information loss along with high energy expenditure. Furthermore, WSN is prone to various attacks, such as blackhole attack, wormhole attack, sinkhole attack, etc. In a blackhole attack, an attacker can physically capture and re-program a set of nodes in the network to block the packets they receive instead of forwarding them towards the BS. Any information that enter in the blackhole region is then compromised, and the information does not reach to the destination. This produces high end-to-end delay, and decrease in packet delivery ratio and network throughput. As a result, the information cannot reach to the destination node within the required time period. In a sinkhole attack, an attacker node first advertises a best possible route (with less hop-distance route) to the destination (BS) to attract its neighbors so that they may fall into this attraction to utilize the advertised route more frequently. The neighbors can then forward their traffic through the efficient advertised route declared by the sinkhole attacker node. The route can also captivate other nodes apart from the neighbor nodes of the sinkhole attacker node, which are closer to the sinkhole than the BS. So, the attacker node has an opportunity to tamper with the data, damage the regular network operations or conduct other serious threats. Sometimes multiple attacks can also be launched in the network using the hybrid anomaly. Hybrid anomaly degrades the network performance rapidly and also trouble the attack specific detection mechanisms. In this situation, it is very crucial to find out what kind of anomaly is activated in the network. In the presence of blackhole attack, sinkhole attack and hybrid anomaly, the confidential information can be delayed, leaked, altered or even can not reach to the destination. Therefore, it becomes essential to provide robust and efficient solutions for the detection and prevention of blackhole attack, sinkhole attack and hybrid anomaly in HWSN. This motivates us to propose new detection and prevention protocols for these types of attacks in HWSN. In this thesis, we propose several intrusion detection and prevention schemes in hierarchical wireless sensor networks (HWSNs). The first contribution is on designing a new efficient group-based scheme for the detection and prevention of multiple blackhole attacker nodes in HWSNs. In our scheme, the entire WSN is divided into several clusters, and each cluster has a resource-rich cluster head (CH) node. Each CH is responsible for the detection and prevention of blackhole attack in the network. For the detection purpose, each CH uses the proposed detection algorithm by which it sends status-cum-data query messages to all its cluster members (sensor nodes). CH then waits for the reply to those messages from its members. Node status messages (e.g. sleeping, working, idle) are merged with data query messages in order to further reduce the total communication cost of the proposed scheme. After the detection of blackhole nodes, CH nodes use the proposed responsibility reassignment algorithm in order to reassign the responsibilities of the blacklisted nodes (blackhole attacker nodes) to the normal nodes. Our scheme is secure against blackhole attacker nodes, which is shown through the analytical and the simulation results using the widely-accepted NS2 simulator. The proposed scheme achieves about 90% detection rate and 3.75% false positive rate, which are significantly better than the existing related schemes. Furthermore, our scheme is efficient, and thus, it is very appropriate for practical applications in HWSNs. In the second contribution, we propose a new detection scheme for the detection of different types of sinkhole nodes in HWSNs. To the best of our knowledge, this is the first attempt to design such a detection scheme in HWSNs which can detect sinkhole message modification nodes (SM D), sinkhole message dropping nodes (SDP ) and sinkhole message delay nodes (SDL) nodes. In our approach, the entire HWSN is divided into several disjoint clusters, and each cluster has a resource-rich high-end sensor node (called a cluster head), which is responsible for the detection of different sinkhole attacker nodes if present in that cluster. Our scheme is secure against sinkhole attacker nodes, which is shown through the ana

Abstract

In remote sensing, multi-sensor data fusion aims at combining strengths of different data sources in a manner such that the resultant fused product preserves the best of the participating sources. From among all the levels at which fusion is done, namely pixel level, feature level and decision/information level, pixel level data fusion is the most elementary form of fusion. The real advantage with pixel level fusion is that it has the actual raw input data at its disposal which could be transferred to the output in a form closest to the input. Hence, pixel level algorithms need to aim at preserving the input properties rather than just improving the overall visual quality. Most of the existing methods are aimed at visual quality improvement and perform fusion using the global distributions, which results in loss of uniqueness of certain objects features. Such objects are either completely lost or their responses are milder in the output. These kind of local distributions can be preserved using a localized method of fusion. Other limitations of fusion methods include distortion of overall spectral distribution of the input data source and their applicability on the type of data being fused. There is a Gaussian Linear Stretch Image (GLSI) technique [1] for images fusion to handle spectral distortion. It is a moving window-based distribution matching and details are transferred in a linear manner. We have enhanced this algorithm to effectively handle two major challenges it faced. One is the detection and matching of class distributions which are inverted in the low and high resolution images and the other being the fusion of pixels belong to class distributions with significantly low sample space. We call it the Enhanced Gaussian Linear Stretch Image (e-GLSI) algorithm. The proposed e-GLSI algorithm is also a window based technique which aims at improving spatial resolution of multispectral bands using higher resolution panchromatic band. e-GLSI fuses the two inputs by transferring the spatial details of the panchromatic band into each multispectral band independently at local window level, hence prevents suppression of local unique features and preserves the spectral distribution of each band as well. In this approach, the local window statistics are used to establish linear relationships between corresponding windows from two different spatial resolution bands. Cases of distribution inversions and unimodal splits are handled using well defined metrics before fusing two distributions together. The algorithm then enhances each band independently without using any derived or aggregated knowledge, thus making this technique suitable to fuse any combination of input images. Results show that the statistical profile of the e-GLSI output bands matches the original multispectral image bands’ statistical profile better than other widely used approaches. In order to evaluate this matching quantitatively, standard deviation of the relative difference in fused pixel value and original multispectral pixel value per band has been computed. e-GLSI output gives a standard deviation of 0.18, 0.12 and 0.26 for red, blue and green bands respectively. Thus showing that it’s able to preserve the spectral characteristics from input images. In Spite of being able to preserve the spectral characteristics of the input images, there are some artifacts introduced in the e-GLSI fusion output. One of them is objects getting split across the moving window of the technique, which introduces a window boundary or sharp edge in some of the objects. A way to handle this could be the use of a variable sized window depending on the area of the image being processed. But this in itself is a challenge as it would require surrounding information and certain heuristics in order to decide the window size at a location in an image. In order to tackle this challenge, we used objects as windows in our approach. We propose an object based fusion algorithm based on GLSI, which we call the Object based GLSI (o-GLSI). Processing blocks for this approach are are the objects segmented from the Pan image. Since pixels being fused are part of a single object, we transfer details in a Gaussian manner in case of o-GLSI. The quantitative evaluation of o-GLSI fusion results clearly show that o-GLSI standard deviation of 0.32, 0.49 and 0.82 for relative difference in fused pixel and ground truth multispectral pixel for red, blue and green bands respectively is successful in keeping the fusion output spectrally closer to the input. Universal Image Quality Index (UIQI) [2] values of 0.9 for each band with respect to the ground truth proves that the quality of image fused by o-GLSI is very good. Since classification is one of the prominent applications for fused images, a superior accuracy rate supports the effectiveness of the fusion algorithm involved. As another measure to evaluate the utility of the fused images, k-means classification accuracy has been co

Abstract

Empathy entails the ability to understand (perspective-taking) and share the affective experiences of others, both processes considered the bedrock of all social interactions and critical for survival. Most empathy related neuroscience studies employ simple paradigms like pain-infliction or static images with emotional context and analysis focuses on either perspective-taking processing or affective experience overlooking the complex interaction among the processes. The initial studies informed how one understands pieces of the whole system with an assumption that when information from the pieces is fit the whole picture can be realized but this methodology has led to constrained theories on empathy process. To address the short-comings in understanding empathy, the focus of affective neuroscience research has shifted to studying this phenomenon as part of naturalistic social cognition. Hence with the goal to mimic real-life situations that evoke empathy, naturalistic or ecologically-valid paradigms are recent inclusions building upon the foundation provided by the earlier simple models. This study focuses specifically on narratives and empathy response. The aim of this study is to explore the neural correlates of narrative empathy using multi-modal visual movie narrative. Towards this, two functional magnetic resonance imaging (fMRI) studies were conducted using a total of 5 commercial full-length short film and movie clips. The first experiment had 3 movie clips (5-8minute long) from a diverse genre (animation, Hollywood and Indian Hindi movie) and the goal of this exploratory investigation was to identify the networks of empathy – cognitive (mentalizing), emotional and motor (affective experiences) and analyzing the dynamic nature of the neural activations as a function of the self-reported level of empathy experienced. The data-driven independent component analysis (ICA) method applied to isolate the underlying brain networks revealed: a) the ability of the fictional narratives induces emotional contagion and the potential to elicit empathy response b) activation of significant, distinct but overlapping empathy networks and c) the dynamic cross-correlation analysis between the task default-mode network (DMN) and individual empathy networks revealed that emotional empathy response displayed lower correlation while cognitive empathy higher correlation with DMN, suggesting the possible role of self-reflection or attention to an internal narrative triggered by external stimuli (Andrews-Hanna et al., 2012; Gusnard et al., 2001). This is a potentially significant finding for understanding the difference in cognitive and emotional empathy in clinical conditions like autism and psychopathy. Developing on the findings from the first study, a second fMRI study was designed to understand specifically the role of context in empathy response. Toward this, two diverse full-length short movies (8 minutes each) with real-life actors of ethnicity dissimilar to the participants were selected as stimuli. Functional imaging data was collected from two sets of extremely short clips (13-20seconds) with neutral and emotional scenes extracted from the full-length movie followed by the whole movie. Using a general linear model approach for data analysis, the contrasts estimated from each condition (activation for the short clips of neutral and emotional scenes versus the same clips in the full-length movie) were compared and the analysis showed that empathy sub-processes (emotional, cognitive and motor) were statistically significant when context is presented. The results also reveal the complex differential empathy processes in real-life social interactions, where responses are a function of context. Overall, the neural correlates of narrative empathy presented in this thesis, the first of the kind, suggest that a complex construct such as empathy should be best studied in with ecologically valid stimuli. The impact of the results, in studying empathy deficiency and interventions using narratives to enhance empathy, needs to be explored in the future.

Abstract

The rst problem is on anonymous communication, specically the so called "Anonymous Leader". A leader is one who has some special function such as decision making, etc. An anonymous leader is a leader who is anonymous, i.e.. no body knows who he is. The problem asks how a leader can perform its special function without revealing itself. We note that communication is the primary constraint to being anonymous. The anonymous leader must be able to send and receive messages without revealing its identitiy. Morever, other nodes/players must not be able to pretend to be the anonymous leader. There are many possible applications of this anonymous leader setting and here we will also describe one such application. Formally, we give denition of anonymous leader and give protocols with proof that enable anonymous communication while satisfying the dual requirements the leader must not be revealed, and other nodes/players should not be able to pretend to be the leader. Interestingly, this is the rst such formalisation in the literature to date; while anonymous communication itself is common, restricting it to only one anonymous node/party has not been attempted before The second problem is on broadcast, specifically ”Simultaneous Broadcast”. There are situations where the broadcasted message is required to reach nodes at the same time. This is especially true in auction bidding and gambling. Admitedly, it is possible to ensure that the time difference between reciepients is small enough to ensure that perfect simultanity is not required. Nevertheless this problem is interesting in its own right, especially considering the difficulty and the lack of any theory in the literature. Here, we try to solve the problem using ”time lock puzzles”. Time lock puzzles enable us to lock messages in such a way that a fixed amount of time is required to be invested to unlock the message. This can be used to setup synchrony in message receipt. However, the constraints and limitations are considerable, forcing this solution to be ultimately purely a first attempt at the problem. Formally, we give definition and protocol for simultaneous broadcast using time lock puzzles.

Abstract

Simultaneous Localization and Mapping (SLAM) refers to the problem of mapping an unknown environment that the robot is operating in and localizing itself in the unknown environment at the same time. Out of the various methods of performing SLAM, using a single monocular camera as the sole sensory input is highly preferred due to its simplicity and low power consumption. Range sensors such as laser range finders, depth cameras etc require much more power to operate and performing SLAM with them is more computationally intensive as compared to SLAM with a single camera. However, when compared to trajectory planning methods using depth-based SLAM, Monocular SLAM in loop does need additional considerations. One main reason being that for a robust optimization of the map and robot trajectory, using Bundle Adjustment (BA) in the case of most monocular SLAM methods, the SLAM system needs to scan the area for a reasonable duration to gather more information about the area to improve the map and pose estimates. Additionally, due to the way monocular SLAM methods work, they do not tolerate large camera rotations between successive views and tend to breakdown. Other reasons for Monocular SLAM failure include ambiguities in decomposition of the Essential Matrix, feature-sparse scenes and more layers of non linear optimization apart from BA. Learning a complex task such as low-level robot manoeuvres while preventing failure of monocular SLAM is a challenging problem for both robots and humans. The data-driven identification of basic motion strategies in preventing monocular SLAM failure is a largely unexplored problem. In this thesis, a computational model is devised for representing and inferring strategies for the problem, formulated as a Markov Decision Process (MDP), where the reward function models the goal of the task as well as information about the strategy. Reinforcement Learning (RL) is used with an intuitive, handcrafted reward function to generates fail safe trajectories wherein the SLAM generated outputs (scene structure and camera motion) do not deviate largely from their true values. This model is expanded on by treating it as an expert and try to learn an underlying true reward function for the given task at hand using Inverse Reinforcement Learning (IRL). Quintessentially, the framework successfully learns the otherwise complex relation between motor actions and perceptual inputs and uses this knowledge to generate trajectories that do not cause failure of SLAM. It also learns how a few chosen parameters affect the task at hand. This complex relation is almost intractable to capture in an obvious mathematical formulation. The framework allows one to identify the way in which a few chosen parameters affect the quality of monocular SLAM estimates. The estimated reward function was able to capture expert demonstration information and the inherent expert strategy and it was possible to give an intuitive explanation to the obtained reward structure. Drastic improvements in the quality of the SLAM map and trajectory are systematically shown in simulations. Better performance is demonstrated over other methods based on maximizing scene overlap, supervised learning based methods and over state-of-the-art methods described in literature that select the next view from a candidate set of views based on the best score calculated according to SLAM quality measures. The proposed method scales effectively across various SLAM frameworks in real world experiments as well as in simulations with a mobile robot.

Abstract

Sentiment analysis has emerged as a field, that has attracted a significant amount of attention over the last decade. This is the area of study to analyze people reviews, songs and attitudes from different types of data and classify whether it is a positive, negative or neutral. Recent advancement of social media which is an enormous ever-growing source has led people to share their views through various modalities such as audio, text and video. This source of information is important to automatically make out the sentiment embedded in the different types of data such as reviews and songs. In this thesis, an improved multimodal approach to detect the sentiment of product reviews and songs based on their multi-modality natures (audio and text) is proposed. The basic goal is to classify the input data as either positive or negative sentiment. Database used in this study are Spanish product reviews, Hindi product reviews and Telugu songs. Most of the existing systems for audio or speech based sentiment analysis use the conventional audio features which are extracted from entire signal, but they are not domain specific features to extract the sentiment. In this work, instead of extracting the features from entire signal, a specific regions of an audio signal have been identified and experiments are performed on these regions by extracting the relevant features. For songs data, experiments are performed over each song, and its beginning and ending regions. For all these cases, Gaussian Mixture Models (GMM) and Support Vector Machine (SVM) classifiers are built using the prosody, temporal, spectral, tempo and chroma features. Experimental results shown that the ate of detecting the sentiment of a song is high at beginning of a song compared to its ending region and over the entire duration of the song. This is because, the instruments and vocals which convey the sentiment for beginning part of the song may or may not sustain throughout the song. For the reviews data, we could not perform these experiments because in such cases the sentiment may not be present at beginning or ending regions of an utterance. So for the reviews data, the stressed and normal regions are identified using the strength of excitation. From the stressed regions, Mel Frequency Cepstral Coefficients (MFCC) features are extracted and GMM classifier is built. Further, experiments are performed by extracting the prosody (energy, pitch and duration) and relative prosody features from both the regions and from the entire audio signal and a GMM classifier is built. From the results, it is observed that, the performance at specific regions is better as compared to the entire signal. It is also observed that, relative prosody features extracted from both the regions has high accuracy of detecting the sentiment compared to the prosody and MFCC features. This is because, the natural variations present in the prosody features are reduced using the relative prosody features. Recently, neural networks have achieved good success on sentiment classification. In this work also different deep learning architectures like Deep Neural Network (DNN) and Deep Neural Network Attention Mechanism (DNNAM) are explored. Here stressed regions concept fail because of less training data. DNN performance depends on the amount of training data. The more the training data, the more accurate it is. So here the experiments are performed by combination of frames which result in better performance because each frame will not carry the sentiment. MFCC features considered are 13-dimensional, 65-dimensional and 130-dimensional feature vectors. From the studies, it is observed that DNNAM classifier gives better results compared to DNN, because the DNN approach is a frame based one where as the DNNAM approach is a utterance level classification there by efficiently making use of the context. For text based sentiment analysis, transcriptions are carried out manually from the audio signal. For song classification, SVM and Naive-Bayes classifiers are built using the textual features which are computed by Doc2Vec vectors. As in the audio, here also experiments are performed at the beginning, the ending and over the entire song. The studies shown that beginning of a song has high accuracy in detecting the sentiment compared to the ending region and over the entire song. As similar experiments could not be carried out with the reviews data, entire document is taken as an input to extract the sentiment. Support Vector Machine (SVM) and Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) classifiers are used to develop a sentiment model with the textual features computed by Doc2Vec and Word2Vec. From the experimental studies, it is observed that LSTM-RNN outperforms the SVM because LSTM-RNN is able to memorize long temporal context. Finally, we combine both the modalities such as audio and text to extract the sentiment. Both the modalities are hypothesized based on

Abstract

Technological advancements have made human beings dependent on machines in unprecedented ways. High precision in many tasks and a vast, tractable memory confirms their presence as an integral part of human lifestyle. Human-machine interaction, then, is the evident next step in developing smarter systems and consequently, a research topic of foremost importance. Effective communication is achieved by means of natural language and hence, language becomes an obvious necessity for human-machine communication too. The vast majority of languages spoken in a multi-ethnic society mandates the necessity of development of multilingual systems. India is one such region which is inhabited by highly diverse multicultural societies and a multitude of ethnic groups living as a nation. An effective Spoken Dialog System (SDS) developed for Indian scenario has to keep in mind the shared characteristics of languages spoken across India. This motivates the dire need to develop multilingual human-machine interfaces which can exploit this commonality. Portability and ever-increasing computing capacity have rendered mobile phones to be the most common form of communication agents. But in absence of network connectivity, they lose a lot of power residing in cloud platforms and have to rely on local resources which are limited and scarce in nature. This problem is ubiquitous because there are several locations and situations when network is not available but basic communication is crucial for information retrieval. One such situation is Healthcare for preventable diseases in rural areas. This thesis is focused to alleviate some of the problems which arise when humans move to interact with hand-held conversational machine agents in a multilingual setting such as India. With the advent of social media, code-mixing became a popular phenomena in spoken language. This work progresses to develop a spoken dialog system for the domain of Healthcare and focuses on problems of 1. recognizing the language of communication within speech segments through large-scale spoken language modeling and 2. spotting keywords in speech signal by creating a robust representation of articulatory gestures in speech. The proposed approaches are based on recurrent and convolutional architectures of neural networks and have been experimentally shown to outperform the state-of-the-art on standard datasets.