Abstract

The amelioration in video capture technology has made recording videos very easy. The introduction of smaller affordable cameras which not only boast high-end specifications but are also capable of capturing videos at very high resolution (4K,8K and even 16K) have made recording of high quality videos accessible to everyone. Although this makes recording of videos very straightforward and effortless, a major part of the video production process - Editing is still labor intensive and requires skill and expertise. This thesis takes a step towards automating the editing process and making it less time consuming. In this thesis, (1) we explore a novel approach of automatically editing stage performances such that both the context of the scene as well as close-up details of the actors are shown. (2) We propose a new method to optimally re-target videos to any desired aspect ratio while retaining the salient regions that are derived using gaze tracking. Recordings of stage performances are easy to capture with a high-resolution camera, but are difficult to watch because the actors’ faces are too small. We present an approach to automatically create a split screen video that transforms these recordings to show both the context of the scene as well as close-up details of the actors. Given a static recording of a stage performance and the tracking information about the actors positions, our system generates videos showing a focus+context view based on computed close-up camera motions using crop-and zoom. The key to our approach is to compute these camera motions such that they are cinematically valid close-ups and to ensure that the set of views of the different actors are properly coordinated and presented. We pose the computation of camera motions as convex optimization that creates detailed views and smooth movements, subject to cinematic constraints such as not cutting faces with the edge of the frame. Additional constraints link the close up views of each actor, causing them to merge seamlessly when actors are close. Generated views are placed in a resulting layout that preserves the spatial relationships between actors. This eliminates the need for manual labour and expertise required for both capturing the performance and later editing it, instead the splitscreen of focus+context views allows the viewer to make an active decision on attending to whatever seems important. We also demonstrate our results on a variety of staged theater and dance performances. When videos are captured they are captured according to a specific aspect ratio keeping in mind the size of target screen in which they are meant to be viewed, this results in a inferior viewing experience when they are not watched on screens with their native aspect ratio. We present an approach to auto-matically retarget any given video to any desired aspect ratio while preserving its most salient regions obtained using gaze tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. The algorithm has two steps in total. The first step uses dynamic programming to find a cropping window path that maximizes gaze inclusion within the window and also tries to find the location of plausible new cuts (if required). The second step performs regularized convex optimization on the path obtained via dynamic programming to produce a smooth cropping window path comprised of piecewise linear, constant and parabolic segments. We test our re-editing algorithm on a diverse collection of movie and theater sequences. A study conducted with 16 users confirms that our retargeting algorithm results in a superior viewing experience as compared to gaze driven re-editing [30] and letterboxing methods, especially for wide-angle static camera recordings.

Abstract

The amelioration in video capture technology has made recording videos very easy. The introduction of smaller affordable cameras which not only boast high-end specifications but are also capable of capturing videos at very high resolution (4K,8K and even 16K) have made recording of high quality videos accessible to everyone. Although this makes recording of videos very straightforward and effortless, a major part of the video production process - Editing is still labor intensive and requires skill and expertise. This thesis takes a step towards automating the editing process and making it less time consuming. In this thesis, (1) we explore a novel approach of automatically editing stage performances such that both the context of the scene as well as close-up details of the actors are shown. (2) We propose a new method to optimally re-target videos to any desired aspect ratio while retaining the salient regions that are derived using gaze tracking. Recordings of stage performances are easy to capture with a high-resolution camera, but are difficult to watch because the actors’ faces are too small. We present an approach to automatically create a split screen video that transforms these recordings to show both the context of the scene as well as close-up details of the actors. Given a static recording of a stage performance and the tracking information about the actors positions, our system generates videos showing a focus+context view based on computed close-up camera motions using crop-and zoom. The key to our approach is to compute these camera motions such that they are cinematically valid close-ups and to ensure that the set of views of the different actors are properly coordinated and presented. We pose the computation of camera motions as convex optimization that creates detailed views and smooth movements, subject to cinematic constraints such as not cutting faces with the edge of the frame. Additional constraints link the close up views of each actor, causing them to merge seamlessly when actors are close. Generated views are placed in a resulting layout that preserves the spatial relationships between actors. This eliminates the need for manual labour and expertise required for both capturing the performance and later editing it, instead the splitscreen of focus+context views allows the viewer to make an active decision on attending to whatever seems important. We also demonstrate our results on a variety of staged theater and dance performances. When videos are captured they are captured according to a specific aspect ratio keeping in mind the size of target screen in which they are meant to be viewed, this results in a inferior viewing experience when they are not watched on screens with their native aspect ratio. We present an approach to automatically retarget any given video to any desired aspect ratio while preserving its most salient regions obtained using gaze tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. The algorithm has two steps in total. The first step uses dynamic programming to find a cropping window path that maximizes gaze inclusion within the window and also tries to find the location of plausible new cuts (if required). The second step performs regularized convex optimization on the path obtained via dynamic programming to produce a smooth cropping window path comprised of piecewise linear, constant and parabolic segments. We test our re-editing algorithm on a diverse collection of movie and theater sequences. A study conducted with 16 users confirms that our retargeting algorithm results in a superior viewing experience as compared to gaze driven re-editing [30] and letterboxing methods, especially for wide-angle static camera recordings.

Abstract

Social media platforms like Twitter and Facebook have become two of the largest platforms for people to communicate and share their views with the people. The casual and informal environment on these platforms leads to more people expressing themselves in their native language which results in a larger amount of code-mixed data that the annotated set of data currently lacks. With access to public opinion on nearly every topic, we can gather a huge amount of user data which could prove to be useful for various companies, thus making tasks like opinion mining and sentiment analysis even more important. Hence understanding users in low resourced languages has become one of the most researched tasks of late. We present two English-Hindi code-mixed datasets and to evaluate these datasets we simultaneously build baseline classification systems to evaluate them. As it takes time to create the datasets we decided to test our classification system on another dataset of Spanish and Catalan tweets on Catalan Independence as Catlan is one of the low resourced languages when seen from the perspective of Natural Language Processing. Thus, we first present a supervised classification system for stance and gender detection in Spanish and Catalan tweets on Catalan Independence. Then we present two English-Hindi code-mixed corpus, one for stance detection and the other for sarcasm detection in code-mixed tweets. The tweets for stance detection are collected for the target ‘Demonetisation’ whereas the tweets for sarcasm detection are collected on various topics such as cricket, bollywood, and politics. Each tweet in their respective datasets is marked for the stance and presence of sarcasm. Each token in the tweets is annotated with a language tag. Finally, we present a classification system developed using these datasets for stance and sarcasm detection. This system uses various word and character level features along with three different classification techniques. 10-fold cross-validation is used for evaluation of this system.

Abstract

Over the last decade, there has been an exponential increase in the popularity of social media platforms such as Facebook, Twitter and other online social networking websites. Apart from offering a fast, easy and reliable method of communication with friends and family, for many people they have also become the primary source of information on current events and happenings around the world. With the rise of these social media networks, it has become extremely easy for anyone and everyone to share text, images and videos with other people. It has also proven to be a boon for creative people, by allowing them to share their work with a lot of people instantly at the press of a button. The advantages of these online social networks are manifold. Unfortunately, some characteristics of user generated content (UGC) poses problems in the way of natural language processing of the data that is shared by users on these websites. UGC typically contains non-standard spellings of words, unusual abbreviations, informal grammatical structure and variations in vocabulary, which are all issues that are not easily mitigated and therefore render current systems ineffective in processing such information. In recent years, a phenomenon called “code-mixing”, which is frequently observed in online social media, has attracted a lot of research and interest from sociolinguists. A sentence is said to be codemixed if it consists of linguistic units, such as phrases, words or morphemes, from multiple languages. Code-mixing is often observed in the utterances of multilingual speakers, and is prevalent in countries where people speak multiple languages with native proficiency such as India. The most prominent effect of code-mixing on everyday life can be seen in UGC on online social networks. Due to the informal nature of UGC, code-mixing is bound to happen especially in text written by multilingual writers. Since this hybrid code-mixed language does not have a formally defined grammar, it becomes a challenging task to process this information and analyze it further. In this thesis, we attempt to understand the phenomenon of code-mixing in a deeper manner by analyzing text from online social media platforms. For deep learning NLP approaches to work on code-mixed text, they require an abundance of appropriate data to train on. Currently, there is a lack of such resources, therefore we start this work with the creation of 6096 English-Hindi code-mixed to monolingual English parallel corpus, which could aid deep learning methods in processing codemixed text. We then proceed to develop a pipeline that can augment existing Machine Translation (MT) systems to enable support for translating code-mixed languages to monolingual. We present the results obtained by experiments conducted that augmented existing statistical and neural network based machine translation systems such as Google Translate, Bing and Moses. This pipeline was also used toaugment the classification of sentiments for code-mixed sentences by existing systems, by translating code-mixed English-Hindi sentences to English using the augmented MT systems and then using an available sentiment analysis system for English. We take our work on code-mixing further by developing a hybrid architecture model that relies on neural networks with Attention mechanism as well as linguistic features to classify the sentiment labels for code-mixed sentences collected from online social networks such as Facebook and Twitter. We report these results which show that our approach is able to outperform previous methods for this task. Finally, we demonstrate that a similar hybrid architecture can be used for other NLP tasks as well, such as Clickbait detection and Aggression identification of online social media posts. Our approach imporved on the results obtained by previous methods for the task of clickbait detection. The results of these experiments are also presented in this thesis.

Abstract

Diabetic retinopathy (DR) is the most common eye disease in people with diabetes. It affects them for significant number of years and can also lead to permanent blindness if left untreated. Early detection and treatment of DR is of utmost importance for the prevention of blindness. Hence, automatic disease detection and classification have been attracting much interest. High performance is critical in adoptionof such systems, which generally rely on training with a wide variety of annotated data. Availability of such varied annotated data in medical imaging is very scarce. The main focus of this thesis is to deal with the sparsity of annotated data and develop computer-aided diagnostic CAD systems which take less annotated data and yet give high accuracies. We propose three different solutions to address this problem. First, we propose a semi-supervised framework which paves way for including unlabeled data in training. A co-training framework is used in which features are extracted from a limited training set and independent models are learnt on each of the features, later the models are used to predict labels for new data. The highly confident labelled images from unlabelled set are added back to the training set and the process is continued, thus expanding the number of known labels. This framework is showcased on retinal neovascularization (NV) which is a critical stage of proliferative DR. The analysis of the results for detection of NV showed that an AUC of 0.985 with sensitivity of 96.2% at specificity of 92.6% which were superior to the existing models. Secondly, we propose crowdsourcing as a solution where we obtain annotations from a crowd and use them for training after refining. We employ a strategy to refine/overcome the noisy nature of crowdsourced annotations by i) assigning a reliability factor for each subject of the crowd based on their performance (at global and local levels) and experience and ii) requiring region of interest (ROI) markings rather than pixel-level markings from the crowd. We also show that these annotations are reliable by training a deep neural net (DNN) for detection of hard exudates which occur in mild non-proliferative DR. Experimental results obtained for hard exudate detection showed that training with refined crowdsourced data is effective as detection performance improves by 25% over training with just expert-markings. Lastly, we explore synthetic data generation as a solution to address this problem. We propose a novel method, based on generative adversarial networks (GAN), to generate images with lesions such that the overall severity level can be controlled. We showcase this approach for hard exudate and haemorrhage detection in retinal images with 4 levels of severity. These vary from mild to severe non-proliferativeDR. The synthetic data were also shown to be reliable for developing a CAD system for DR detection. Hard exudate/ haemorrhage detection was found to improve with inclusion of synthetic data in thetraining set with improvement in sensitivity of about 25% over training with just expert marked data.

Abstract

Road Networks are the most preferred and important mode of transportation due to its accessibility and connectivity thereby influencing the social and economic activity patterns. Therefore, understanding road network behaviour is crucial for the economic and social development of the country. Geographical Information System(GIS) is designed to store, analyze and manipulate ge- ographical data. Since road network data forms a major part of geographical data, GIS can be employed in understanding and utilizing road network behaviour for a variety of applications. Due to the rapidly increasing road network information and its complexity, road network data management for GIS applications is becoming increasingly difficult. Navigation and transportation planning are two most important GIS applications. The challenges in navigation is the processing and storage of large road network data and design of cost metric to be minimized while computing minimum cost path. Sometimes people prefer a path used by most of the travelers rather than the optimal path. In such a case understanding road networkbehaviour may provide better insights on the travel patterns leading to improved efficiency of navigation applications. Traffic flow analysis is one of the important goals of transportation planning. It involves monitoring the entire road network for the collection, storage and processing of traffic data, which may be a huge overhead in case of large road networks. Proper understanding of the road network behaviour leads to efficient traffic flow analysis even in the absence of traffic data while reducing the amount of data to be collected and analyzed. The existing methods in the context of road network management as a standalone might work but may require additional processing when trying to perform some important operations in GIS applications. In order to understand the road network behaviour and achieve better road network data management, we propose Skeleton Network of a road network extracted using map generalization tech- niques by exploiting the inherent node/edge characteristics. The Skeleton Network is the most significant and representative network subset capturing the overall connectivity and structure of the road network. In order to understand its significance in terms of capturing road user behaviour the Skeleton Network of Melbourne, Australia is evaluated against the Annual Average Daily Traf-fic(AADT) data . Observations indicate that the Skeleton Network mimics the traffic volume data thus capturing the road user behaviour. The utility of Skeleton Network in efficient road network data management is assessed by applying it in the previously mentioned GIS applications namely navigation and transportation planning. In order to assess the applicability of the Skeleton Network in navigation, a model has been proposed that incorporates the Skeleton Network in pgRouting, a SQL based geospatial routing library. The model is tested on road networks of varying size and topology and the observations indicate that significant gains in path computation are achieved. Moreover path computed using the proposed model mimics the natural traveling pattern. In addition to this, the proposed model also resulted in structured storage of the road network in a relational database enabling efficient querying of road network data. Furthermore, the applicability of Skeleton Network in transportation planning is assessed using an interesting approach that generates a well connected hybrid road network subset by integrating the Skeleton Network and the traffic flow(AADT) data. The generated hybrid network being significantly smaller than the original network comprises of all the high traffic roads and the connectivities between them, leading to efficient analysis of traffic flow. For the ease of understanding and appreciating the outcomes of this work, the proposed methods have been applied to three city level networks(Chandigarh,Hyderabad, Melbourne), one state level network(New York) and a country level network(Belgium). Given the availability of appropriate datasets required for the study, the proposed method can easily be applied to much larger road networks.

Abstract

Data scientists very often find that a central step in their work, is to implement an appropriate transformation restructuring the originally given data into a new and more revealing form. Although specific domain knowledge can be used to help design representations, data-driven learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. In order to make it possible to apply machine learning to different domains, it is very important to make learning algorithms less dependent on manual feature engineering, so that novel AI based applications could be developed faster. The work in this thesis focuses not only on the mapping from representation to output but also the representation itself by mining hidden patterns from available data. Keeping this in mind, we propose a learning model to alleviate difficulties involved in feature engineering through automation. We develop a learning model, based on regression between feature pairs, that discovers underlying patterns and its variations in the data, by the way features are related to each other and selects a very small number of new features to create a significant improvement in predictive performance. Because this model takes into account the inherent feature structures, we took our next motivating example from bioinformatics. The features in this domain have some natural spatial order (for instance, in the medical domain, genes are often associated with different types of clinical features) and thus incorporating such structure can help select more important features and achieve more accurate classification accuracy. It also provides better readability and interpretability to the models. The final component of the work focuses on the problem of fake news detection. The low cost, easy access and rapid circulation of information over the internet has encouraged more people than ever, to seek out and consume news from online sources or social media rather than traditional news organizations. While the traditional (count-based) feature engineering strategies for textual data are effective methods for extracting features from text, due to the inherent nature of the model being just a bag of unstructured words, we lose additional information like the semantics, structure, sequence and context around nearby words in each text document. This formed as a motivation for us to explore more sophisticated models which can capture this information and give us features which are vector representation of words. Keeping in mind that fake news is intentionally written to mislead readers to believe false information, it can be difficult to detect it, solely based on news content. Therefore, we need to include auxiliary information, such as user social engagements on social media, to help make adetermination. We therefore design, a novel attention based hybrid network which integrates the article information along with the meta data and claims of the news articles.

Abstract

Autonomous Driving is one of the hot areas of research in the current world. Deep Machine Learning and Computer Vision together have shown promising results in this field. A relatively new domain is Reinforcement Learning. RL is associated with behavior based learning. An RL agent learns from experiences, by exploring it action and state space. In this work, a Deep Reinforcement Learning algorithm, Deep Deterministic Policy Gradient(DDPG) is used to learn various driving behaviors. DDPG is an off-policy, continuous control framework. It is the off-policy nature of this algorithm which enables it to efficiently explore the environment. Curriculum learning and Intrinsic Motivation are used to extract advantage of the off-policy nature. Different behavior based agents are trained. The important ones being Overtaking in highways and Opportunistic agent in dense unstructured traffic scenes. For overtaking in highways, the agent learns in a Curriculum based approach. First, it learns to drive on an empty road and next it learns overtaking strategies. The reward function is handcrafted in a manner that the desired behavior is learned in least number of training episodes. Various experiments were conducted to handcraft the reward, a detailed analysis of which have been provided. The learned agent is able to overtake not only on straight but curved roads as well. Another behavior that was learned was blocking. Blocking is an hostile behavior and should not be practiced in real life. By slightly changing the reward function for Overtaking on highways and by positioning the learning agent ahead of the opponent, blocking behavior was learned. The approach for overtaking and blocking is compared with existing RL based approach for the same. For dense traffic scenes, we learn two behaviors: Opportunistic and Defensive. Opportunistic agent actively looks for free space ahead of it and navigates itself there, while defensive stays in its own lane and changes its speed each time any vehicle approaches it, to avoid collisions. There is no prior work which deals with end-to-end driving in dense unstructured traffic. Learning the behavior for defensive agent was not easy to achieve by simply using noise. We used intrinsic motivation based approach by explicitly showing it the rewarding actions. For all of the behaviors the learning is scalable and robust in terms of the speeds of opponent vehicles and the number of cars in the surroundings.

Abstract

Humour is an intricate part of human behaviour and a valued quality when developing interpersonal relationships. It is an interesting and challenging domain to work in owing to its subjectivity. Humour is one of the most sophisticated forms of human intelligence and creativity, intimately related with factors such as world knowledge, reasoning and perception. With our AI systems becoming increasingly intelligent, there is now a demand for systems which are more human-like than ever before. No AI system could be complete without the ability to to process and use an integral human phenomenon like humour. Humour is often expressed through language in various forms such as funny stories/anecdotes, jokes, witty sayings, puns etc. This verbally expressed humour is of interest to NLP researchers because it deals with spoken and textual language. The subfield of Computational Linguistics and Artificial Intelligence which aims to develop systems which can use and understand humour like humans do is called Computational Humour. A complete computational humour system has to contain two main modules - a humour interpreter, and a humour generator. Both these modules could have their independent uses in a variety of applications, but when it comes to interactive and comprehensive AI systems, both are equally important. In this thesis, we present a first ever work for computational humour for Hindi. We consider both sub-fields in computational humour - Humour Understanding and Humour Generation, and work on specific applications in both. We study Hindi-English code mixed puns by focusing on automatic recovery of the target words/phrases in them. We define code-mixed puns, analyse the type of structures present in them, and classify them into two main categories - intra-sentential and intra-word code-mixed puns. We then propose a four-step methodology to recover intra-sentential code-mixed puns. On evaluating our model on a test dataset of such puns, we observe that we are able to successfully recover 67%. We recognise several limitations of our system, like the inability to deal with highly unusual and creative language use. For the problem of computational generation of humour, we consider two tasks. First, we build a prototype system which automatically generates a simple form of Dur-se-Dekha jokes in Hindi using a small lexicon and handcrafted rules. Second, we design an algorithm which can induce humour in a short non-humorous text by performing single word subsitutions on the basis of phonetic similarity and sentiment opposition. We perform human evaluation on the results of both and observe that we are successfully able to generate funny instances in both these cases.

Abstract

The major contribution of this thesis is creation and enrichment of resources (a lexicon) for a resource poor language viz. Telugu. This thesis also describes the enrichment of OntoSenseNet-a verb-centric lexical resource. Our work aims to preserve and enhance the usage of an authentic Telugu dictionary by developing a computational version of the same. This enables the native speakers of the language to actively involve in the research as most of the computer science experts or algorithms work on top of the annotated language data. With the aid of developed Telugu dictionary, native speakers can perform better annotations as both the word and its meaning are in a language they are familiar with. Hence, efforts are made to develop the aforementioned Telugu lexical resource and numerous annotations are done manually by language experts. Primarily, we attempted two types of annotations 1) Ontological classification of verbs, adverbs and adjectives; 2) Annotation of unigrams and bigrams for sentiment polarity. Based on the proposed ontological classification, the manually annotated gold standard corpus consists of 8483 unique verbs and 253 unique adverbs. Annotations are done by native language speakers according to the set of provided annotation guidelines. We discuss annotation procedure in detail and present the validation of the developed resource through inter-annotator agreement as a measure. Additional words extracted from Telugu WordNet are combined with our resource and annotations are done. Furthermore, we discuss the enrichment of this manually developed resource of Telugu lexicon, OntoSenseNet. OntoSenseNet is a ontological-sense annotated lexicon that classifies each verb into 7 sense-types and adverbs into 4 sense-classes. The developed OntoSenseNet for Telugu has primary and secondary sense-types identified for the verbs and primary sense-class tag for adverbs. The area of research is relatively recent but has a large scope of development. We provide an introductory work to enrich the OntoSenseNet to promote further research in Telugu. Classifiers are adopted to learn the sense-type of the words in the resource and thus, we automate the tagging of sense-types for verbs. We perform a comparative analysis of different classifiers applied on OntoSenseNet. The results of the experiment prove that automated enrichment of the resource is effective using SVM classifiers and Adaboost ensemble. However, the accuracy is low compared to the task of manual annotations. To perform manual annotations more extensively, we have developed a tool for crowd-sourcing the task of annotating. Access to contribute to the resource is given only to certified individuals after preliminary assessment.This tool consists of guidelines for annotations and list of words that are to be annotated and the annotator is given the freedom to choose “uncertain” option in case of an unclear judgment. Mechanisms adopted to minimize the disagreements and measures are taken while adding these annotations to our resource are discussed in this thesis in detail. Additionally, we discuss the potential applications of this ontological resource. Moreover, efforts have been put to enhance the sentiment analysis task through phrase-level annotations. We developed a systematically annotated corpus that can support the enhancement of sentiment analysis tasks in Telugu by annotating bigrams in Telugu. These are the second kind of annotations that are mentioned before. The developed polarity-annotated corpus is called ‘BCSAT’. From the developed Telugu dictionary, we extracted 11,000 adjectives, 253 adverbs, 8483 verbs. We extracted words from SentiWordNet and bigrams from target corpus. Sentiment based polarity annotations for these extracted lexemes are done by language experts. We discuss the methodology followed for the polarity annotations and provide validation for the developed resource. The fundamental aim is to validate and study the possibility of utilizing phrase-level sentiment annotations in the task of automated sentiment identification. This work aims at developing a benchmark corpus, as an extension to SentiWordNet, and baseline accuracy for a model where phrase-level sentiment annotations are applied for sentiment predictions. The method we present outperforms all known methods when tested on the recognized and standard benchmarks for sentiment analysis task in Telugu.