Abstract

Parsing natural scenes into semantically meaningful entities is one of the open problems in computer vision. Due to complexity present at various levels ranging from scene-object, object-object and objectlayout, the problem becomes challenging. Restricting the problem to indoor scenes makes it slightly easier to track. Holistic understanding of indoor scene involves detection of object, recovery of 3D geometry of the object, estimating spatial layout of the scene and classification of the indoor scene. It helps in executing the high-level task such as navigation, free space estimation, object placement and manipulation. In this thesis, we integrate the information at the various levels in the cluttered indoor environments for efficient semantic segmentation. We use appearance and geometric properties of different entities for estimating free space and localising objects in the given indoor scene. We believe that this work can enable a variety of applications where the semantic understanding of indoor scene is required. For example, mobility assessment, robot navigation, path planning and surveillance, object manipulation, grasping, learning object support order, visual search and 3D reconstruction. In this thesis, we first attempt the problem of learning and estimating free space i.e., floor regions in indoor scenes from a single image. Estimating free space is challenging due to high appearance variability within the floor and non-floor regions. It is even harder to segment floor regions when clutter, specular reflections, shadows and textured floors are present within the scene. We propose a framework which utilises a generic classifier of appearance cues as well as floor density estimates, both trained from a variety of indoor images. The result of the classifier is then adapted to a specific test image where we integrate appearance, position and geometric cues in an iterative framework. A Markov Random Field framework is used to integrate the cues to segment floor regions. The proposed approach is also flexible in situations where scene avoids assumptions like Manhattan world scene or restricting clutter only to wall-floor boundaries. Moving from detecting free space or floor regions, we use the appearance and geometric properties to estimate more general entities in the cluttered scene. These entities or objects are the basic units of any indoor scene, free space being one of them. For example different configurations of the set of objects in scenarios ranging from indoor scenes with cluttered tabletops to indoor scenes of offices, home, corridors, classrooms etc. Due to high variability in indoor scenes itself and complexity due to intraclass variability within objects, we restricted our attention to table top scenarios with known objects. The problem of estimating the layout of table top scenes is challenging due to the presence of clutter, objects of homogeneous appearance with that of the table surface, object-object occlusions and objects having irregular shapes and sizes. We train an ensemble of classifiers over appearance cues from various images of the known objects with different poses. We learn the meta-data (pose, shape) associated with each object and try to estimate its pose and shape in a given cluttered scene. The approach predicts the detailed layout of the objects present and free space on the table top where the objects can be placed. We created two datasets for the above-mentioned work. We first created an RGB based ”CVIT Indoor Scene dataset” of 110 images from various buildings in our campus that included cluttered floor regions. The images contained the wide variety of indoor scenes including classrooms, living rooms, library, corridors, two or three visible walls, etc. It also consisted of images with varied texture within the floor, specular highlights, shadows and scenes with cluttered floors due to furniture or other obstacles, where the clutter is not just confined to image boundaries. We also created a second RGBD dataset ”3DMOS” of 50 objects with different shapes and sizes. Each object had 15 images of different pose and view. It also had 10 cluttered table top scenes from sparsely cluttered to densely cluttered. We have made the dataset publicly available for the research community. The proposed approaches successfully demonstrate the robustness and efficiency on the various mentioned complex situations in indoor scenes. We believe that this work can play a significant role in the true understanding of an indoor scene and its semantics. It could also help in increased interaction of robotic agents or humans with the surrounding environment when this information is given to them. Keywords: Semantic Segmentation, Indoor Scene, Space Estimation, Object Manipulation, Cognitive Vision.

Abstract

Diabetic Retinopathy (DR) is an eye-aliment, which is one of the leading causes of blindness all over the world. With timely diagnosis and proper care, it is possible to manage this condition and control its severity. The sooner this condition is diagnosed, the better are the chances of preventing blindness and causing any further structural damage associated with DR. In this thesis, we address the problem of detecting DR from colour retinal images, by employing some biologically inspired approaches in a machine learning setting. DR detection entails the detection of a variety of lesion, among which we restrict our attention to bright lesions such as hard exudates, cotton wool spots and dark lesions such as haemorrhages. Microaneurysms, which are the smallest of all types of retinal lesions are not covered in this study, since due to their extremely small size, they are generally studied and analysed separately. In this thesis, we approach the problem of DR lesion detection via visual saliency, which is a basic and important biologically motivated concept. We show the relevance of visual saliency in bright lesion detection by demonstrating that visual saliency models, do capture hard exudates. We compare three popular visual saliency models, namely, Itti-Koch, Graph Based Visual Saliency (GBVS) and Spectral Residual (SR) and show that top 90% of the most salient locations provided by these models, are able to correspond with an average of 70% of all true hard exudates present in images. We propose a multi-scale version of the SR model, called as Extended SR (ESR) model and show its relative performance to be about 28.7% better than the previous models in detecting true hard exudates. This ESR model is used as a basis to design a bright lesion detector, which provides at the image level, high sensitivity (0.93) and specificity (0.78), averaged over 6 public datasets. We next propose two systems, one aimed as an assistive system for an annotator and another for detecting both bright and dark lesions. Both are based on a biologically inspired novel, lesion enhancing pipeline. The proposed system has been evaluated over 5 public datasets and is found to outperform existing systems with sensitivity/specificity values of 0.92/0.86 for bright lesions and 0.86/0.84 for dark lesions. We also evaluate the regional level lesion detection on one public dataset and obtain a performance figure of (0.91/0.88) for bright lesions and (0.85/0.83) for dark lesions. These underscore the potential of biologically inspired approaches to computer aided diagnostic algorithm development.

Abstract

Visual Attributes are properties observable in images that have human-designated names ( e.g., smiling, natural) and they are valuable as a new semantic cue in various vision problems like facial verification, object recognition, generating description of unfamiliar objects and to facilitate zero shot transfer learning etc. While most of the work on attributes focuses on binary attributes (indicating the presence or absence of attribute) the notion of relative attributes as introduced by Parikh and Grauman in ICCV 2011 provides an appealing way of comparing two images based on their visual properties than the binary attributes. Relative visual properties are a semantically rich way by which humans describe and compare objects in the world. They are necessary, for instance, to refine an identifying description (the rounder pillow; the same except bluer), or to situate with respect to reference objects (brighter than a candle; dimmer than a flashlight). Furthermore, they have potential to enhance active and interactive learning, for instance, offering a better guide for a visual search (find me similar shoes, but shinier or refine the retrieved images of downtown Chicago to those taken on sunnier days). For learning relative attributes a ranking svm based formulation was proposed that uses globally represented pairs of annotated images. In this thesis, we extend this idea towards learning relative attributes using local parts that are shared across categories. First we propose a part based representation that jointly represents a pair of images. For facial attributes, part corresponds to a block around a landmark point detected using a domain specific method. This representation explicitly encodes correspondences among parts, thus better capturing minute differences in parts that make an attribute more prominent in one image than another as compared to global representation. Next we update this part based representation by additionally learning weights corresponding to each part that denote their contribution towards predicting the strength of a given attribute. We call these weights as significance coefficients of parts. For each attribute the significance coefficients are learned in a discriminative manner simultaneously with a max-margin ranking model. Thus the best parts for predicting relative attribute more smiling will be different from those from predicting more eyes open. We compare the baseline method of Parikh and Grauman with the proposed method under various settings. We have collected a new dataset of 10000 pair wise attribute level annotations using images from labeled faces in the wild ( LFW) dataset particularly focusing on large variety of samples in terms of poses, lightning conditions etc and completely ignoring the category information while collecting attribute annotation . Extensive experiments demonstrate that the new method significantly improves prediction accuracy as compared to the baseline method. Moreover the learned parts also compare favorably with human selected parts, thus indicating the intrinsic capacity of the proposed framework for learning attribute specific semantic parts. Additionally we illustrate the advantage of the proposed method with interactive image search using relative attribute based feedback. In this work, we also propose relational attributes, which provide a more natural way of comparing two images based on some given attribute than relative attributes. Relational attributes consider not only the content of a given pair of images, but also take into account its relationship with other pairs, thus making the comparison more robust.

Abstract

Malayalam is an agglutinative and morphologically rich language as any other Dravidian language. Computational processing of Dravidian languages is not trivial because two or more words can join to form a string of words with a morpho-phonemic change at the point of joining. This process known as “Sandhi”, in turn complicates the individual word identification. The current work is an attempt to break the barrier of word segmentation and to create a shallow parser for Malayalam, which facilitates non-recursive phrase identification given an input. In this work, Shallow Parser has mainly 3 modules namely Sandhi Splitter, POS Tagger and Chunker. Since words are the basic components in the sentence, not identifying the individual words in a sentence will affect the output of a shallow parser. Hence in order to tackle the problem of “Sandhi”, after attempting a few rule-based approaches, we arrived at a hybrid “Sandhi Splitter” which gave an overall accuracy of 87% . This system uses Naive Bayes classifier to identify the split point and hand crafted character-level rules to induce morpho-phonemic changes. A CRF based Parts-Of-Speech tagger has been employed for the identification of grammatical category of words. Various experiments with different templates of features showed that Malayalam has more dependence over word-internal features like prefix and suffix information than word-external features like the position of a word in a sentence. The highest overall accuracy we obtained is 91.25%. The final module “chunker” has been employed to find out the non-recursive phrases based on the Parts-Of-Speech tags of the words. This CRF based chunker gave an overall accuracy of 94.33%. Error propagation is a problem in shallow parsing. Errors created by each module affect the subsequent modules. When each module is put in a pipeline in such a way that the output of the previous module will be the input of the next module, overall accuracy of shallow parser came down to 71% from 92%. This shows the need and importance of a highly accurate sandhi splitter which is the main source of error. It is important to note that the experiments clearly show that morphology is the key factor for processing Malayalam.

Abstract

Expecting a shipment of 1 billion Android devices in 2017, cyber criminals have naturally extended their vicious activities towards Google’s mobile operating system: threat researchers are reporting an alarming increase of detected Android malware from 2014 to 2015. In order to have some control over the estimated 700 new Android applications that are being released every day, there is need for a form of automated analysis to quickly detect and isolate new malware instances. Android is an open source Linux-based mobile operating system distributed by Google. According to the latest statistics, android powers hundreds of thousands mobile devices over 190 countries [26]. Google Play [51] is the official android centralized market place maintained by Google, where any independent application developer can submit his/her android app and make it available to the users. The growing popularity of this android ecosystem also is becoming a worthy target for security and privacy violations. Highly sensitive and confidential information such as text messages, private and business contacts, calendar data, etc may be leaked through an application. Sensors such as GPS present in the phones allow applications to provide context-sensitive user experience, they also create additional privacy concerns it can exploit the data for tracking or monitoring. Apart from these issues, smart phones are also susceptible to various malware threats such as viruses, Trojan horses, worms, etc. [50] Android security model relies highly on permission-based mechanism. There are about 130 per- missions that govern access to different resources. Whenever an user tries to install a new application, he/she is prompted to approve or reject all the permissions requested by the application. The application will be installed only after the user accepts all the necessary permissions requested by it. In this work, we use the permissions and api level information from the apps as the features to detect malicious applications. Further we observe that, android store [51] defines a category for every published application. We have done extensive studies and discovered that, certain categories are highly prone to malicious acts compared to other categories. We explicitly incorporate this information in our model and learn a naive bayes classifier for each category using the features that encode information about permissions and api calls. Given a new test application with a known category, we apply an appropriate classifier to detect if the application is malicious. We created a large data set of android applications and achieve an improvement of 3 4% by incorporating category level information. Secondly, we combine the association rule mining and classification rule mining techniques to build a classifier. The integration is done by focusing on mining a special subset of association rules, called class association rules (CARs). To select the best features that distinguish between malware from benign apps, we rely on API level information within the bytecode since it conveys substantial semantics about the apps behaviour. More specifically, we focus on critical API calls and their package level information. Rather than simply treating the individual api calls as items, we represent an item as a combination of caller and callee api. We capture one level of control flow and context between caller and callee. Each item in our model is of the form A%B, where A is the caller and B is the callee. We use Androguard [8], a reverse engineering tool to perform API level feature extraction and data flow analysis. In summary, • combining association rule mining and classification rule mining for Android malware detection. • We achieved a detection rate of 85% over the baseline classifier of 0.69%.

Abstract

This thesis discusses sophisticated navigation frameworks for autonomous navigation of quadcopter in the indoor and outdoor environment using the frontal monocular camera as its primary exteroceptive sensor. Indoor and outdoor environment vary in their views, composition and constraints thus requiring different sophisticated approaches for navigation. Therefore, individual autonomous navigation frameworks for the indoor and outdoor environment are developed and presented in the thesis. The navigation frameworks for indoor and outdoor environment consists of three major components – perception, planning and control. Initially, navigation framework for autonomous navigation of quadcopter in GPS denied unknown indoor environment is presented. The perception module of navigation framework estimates dense depth map of the environment in real time from video stream obtained from the frontal monocular camera of the quadcopter using our novel supervised Hierarchical Structured Learning (HSL) approach. The proposed HSL approach discretizes the overall depth range into multiple sets. It structures these sets hierarchically and recursively through partitioning the set of classes into two subsets with subsets representing apportioned depth range of the parent set, forming a binary tree. The binary classification method is applied to each internal node of binary tree separately using Support Vector Machine (SVM). Moreover, the depth estimation of each pixel of the image starts from the root node in the top-down approach, classifying repetitively till it reaches any of the leaf node representing its estimated depth. The generated depth map is provided as an input to planning module based on Convolutional Neural Network (CNN), which generates flight planning commands. Finally, control module employs a convex programming technique to generate collision-free minimum time trajectory which follows these flight planning commands and produces appropriate control inputs for the quadcopter. Thereafter, outdoor autonomous navigation framework of quadcopter along with its application in precision agriculture is presented. The proposed system is able to perform plantation monitoring and yield estimation using supervised learning approach while autonomously navigating through the interrow path of pomegranate plantation. A new ‘pomegranate dataset’ comprising of plantation surveillance video and annotated frames capturing the varied stages of pomegranate growth along with the navigation framework are being delivered as a part of this thesis. The capabilities of the proposed system could be described as a twofold framework:(1) plantation monitoring and yield estimation: life cycle stage grad- ing and yield estimation of pomegranate while autonomously navigating through inter-row paths of the plantation using a monocular camera as its primary sensor. (2) Autonomous outdoor navigation framework: a novel navigation framework, which uses GPS way-points and vanishing point based planner and employs convex optimization for generating minimum time trajectory and control. The framework is implemented over Robot Operating System (ROS) middle-ware and ” first to release in Open Source”. A majority of the work in this thesis is attributed to developing an autonomous navigation framework for generic quadcopter which incorporates various theoretical concepts into practical implementation with Robot Operating System ( ROS ) as underlying middle-ware. Repeatable flights of successful nature, confirm the efficacy of the proposed navigation frameworks. The thesis is concluded by demonstrating the applicability of developed navigation framework on a low-cost commercial quadcopter P arrot T M Bebop.

Abstract

Cinema, being a newest of art forms, has introduced daunting ontological challenge in accounting for the phantasm of ‘cinematic reality’ and a narrative theorist challenge in accounting for cinematic expressivity or telling. Furthermore, cinema has become entirely digital. There is a need for the com- putable ontology of cinema and in particular narrative computing of cinematic story telling. Research in film theory and even future film making will benefit from building of a semantically searchable archive of digital films based on such formal ontology and narrative computing. The thesis depicts any digital cinema to be a hypergraph of various layers of discretist ontology. In particular, we develop automatic determination of Plot Units of the cinema narrative and depict it as hypergraph. We elaborate on the discrete ontology of cinema as hypergraph and finally suggest prototype of a film archive CinemaScope based on HyperGraph ontology of cinema. We give an initial report on the potential of narrative theory driven graph based representations of movies for narrative analysis and summarization. This is done using automatic generation of Plot Units and determination of affect states for characters in movies. The Plot Units are generated from movie scripts which are textual in nature. The movie scripts are parsed into scenes using a simple grammar. We then use some simple heuristics, rules and techniques to programatically generate Plot Units from these parsed movie scripts. We also present results of character analysis, related events detection and summarization from the generated Plot Units for movies. We use these results to argue that a graph data model for the content of movies could better capture its underlying semantics. We then show that the various graph representations for movies can be made to fit into the HyperGraph ontology. We also present a prototype for a digital film archive called CinemaScope based on HyperGraph ontology for films. Digital film archives in addition to preserving cinema should also accommodate and provide provisions for the computation of its cinematic elements by means of the metadata provided to represent the content in them. The provision for the computation of cinema could lead to various applications like semantic search, automatic genre clustering and classification, special visualization tools and robust comparisons of movies. This prototype has been designed to reduce the content and semantics of a movie to a hypergraph of tags. The prototype is very much in its preliminary stage but could serve as a foundation upon which more robust systems for film archives using HyperGraph ontology could be built upon and used for various research and academic analyses.

Abstract

A dictionary data structure is a key-value store which supports insert, search and delete operations. Conventional data structures such as balanced binary search trees and hash tables are good both in theory and practice. However, these dictionaries are not adaptive as they do not exploit the temporal locality in key access patterns over time. On the other hand dictionaries like splay trees and working-set trees adapt to the changing key access patterns by reorganizing themselves with every key access. Consequently, the set of most frequently accessed key, i.e. the working-set, can be retrieved quickly in these dictionaries. With the gaining popularity of GPU based accelerator platforms for general purpose computing, several data structures such as B-tees and hash tables have been successfully ported to these platforms. In this thesis, we propose heterogeneous (CPU+GPU) hash tables, that optimizes operations for the frequently accessed keys. The basic idea is to maintain a dynamic set of most frequently accessed keys in the GPU memory and the rest of the keys in the CPU main memory. Further, queries are processed in batches of fixed size. We measured the query throughput of our hash tables using Millions of Queries Processed per Second (MQPS) as a metric, on different key access distributions. On distributions, where some keys are queried more frequently than others, we achieved on an average 10x higher MQPS when compared to a highly tuned serial hash table in the C++ Boost library; and 5x higher MQPS against a state of the art concurrent lock free hash table. The maximum load factor on the hash tables was set to 0.9. On uniform random query distributions, as expected our hash tables do not out perform concurrent lock free hash tables, nevertheless matches their performance. The second main contribution of this thesis is the design and implementation of self organizing Cache Oblivious B-trees (CO B-tree) which use the memory hierarchy effectively to reduce the number of cache misses incurred while processing a query. A cache oblivious B-tree (CO B-tree) incurs optimal number of cache block transfers across any two levels in a multi level memory hierarchy. In this work, we propose two variants of the dynamic CO B-tree, a working set CO B-tree and a hybrid CO B-tree. They reorganize themselves with every key access facilitating faster retrieval of the frequently accessed keys with the minimum number of cache misses. Theoretically, we prove that a self organizing CO B-tree can be designed with a small additive factor overhead to the already proven bounds for the data structure. In empirical experiments we compare both our data structures to a B-tree, an AVL tree, a top down splay tree and a static/dynamic CO B-tree. Our CO B-trees, on average, have 1.5 to 2 times lesser cache misses compared to the B-tree and the top down splay tree respectively. For higher degree of temporal locality in access patterns, the hybrid CO B-tree has better query throughput (queries processed per unit time) and lesser number of cache misses compared to the rest of the data structures including a static CO B-tree.

Abstract

Relation between reality and language is an age old problem. The term ‘meaning’ embodies hordes of issues which are found factored into the problem. One contemporary aspect of the problem is how to computationally specify ‘meaning’ of a word. To conceive a web resource for the ‘meaning’ of a word involves ontological as well as linguistic considerations. Usually ‘meaning’ of a word is specified using other words like in dictionaries, thesauruses etc. Ontologically ‘meaning’ of a word is thought to be specified in terms of it participating in classes, relations and events. Both these accounts seem extensional as they refer to objects outside a word to determine its ‘meaning’. For both of these accounts words ‘mean’ only in the company of other words or in association with the outlying reality which words represent. We take a view that ‘meaning’ of a single word has a necessary aspect of intension. This intension is in situ formations embodied in the ‘meaning’ of a word. The in situ forms are present intrinsically in the ‘meaning’ of words and are the causes for the association of words as well as assertion of knowledge. The work presents a new approach to formal ontology of natural language based on in situ formations in the ‘meanings’ of words and pairings of words. The basic motivation is to computationally manipu- late language at the level of intrinsic lexical meanings. Lexical meanings, in the transaction of language, have discrete intrinsic forms of types and classes. Intrinsic sense-types and sense-classes have been iden- tified for 3867 verbs,1980 adverbs and 300 adjectives. Identification of sense-types and sense-classes for adjectives and nouns is in progress. These types and classes are unambiguously locatable in parts of speech through collective introspective inquiry first and then enriched with the help of computational methods of corpus study. This work reports on the construction of a web resource, in which ‘meanings’ of English words are given in terms of formal ontology of language inspired by Leibniz, Patanjali and Bhartrihari. The experimental results on testing data of SemEval - 2010 (word sense induction and disambiguation task), the measure of synonimity of verbs and nouns and the inferences drawn from the sense-class and sense-type distributions show the potentiality of the resource. Integration of resource of sense-types and sense-classes with lexical grammar of cases, interpreted as verb-noun pairs, leads to paraphrasing of English sentences into graphs of formal senses. This gives confidence that the resource can be enriched through these studies and resource exploration framework can be made such that as a resultant resource itself is continuously enriched. The proposed ontology of verbs, adverbs, adjectives and nouns, if can be implemented for large numbers of words, would make the resource adventitious in knowledge computing. We aim at extensive lexical coverage of language to show that our approach is non-reductionist unlike semantic prime approaches. Rigorous formal underpinning of the ontology resource will enhance capability, possibility and accuracy of ontology driven NLP and AI applications. It is a growing resource and as yet it covers only portion of English language. However, the resource framework is designed to include resource enriching tasks and revisionary studies.

Abstract

The thesis posits that analytical studies relating to knowledge based industries (especially social media) as rent seeking organization must be revisited. Proponents of cognitive capitalism, especially Autonomist Marxists attribute sources of revenue in knowledge based industries to “rent seeking be- havior” departing from Marx’s labor theory of value. Industrial production process requires capitalist’s active supervision for efficient production and value of labor is calculated with respect to socially ab- stract labor time invested in production process. Autonomist Marxists contest that after the crisis of Workerism(resulting in reduction in working time) and rise of knowledge as productive force, capital has become external to production process while extracting collective knowledge as a rent for provisions to access means of production. Knowledge has always been central to organization of societies, although Autonomist Marxists and economists indicate a renewed interest in knowledge (in its technological ca- pacity) owing for its effects on growth and productivity in world economy. We claim that production process must be analyzed in totality to fully apprehend the realities of today’s political economy and not just limited to advanced capitalist countries, which Autonomist Marxists neglect in their theories of cognitive capitalism. Our criticism is two fold. Firstly we present our critique to cognitive capitalist theories by analyzing Facebook: an epitome of post-Fordist management practices, and claim that Face- book is not a rent seeking organization. If Facebook is indeed a rent seeking organization, we argue that it undermines the contributions of software creators and contributors who maintain the large scale oper- ations of the firm. Secondly knowledge labors represent only one aspect of the class of multitude(term popularized by Hardt and Negri in Empire, refers to the class of collective social subjectivities present today) while other, which is often neglected, is the rise of precarious labor in global south. Marx argued that opening up of luxury industry stands on the shoulder of relative surplus population, “a population often made available owing to the preponderance of constant capital in other branches of production; these base themselves in turn on a preponderance of the element of living labor, and only gradually pass through the same trajectory as other branches”. Knowledge industries relies heavily on physical infrastructure which increases the organic composition of firm. Organic composition of firm is the ratio of constant capital(machines and raw materials) to variable capital(value of labor). The crisis of value is not because of cognitive capitalism but because of inherent antagonism between capital-labor relation- ship at play in global value chain. For Knowledge based industries, in order to recover the tendential fall of rate of profits (owing to relative increase in constant capital to variable capital) low organic firms are introduced in economy so as to shift the surplus to advanced level of production. Lastly we part from the critique of cognitive capitalism and asses the tragedy of commons evidently indicated by rising privatization of modes of production which is central to immaterial production