Abstract

Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting highlevel semantic information present in images as textual cues in the Visual Question Answering process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research

Abstract

As vision-based autonomous systems, such as selfdriving vehicles, become a reality, there is an increasing need for large annotated datasets for developing solutions to vision tasks. One important task that has seen significant interest in recent years is semantic segmentation. However, the cost of annotating every pixel for semantic segmentation is immense, and can be prohibitive in scaling to various settings and locations. In this paper, we propose a region-based active learning method for efficient labeling in semantic segmentation. Using the proposed active learning strategy, we show that we are able to judiciously select the regions for annotation such that we obtain 93.8% of the baseline performance (when all pixels are labeled) with labeling of 10% of the total number of pixels. Further, we show that this approach can be used to transfer annotations from a model trained on a given dataset (Cityscapes) to a different dataset (Mapillary), thus highlighting its promise and potential

Abstract

Do we know what the different filters of a face network represent? Can we use this filter information to train other tasks without transfer learning? For instance, can age, head pose, emotion and other face related tasks be learned from face recognition network without transfer learning? Understanding the role of these filters allows us to transfer knowledge across tasks and take advantage of large data sets in related tasks. Given a pretrained network, we can infer which tasks the network generalizes for and the best way to transfer the information to a new task. We demonstrate a computationally inexpensive algorithm to reuse the filters of a face network for a task it was not trained for. Our analysis proves these attributes can be extracted with an accuracy comparable to what is obtained with transfer learning, but 10 times faster. We show that the information about other tasks is present in relatively small number of filters. We use these insights to do task specific pruning of a pretrained network. Our method gives significant compression ratios with reduction in size of 95% and computational reduction of 60%

Abstract

Our goal is to spot words in silent speech videos without explicitly recognizing the spoken words, where the lip motion of the speaker is clearly visible and audio is absent. Existing work in this domain has mainly focused on recognizing a fixed set of words in word-segmented lip videos, which limits the applicability of the learned model due to limited vocabulary and high dependency on the model’s recognition performance. Our contribution is twofold: (1) we develop a pipeline for recognition-free retrieval and show its performance against recognition-based retrieval on a large-scale dataset and another set of out-of-vocabulary words. (2) We introduce a query expansion technique using pseudo-relevant feedback and propose a novel re-ranking method based on maximizing the correlation between spatiotemporal landmarks of the query and the top retrieval candidates. Our word spotting method achieves 35% higher mean average precision over recognition-based method on large-scale LRW dataset. We also demonstrate the application of the method by word spotting in a popular speech video (“The great dictator” by Charlie Chaplin) where we show that the word retrieval can be used to understand what was spoken perhaps in the silent movies. Finally, we compare our model against ASR in a noisy environment and analyze the effect of the performance of underlying lip-reader and input video quality on the proposed word spotting pipeline

Abstract

This document describes the machine translation system used in the submissions of IIIT-Hyderabad (CVIT-MT) for the WAT-2018 English-Hindi translation task. Performance is evaluated on the associated corpus provided by the organizers. We experimented with convolutional sequence to sequence architectures. We also train with additional data obtained through backtranslation.

Abstract

Abstract— For an autonomous corridor following task where the environment is continuously changing, several forms of environmental noise prevent an automated feature extraction procedure from performing reliably. Moreover, in cases where pre-defined features are absent from the captured data, a well defined control signal for performing the servoing task fails to get produced. In order to overcome these drawbacks, we present in this work, using a convolutional neural network (CNN) to directly estimate the required control signal from an image, encompassing feature extraction and control law computation into one single end-to-end framework. In particular, we study the task of autonomous corridor following using a CNN and present clear advantages in cases where a traditional method used for performing the same task fails to give a reliable outcome. We evaluate the performance of our method on this task on a Wheelchair Platform developed at our institute for this purpose.

Abstract

Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the VQA process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research

Abstract

Understanding videos of people speaking across internationalborders is hard as audiences from different demographies donot understand the language. Such speech videos are oftensupplemented with language subtitles. However, these ham-per the viewing experience as the attention is shared. Simpleaudio dubbing in a different language makes the video appearunnatural due to unsynchronized lip motion. In this paper, wepropose a system for automated cross-language lip synchro-nization for re-dubbed videos. Our model generates superiorphotorealistic lip-synchronization over original video in com-parison to the current re-dubbing method. With the help of auser-based study, we verify that our method is preferred overunsynchronized videos

Abstract

Abstract— Driver inattention is one of the leading causesof vehicle crashes and incidents worldwide. Driver inattentionincludes driver fatigue leading to drowsiness and driver distrac-tion, say due to use of cellphone or rubbernecking, all of whichleads to a lack of situational awareness. Hitherto, techniquespresented to monitor driver attention evaluated factors suchas fatigue and distraction independently. However, in orderto develop a robust driver attention monitoring system allthe factors affecting driver’s attention needs to be analyzedholistically. In this paper, we presentAutoRate, a system thatleverages front camera of a windshield-mounted smartphoneto monitor driver’s attention by combining several features.We derive a driver attention rating by fusing spatio-temporalfeatures based on the driver state and behavior such as headpose, eye gaze, eye closure, yawns, use of cellphones, etc.We perform extensive evaluation ofAutoRateon real-world driving data and also data from controlled, static vehiclesettings with 30 drivers in a large city. We compareAutoRate’sautomatically-generated rating with the scores given by 5human annotators. Further, we compute the agreement betweenAutoRate’s rating and human annotator rating using kappacoefficient.AutoRate’s automatically-generated rating has anoverall agreement of 0.87 with the ratings provided by 5 humanannotators on the static dataset.

Abstract

AbstractFully supervised deep learning-based methods have created a profound im-pact in various fields of computer science. Compared to classical methods, superviseddeep learning-based techniques face scalability issues as they require huge amountsof labeled data and, more significantly, are unable to generalize to multiple domainsand tasks. In recent years, a lot of research has been targeted towards addressingthese issues within the deep learning community. Although there have been extensivesurveys on learning paradigms such as semi-supervised and unsupervised learning,there are few timely reviews after the emergence of deep learning. In this paper, weprovide an overview of the contemporary literature surrounding alternatives to fullysupervised learning in the deep learning context. First, we summarize the relevanttechniques that fall between the paradigm of supervised and unsupervised learning.Second, we take autonomous navigation as a running example to explain and com-pare different models. Finally, we highlight some shortcomings of current methodsand suggest future directions.