Abstract

Mobile image retrieval allows users to identify visual information about their environment by transmitting image queries to an online image database that has associated annotations (e.g. location, product information, etc.) with the images. However, this is reliant on a network connection to transmit the query and retrieve the information. This chapter examines mobile image retrieval for offline use when the data net- work connection is limited or not available. In this scenario, the entire visual search index must reside on the mobile device itself. More specifically, we are interested in ‘instance retrieval’, where the annotations associated with the images (e.g. building’s name, object information) are returned by the query and not the images themselves. Figure 1 shows an example use case where mobile camera photos are used to identify buildings and landmarks without the need for a network connection. While our targeted instance retrieval does not need to store the images, the entire visual index structure needs to fit on a mobile device, ideally within a small footprint (e.g. 16–32 MB). This small memory footprint serves two purposes. First, while mobile phones have up to 16–32 GB of storage, this is mainly in the form of slower flash memory that is an order of magnitude slower than RAM. Having the entire index within tens of MBs makes it possible for use in a resident application on the phone’s RAM. Second, this small size is inline with common practices for mobile applications; e.g. the iPhone average application size is currently 23 MB. Addition- ally, iPhone apps less than 50 MB can be downloaded using 3G/4G, anything larger

Abstract

Building bilateral semantic associations between images and texts is among the fundamental problems in computer vision. In this paper, we study two complementary cross-modal prediction tasks: (i) predicting text(s) given a query image (“Im2Text”), and (ii) predicting image(s) given a piece of text (“Text2Im”). We make no assumption on the specific form of text; i.e., it could be either a set of labels, phrases, or even captions. We pose both these tasks in a retrieval framework. For Im2Text, given a query image, our goal is to retrieve a ranked list of semantically relevant texts from an independent text-corpus (i.e., texts with no corresponding images). Similarly, for Text2Im, given a query text, we aim to retrieve a ranked list of semantically relevant images from a collection of unannotated images (i.e., images without any associated textual meta-data). We propose a novel Structural SVM based unified framework for these two tasks, and show how it can be efficiently trained and tested. Using a variety of loss functions, extensive experiments are conducted on three popular datasets (two medium-scale datasets containing few thousands of samples, and one web-scale dataset containing one million samples). Experiments demonstrate that our framework gives promising results compared to competing baseline cross-modal search techniques, thus confirming its efficacy. Keywords: Image search; Image description; Cross-media analysis

Abstract

Human pose as a query modality is an alternative and rich experience for image and video retrieval. It has interesting retrieval applications in domains such as sports and dance databases. In this work we propose two novel ways for representing the image of a person striking a pose, one looking for parts and other looking at the whole image. These representations are then used for retrieval. Both the representations are obtained using deep learning methods. In the first method, we make the following contributions: (a) We introduce ‘deep poselets’ for pose- sensitive detection of various body parts, built on convolutional neural network (CNN) features. These deep poselets significantly outperform previous instantiations of Berkeley poselets [6], and (b) Using these detec- tor responses, we construct a pose representation that is suitable for pose search, and show that pose retrieval performance is on par with the previous methods. In the second method, we make the follow- ing contributions: (a) We design an optimized neural network which maps the input image to a very low dimensional space where similar poses are close by and dissimilar poses are farther away, and (b) We show that pose retrieval system using these low dimensional representation is on par with the deep poselet representation and is on par with the previous methods. The previous works with which the above two methods are compared include bag of visual words [44], Berkeley poselets [6] and human pose estimation algorithms [52]. All the methods are quantitatively eval- uated on a large dataset of images built from a number of standard benchmarks together with frames from

Abstract

Building robust recognizers for Arabic has always been challenging. We demonstrate the effectiveness of an end-to- end trainable CNN - RNN hybrid architecture in recognizing Arabic text in videos and natural scenes. We outperform previous state- of-the-art on two publicly available video text datasets - ALIF and ACTIV . For the scene text recognition task, we introduce a new Arabic scene text dataset and establish baseline results. For scripts like Arabic, a major challenge in developing robust recognizers is the lack of large quantity of annotated data. We overcome this by synthesizing millions of Arabic text images from a large vocabulary of Arabic words and phrases. Our implementation is built on top of the model introduced here [37] which is proven quite effective for English scene text recognition. The model follows a segmentation-free, sequence to sequence transcription approach. The network transcribes a sequence of convolutional features from the input image to a sequence of target labels. This does away with the need for segmenting input image into constituent characters/glyphs, which is often difficult for Arabic script. Further, the ability of RNN s to model contextual dependencies yields superior recognition results.

Abstract

Inspired by the success of Deep Learning based approaches to English scene text recognition, we pose and bench- mark scene text recognition for three Indic scripts - Devanagari, Telugu and Malayalam. Synthetic word images rendered from Unicode fonts are used for training the recognition system. And the performance is bench-marked on a new - IIIT-ILST dataset comprising of hundreds of real scene images containing text in the above mentioned scripts. We use a segmentation free, hybrid but end-to-end trainable CNN - RNN deep neural network for transcribing the word images to the corresponding texts. The cropped word images need not be segmented into the sub- word units and the error is calculated and backpropagated for the the given word image at once. The network is trained using CTC loss, which is proven quite effective for sequence- to-sequence transcription tasks. The CNN layers in the network learn to extract robust feature representations from word images. The sequence of features learnt by the convolutional block is transcribed to a sequence of labels by the RNN + CTC block. The transcription is not bound by word length or a lexicon and is ideal for Indian languages which are highly inflectional.

Abstract

This paper explores the effectiveness of statistical language model ( SLM ) and dictionary based methods for detec- tion and correction of errors in Indic OCR output. In SLM , we use unicode level ngrams for building the language model. We compare its performance with akshara level ngrams and find that akshara level ngrams perform better in detecting the errors when compared to unicode level ngrams. We experimentally analyze the performance of Indic OCR post-processing using dictionary method, compare the performance with English and analyze the reasons for the under-performance in Indic scripts. We use four major Indian languages for our experiments, namely Hindi, Gurumukhi, Telugu and Malayalam.

Abstract

The power of ConvNets has been demonstrated in a wide variety of vision tasks including pose estimation. But they often produce absurdly erroneous predictions in videos due to unusual poses, challenging illumination, blur, self-occlusions etc. These erroneous predictions can be re- fined by leveraging previous and future predictions as the temporal smoothness constrain in the videos. In this paper, we present a generic approach for pose correction in videos using sequence learning that makes minimal assumptions on the sequence structure. The proposed model is generic, fast and surpasses the state-of-the-art on benchmark datasets. We use a generic pose estimator for initial pose estimates, which are further refined using our method. The proposed architecture uses Long Short-Term Memory ( LSTM ) encoder- decoder model to encode the temporal context and refine the estimations. We show 3.7% gain over the baseline Yang & Ramanan ( YR ) [1] and 2.07% gain over Spatial Fusion Network ( SFN ) [2] on a new challenging YouTube Pose Subset dataset [3]. Keywords-Pose estimation; sequence to sequence learnin

Abstract

We look at the problem of developing a compact and accurate model for gesture recognition from videos in a deep-learning framework. Towards this we propose a joint 3DCNN-LSTM model that is end-to-end trainable and is shown to be better suited to capture the dynamic information in actions. The solution achieves close to state-of-the-art accuracy on the ChaLearn dataset, with only half the model size. We also explore ways to derive a much more compact representation in a knowledge distillation framework followed by model compression. The final model is less than 1 MB in size, which is less than one hundredth of our initial model, with a drop of 7% in accuracy, and is suitable for real-time gesture recognition on mobile devices.

Abstract

We will present our work on controlling nuclear and electronic motion in molecules using laser pulses designed using Optimal control theory (OCT). The control problem is formulated as the maximization of cost functional made of an objective function along with the constraints on the field fluence. The objective function has the prescribed dynamical goals. A detailed analysis of the optimal pulse reveals complex path-ways to arrive at seemingly simple goals. We have used OCT to obtain infrared laser pulses for the selective vibrational excitation of several different systems, from a simple diatomic to triatomic systems located in complex molecular environments. Controlled initiation of intramolecular H-transfer in malonaldehyde using ultrashort uv-laser pulses through an optically bright electronic excited state as a mediator will be discussed. Finally, our attempts to control the quantum dynamics of N-H photodissociation of pyrrole via a counterintuitive coupling of electronic states using uv-laser pulses will be discussed.

Abstract

A first principles quantum dynamics study of N–H photodissociation of pyrrole on the coupled electronic states is carried out with the aid of an optimally designed UV-laser pulse. A new photodissociation path, as compared to the conventional barrier crossing on the state, opens up upon electronic transitions under the influence of pump-dump laser pulses, which efficiently populate both the dissociation channels. The interplay of electronic transitions due both to vibronic coupling and the laser pulse is observed in the control mechanism and discussed in detail. The proposed control mechanism seems to be robust, and not discussed in the literature so far, and is expected to trigger future experiments on the photochemistry of molecules of chemical and biological importance. The design of the optimal pulses and their application to enhance the overall dissociation probability is carried out within the framework of optimal control theory. The quantum dynamics of the system in the presence of pulse is treated by solving the time-dependent Schr ̈ odinger equation in the semi-classical dipole approximation