Abstract

n this paper, we present a solution towards building a retrieval system over handwritten document images that i) is recognition-free, ii) allows text-querying, iii) can retrieve at sub-word level, iv) can search for out-of-vocabulary words. Unlike previous approaches that operate at either character or word lev-els, we use character n-gram images (CNG-img) as the retrieval primitive.CNG-img are sequences of character segments, that are represented and matched in the image-space. The word-images are now treated as a bag-of-CNG-img, that can be indexed and matched in the feature space. This allows for recognition-free search (query-by-example), which can retrieve morphologically similar words that have matching sub-words. Further, to enable query-by-keyword, we build an automated scheme to generate labeled exemplars for characters and character n-grams, fromun constrained handwritten documents. We pose this problem as one of weakly-supervised learning, where character/n-gram labeling is obtained automatically from the word labels. There sulting retrieval system can answer queries from an unlimited vocabulary. The approach is demonstrated on the George Wash-ington collection, results show major improvement in retrieval performance as compared to word-recognition and word-spotting methods

Abstract

Optical Character Recognition (OCR) problems areoften formulated as isolated character (symbol) classification task followed by a post-classification stage (which contains modules like Unicode generation, error correction etc. ) to generate the textual representation, for most of the Indian scripts. Such approaches are prone to failures due to (i) difficulties in designing reliable word-to-symbol segmentation module that can robustly work in presence of degraded (cut/fused) images and (ii) convert-ing the outputs of the classifiers to a valid sequence of Unicodes.In this paper, we propose a formulation, where the expectations on these two modules is minimized, and the harder recognition task is modelled as learning of an appropriate sequence to sequence translation scheme. We thus formulate the recognition as a direct transcription problem. Given many examples of feature sequences and their corresponding Unicode representations, our objective is to learn a mapping which can convert a word directly in to a Unicode sequence. This formulation has multiple practical advantages: (i) This reduces the number of classes significantly for the Indian scripts. (ii) It removes the need for a reliable word-to-symbol segmentation. (ii) It does not require strong annotation of symbols to design the classifiers, and (iii) It directly generates a valid sequence of Unicodes. We test our method on more than 6000 pages of printed Devanagari documents from multiple sources. Our method consistently outperforms other state of the art implementations.

Abstract

Bag of words (BoW) based retrieval is an efficient method to compare the visual similarity between two images.Recognition free methods based on BoW have shown to out-perform OCR based methods. We further improve the perfor-mance by defining a document specific sparse coding scheme for representing visual words (interest points) in document images.Our method is motivated by the successful use of sparsity insignal representation by exploiting the neighbourhood properties.In addition to providing insights into the design of the coding scheme, we also verify the method on two data sets and compare with the recent methods. We have also developed text query based search solution, and we report performance comparable to image based search

Abstract

Recognizing text in images taken in the wild is achallenging problem that has received great attention in recentyears. Previous methods addressed this problem by first detectingindividual characters, and then forming them into words. Suchapproaches often suffer from weak character detections, due tolarge intra-class variations, even more so than characters fromscanned documents. We take a different view of the problemand present a holistic word recognition framework. In this,we first represent the scene text image and synthetic imagesgenerated from lexicon words using gradient-based features. Wethen recognize the text in the image by matching the scene andsynthetic image features with our novel weighted Dynamic TimeWarping (wDTW) approach.We perform experimental analysis on challenging publicdatasets, such as Street View Text and ICDAR 2003. Ourproposed method significantly outperforms our earlier work inMishraet al.(CVPR 2012), as well as many other recent works,such as Novikovaet al.(ECCV 2012), Wanget al.(ICPR 2012),Wanget al.(ICCV 2011)

Abstract

Performance of the recognition free approaches fordocument retrieval, heavily depends on the exact or approximatematching of images (in some feature space) to retrieve documentscontaining the same word. However, the harder problem in infor-mation retrieval is to effectively bring semantics into the retrievalpipeline. This is further challenging when the matching is basedon visual features. In this work, we investigate this problem,and suggest a solution by directly transferring the semanticsfrom the textual domain. Our retrieval framework uses (i) thelanguage resources like WordNet and (ii) an annotated corpus ofdocument images, to retrieve semantically relevant words from alarge word image database. We demonstrate the method on twolanguages — English and Hindi, and quantitatively evaluate theperformance on annotated word image databases of more thana Million images

Abstract

Many documents are created by Cut-And-Paste(CAP) of existing documents. In this paper, we proposed a noveltechnique to detect CAP in document images. This can helpin detecting unethical CAP in document image collections. Oursolution is recognition free, and scalable to large collection ofdocuments. Our formulation is also independent of the imagingprocess (camera based or scanner based) and does not use anylanguage specific information for matching across documents. Wemodel the solution as finding a mixture of homographies, anddesign a linear programming (LP) based solution to compute thesame. Our method is presently limited by the fact that we do notsupport detection of CAP in documents formed by editing of thetextual content.Our experiments demonstrate that without loss of generality(i.e. without assuming the number of source documents), we cancorrectly detect and match the CAP content in a questioneddocument image by simultaneously comparing with large numberof images in the database. We achieve the CAP detection accuracyof as high as 90%, even when the spatial extent of the CAP contentin a document image is as small as 15% of the entire image area.

Abstract

Parsing for clothes in images and videos is a critical step towards understanding the human appearance. In this work, we propose a method to segment clothes in settings where there is no restriction on number and type of clothes, pose of the person, viewing angle, occlusion and number of people. This is a challenging task as clothes, even of the same category, have large variations in color and texture. The presence of human joints is the best indicator for cloth types as most of the clothes are consistently worn around the joints. We incorporate the human joint prior by estimating the body joint distributions using the detectors and learning the cloth-joint co-occurrences of different cloth types with respect to body joints. The cloth-joint and cloth-cloth co-occurrences are used as a part of the conditional random field framework to segment the image into different clothing. Our results indicate that we have outperformed the recent attempt [16]on H3D [3], a fairly complex dataset.

Abstract

We address the problem of automatic image annotation in large vocabulary datasets.In such datasets, for a given label, there could be several other labels that act as its confusing labels. Three possible factors for this are (i) incomplete-labeling (“cars”vs.“vehicle”), (ii) label-ambiguity (“flowers”vs.“blooms”), and (iii) structural-overlap(“lion”vs.“tiger”). While previous studies in this domain have mostly focused on nearest-neighbour based models, we show that even the conventional one-vs-rest SVM significantly outperforms several benchmark models. We also demonstrate that with a simple modification in the hinge-loss of SVM, it is possible to significantly improve its performance. In particular, we introduce a tolerance-parameter in the hinge-loss. Thismakes the new model more tolerant against the errors in the classification of samples tagged with confusing labels as compared to other samples. This tolerance parameter is automatically determined using visual similarity and dataset statistics. Experimental evaluations demonstrate that our method (referred to as SVM with Variable Tolerance or SVM-VT) shows promising results on the task of image annotation on three challenging datasets, and establishes a baseline for such models in this domain.

Abstract

Understanding positional semantics of the environment plays an important role in manipulating an object in clutter. The interaction with surrounding objects in the environment must be considered in order to perform the task without causing the objects fall or get damaged. In this paper, we learn the semantics in terms of support relationship among different objects in a cluttered environment by utilizing various photometric and geometric properties of the scene. To manipulate an object of interest, we use the inferred support relationship to derive a sequence in which its surrounding objects should be removed while causing minimal damage to the environment. We believe, this work can push the boundary of robotic applications in grasping, object manipulation and picking-from-bin, towards objects of generic shape and size and scenarios with physical contact and overlap. We have created an RGBD dataset that consists of various objects used in day-to-day life present in clutter. We explore many different settings involving different kind of object-object interaction. We successfully learn support relationships and predict support order in these settings.

Abstract

Detecting compact overlapping communities in large networks is an important pattern recognition proble mwith applications in many domains. Most community de-tection algorithms trade-off between community sizes, their compactness and the scalability of finding communities. Clique Percolation Method (CPM) [1] and Local Fitness Maximization(LFM) [2] are two prominent and commonly used overlapping community detection methods that scale with large networks.However, significant number of communities found by the mare large, noisy, and loose. In this paper, we propose a general algorithm that takes such large and loose communities generated by any method and refines them into compact com-munities in a systematic fashion. We define a new measure of community-ness based on eigenvector centrality, identify loose communities using this measure and propose an algorithm for partitioning such loose communities into compact communities.We refine the communities found by CPM and LFM using our method and show their effectiveness compared to the original communities in a recommendation engine task.