Abstract

Abstract—This paper presents final results of ICDAR 2019Scene Text Visual Question Answering competition (ST-VQA).ST-VQA introduces an important aspect that is not addressedby any Visual Question Answering system up to date, namelythe incorporation of scene text to answer questions asked aboutan image. The competition introduces a new dataset comprising23,038images annotated with31,791question / answer pairswhere the answer is always grounded on text instances presentin the image. The images are taken from7different publiccomputer vision datasets, covering a wide range of scenarios.The competition was structured in three tasks of increas-ing difficulty, that require reading the text in a scene andunderstanding it in the context of the scene, to correctlyanswer a given question. A novel evaluation metric is presented,which elegantly assesses both key capabilities expected from anoptimal model: text recognition and image understanding.A detailed analysis of results from different participants isshowcased, which provides insight into the current capabilitiesof VQA systems that can read. We firmly believe the datasetproposed in this challenge will be an important milestone toconsider towards a path of more robust and general models thatcan exploit scene text to achieve holistic image understanding.

Abstract

Cross-modal retrieval methods have been significantly improvedin last years with the use of deep neural networks and large-scaleannotated datasets such as ImageNet and Places. However, collect-ing and annotating such datasets requires a tremendous amountof human effort and, besides, their annotations are limited to dis-crete sets of popular visual classes that may not be representativeof the richer semantics found on large-scale cross-modal retrievaldatasets. In this paper, we present a self-supervised cross-modalretrieval framework that leverages as training data the correlationsbetween images and text on the entire set of Wikipedia articles.Our method consists in training a CNN to predict: (1) the semanticcontext of the article in which an image is more probable to appearas an illustration, and (2) the semantic context of its caption. Ourexperiments demonstrate that the proposed method is not only ca-pable of learning discriminative visual representations for solvingvision tasks like classification, but that the learned representationsare better for cross-modal retrieval when compared to supervisedpre-training of the network on the ImageNet dataset.

Abstract

Histopathological images contain morphological markers of disease progression that have diagnostic and predictive values. In this study, we demonstrate how deep learning framework can be used for an automatic classification of Renal Cell Carcinoma (RCC) subtypes, and for identification of features that predict survival outcome from digital histopathological images. Convolutional neural networks (CNN’s) trained on whole-slide images distinguish clear cell and chromophobe RCC from normal tissue with a classification accuracy of 93.39% and 87.34%, respectively. Further, a CNN trained to distinguish clear cell, chromophobe and papillary RCC achieves a classification accuracy of 94.07%. Here, we introduced a novel support vector machine-based method that helped to break the multi-class classification task into multiple binary classification tasks which not only improved the performance of the model but also helped to deal with data imbalance. Finally, we extracted the morphological features from high probability tumor regions identified by the CNN to predict patient survival outcome of most common clear cell RCC. The generated risk index based on both tumor shape and nuclei features are significantly associated with patient survival outcome. These results highlight that deep learning can play a role in both cancer diagnosis and prognosis.

Abstract

We present a simple, yet effective, Neural Ma-chineTranslationsystemforIndianlanguages.Wedemonstratethefeasibilityformultiplelan-guage pairs, and establish a strong baseline forfurther research.

Abstract

A Cost Efficient Approach to Correct OCR Errors in Large Document CollectionsDeepayan Das, Jerin Philip, Minesh Mathew and C. V. JawaharCenter for Visual Information Technology, IIIT Hyderabad, India.{deepayan.das, jerin.philip, minesh.mathew}@research.iiit.ac.in, jawahar@iiit.ac.inAbstract—Word error rate of anOCRis often higher thanits character error rate. This is especially true whenOCRsaredesigned by recognizing characters. High word accuracies arecritical for many practical applications like content creationand text-to-speech systems. In order to detect and correct themisrecognised words, it is common for anOCRto employ apost-processor module to improve the word accuracy. However,conventional approaches to post-processing like looking up adictionary or using a statistical language model (SLM), are stilllimited. In many such scenarios, it is often required to removethe outstanding errors manually.We observe that the traditional post-processing schemes lookat error words sequentially sinceOCRs process documents oneat a time. We propose a cost efficient model to address the errorwords in batches rather than correcting them individually.We exploit the fact that a collection of documents (eg. abook), unlike a single document, has a structure leading torepetition of words. Such words, if efficiently grouped togetherand corrected together, can lead to significant reduction in theeffort. Error correction can be fully automatic or with a humanin the loop. We compare the performance of our method withvarious baseline approaches including the case where all theerrors are removed by a human. We demonstrate the efficacy ofour solution empirically by reporting more than70%reductionin the human effort with near perfect error correction. Wevalidate our method on books in both English and Hindi.

Abstract

bstract—Reading of mathematical expression or equationin the document images is very challenging due to the largevariability of mathematical symbols and expressions. In thispaper, we pose reading of mathematical equation as a taskof generation of the textual description which interprets theinternal meaning of this equation. Inspired by the naturalimage captioning problem in computer vision, we presenta mathematical equation description (MED) model, a novelend-to-end trainable deep neural network based approachthat learns to generate a textual description for readingmathematical equation images. OurMEDmodel consists of aconvolution neural network as an encoder that extracts featuresof input mathematical equation images and a recurrent neuralnetwork with attention mechanism which generates descriptionrelated to the input mathematical equation images. Due tothe unavailability of mathematical equation image data setswith their textual descriptions, we generate two data sets forexperimental purpose. To validate the effectiveness of ourMEDmodel, we conduct a real-world experiment to see whether thestudents are able to write equations by only reading or listeningtheir textual descriptions or not. Experiments conclude that thestudents are able to write most of the equations correctly byreading their textual descriptions only.

Abstract

bstract—Graphical elements: particularly tables and fig-ures contain a visual summary of the most valuable infor-mation contained in a document. Therefore, localization ofsuch graphical objects in the document images is the initialstep to understand the content of such graphical objects ordocument images. In this paper, we present a novel end-to-endtrainable deep learning based framework to localize graphicalobjects in the document images called as Graphical ObjectDetection (GOD). Our framework is data-driven and does notrequire any heuristics or meta-data to locate graphical objectsin the document images. TheGODexplores the concept oftransfer learning and domain adaptation to handle scarcity oflabeled training images for graphical object detection task inthe document images. Performance analysis carried out on thevarious public benchmark data sets:ICDAR-2013,ICDAR-POD2017 andUNLVshows that our model yields promising resultsas compared to state-of-the-art techniques

Abstract

Abstract—Automated evaluation of handwritten answers hasbeen a challenging problem for scaling the education systemfor many years. Speeding up the evaluation remains as themajor bottleneck for enhancing the throughput of instructors.This paper describes an effective method for automaticallyevaluating the short descriptive handwritten answers fromthe digitized images. Our goal is to evaluate a student’shandwritten answer by assigning an evaluation score that iscomparable to the human-assigned scores. Existing works inthis domain mainly focused on evaluating handwritten essayswith handcrafted, non-semantic features. Our contribution istwo-fold: 1) we model this problem as a self-supervised, feature-based classification problem, which can fine-tune itself for eachquestion without any explicit supervision. 2) We introduce theusage of semantic analysis for auto-evaluation in handwrittentext space using the combination of Information Retrievaland Extraction (IRE) and, Natural Language Processing (NLP)methods to derive a set of useful features. We tested our methodon three datasets created from various domains, using the helpof students of different age groups. Experiments show that ourmethod performs comparably to that of human evaluators.

Abstract

Abstract—The ICDAR 2019 Challenge on “Scanned receiptsOCR and key information extraction” (SROIE) covers importantaspects related to the automated analysis of scanned receipts.The SROIE tasks play a key role in many document analysissystems and hold significant commercial potential. Although a lotof works have been published over the years on administrativedocument analysis, the community has advanced relatively slowly,as most datasets have been kept private. One of the keycontributions of SROIE to the document analysis community is tooffer a first, standardized dataset of 1000 whole scanned receiptimages and annotations, as well as an evaluation procedure forsuch tasks. The Challenge is structured around three tasks,namely Scanned Receipt Text Localization (Task 1), ScannedReceipt OCR (Task 2) and Key Information Extraction fromScanned Receipts (Task 3). The competition opened on 10thFebruary, 2019 and closed on 5th May, 2019. We received 29,24 and 18 valid submissions received for the three competitiontasks, respectively. This report presents the competition datasets,define the tasks and the evaluation protocols, offer detailedsubmission statistics, as well as an analysis of the submittedperformance. While the tasks of text localization and recognitionseem to be relatively easy to tackle, it is interesting to observethe variety of ideas and approaches proposed for the informationextraction task. According to the submissions’ performance webelieve there is still margin for improving information extractionperformance, although the current dataset would have to growsubstantially in following editions. Given the success of theSROIE competition evidenced by the wide interest generatedand the healthy number of submissions from academic, researchinstitutes and industry over different countries, we consider thatthe SROIE competition can evolve into a useful resource for thecommunity, drawing further attention and promoting researchand development efforts in this field

Abstract

Abstract—Document figure classification (DFC) is an im-portant stage of a document figure understanding system.The design of a DFC system required a well defined figurecategories and dataset. To the best of the author’s knowledge,the existing datasets related to classification of figures in thedocument images are limited with respect to their size andcategories [1]–[3]. In this paper, we introduce a scientificfigure classification dataset, named asDocFigure. The datasetconsists of33K annotated figures of28different categoriespresent in the document images which correspond to scientificarticles published in theCVPR,ECCV,ICCV, etc. conferences inlast several years. Manual annotation of such a large number(33K) of figures is time consuming and cost ineffective. Inthis article, we design a web based annotation tool which canefficiently assign category labels to large number of figures withthe minimum efforts of human annotators. To benchmark ourgenerated dataset on classification task, we propose three base-line classification techniques using deep feature, deep texturefeature and combination of both. In our analysis, we found thatthe combination of both deep feature and deep texture featureis more effective for document figure classification task thanthe individual features. The dataset and the code are publiclyavailable at https://researchweb.iiit.ac.in/∼jobin.kv/projects