Abstract

In this paper, we have considered a real world information synthesis task, generation of a fixed length multi document summary which satisfies a specific information need. This task was mapped to a topic-oriented, informative multi-document summarization. We also tried to estimate, given the human written reference summaries and the document set, the maximum performance (ROUGE1 scores) that can be achieved by an extraction-based summarization technique. Motivated by the observation that the current approaches are far behind the estimated maximum performance, we have looked at Information Retrieval techniques to improve the relevance scoring of sentences towards information need. Following information theoretic approach we have identified a measure to capture the notion of importance or prior of a sentence. Following a different decomposition of Probability Ranking Principle, the calculated importance/prior is incorporated into the final sentence scoring by weighted linear combination. In order to evaluate the performance, we have explored information sources like WWW and encyclopedia in computing the information measure in a set of different experiments. The t-test analysis of the improvement on DUC2 2005 data set is found to be significant (p∼ 0.05). The same system has outperformed rest of the systems at DUC 2006 challenge in terms of ROUGE scores with a significant margin over the next best system.

Abstract

A large annotated corpus is critical to the development of robust optical character recognizers (OCRs). However, creation of annotated corpora is a tedious task. It is laborious, especially when the annotation is at the character level. In this paper, we propose an efficient hierarchical approach for annotation of large collection of printed document images. We align document images with independently keyed-in text. The method is model-driven and is intended to annotate large collection of documents, scanned in three different resolutions, at character level. We employ an XML representation for storage of the annotation information. APIs are provided for access at content level for easy use in training and evaluation of OCRs and other document understanding tasks.

Abstract

In Africa around 2,500 languages are spoken. Some of these languages have their own indigenous scripts. Accordingly, there is a bulk of printed documents available in libraries, information centers, museums and offices. Digitization of these documents enables to harness already available language technologies to local information needs and developments. This paper presents an Optical Character Recognition (OCR) system for converting digitized documents in local languages. An extensive literature survey reveals that this is the first attempt that reports the challenges towards the recognition of indigenous African scripts and a possible solution for Amharic script. Research in the recognition of African indigenous scripts faces major challenges due to (i) the use of large number characters in the writing and (ii) existence of large set of visually similar characters. In this paper, we employ a novel feature extraction scheme using principal component and linear discriminant analysis, followed by a decision directed acyclic graph based support vector machine classifier. Recognition results are presented on real-life degraded documents such as books, magazines and newspapers to demonstrate the performance of the recognizer

Abstract

Search engines today often return a large volume of results with possibly a few relevant results. The notion of relevance is subjective and depends on the user and context. Re-ranking of the results to reflect the most relevant results to the user using the relevance feedback has received wide attention in information retrieval in recent years. Also, sharing of information among users having similar interests using collaborative filtering techniques has achieved wide success in recommendation systems. In this paper, we propose a novel approach for re-ranking of the search results using collaborative filtering techniques using relevance feedback of a given user as well as the other users. Our approach is to learn the profiles of the users using machine learning techniques making use of past browsing histories including queries posed and documents found relevant or irrelevant. Re-ranking of the results is done using collaborative filtering techniques. First, the context of the query is inferred from the query category. The user’s community is determined dynamically in the context of the query by using the user profiles. The rank of a document is calculated using the user’s profile as well profiles of the other users in the community

Abstract

Automatic annotation is an elegant alternative to explicit recognition in images. In annotation, the image is matched with keyword models, and the most relevant keywords are assigned to the image. Using existing techniques, the annotation time for large collections is very high, while the annotation performance degrades with increase in number of keywords. Towards the goal of large scale annotation, we present an approach called “Reverse Annotation”. Unlike traditional annotationwhere keywords are identified for a given image, in Reverse Annotation, the relevant images are identified for each keyword. With this seemingly simple shift in perspective, the annotation time is reduced significantly. To be able to rank relevant images, the approach is extended to Probabilistic Reverse Annotation. Our framework is applicable to a wide variety of multimedia documents, and scalable to large collections. Here, we demonstrate the framework over a large collection of 75,000 document images, containing 21 million word segments, annotated by 35000 keywords. Our image retrieval system replicates text-based search engines, in response time.

Abstract

An indexing model is the heart of an Information Retrieval (IR) system. Data structures such as term based inverted indices have proved to be very effective for IR using vector space retrieval models. However, when functional aspects of such models were tested, it was soon felt that better relevance models were required to more accurately compute the relevance of a document towards a query. It was shown that language modeling approaches [1] in monolingual IR tasks improve the quality of search results in comparison with TFIDF [2] algorithm. The disadvantage of language modeling approaches when used in monolingual IR task as suggested in [1] is that they would require both the inverted index (term-todocument) and the forward index (document-to-term) to be able to compute the rank of document for a given query. This calls for an additional space and computation overhead when compared to inverted index models. Such a cost may be acceptable if the quality of search results are significantly improved. In a Cross-lingual IR (CLIR) task, we have previously shown in [3] that using a bilingual dictionary along with term co-occurrence statistics and language modeling approach helps improve the functional IR performance. However, no studies exist on the performance overhead in a CLIR task due to language modeling. In this paper we present an augmented index model which can be used for fast retrieval while having the benefits of language modeling in a CLIR task. The model is capable of retrieval and ranking with or without query expansion techniques using term collocation statistics of the indexed corpus. Finally we conduct performance related experiments on our indexing model to determine the cost overheads on space and time

Abstract

This chapter describes the role of ontologies and corporate taxonomies in managing the content and knowledge within organizations. Managing content in a reusable and effective manner is becoming increasingly important in knowledge centric organizations as the amount of content generated, both text based and rich media, is growing exponentially. Search, categorization and document characterization, content staging and content delivery are the key technology challenges in knowledge management systems. This chapter describes how corporate taxonomies and ontologies can help in making sense of huge amount of content that gets generated across the locations in different languages and formats Different information silos can be connected and workflow and collaboration can be achieved using ontologies. As the KM solutions are moving from a centralized approach to a distributed approach, a framework where multiple taxonomies and ontologies can co-exist with uniform interfaces is needed.

Abstract

Commonly used vocabulary in Indian language documents found on the web contain a number of words that have Sanskrit, Persian or English origin. However, such words may be written in different scripts with slight variations in spelling and morphology. In this paper we explore fuzzy string matching techniques to exploit this situation of relatively large number of cognates among Indian languages, which are higher when compared to an Indian language and a non-Indian language. We present an approach to identify cognates and make use of them for improving dictionary based CLIR when the query and documents both belong to two different Indian languages. We conduct experiments using a Hindi document collection and a set of Telugu queries and report the improvement due to cognate recognition and translation.

Abstract

The twin challenges of massive information overload via the web and ubiquitous computers present us with an unavoidable task: developing techniques to handle multilingual information robustly and efficiently, with as high quality performance as possible. Previous research activities on multilingual information access systems have studied cross-language information retrieval (CLIR), information extraction and factoid based question answering tasks in detail. It is believed by the research community and also previously acknowledged in a US-NSF report [Hovy et al., 1999] that a cross-language query focused summarization could play a vital role in multilingual information access, as a bridge between CLIR and machine translation. Surprisingly enough, no detailed studies exist yet on the effects of cross-linguality to query focused summarization. In this paper we study the effects of adding a cross-language dimension to query focused multi-document summarization for the Telugu-English language pair. We use a cross-lingual relevance based language modeling approach to generate extraction based summary. We evaluate the system using DUC1 2005 dataset using ROUGE2 metrics and compare with the mono-lingual baseline which uses relevance based language modeling in mono-lingual setting.

Abstract

This paper reports on the first Oromo-English CLIR system that is based on dictionary-based query translation techniques. The basic objective of the study is to design and develop an Oromo-English CLIR system with a view to enable Afaan Oromo speakers to access and retrieve the vast online information sources that are available in English by using their own (native) language queries. We describe the major approaches and procedures that have been used in designing and developing Oromo-English CLIR system together with the information retrieval evaluation experiments recently conducted at CLEF 2006 ad hoc track. The purpose of the current initial evaluation experiments was to assess the over all performance of the Oromo-English CLIR system by using different fields of Afaan Oromo topics. Thus we submitted three official runs (experiments) that differed in terms of utilized fields in the topic set, ie title run (OMT), title and description run (OMTD), and title, description and narration run (OMTDN) to CLEF 2006. Since appropriate online language processing resources and information retrieval tools are not available for Afaan Oromo, only limited linguistic resources such as Oromo-English dictionary and Afaan Oromo stemmer that had been designed and developed at our research center were used in conducting the experiments. Yet we found the performances of our evaluation experiments at CLEF 2006 very encouraging which is a good news for development and application of CLIR systems in other indigenous African languages.