Abstract

In this paper we present a statistical transliteration technique that is language independent. This technique uses Hidden Markov Model (HMM) alignment and Conditional Random Fields (CRF), a discriminative model. HMM alignment maximizes the probability of the observed (source, target) word pairs using the expectation maximization algorithm and then the character level alignments (n-gram) are set to maximum posterior predictions of the model. CRF has efficient training and decoding processes which is conditioned on both source and target languages and produces globally optimal solutions. We apply this technique for Hindi-English transliteration task. The results show that our technique perfoms better than the existing transliteration system which uses HMM alignment and conditional probabilities derived from counting the alignments.

Abstract

This paper presents the experiments of Language Technologies Research Centre (LTRC) as part of their participation in CLEF2 2007 Indian language to English ad-hoc cross language document retrieval task. In this paper we discuss our Hindi and Telugu to English CLIR system and the experiments using CLEF 2007 dataset. We used a variant of TFIDF algorithm in combination with a bilingual lexicon for query translation. We also explored the role of a document summary in fielded queries and two different boolean formulations of query translations. We find that a hybrid boolean formulation using a combination of boolean AND and boolean OR operators improves ranking of documents. We also find that simple disjunctive combination of translated query keywords results in maximum recall.

Abstract

Web search personalization has been well studied in the recent few years. Relevance feedback has been used in various ways to improve relevance of search results. In this paper, we propose a novel usage of relevance feedback to effectively model the process of query formulation and better characterize how a user relates his query to the document that he intends to retrieve using a noisy channel model. We model a user profile as the probabilities of translation of query to document in this noisy channel using the relevance feedback obtained from the user. The user profile thus learnt is applied in a re-ranking phase to rescore the search results retrieved using an underlying search engine. We evaluate our approach by conducting experiments using relevance feedback data collected from users using a popular search engine. The results have shown improvement over baseline, proving that our approach can be applied to personalization of web search. The experiments have also resulted in some valuable observations that learning these user profiles using snippets surrounding the results for a query gives better performance than learning from entire document collection

Abstract

This paper discourses our CLIR experiments performed for the FIRE1 workshop. We had submitted our runs for Adhoc monolingual document retrieval in Hindi and English, and Ad-hoc cross-lingual document retrieval from Hindi to English, and English to Hindi. In this paper, we describe our English to Hindi and Hindi to English CLIR systems and the experiments conducted on them using the FIRE-2008 dataset. We had used a dictionary-based approach of query translation and transliteration of named entities in the queries using a mapping-based, Compressed Word Format (CWF) algorithm [1]. Disjunctive Query formulation with different scoring weights based on the part of the topic from which they originated gave us overall better performance.

Abstract

Named Entity Recognition (NER) is the task of identifying and classifying tokens in a text document into predefined set of classes. In this paper we show our experiments with various feature combinations for Telugu NER. We also observed that the prefix and suffix information helps a lot in finding the class of the token. We also show the effect of the training data on the performance of the system. The best performing model gave an Fβ= 1 measure of 44.91. The language independent features gave an Fβ= 1 measure of 44.89 which is close to Fβ= 1 measure obtained even by including the language dependent features.

Abstract

Text classification techniques such as Bayesian classifiers have been proved to be giving as good or better results than much more sophisticated approaches to induction. These algorithms depend on the conditional independence of the attributes given the class and term based statistical representations of documents. However, one of the main shortcomings of such methods is that they largely disregard lexical semantics and, as a consequence, are not sufficiently robust with respect to variations in word usage. In this paper we investigate the use of word sense disambiguation and other lexical semantic based techniques to achieve consistent improvement with Bayesian classifiers. Finally, we discuss the test cases experimented with Rainbow-a standard Bayesian classifier.

Abstract

The impact and importance of software has come a long way. And yet, a new generation of software developers must meet many of the same challenges that faced earlier generations. This paper aims to identify some of the most fundamental issues, challenges and opportunities for research in Software Engineering. The paper starts by examining the past, current, and future states of software engineering. The paper then examines the critical technical issues in software engineering including complexity, structure, and evolution of software systems; economics of software engineering, and measurement of software engineering products and processes, as well as the critical people and organizational issues including learning, motivation and performance improvement.

Abstract

In this paper we report our performance at DUC 2007 summarization tasks. We participated both in the query-focused multidocument summarization main task and in a pilot update summary generation tasks. This year we used a term clustering approach to better estimate a sentence prior. We used only the sentence prior which is query independent, in the update summarization task and found that it’s performance is comparable with the top performing systems. In the main task our system ranked 1 in ROUGE-2, ROUGE-SU4 and ROUGE-BE scores as well as in pyramid scores.

Abstract

Commonly used vocabulary in Indian language documents found on the web contain a number of words that have Sanskrit, Persian or English origin. However, such words may be written in different scripts with slight variations in spelling and morphology. In this paper we explore approximate string matching techniques to exploit this situation of relatively large number of cognates among Indian languages, which are higher when compared to an Indian language and a non-Indian language. We present an approach to identify cognates and make use of them for improving dictionary based CLIR when the query and documents both belong to two different Indian languages. We conduct experiments using a Hindi document collection and a set of Telugu queries and report the improvement due to cognate recognition and translation.

Abstract

Named Entity Recognition (NER) is the task of identifying and classifying all proper nouns in a document as person names, or—ganization names, location names, date & time expressions and miscellaneous. Previ—ous work (Cucerzan and Yarowsky, 1999) was done using the complete words as fea—tures which suffers from a low recall prob—lem. Character n—gram based approach (Klein et al., 2003) using generative mod—els, was experimented on English language and it proved to be useful over the word based models. Applying the same technique on Indian Languages, we experimented with Conditional Random Fields (CRFs), a dis—criminative model, and evaluated our sys—tem on two Indian Languages Telugu and Hindi. The character n—gram based models showed considerable improvement over the word based models. This paper describes the features used and experiments to increase the recall of Named Entity Recognition Systems which is also language independent