Search T-Space Advanced Search
Home

Browse
Communities
& Collections

Issue Date
Author
Title
Subject

Sign on to:
Receive email
updates

My Account
authorized users

Edit Profile

Help
About T-Space
 Please use this identifier to cite or link to this item: http://hdl.handle.net/1807/11238

 Title: Measuring Semantic Distance using Distributional Profiles of Concepts Authors: Mohammad, Saif Advisor: Hirst, Graeme Department: Computer Science Keywords: Computational LinguisticsNatural Language ProcessingLexical semanticssemantic distancedistributional similaritysemantic similaritysemantic relatednessword concept co-occurrence matrixdistributional profiles of conceptsthesauruscorpus-based techniquesword sensescross-lingual techniquesword sense dominanceword sense disambiguationwordnet Issue Date: 1-Aug-2008 Abstract: Semantic distance is a measure of how close or distant in meaning two units of language are. A large number of important natural language problems, including machine translation and word sense disambiguation, can be viewed as semantic distance problems. The two dominant approaches to estimating semantic distance are the WordNet-based semantic measures and the corpus-based distributional measures. In this thesis, I compare them, both qualitatively and quantitatively, and identify the limitations of each. This thesis argues that estimating semantic distance is essentially a property of concepts (rather than words) and that two concepts are semantically close if they occur in similar contexts. Instead of identifying the co-occurrence (distributional) profiles of words (distributional hypothesis), I argue that distributional profiles of concepts (DPCs) can be used to infer the semantic properties of concepts and indeed to estimate semantic distance more accurately. I propose a new hybrid approach to calculating semantic distance that combines corpus statistics and a published thesaurus (Macquarie Thesaurus). The algorithm determines estimates of the DPCs using the categories in the thesaurus as very coarse concepts and, notably, without requiring any sense-annotated data. Even though the use of only about 1000 concepts to represent the vocabulary of a language seems drastic, I show that the method achieves results better than the state-of-the-art in a number of natural language tasks. I show how cross-lingual DPCs can be created by combining text in one language with a thesaurus from another. Using these cross-lingual DPCs, we can solve problems in one, possibly resource-poor, language using a knowledge source from another, possibly resource-rich, language. I show that the approach is also useful in tasks that inherently involve two or more languages, such as machine translation and multilingual text summarization. The proposed approach is computationally inexpensive, it can estimate both semantic relatedness and semantic similarity, and it can be applied to all parts of speech. Extensive experiments on ranking word pairs as per semantic distance, real-word spelling correction, solving Reader's Digest word choice problems, determining word sense dominance, word sense disambiguation, and word translation show that the new approach is markedly superior to previous ones. URI: http://hdl.handle.net/1807/11238 Appears in Collections: DoctoralDepartment of Computer Science - Doctoral theses

Files in This Item:

File Description SizeFormat
Mohammad_Saif_M_200806_PhD_thesis.pdf1.23 MBAdobe PDF
View/Open

This item is licensed under a Creative Commons License

Items in T-Space are protected by copyright, with all rights reserved, unless otherwise indicated.