Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model Siva Reddy Artificial Intelligence Group Department of Computer Science University of York
Research Student Seminar Jan 24 2011
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Outline
1
Multi-word
2
Compositionality
3
Problem Definition
4
Background
5
Previous Approaches
6
My Observations
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Multi-word
A sequence of two or more words describing a meaning together. Compound Nouns credit card leather jacket
Phrasal Verbs look up get over
Idiomatic expressions kick the bucket spill the beans
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Compositionality
Given meanings of Think Water Tank Can we interpret the meanings of Think Tank Water Tank
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Compositionality
Given meanings of Think Water Tank Can we interpret the meanings of Think Tank Water Tank
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Compositionality
A multi-word “A B” is compositional if meaning(A B) = meaning(A) + meaning(B) e.g. Water Tank e.g. Post Man Non-Compositional Think Tank Smoking Gun Apple Polisher
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Compositionality
A multi-word “A B” is compositional if meaning(A B) = meaning(A) + meaning(B) e.g. Water Tank e.g. Post Man Non-Compositional Think Tank Smoking Gun Apple Polisher
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Problem Definition
Given: Large Web text of a language English: 15 billion word corpora German: 1 billion word corpora Telugu: 10 million word corpora Goal: Identify compositional and non-compositional multi-words. My focus is on compound nouns A sequence of nouns is treated as a multi-word
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Problem Definition
Given: Large Web text of a language English: 15 billion word corpora German: 1 billion word corpora Telugu: 10 million word corpora Goal: Identify compositional and non-compositional multi-words. My focus is on compound nouns A sequence of nouns is treated as a multi-word
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Importance
Dictionary Building A good dictionary does not miss non-compositional multi-words Machine Translation Non-compositional words should be treated as a single word goose egg 6= Gänseei goose egg → unwichtig Word Tokenization Search engines
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Background: Foundations of Semantics
Distributional Hypothesis (Harris, 1954) Words that occur in similar contexts tend to have similar meanings e.g. Tree and Plant, Tea and Coffee, Bus and Vehicle Bag of words hypothesis: Two documents tend to be similar if they have same distribution of similar words You shall know a word by the company it keeps (Firth, 1957)
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Vector Space Models (VSMs) of Semantics
Interpret semantics using VSM Backbone: Distributional Hypothesis
Text entity (we are interested in) as a Vector (point) in dimensional space. Existing methods representation term-document term-context
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Term-Document: (Salton et al., 1975)
1
d1: Human machine interface for Lab ABC computer applications
1
Image courtesy: (Landauer et al., 1998)
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Term-Document: (Salton et al., 1975)
2
Document similarity can be found using Cosine similarity D1.D2 sim(D1, D2) = kD1 kkD2k
2
Image courtesy: (Salton et al., 1975)
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Term-Document: (Salton et al., 1975)
2
Document similarity can be found using Cosine similarity D1.D2 sim(D1, D2) = kD1 kkD2k
2
Image courtesy: (Salton et al., 1975)
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Term-Context: Word Space Model
Words are represented as a vector build from context words I rent a house. I bought an apartment. I booked a room. Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Previous Approaches: Lin (1999)
Water Tank substituting thesaurus entries of water Aqua Tank Liquid Tank Petrol Tank Think Tank Cognition Tank Fails Drink Tank 15.7 % accuracy reported
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Previous Approaches: Lin (1999)
Water Tank substituting thesaurus entries of water Aqua Tank Liquid Tank Petrol Tank Think Tank Cognition Tank Fails Drink Tank 15.7 % accuracy reported
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Baldwin et al. (2003) Build distributional vectors of WaterTank ThinkTank Tank Measure sim(WaterTank, Tank) and sim(ThinkTank, Tank) if sim > thrsh: multi-word is compositional else: multi-word is non-compositional Pitfalls Was able to capture type-of relations only Negative examples: agony aunt
Skewed nature of senses Threshold highly varies River Bank is not similar to Bank
Moderate results: 51 % accuracy Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Katz and Giesbrecht (2006)
Build distributional vectors of WaterTank ThinkTank Water, Think, Tank
Compositional Vector: WaterTank_comp = Water + Tank sim (WaterTank, WaterTank_comp) if sim > thrsh: multi-word is compositional else: multi-word is non-compositional
Pitfalls: Threshold highly varies 48 % accuracy
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
All my observations are based on UKWaC corpus (UK Web as Corpus) Traffic: 102847 examples Light: 248029 examples TrafficLight: 1070 examples sim(TrafficLight, Traffic)= 0.577 sim(TrafficLight, Light)= 0.282 sim(TrafficLight, TrafficLight_comp)= 0.579
Context Sensitive Vectors TrafficLight : 42160 examples LightTraffic : 73544 example sim(TrafficLight, TrafficLight LightTraffic _comp)= 0.705
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
All my observations are based on UKWaC corpus (UK Web as Corpus) Traffic: 102847 examples Light: 248029 examples TrafficLight: 1070 examples sim(TrafficLight, Traffic)= 0.577 sim(TrafficLight, Light)= 0.282 sim(TrafficLight, TrafficLight_comp)= 0.579
Context Sensitive Vectors TrafficLight : 42160 examples LightTraffic : 73544 example sim(TrafficLight, TrafficLight LightTraffic _comp)= 0.705
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
All my observations are based on UKWaC corpus (UK Web as Corpus) Traffic: 102847 examples Light: 248029 examples TrafficLight: 1070 examples sim(TrafficLight, Traffic)= 0.577 sim(TrafficLight, Light)= 0.282 sim(TrafficLight, TrafficLight_comp)= 0.579
Context Sensitive Vectors TrafficLight : 42160 examples LightTraffic : 73544 example sim(TrafficLight, TrafficLight LightTraffic _comp)= 0.705
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
All my observations are based on UKWaC corpus (UK Web as Corpus) Traffic: 102847 examples Light: 248029 examples TrafficLight: 1070 examples sim(TrafficLight, Traffic)= 0.577 sim(TrafficLight, Light)= 0.282 sim(TrafficLight, TrafficLight_comp)= 0.579
Context Sensitive Vectors TrafficLight : 42160 examples LightTraffic : 73544 example sim(TrafficLight, TrafficLight LightTraffic _comp)= 0.705
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
Threshold is relative High: 495187 examples Jump: 27899 examples HighJump: 1158 sim(HighJump, High)= 0.120 sim(HighJump, Jump)= 0.333 sim(HighJump, HighJump_comp)= 0.333 Is HighJump compositional?
Context Sensitive Vectors HighJump : 84935 examples JumpHigh : 3093 examples sim(HighJump, HighJump JumpHigh _comp)= 0.481
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
Threshold is relative High: 495187 examples Jump: 27899 examples HighJump: 1158 sim(HighJump, High)= 0.120 sim(HighJump, Jump)= 0.333 sim(HighJump, HighJump_comp)= 0.333 Is HighJump compositional?
Context Sensitive Vectors HighJump : 84935 examples JumpHigh : 3093 examples sim(HighJump, HighJump JumpHigh _comp)= 0.481
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
My Observations
Threshold is relative High: 495187 examples Jump: 27899 examples HighJump: 1158 sim(HighJump, High)= 0.120 sim(HighJump, Jump)= 0.333 sim(HighJump, HighJump_comp)= 0.333 Is HighJump compositional?
Context Sensitive Vectors HighJump : 84935 examples JumpHigh : 3093 examples sim(HighJump, HighJump JumpHigh _comp)= 0.481
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Summary
Compositionality detection is tough Vector Space Model for Compositionality Detection Role of context Sensitive vectors Relative thresholds
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Suggestions/Questions? Thank You
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Bibliography I Baldwin, T., Bannard, C., Tanaka, T., and Widdows, D. (2003). An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment - Volume 18, pages 89–96, Morristown, NJ, USA. Association for Computational Linguistics. Firth, J. R. (1957). A synopsis of linguistic theory 1930-55. 1952-59:1–32. Harris, Z. (1954). Distributional structure. Word, 10(23):146–162. Katz, G. and Giesbrecht, E. (2006). Automatic identification of non-compositional multi-word expressions using latent semantic analysis. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties, MWE ’06, pages 12–19, Morristown, NJ, USA. Association for Computational Linguistics. Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
Multi-word
Compositionality
Problem Definition
Background
Previous Approaches
My Observations
Summary
References
Bibliography II
Landauer, T. K., Foltz, P. W., and Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25:259–284. Lin, D. (1999). Automatic identification of non-compositional phrases. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, ACL ’99, pages 317–324, Stroudsburg, PA, USA. Association for Computational Linguistics. Salton, G., Wong, A., and Yang, C. S. (1975). A vector space model for automatic indexing. Commun. ACM, 18:613–620.
Siva Reddy (UoY)
Think Tank vs Water Tank: Compositionality Detection using Vector Space Model
