Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based...
Transcript of Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based...
![Page 1: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/1.jpg)
Recursive Autoencoders for ITG-based TranslationPeng Li Tsinghua University [email protected]
(Joint work with Yang Liu and Maosong Sun)
![Page 2: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/2.jpg)
• Phrase reordering model is a critical problem in machine translation (MT), and is NP-complete
Overview
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
���2(Knight, 1999)
![Page 3: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/3.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
![Page 4: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/4.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
d=0
![Page 5: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/5.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
d=0
+5
![Page 6: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/6.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
d=0 d=5
+5
![Page 7: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/7.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
d=0 d=5
+5-7
![Page 8: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/8.jpg)
• Distortion models: penalize relative displacement of source phrases
Distortion Models
���3
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Koehn et al., 2003; Och and Ney, 2004)
d=0 d=5 d=7
+5-7
![Page 9: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/9.jpg)
Lexicalized Reordering Models
• Lexicalized reordering models: penalize reordering conditioned on both the source and target phrases
���4(Koehn et al., 2007)
与 沙⻰龙 举⾏行 了 会谈
Bush hold a talk with Sharon
布什
![Page 10: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/10.jpg)
Lexicalized Reordering Models
• Lexicalized reordering models: penalize reordering conditioned on both the source and target phrases
���4(Koehn et al., 2007)
与 沙⻰龙 举⾏行 了 会谈
Bush hold a talk with Sharon
布什
M
![Page 11: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/11.jpg)
Lexicalized Reordering Models
• Lexicalized reordering models: penalize reordering conditioned on both the source and target phrases
���4(Koehn et al., 2007)
与 沙⻰龙 举⾏行 了 会谈
Bush hold a talk with Sharon
布什
M D
![Page 12: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/12.jpg)
Lexicalized Reordering Models
• Lexicalized reordering models: penalize reordering conditioned on both the source and target phrases
���4(Koehn et al., 2007)
与 沙⻰龙 举⾏行 了 会谈
Bush hold a talk with Sharon
布什
M D S
![Page 13: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/13.jpg)
Lexicalized Reordering Models
• Lexicalized reordering models: penalize reordering conditioned on both the source and target phrases
���5
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
M D D M D
(Koehn et al., 2007)
![Page 14: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/14.jpg)
Block Merging
• Reordering as block merging
���6
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 15: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/15.jpg)
Block Merging
• Reordering as block merging
���6
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
straight
(Wu, 1997; Xiong et al., 2006)
![Page 16: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/16.jpg)
Block Merging
• Reordering as block merging
���7
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 17: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/17.jpg)
Block Merging
• Reordering as block merging
���7
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
invert
(Wu, 1997; Xiong et al., 2006)
![Page 18: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/18.jpg)
Block Merging
• Reordering as block merging
���8
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 19: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/19.jpg)
Block Merging
• Reordering as block merging
���8
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
straight
(Wu, 1997; Xiong et al., 2006)
![Page 20: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/20.jpg)
Block Merging
• Reordering as block merging
���9
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 21: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/21.jpg)
Block Merging
• Reordering as block merging
���9
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
straight
(Wu, 1997; Xiong et al., 2006)
![Page 22: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/22.jpg)
Block Merging
• Reordering as block merging
���10
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 23: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/23.jpg)
Block Merging
���11
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 24: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/24.jpg)
Block Merging
���11
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 25: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/25.jpg)
Block Merging
���11
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 26: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/26.jpg)
Block Merging
���11
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 27: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/27.jpg)
Block Merging
���11
我 有 ⼀一个 从 没有 ⻅见 过 的 ⼥女性 朋友 。
I have a female friend never seen before .
(Wu, 1997; Xiong et al., 2006)
![Page 28: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/28.jpg)
Block Merging
• Can you find a counter example?
���12(Huang et al., 2009)
![Page 29: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/29.jpg)
Block Merging
• Can you find a counter example?
���12(Huang et al., 2009)
进⼀一步 就 中东 危机 会谈
hold further talk on the Mideast crisis...
... 举⾏行
![Page 30: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/30.jpg)
Block Merging
���13
“inside-outside”
(Wu, 1997)
![Page 31: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/31.jpg)
ITG
• Inversion transduction grammar (ITG)
���14
X ! [X1, X2]
X ! hX1, X2iX ! f/e
: straight rule
: inverted rule
: lexical rules
(Wu, 1997)
![Page 32: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/32.jpg)
ITG-based Reordering Model
• Type 1: Incorporating ITG into left-to-right decoding to constrain the reordering space (e.g., Zens et al., 2004; Feng et al., 2010)
• Type II: Translation as ITG parsing, e.g.
• Max-Ent ITG reordering model: using maximum entropy (MaxEnt) model to predict which rule to use (Xiong et al., 2006)
���15
![Page 33: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/33.jpg)
MaxEnt ITG Reordering Model
Potentially alleviates the data sparseness problem
How to extract features from training examples?
• Which words are representative for predicting reordering?
• Xiong et al. (2006) only use boundary words
���16
![Page 34: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/34.jpg)
This Work
• We propose an ITG reordering classifier based on recursive autoencoders (RAE)
• Our model considers the whole phrases
• RAEs can produce vector space representations for arbitrary strings
• Our system achieves1.07 BLEU points improvement on NIST 2008 dataset
���17
![Page 35: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/35.jpg)
Neural ITG Reordering Model
���18
“never seen before” v.s. “seen before never”
![Page 36: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/36.jpg)
Neural ITG Reordering Model
���19
RAE
Real-valued vector
![Page 37: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/37.jpg)
Neural ITG Reordering Model
���20
straight inverted
Softmax layer
![Page 38: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/38.jpg)
Translation
���21
![Page 39: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/39.jpg)
Translation
���21
![Page 40: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/40.jpg)
Translation
���21
![Page 41: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/41.jpg)
Translation
���21
![Page 42: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/42.jpg)
Translation
���21
![Page 43: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/43.jpg)
Translation
���21
![Page 44: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/44.jpg)
Translation
���22
![Page 45: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/45.jpg)
Translation
���22
![Page 46: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/46.jpg)
Translation
���22
![Page 47: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/47.jpg)
Translation
���22
![Page 48: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/48.jpg)
Autoencoders
• Each word is represented as a vector, e.g.
• “female” ➤ [0.1 0.8 0.4]T
• “friend” ➤ [0.7 0.1 0.5]T
• What is the vector representation of “female friend”?
���23
![Page 49: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/49.jpg)
• Encoding
!
• Decoding
!
• What about multi-word strings?
Autoencoders
���24
p = f (1)(W (1)[c1; c2] + b(1))
[c01; c02] = f (2)(W (2)p+ b(2))
![Page 50: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/50.jpg)
Recursive Autoencoders
���25(Socher et. al, 2011)
![Page 51: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/51.jpg)
Training
���26
Reconstruction error: how well the learned vector space representations represent the corresponding strings?
Reordering error: how well the classifier predicts
the merging order?
![Page 52: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/52.jpg)
Reconstruction Error
• Reconstruction error
!
• Source side average reconstruction error
!
• Total reconstruction error
���27
Erec([c1; c2]; ✓) =1
2||[c1; c2]� [c01; c
02]||2
Erec,s(S; ✓) =1
Ns
X
i
X
p2T ✓R(ti,s)
Erec([p.c1, p.c2]; ✓)
Erec(S; ✓) = Erec,s(S; ✓) + Erec,t(S; ✓)
![Page 53: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/53.jpg)
Reordering Error
• Average cross-entropy error
!
• Joint training objective
���28
E
reo
(S; ✓) =1
|S|X
i
�X
o
d
ti(o) · log(P✓
(o|ti
))
!
J = ↵Erec
(S; ✓) + (1� ↵)Ereo
(S; ✓) +R(✓)
R(✓) =�L
2||✓
L
� ✓L0 ||2 +
�rec
2||✓
rec
||2 + �reo
2||✓
reo
||2
![Page 54: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/54.jpg)
Optimization
• Hyper-parameters optimization
•
• Optimized by random search (Bergstra and Bengio, 2012)
• Training objective optimization: L-BFGS
• Using backpropagation through structures to compute gradients (Goller and Kuchler, 1996)
���29
↵,�L
,�rec
,�reo
![Page 55: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/55.jpg)
Experiments
• Training corpus: 1.23M sentence pairs
• Language model: 4-gram language model trained on the Xinhua portion of the GIGAWORD corpus
• Dev. set: NIST 2006 MT dataset
• Test set: NIST 2008 MT dataset
• Metric: case-insensitive BLEU-4 score
���30
![Page 56: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/56.jpg)
BLEU-4
���31
System NIST06 (dev) NIST08 (tst)
maxent 30.40 23.75
neural 31.61* 24.82*
*: significantly better (p < 0.01)
![Page 57: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/57.jpg)
BLEU-4
���32
Sentence!Length > = <
[1, 10] 43 121 57[11, 20] 181 67 164[21, 30] 170 11 152[31, 40] 105 3 90[41, 50] 69 1 53[51, 119] 40 0 30
![Page 58: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/58.jpg)
Classification Accuracy
���33
![Page 59: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/59.jpg)
Conclusion
• We have presented an ITG reordering classifier based on RAEs
• Feature work
• Combine linguistically-motivated labels with recursive neural networks
• Investigate more efficient decoding algorithms
• Apply our method to other phrase-based and even syntax-based systems
���34
![Page 60: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/60.jpg)
Reference• Yang Feng, Haitao Mi, Yang Liu, and Qun Liu. 2010. An efficient
shift-reduce decoding algorithm for phrased- based machine translation. In Proceedings of COLING 2010: Posters, pp. 285–293.
• Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of IJCNN 1996, pp. 347–352.
• Liang Huang, Hao Zhang, Daniel Gildea, Kevin Knight. Binarization of synchronous context-free grammars. Computational Linguistics, 35(4), pp. 559–595.
• Kevin Knight. 1999. Decoding complexity in word- replacement translation models. Computational Linguistics, 25(4):607–615.
���35
![Page 61: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/61.jpg)
Reference• Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch,
Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of ACL 2007, pp. 177–180.
• Philipp Koehn, Franz Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT-NAACL 2003, pp. 48–54.
• Franz Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449.
���36
![Page 62: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/62.jpg)
Reference• Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and
Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of EMNLP 2011, pp. 151–161.
• Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377–403.
• Deyi Xiong, Qun Liu, and Shouxun Lin. 2006. Maximum entropy based phrase reordering model for statistical machine translation. In Proceedings of COLING/ACL 2006, pp. 521–528.
• Richard Zens, Hermann Ney, Taro Watanabe, and Eiichiro Sumita. 2004. Reordering constraints for phrase-based statistical machine translation. In Proceedings of COLING 2004, pp. 205–211.
���37
![Page 63: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/63.jpg)
Thanks!
![Page 64: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/64.jpg)
Backup Slides
![Page 65: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/65.jpg)
Training Data Size
# of examples NIST06 (dev) NIST08 (tst)
100,000 30.88 23.78200,000 30.75 23.89300,000 30.80 24.35400,000 31.01 24.45
6,004,441 31.61 24.82
![Page 66: Recursive Autoencoders for ITG-based Translation · Recursive Autoencoders for ITG-based Translation Peng Li Tsinghua University! pengli09@gmail.com (Joint work with Yang Liu and](https://reader034.fdocument.pub/reader034/viewer/2022042513/5f7315321b5a6779b1008f3b/html5/thumbnails/66.jpg)
Cluster Examples
Cluster 1 Cluster 2 Cluster 3
works for verify on tunnels from transparency in opinion at
these people who the reasons why the story of how the system which the trend towards
of the three on the fundamental over the entire through its own with the best