AttSum: Joint Learning of Focusing and Summarization

with Neural AttentionZiqiang Cao, Wenjie Li, Sujian Li, Furu Wei and Yanran Li

Coling 2016

発表者：小平　知範

1

Abstract• Task: Extractive query-focused summarization

- query relevance ranking and sentence saliency ranking

• Main contributions:- They apply the attention mechanism that tries to simulate human attentive reading behavior for query-focused summarization.- They propose a joint neural network model to learn query relevance ranking and sentence saliency ranking simultaneously

2

Query-Focused Sentence Ranking

CNN Layer

Pooling Layer

Ranking Layer

3

1. CNN Layer3. Query-Focused Sentence Ranking

4

v(wi: wi+j) = concatenation

Convolution Layer Max-over-time pooling

f (●) = non-linear function

Wth ∈ Rl x hk h = window size k = embeding size cth ∈ Rl

2. Pooling Layer

• Query relevance:

• The document Embedding:

3. Query-Focused Sentence Ranking

5

M ∈ Rl x l

3. Ranking Layer

• rank a sentence according to cosine similarity

• Cosine Similarity:

3. Query-Focused Sentence Ranking

6

ex. Training Process

• Cost Function:

• s+: High ROUGE score, s-: rest

• Ω is margin threthold

3. Query-Focused Sentence Ranking

7

Sentence Selection• 1. discard sentences less than 8 words

• 2. sort descending order

• 3. They iteratively dequeue the top-ranked sentence, and append it to the current summary if it is non-redundant.

• non-redundant: new bi-gram ratio > .5

8

Dataset• DUC 2005 ~ 2007, query-focused multi-document

summarization task.

• Preprocessing: StanfordCoreNLP (ssplit, tokenize)

• Summary: the length limit of 250 words

• Validation: 3-fold cross-validation

4. Experiments

9

Model Setting

• Word Embedding: (50 dimention, trained on News corpus)- don’t update word embeddings in the training process

• word window size h = 2

• Convolution output l = 50

• margin Ω = 0.5

10

4. Experiments

Evaluation Metrics

• ROUGE-2

11

4. Experiments

Baselines• LEAD

• QUERY_SIM

• MultiMR (Wan and iao, 2009)

• SVR (Ouyang et al., 2011)

• DocEmb (Kobayashi et al., 2015)

• ISOLATION: AttSum w/o attention mechanism

12

4. Experiments

Summarization Performance

13

4. Experiments

Conclusion

• Propose a novel query-focuesed summarization system called AttSum, which jointly handles saliency ranking and relevance ranking.

14