Get to the point! Summarization with pointer-generator networks

Loading...

En provenance du cours de National Research University Higher School of Economics

Natural Language Processing

383 notes

National Research University Higher School of Economics

Natural Language Processing

383 notes

Cours 6 sur 7 dans Specialization Advanced Machine Learning

This course covers a wide range of tasks in Natural Language Processing from basic to advanced: sentiment analysis, summarization, dialogue state tracking, to name a few. Upon completing, you will be able to recognize NLP tasks in your day-to-day work, propose approaches, and judge what techniques are likely to work well. The final project is devoted to one of the most hot topics in today’s NLP. You will build your own conversational chat-bot that will assist with search on StackOverflow website. The project will be based on practical assignments of the course, that will give you hands-on experience with such tasks as text classification, named entities recognition, and duplicates detection. Throughout the lectures, we will aim at finding a balance between traditional and deep learning techniques in NLP and cover them in parallel. For example, we will discuss word alignment models in machine translation and see how similar it is to attention mechanism in encoder-decoder neural networks. Core techniques are not treated as black boxes. On the contrary, you will get in-depth understanding of what’s happening inside. To succeed in that, we expect your familiarity with the basics of linear algebra and probability theory, machine learning setup, and deep neural networks. Some materials are based on one-month-old papers and introduce you to the very state-of-the-art in NLP research.

À partir de la leçon

Sequence to sequence tasks

Nearly any task in NLP can be formulates as a sequence to sequence task: machine translation, summarization, question answering, and many more. In this module we will learn a general encoder-decoder-attention architecture that can be used to solve them. We will cover machine translation in more details and you will see how attention technique resembles word alignment task in traditional pipeline.

Sequence to sequence learning: one-size fits all?10:53

Get to the point! Summarization with pointer-generator networks12:36

Rencontrer les enseignants

Anna Potapenko
Researcher
HSE Faculty of Computer Science
Alexey Zobnin
Accosiate professor
HSE Faculty of Computer Science
Anna Kozlova
Team Lead
Yandex
Sergey Yudin
Analyst-developer
Yandex
Andrei Zimovnov
Senior Lecturer
HSE Faculty of Computer Science

Hey, in this video, I'm going to cover one nice paper about summarization.

This is a very recent paper from Chris Manning Group,

and it is nice because it tells us that on the one hand,

we can use encoder-decoder architecture, and it will work somehow.

On the other hand, we can think a little bit and improve a lot.

So, the improvement will be based on pointer networks,

which are also a very useful tool to be aware of.

Also sometimes, we have

rather hand-wavy explanations of the architectures with the pictures.

Sometimes, it is good to go into details and to see some actual formulas.

That's why I want to be very precise in this video,

and in the end of this video,

you will be able to understand all the details of the architecture.

So, this is just a recap,

first of all that we have usually some encoder,

for example bidirectional LSTM and then we have some attention mechanism,

which means that we produce some probabilities that tells us

what are the most important moments in our input sentence.

Now, you see there is some arrow on the right of the slide.

Do you have any idea what does this arrow means?

Where does it comes from?

Well, the attention mechanism is about the important moments of

the encoder based on the current moment of the decoder.

So, now we definitely have the yellow part which is decoder,

and then the current state of this decoder tells us how to compute attention.

Just to have the complete scheme,

we can say that we use this attention mechanism to

generate our distribution or vocabulary.

Awesome. So, this is just a recap of encoder-decoder attention architecture.

Let us see how it works.

So, we have some sentences,

and we try to get a summary.

So, the summary would be like that.

First, we see some UNK tokens because the vocabulary is not big enough.

Then, we also have some problems in this paragraph that we will try to improve.

One problem is that the model is abstractive,

so the model generates a lot,

but it doesn't know that sometimes,

it will be better just to copy something from the input.

So, the next architecture will tell us how to do it.

Let us have a closer look into the formulas and then see how we can improve the model.

So, first, attention distribution.

Do you remember notation?

Do you remember what is H and what is S?

Well, H is the encoder states and S is the decoder states.

So, we use both of them to compute the attention weights,

and we apply softmax to get probabilities.

Then, we use these probabilities to weigh encoder states and get v_j.

v_j is the context vector specific for the position j over the decoder.

Then how do we use it?

We have seen in some other videos that we can use it

to compute the next state of the decoder.

In this model, we will go in a little bit more simple way.

Our decoder will be just normal RNN model,

but we will take the state of this RNN model s_j and

concatenate with v_j and use it to produce the probabilities of the outcomes.

So, we just concatenate them, apply some transformations,

and do softmax to get the probabilities of the words in our vocabulary.

Now, how can we improve our model?

We would want to have some copy distribution.

So, this distribution should tell us that

sometimes it is nice just to copy something from the input.

How can we do this?

Well, we have attention distribution that already

have the probabilities of different moments in the input.

What if we just sum them by the words?

So, for example, we have seen as two times in our input sequence.

Let us say the probability of as should be equal to the sum of those two.

And in this way,

we'll get some distribution over words that occurred in our input.

Now, the final thing to do will be just to have a mixture of those two distributions.

So, one is this copy distribution that tells that some words from the input are good,

and another distribution is our generative model that we have discussed before.

So just a little bit more formulas.

How do we weigh these two distributions?

We weigh them with some probability p generation here,

which is also sum function.

So every thing which is in green on this slide is some parameters.

So, you just learn these parameters and you learn to

produce this probability to weigh two kinds of distributions.

And this weighting coefficient depends on everything that you have,

on the context vector v_j,

on the decoder state s_j,

on the current inputs to the decoder.

So you just apply transformations

to everything that you have and then sigmoid to get probability.

The training objective for our model would be,

as usual, cross-entropy loss with this final distribution.

So, we will try to predict those words that we need to predict.

This is similar to likelihood maximization,

and we will need to optimize the subjective.

Now, this is just the whole architecture, just once again.

We have encoder with attention,

we have yellow decoder,

and then we have two kinds of distributions that we

weigh together and get the final distribution on top.

Let us see how it works.

This is called pointer-generation model because it has two pieces,

generative model and pointer network.

So this part about copying

some phrases from the input would be called pointer network here.

Now, you see that we are good,

so we can learn to extract some pieces from the text,

but there is one drawback here.

So you see that the model repeats some sentences or some pieces of sentences.

We need one more trick here,

and the trick will be called coverage mechanism.

Remember you have attention probabilities.

You know how much attention you give to every distinct piece of the input.

Now, let us just accumulate it.

So at every step,

we are going to sum all those attention distributions to some coverage vector,

and this coverage vector will know that certain pieces

have been attended already many times.

How do you compute the attention then?

Well, to compute attention,

you would also need to take into account the coverage vector.

So the only difference here is that you have one more term there,

the coverage vector multiplied by some parameters,

green as usual, and this is not enough.

So you also need to put it to the loss.

Apart from the loss that you had before,

you will have one more term for the loss.

It will be called coverage loss and the idea is to

minimize the minimum of the attention probabilities and the coverage vector.

Take a moment to understand that.

So imagine you want to attend some moment that has been already attended a lot,

then this minimum will be high and you will want to minimize it.

And that's why you will have to have small attention probability at this moment.

On the opposite, if you have some moment with low coverage value,

then you are safe to try to have

high attention weight here because the minimum will be still the low coverage value,

so the loss will not be high.

So this loss motivates you to attend those places that haven't been attended a lot yet.

Let us see whether the model works nice and whether

the coverage trick helps us to avoid repetitions.

We can compute the ratio of duplicates in our produced outcomes,

and also we can compute the same ratio for human reference summaries,

and you can see that it is okay to duplicate unigrams,

but it is not okay to duplicate sentences

because the green level there is really low, it is zero.

So the model before coverage, the red one,

didn't know that and it duplicated a lot of three-grams and four-grams and sentences.

The blue one doesn't duplicate that,

and this is really nice.

However, we have another problem here.

The summary becomes really extractive,

which means that we do not generate new sentences,

we just extract them from our input.

Again, we can try to compare what we have with reference summaries.

Let us compute the ratio of those n-grams that are novel.

And you can see that for the reference summaries,

you have rather high bars for all of them.

So, the model with coverage mechanism has

sufficiently lower levels than the model without the coverage mechanism.

So in this case, our coverage spoils a model a little bit.

And again for the real example,

this is the summary generated by pointer-generator network plus coverage,

and actually let us see.

Somebody says he plans to something.

And here in the original text,

we see exactly the same sentences but they are somehow linked.

So, we just link them with he says that and so on.

Otherwise, it is just extractive model that extracts these three important sentences.

Now, I want to show you quantitative comparison of different approaches.

ROUGE score is an automatic measure for summarization.

You can think about it as something as BLEU,

but for summarization instead of machine translation.

Now, you can see that pointer-generator networks

perform better than vanilla seq2seq plus attention,

and coverage mechanism improves the system even more.

However, all those models are not that good if we compare them to some baselines.

One very competitive baseline would be just to take first three sentences over the text.

But it is very simple and extractive baseline,

so there is no idea how to improve it.

I mean, this is just something that you get out of this very straightforward approach.

On the contrary, for those models for attention and coverage,

there are some ideas how to improve them even more,

so in future everybody hopes that neural systems will be able to improve on that,

and it is absolutely obvious that in a few years,

we will be able to beat those baselines.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play