Explore
For Enterprise
Join for Free
Log In

Arts et humanités
Business
Informatique
Science des données
Technologies de l'information
santé
Mathématiques et logique
Développement personnel
Sciences physiques et ingénierie
Sciences sociales
Apprentissage des langues
Diplômes et Certificats

Explorez l'intégralité de Coursera

Parcourir
Rechercher
For Enterprise
Connexion
Inscrivez-vous gratuitement

Trading Off Precision and Recall

Loading...

Apprentissage automatique

Université de Stanford

4.9 (104,909 notes) | 2.3M étudiants inscrits

Visualiser le programme de cours

Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it. Many researchers also think it is the best way to make progress towards human-level AI. In this class, you will learn about the most effective machine learning techniques, and gain practice implementing them and getting them to work for yourself. More importantly, you'll learn about not only the theoretical underpinnings of learning, but also gain the practical know-how needed to quickly and powerfully apply these techniques to new problems. Finally, you'll learn about some of Silicon Valley's best practices in innovation as it pertains to machine learning and AI. This course provides a broad introduction to machine learning, datamining, and statistical pattern recognition. Topics include: (i) Supervised learning (parametric/non-parametric algorithms, support vector machines, kernels, neural networks). (ii) Unsupervised learning (clustering, dimensionality reduction, recommender systems, deep learning). (iii) Best practices in machine learning (bias/variance theory; innovation process in machine learning and AI). The course will also draw from numerous case studies and applications, so that you'll also learn how to apply learning algorithms to building smart robots (perception, control), text understanding (web search, anti-spam), computer vision, medical informatics, audio, database mining, and other areas.

Compétences que vous apprendrez

Logistic Regression, Artificial Neural Network, Machine Learning (ML) Algorithms, Machine Learning

Avis

4.9 (104,909 notes)

5 stars
97,244 ratings
4 stars
7,066 ratings
3 stars
443 ratings
2 stars
74 ratings
1 star
82 ratings

MN

Jun 15, 2016

Excellent starting course on machine learning. Beats any of the so called programming books on ML. Highly recommend this as a starting point for anyone wishing to be a ML programmer or data scientist.

PT

Sep 01, 2018

Sub title should be corrected. Since I'm not that good in English but I know when there're mis-traslated or wrong sub title. If you fix this problems , I thin it helps many students a lot. Thanks!!!!!

À partir de la leçon

Machine Learning System Design

To optimize a machine learning algorithm, you’ll need to first understand where the biggest improvements can be made. In this module, we discuss how to understand the performance of a machine learning system with multiple parts, and also how to deal with skewed data.

Error Metrics for Skewed Classes11:35

Trading Off Precision and Recall14:05

Enseigné par

Andrew Ng
CEO/Founder Landing AI; Co-founder, Coursera; Adjunct Professor, Stanford University; formerly Chief Scientist,Baidu and founding lead of Google Brain

In the last video, we talked about precision and recall

as an evaluation metric for classification problems with skewed constants.

For many applications,

we'll want to somehow control the trade-off between precision and recall.

Let me tell you how to do that and also show you some even more effective ways to

use precision and recall as an evaluation metric for learning algorithms.

As a reminder, here are the definitions of precision and

recall from the previous video.

Let's continue our cancer classification example,

where y equals 1 if the patient has cancer and y equals 0 otherwise.

And let's say we're trained in logistic regression classifier which outputs

probability between 0 and 1.

So, as usual, we're going to predict 1, y equals 1,

if h(x) is greater or equal to 0.5.

And predict 0 if the hypothesis outputs a value less than 0.5.

And this classifier may give us some value for precision and some value for recall.

But now, suppose we want to predict that the patient has cancer

only if we're very confident that they really do.

Because if you go to a patient and you tell them that they have cancer,

it's going to give them a huge shock.

What we give is a seriously bad news, and

they may end up going through a pretty painful treatment process and so on.

And so maybe we want to tell someone that we think they have cancer only if they

are very confident.

One way to do this would be to modify the algorithm, so

that instead of setting this threshold at 0.5, we might instead say

that we will predict that y is equal to 1 only if h(x) is greater or equal to 0.7.

So this is like saying, we'll tell someone they have cancer only if we think

there's a greater than or equal to, 70% chance that they have cancer.

And, if you do this,

then you're predicting someone has cancer only when you're more confident and

so you end up with a classifier that has higher precision.

Because all of the patients that you're going to and saying,

we think you have cancer, although those patients are now ones that you're

pretty confident actually have cancer.

And so a higher fraction of the patients that you predict have

cancer will actually turn out to have cancer because making those predictions

only if we're pretty confident.

But in contrast this classifier will have lower recall because now we're going to

make predictions, we're going to predict y = 1 on a smaller number of patients.

Now, can even take this further.

Instead of setting the threshold at 0.7, we can set this at 0.9.

Now we'll predict y=1 only if we are more than 90% certain that

the patient has cancer.

And so, a large fraction of those patients will turn out to have cancer.

And so this would be a higher precision classifier will have lower recall because

we want to correctly detect that those patients have cancer.

Now consider a different example.

Suppose we want to avoid missing too many actual cases of cancer, so

we want to avoid false negatives.

In particular, if a patient actually has cancer, but

we fail to tell them that they have cancer then that can be really bad.

Because if we tell a patient that they don't have cancer,

then they're not going to go for treatment.

And if it turns out that they have cancer, but we fail to tell them they have cancer,

well, they may not get treated at all.

And so that would be a really bad outcome because

they die because we told them that they don't have cancer.

They fail to get treated, but it turns out they actually have cancer.

So, suppose that, when in doubt, we want to predict that y=1.

So, when in doubt, we want to predict that they have cancer so

that at least they look further into it, and

these can get treated in case they do turn out to have cancer.

In this case, rather than setting higher probability threshold,

we might instead take this value and instead set it to a lower value.

So maybe 0.3 like so, right?

And by doing so, we're saying that, you know what, if we think there's more

than a 30% chance that they have cancer we better be more conservative and

tell them that they may have cancer so that they can seek treatment if necessary.

And in this case what we would have is going to be a higher recall

classifier, because we're going to be correctly

flagging a higher fraction of all of the patients that actually do have cancer.

But we're going to end up with lower precision

because a higher fraction of the patients that we said have cancer,

a high fraction of them will turn out not to have cancer after all.

And by the way, just as a sider, when I talk about this to other students,

I've been told before, it's pretty amazing,

some of my students say, is how I can tell the story both ways.

Why we might want to have higher precision or

higher recall and the story actually seems to work both ways.

But I hope the details of the algorithm is true and

the more general principle is depending on where you want,

whether you want higher precision- lower recall, or higher recall- lower precision.

You can end up predicting y=1 when h(x) is greater than some threshold.

And so in general, for most classifiers there is going

to be a trade off between precision and recall, and

as you vary the value of this threshold that we join here,

you can actually plot out some curve that trades off precision and recall.

Where a value up here, this would correspond to a very high value of

the threshold, maybe threshold equals 0.99.

So that's saying, predict y=1 only if we're more than 99% confident,

at least 99% probability this one.

So that would be a high precision, relatively low recall.

Where as the point down here,

will correspond to a value of the threshold that's much lower,

maybe equal 0.01, meaning, when in doubt at all, predict y=1, and if you do that,

you end up with a much lower precision, higher recall classifier.

And as you vary the threshold, if you want you can actually trace of a curve for your

classifier to see the range of different values you can get for precision recall.

And by the way, the precision-recall curve can look like many different shapes.

Sometimes it will look like this, sometimes it will look like that.

Now there are many different possible shapes for

the precision-recall curve, depending on the details of the classifier.

So, this raises another interesting question which is,

is there a way to choose this threshold automatically?

Or more generally, if we have a few different algorithms or a few different

ideas for algorithms, how do we compare different precision recall numbers?

Concretely, suppose we have three different learning algorithms.

So actually, maybe these are three different learning algorithms, maybe

these are the same algorithm but just with different values for the threshold.

How do we decide which of these algorithms is best?

One of the things we talked about earlier is the importance of a single real number

evaluation metric.

And that is the idea of having a number that just tells you how well is your

classifier doing.

But by switching to the precision recall metric we've actually lost that.

We now have two real numbers.

And so we often,

we end up face the situations like if we trying to compare Algorithm 1 and

Algorithm 2, we end up asking ourselves, is the precision of 0.5 and

a recall of 0.4, was that better or worse than a precision of 0.7 and recall of 0.1?

And, if every time you try out a new algorithm you end up having to sit around

and think, well, maybe 0.5/0.4 is better than 0.7/0.1, or maybe not, I don't know.

If you end up having to sit around and think and

make these decisions, that really slows down your decision making process for

what changes are useful to incorporate into your algorithm.

Whereas in contrast, if we have a single real number evaluation metric

like a number that just tells us is algorithm 1 or is algorithm 2 better,

then that helps us to much more quickly decide which algorithm to go with.

It helps us as well to much more quickly evaluate different

changes that we may be contemplating for an algorithm.

So how can we get a single real number evaluation metric?

One natural thing that you might try is to look at the average precision and recall.

So, using P and R to denote precision and recall, what you could do is just compute

the average and look at what classifier has the highest average value.

But this turns out not to be such a good solution, because similar to the example

we had earlier it turns out that if we have a classifier that

predicts y=1 all the time, then if you do that you can get a very high recall,

but you end up with a very low value of precision.

Conversely, if you have a classifier that predicts y equals zero, almost

all the time, that is that it predicts y=1 very sparingly, this corresponds to

setting a very high threshold using the notation of the previous y.

Then you can actually end up with a very high precision with a very low recall.

So, the two extremes of either a very high threshold or

a very low threshold, neither of that will give a particularly good classifier.

And the way we recognize that is by seeing that we end up with a very low

precision or a very low recall.

And if you just take the average of (P+R)/2 from this example,

the average is actually highest for Algorithm 3, even though you can get that

sort of performance by predicting y=1 all the time and

that's just not a very good classifier, right?

You predict y=1 all the time, just normal useful classifier, but

all it does is prints out y=1.

And so Algorithm 1 or Algorithm 2 would be more useful than Algorithm 3.

But in this example, Algorithm 3 has a higher average value of

precision recall than Algorithms 1 and 2.

So we usually think of this average of precision and

recall as not a particularly good way to evaluate our learning algorithm.

In contrast, there's a different way for combining precision and recall.

This is called the F Score and it uses that formula.

And so in this example, here are the F Scores.

And so we would tell from these F Scores, it looks like Algorithm 1 has the highest

F Score, Algorithm 2 has the second highest, and Algorithm 3 has the lowest.

And so, if we go by the F Score we would pick probably Algorithm 1 over the others.

The F Score, which is also called the F1 Score, is usually written F1 Score that I

have here, but often people will just say F Score, either term is used.

Is a little bit like taking the average of precision and

recall, but it gives the lower value of precision and

recall, whichever it is, it gives it a higher weight.

And so, you see in the numerator here

that the F Score takes a product of precision and recall.

And so if either precision is 0 or recall is equal to 0,

the F Score will be equal to 0.

So in that sense, it kind of combines precision and recall, but for

the F Score to be large, both precision and recall have to be pretty large.

I should say that there are many different possible formulas for

combing precision and recall.

This F Score formula is really maybe a,

just one out of a much larger number of possibilities, but historically or

traditionally this is what people in Machine Learning seem to use.

And the term F Score, it doesn't really mean anything, so

don't worry about why it's called F Score or F1 Score.

But this usually gives you the effect that you want because if either

a precision is zero or recall is zero, this gives you a very low F Score,

and so to have a high F Score, you kind of need a precision or recall to be one.

And concretely, if P=0 or R=0,

then this gives you that the F Score = 0.

Whereas a perfect F Score, so if precision equals one

and recall equals 1, that will give you an F Score,

that's equal to 1 times 1 over 2 times 2, so

the F Score will be equal to 1, if you have perfect precision and perfect recall.

And intermediate values between 0 and

1, this usually gives a reasonable rank ordering of different classifiers.

So in this video,

we talked about the notion of trading off between precision and recall, and

how we can vary the threshold that we use to decide whether to predict y=1 or y=0.

So it's the threshold that says, do we need to be at least 70% confident or

90% confident, or whatever before we predict y=1.

And by varying the threshold, you can control a trade off between precision and

recall.

We also talked about the F Score, which takes precision and recall, and again,

gives you a single real number evaluation metric.

And of course, if your goal is to automatically set that threshold to decide

what's really y=1 and y=0, one pretty reasonable way to do that would

also be to try a range of different values of thresholds.

So you try a range of values of thresholds and evaluate these different thresholds

on, say, your cross-validation set and then to pick whatever value of

threshold gives you the highest F Score on your crossvalidation [INAUDIBLE].

And that be a pretty reasonable way to automatically choose the threshold for

your classifier as well.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play

Coursera
À propos
Direction
Carrières
Catalogue
Certificats
Certificats MasterTrack™
Diplômes
For Enterprise
For Government

Communauté
Learners
Partners
Developers
Beta Testers
Translators

Se connecter
Blog
Facebook
LinkedIn
Twitter
Instagram
YouTube
Blog Tech

Plus
Conditions
Confidentialité
Aide
Accessibilité
Presse
Contact
Répertoire
Filiales