Bagging

Loading...

En provenance du cours de Johns Hopkins University

Practical Machine Learning

2029 notes

Johns Hopkins University

Practical Machine Learning

2029 notes

Cours 8 sur 10 dans Specialization Data Science

One of the most common tasks performed by data scientists and data analysts are prediction and machine learning. This course will cover the basic components of building and applying prediction functions with an emphasis on practical applications. The course will provide basic grounding in concepts such as training and tests sets, overfitting, and error rates. The course will also introduce a range of model based and algorithmic machine learning methods including regression, classification trees, Naive Bayes, and random forests. The course will cover the complete process of building prediction functions including data collection, feature creation, algorithms, and evaluation.

À partir de la leçon

Week 3: Predicting with trees, Random Forests, & Model Based Predictions

This week we introduce a number of machine learning algorithms you can use to complete your course project.

Predicting with trees12:51

Random Forests6:49

Model Based Prediction11:39

Rencontrer les enseignants

Jeff Leek, PhD
Associate Professor, Biostatistics
Bloomberg School of Public Health
Roger D. Peng, PhD
Associate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhD
Professor, Biostatistics
Bloomberg School of Public Health

This lecture is about bagging, which is short for bootstrap aggregating.

The basic idea is that when you fit complicated models,

sometimes if you average those models together, you get a

smoother model fit, that gives you a better balance between

potential bias in your fit and variance in your fit.

So bootstrap aggregating has a very simple idea.

The basic idea is take your data and take resamples of the data set.

So, this is the similar to the idea of bootstrapping, which you would have

learned about in the inference class that

is part of the data science specialization.

After you resample the cases with replacement, then

you recalculate your prediction function on that resampled data.

And then you either average the

predictions from all these repeated predictors that

you built or you majority vote or

something like that when you're doing classification.

The thing is that you get a similar bias that you would get from fitting any one

of those models individually, but a reduced variability

because you've averaged a bunch of different predictors together.

This is most useful for non-linear functions.

So, we'll show an example with smoothing, but it's

also very useful for things like predicting with trees.

So I'm going to go back to the ozone data, so it's in the ElemStatLearn package.

And I load the the ozone data set.

I then order for the purposes of showing you how this works,

I'm going to order the data set by the outcome, the ozone variable here.

And then I look at the data set and I

can see it has four variables, ozone, radiation, temperature, and wind.

So the idea is that I'm going to try

to predict temperature as a function of ozone.

[BLANK_AUDIO]

So the first thing that we can do is just show you an example of how this works.

So the basic idea is, I'm going to create a matrix

here and it's going to have 10 rows and a 155 columns.

Then what I'm going to do is, I'm going to resample the data set.

In for ten different times, so then a

loop over ten different samples of the data set.

Each time I'm going to sample width replacement from the entire data set.

Then I'm going to create a new data set, ozone0, which is

the resample data set for that particular element of the loop.

And that's just the subset of the data set corresponding to our random sample.

Then I'm going to reorder the data set every time by

the ozone variable, and you'll see why in just a minute.

Then I fit a loess curve each time, so a loess is

kind of a smooth curve that you can fit through the data.

It's very similar to the sublime model fits that we

saw in a previous example with modeling with linear regression.

And so the basic idea is we're fitting

a smooth curve relating temperature, to the ozone variables.

So temperature is the outcome, and ozone is the predictor, and each time I

use the resample data set as the data set I'm building that predictor on.

And I use a common span for each time, the

span being a measure of how smooth that fit will be.

I then predict for every single loess curve the outcome

for a new data set for the exact same values.

I always predict for ozone values 1 to 155.

So the ith row of this ll object is now the prediction from

the loess curve, from the ith resample of the date ozone.

So what I've done here?

I've resampled my data set ten different times, fit a smooth curve through it those

ten different times, and then what I'm

going to do is I'm going to average those values.

So, here's what it looks like in this plot.

So, here I've plotted ozone on the x

axis, these are the observed ozone values versus temperature

on the y axis, those are the observed

temperature values, and each black dot represents an observation.

Each gray line here represents the fit with one resampled data set.

So you can see the gray lines that have a lot of curviness to them.

They capture a lot of the variability in the data set.

But they also maybe over-capture some of the variability.

They're little bit too curvy.

Once I've averaged those lines together I get something that's a little

bit smoother and is closer to the middle of the data set.

That's the red line.

So the red line is the bagged loess curve.

It's basically the average of multiple fitted loess curves,

the same data set where I've resampled it every time.

There's a proof that shows that the bagging estimate will always have

lower variability but similar bias to the individual model fits that you do.

In the caret package there's some models that already perform bagging for you.

So if you're using the train function you

could set method to be bagEarth, treebag, or bagFDA.

And those are specific bagged models that the the

model that the caret package will fit for you.

Alternatively, you can actually build your own bagging function in caret.

This is a bit of an advanced use and so I recommend that you

read the documentation carefully if you're going to be trying to do that yourself.

The idea here though is you basically are going to

take your predictor variable and put it into one data frame.

So I'm going to make the predictors be a data frame that contains the ozone data.

Then you have your outcome variable.

Here's it's going to be just a temperature variable from the data set.

And I pass this to the bag function in caret package.

So I tell it, I want to use the predictors

from that data frame, this is my outcome, this

is the number of replications with the number of

sub samples I'd like to take from the data set.

And then bagControl tells me something about how I'm going to fit the model.

So fit is the function that's going to be applied to fit the model every time.

This could be a call to the train function in the caret package.

Predict is a the way that given a particular

model fit, that we'll be able to predict new values.

So this could be, for example, a call to the predict function from a trained model.

And then aggregate is the way that we'll put the var, the predictions together.

So for example it could average the

predictions across all the different replicated samples.

You can see that if you look at this

custom bag version of the conditional regression trees, you can

see that it gets some of the benefit that I

was showing you in the previous slide with bag loess.

So the idea here is I'm plotting ozone

again on the x-axis versus temperature on the y-axis.

The little grey dots represent actual observed values.

The red dots represent the fit from a single conditional regression tree.

And so you can see that for example, it capture, it doesn't capture the

trend that's going on down here very well, the red line is just flat.

Even though there appears to be a trend upward in the data points here.

But when I average over ten different bagged

model model fits with these conditional regression trees.

I see that there's an increase here in the values in

the blue fit, which is the fit from the bagged regression.

So we're going to look a little bit

at those different parts of the bagging function.

So in this particular case I'm using the ctreeBag function, which you

can look at in, if you've loaded the caret package in R.

So, for the fit part it takes the data frame

that we've passed and the predict, and the outcome that

we've passed, and it basically uses the ctree function to

train a tree, conditional regression tree on the data set.

This is the last command that's called the ctree command.

So it returns this model fit from the ctree function.

The prediction takes in the object.

So this is going to be an object from the ctree model fit.

And a new data set x, and it's going to get a new prediction.

So what you can see here is it basically calculates each time

the tree response or the outcome from the object and the new data.

It then calculates this probability matrix and

returns either the actually the observed levels that

it predicts or it actually re, just returns

the response, the predicted response from the variable.

The aggregation then takes those values and averages

them together or puts them together in some way.

So here what this is doing is

it's basically getting the prediction from every

single one of these model fits, so that' s across a large number of observations.

And then it binds them together into one data matrix by with

each row being equal to the prediction from one of the model predictions.

And then it takes the median at every value.

So in other words it takes the median prediction from

each of the different model fits across all the bootstrap samples.

So bagging is very useful for nonlinear models, and it's widely used.

It's often used with trees.

And you can think of an extension to this as

being random forest, which we'll talk about in a future lecture.

Several models use bagging and caret's main train

function, like I told you about in previous slide.

And you can also build your own specific bagging functions, for any

classification or prediction algorithm that you'd like to take a look at.

For further resources, I've linked to a couple

of different tutorials on bagging and boosting, as

well as the Elements of Statistical Learning which

has a lot more details about how bagging works.

But remember that the basic idea is to basically resample

your data, refit your nonlinear model, then average those model

fits together over resamples to get a smoother model fit,

than you would've got from any individual fit on its own

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play