Linear Regression: Ridge, Lasso, and Polynomial Regression

Loading...

En provenance du cours de University of Michigan

Applied Machine Learning in Python

3220 notes

University of Michigan

Applied Machine Learning in Python

3220 notes

Cours 3 sur 5 dans Specialization Applied Data Science with Python

This course will introduce the learner to applied machine learning, focusing more on the techniques and methods than on the statistics behind these methods. The course will start with a discussion of how machine learning is different than descriptive statistics, and introduce the scikit learn toolkit through a tutorial. The issue of dimensionality of data will be discussed, and the task of clustering data, as well as evaluating those clusters, will be tackled. Supervised approaches for creating predictive models will be described, and learners will be able to apply the scikit learn predictive modelling methods while understanding process issues related to data generalizability (e.g. cross validation, overfitting). The course will end with a look at more advanced techniques, such as building ensembles, and practical limitations of predictive models. By the end of this course, students will be able to identify the difference between a supervised (classification) and unsupervised (clustering) technique, identify which technique they need to apply for a particular dataset and need, engineer features to meet that need, and write python code to carry out an analysis. This course should be taken after Introduction to Data Science in Python and Applied Plotting, Charting & Data Representation in Python and before Applied Text Mining in Python and Applied Social Analysis in Python.

À partir de la leçon

Module 2: Supervised Machine Learning - Part 1

This module delves into a wider variety of supervised learning methods for both classification and regression, learning about the connection between model complexity and generalization performance, the importance of proper feature scaling, and how to control model complexity by applying techniques like regularization to avoid overfitting. In addition to k-nearest neighbors, this week covers linear regression (least-squares, ridge, lasso, and polynomial regression), logistic regression, support vector machines, the use of cross-validation for model evaluation, and decision trees.

Introduction to Supervised Machine Learning17:09

Overfitting and Underfitting12:22

Supervised Learning: Datasets4:58

K-Nearest Neighbors: Classification and Regression13:26

Linear Regression: Least-Squares17:37

Linear Regression: Ridge, Lasso, and Polynomial Regression19:09

Logistic Regression12:49

Linear Classifiers: Support Vector Machines13:43

Multi-Class Classification6:50

Kernelized Support Vector Machines18:53

Cross-Validation9:06

Decision Trees19:40

Rencontrer les enseignants

Kevyn Collins-Thompson
Associate Professor
School of Information

Now we'll look at another way to estimate WNB for linear model, called Ridge Regression.

Ridge regression uses the same least-squares criterion,

but with one difference.

During the training phase,

it adds a penalty for feature weights,

the WI values that are too large as shown in the equation here.

You'll see that large weights means mathematically that

the sum of their squared values is large.

Once ridge regression has estimated the WNB parameters for the linear model,

the prediction of Y values for new instances is exactly the same as in least squares.

You just plug in your input feature values,

the XIs and compute the sum of

the weighted feature values plus B with the usual in your formula.

So why would something like ridge regression be useful?

This addition of a penalty term to

a learning algorithm's objective function is called Regularisation.

Regularisation is an extremely important concept in machine learning.

It's a way to prevent overfitting, and thus,

improve the likely generalization performance of a model,

by restricting the models possible parameter settings.

Usually the effect of this restriction from regularisation,

is to reduce the complexity of the final estimated model.

So how does this work with linear regression?

The addition of the sum of squared parameter values that's shown in the box,

to the least-squares objective means that models with

larger feature weights (w) add more to the objective functions overall value.

Because our goal is to minimize the overall objective function,

the regularisation term acts as a penalty of

models with lots of large feature weight values.

In other words, all things being equal,

if ridge regression finds

two possible linear models that predict the training data values equally well,

it will prefer the linear model that has

a smaller overall sum of squared feature weights.

The practical effect of using ridge regression,

is to find the feature weights WI that fit the data well in at least square sense,

and that set lots of the feature weights two values that are very small.

We don't see this effect with a single variable linear regression example,

but for regression problems with dozens or hundreds of features,

the accuracy improvement from using

regularized linear regression like ridge regression could be significant.

The amount of regularisation to apply is controlled by the alpha parameter.

Larger alpha means more regularization and

simpler linear models with weights closer to zero.

The default setting for alpha is 1.0.

Notice that setting alpha to zero corresponds to the special case of

ordinary least-squares linear regression that we saw earlier,

that minimizes the total square here.

In scikit learn, you use rich regression by importing

the ridge class from sklearn.linear model.

And then use that estimate or object just as you would for least-squares.

The one difference is that you can specify

the amount of the ridge regression regularisation penalty,

which is called the L2 penalty,

using the alpha parameter.

Here, we're applying ridge regression to the crime data set.

Now, you'll notice here that the results are not that impressive.

The R-squared score on the test set is pretty

comparable to what we got for least-squares regression.

However there is something we can do in

applying ridge regression that will improve the results dramatically.

So now is the time for a brief digression about the need for

feature preprocessing and normalization.

Let's stop and think for a moment intuitively,

what ridge regression is doing.

It's regularizing the linear regression by imposing

that sum of squares penalty on the size of the W coefficients.

So the effect of increasing alpha is to

shrink the AW coefficients toward zero and towards each other.

But if the input variables, the features,

have very different scales,

then when this shrinkage happens of the coefficients,

input variables with different scales will have

different contributions to this L2 penalty,

because the L2 penalty is a sum of squares of all the coefficients.

So transforming the input features,

so they're all on the same scale,

means the ridge penalty is in some sense applied more fairly to all features

without unduly weighting some more than others,

just because of the difference in scales.

So more generally, you'll see as we proceed through the course that feature

normalization is important to perform for a number of different learning algorithms,

beyond just regularized regression.

This includes K-nearest neighbors,

support vector machines, neural networks and others.

The type of feature preprocessing and

normalization that's needed can also depend on the data.

For now, we're going to apply a widely used form of

future normalization called MinMax Scaling,

that will transform all the input variables,

so they're all on the same scale between zero and one.

To do this, we compute

the minimum and maximum values for each feature on the training data,

and then apply the minmax transformation for each feature as shown here.

Here's an example of how it works with two features.

Suppose we have one feature "height" whose values fall in

a fairly narrow range between 1.5 and 2.5 units.

But a second feature,

"width" has a much wider range between five and 10 units.

After applying minmax scaling,

values for both features are transformed because they are on the same scale,

with the minimum value getting mapped to zero,

and the maximum value being transformed to one.

And everything else getting transformed to a value between those two extremes.

To apply minmax scaling,

in scikit-learn, you import the minmax scalar object from sklearn.preprocessing.

To prepare the scalar object for use, you create it,

and then call the fit method using the training data Xtrain.

This will compute the min and max feature values

for each feature in this training dataset.

Then to apply the scalar,

you call it transform method,

and pass in the data you want to rescale.

The output will be the scale version of the input data.

In this case, we want to scale the training data and save it

in a new variable called Xtrain scaled.

And the test data,

saving that into a new variable called X-Tests-Scaled.

Then, we just use

these scaled versions of the feature data instead of the original feature data.

Note that it could be more efficient to perform

fitting and transforming in a single step on the training set,

by using the scalers fit transform method as shown here.

There's one last but very important point here,

about how to apply minmax scaling or any kind of

feature normalization in a learning scenario with training and test sets.

You may have noticed two things here.

First, that we're applying the same scalar object to both the training and the testing.

And second, that we're training

the scalar object on the training data and not on the test data.

These are both critical aspects to feature normalization.

If you don't apply the same scaling to training and test sets,

you'll end up with more or less random data skew,

which will invalidate your results.

If you prepare the scaler or other normalization method by

showing it the test data instead of the training data,

this leads to a phenomenon called Data Leakage,

where the training phase has information that is leaked from the test set.

For example, like the distribution of extreme values for each feature in the test data,

which the learner should never have access to during training.

This in turn can cause the learning method to give

unrealistically good estimates on the same test set.

We'll look more at the phenomenon of data leakage later in the course.

One downside to performing feature normalization is that

the resulting model and the transformed features may be harder to interpret.

Again, in the end,

the type of feature normalization that's best to apply,

can depend on the data set,

learning task and learning algorithm to be used.

We'll continue to touch on this issue throughout the course.

Okay, let's return to

ridge regression after we've added the code for minmax scaling of the input features.

We can see the significant effect of

minmax scaling on the performance of ridge regression.

After the input features have been properly scaled,

ridge regression achieves significantly better model fit

with an R-squared value on the test set of about 0.6.

Much better than without scaling,

and much better now than ordinary least-squares.

In fact if you apply the same minmax scaling with ordinary least-squares regression,

you should find that it doesn't change the outcome at all.

In general, regularisation works especially

well when you have relatively small amounts of

training data compared to the number of features in your model.

Regularisation becomes less important as the amount of training data you have increases.

We can see the effect of varying the amount of

regularisation on the scale to training and

test data using different settings for alpha in this example.

The best R-squared value on the test set is achieved with an alpha setting of around 20.

Significantly larger or smaller values of alpha,

both lead to significantly worse model fit.

This is another illustration of the general relationship between

model complexity and test set performance that we saw earlier in this lecture.

Where there's often an intermediate best value of a model of

complexity parameter that does not lead to either under or overfitting.

Another kind of regularized regression that you

could use instead of ridge regression is called Lasso Regression.

Like ridge regression, lasso regression adds

a regularisation penalty term to the ordinary least-squares objective,

that causes the model W-coefficients to shrink towards zero.

Lasso regression uses a slightly different regularisation term called an L1 penalty,

instead of ridge regression's L2 penalty as shown here.

The L1 penalty looks kind of similar to the L2 penalty,

in that it computes a sum over the coefficients but it's

some of the absolute values of the W-coefficients instead of a sum of squares.

And the results are noticeably different.

With lasso regression, a subset of the coefficients are forced to be precisely zero.

Which is a kind of automatic feature selection,

since with the weight of zero the features are

essentially ignored completely in the model.

This sparse solution where only a subset of

the most important features are left with non-zero weights,

also makes the model easier to interpret.

In cases where there are more than a few input variables.

Like ridge regression, the amount of regularisation for

the lasso regression is controlled by the parameter alpha,

which by default is zero.

Also like ridge regression,

the purpose of using lasso regression is to estimate the WNB model coefficients.

Once that's done, the prediction model formula is the same as for ordinary least-squares,

you just use the linear model.

In general, lasso regression is most helpful if you think there are

only a few variables that have a medium or large effect on the output variable.

Otherwise if there are lots of variables that contribute small or medium effects,

ridge regression is typically the better choice.

Let's take a look at lasso regression in scikit-learn using the notebook,

using our communities in crime regression data set.

To use lasso regression,

you import the lasso class from sklearn.linear model,

and then just use it as you would use an estimator like ridge regression.

With some data sets you may occasionally get a convergence warning,

in which case you can set the max_iter attribute to a larger value.

So typically at least 20,000, or possibly more.

Increasing the max inter-parameter will increase the computation time accordingly.

In this example, we're applying lasso to

a minmax scale version of the crime data set as we did for ridge regression.

You can see that for Alpha set to 2.0,

only 20 features with non-zero weights remain because with lasso regularisation,

most of the features are set to have weights of exactly zero.

I've listed the features with non-zero weights in

order of their descending magnitude from the output.

Although we need to be careful in interpreting

any results for data on a complex problem like crime,

the lasso regression results do help us see some of

the strongest relationships between

the input variables and outcomes for this particular data set.

For example, looking at the top five features with

non-zero weight that are found by lasso regression,

we can see that location factors like percentage of people in dense housing,

which indicates urban areas and socio economic variables like

the fraction of vacant houses in an area are positively correlated with crime.

And other variables like the percentage of families with

two parents is negatively correlated.

Finally, we can see the effect of tuning

the regularisation parameter alpha for lasso regression.

Like we saw with ridge regression,

there's an optimal range for alpha that gives

the best test set performance that neither under or over fits.

Of course this best alpha value will be different for different data sets,

and depends on various other factors such as the feature

preprocessing methods being used.

Let's suppose for a moment that we had a set of

two-dimensional data points with features X0 and X1.

Then we could transform each data point by adding additional features that

were the three unique multiplicative combinations of X0 and X1.

So, X0 squared, X0,

X1 and X1 squared.

So we've transformed our original two-dimensional points into a set of

five-dimensional points that rely only on the information in the two-dimensional points.

Now we can write a new regression problem that tries to predict

the same output variable y-hat but using these five features instead of two.

The critical insight here is that this is still a linear regression problem.

The features are just numbers within a weighted sum.

So we can use the same least-squares techniques to estimate

the five model coefficients for these five features that we

used in these simpler two-dimensional case.

Now, why would we want to do this kind of transformation?

Well, this is called polynomial future transformation that

we can use to transform a problem into a higher dimensional regression space.

And in effect, adding these extra polynomial features allows us a

much richer set of complex functions that we can use to fit to the data.

So you can think of this intuitively as allowing

polynomials to be fit to the training data instead of simply a straight line,

but using the same least-squares criterion that minimizes mean squared error.

We'll see later that this approach of adding

new features like polynomial features is also very effective with classification.

And we'll look at this kind of transformation

again in kernelized support vector machines.

When we add these new polynomial features,

we're essentially adding to the model's ability to capture interactions

between the different variables by adding them as features to the linear model.

For example, it may be that housing prices vary as

a quadratic function of both the lat size that a house sits on,

and the amount of taxes paid on the property as a theoretical example.

A simple linear model could not capture this nonlinear relationship,

but by adding nonlinear features like polynomials to the linear regression model,

we can capture this nonlinearity.

Or generally, we can use other types of

nonlinear feature transformations beyond just polynomials.

This is beyond the scope of this course but technically these are

called nonlinear basis functions for regression,

and are widely used.

Of course, one side effect of adding lots of new features

especially when we're taking every possible combination of K variables,

is that these more complex models have the potential for overfitting.

So in practice, polynomial regression is

often done with a regularized learning method like ridge regression.

Here's an example of polynomial regression using scikit-learn.

There's already a handy class called polynomial features in the

sklearn.preprocessing module that will generate these polynomial features for us.

This example shows three regressions on a more complex regression dataset,

that happens to have some quadratic interactions between variables.

The first regression here,

just uses least-squares regression without the polynomial feature transformation.

The second regression creates the polynomial features object with degrees set to two,

and then calls the fit transform method of

the polynomial features object on the original XF1 features,

to produce the new polynomial transform features XF1 poly.

The code then calls ordinary least-squares linear regression.

You can see indications of overfitting on this expanded feature representation,

as the models r-squared score on the training set is

close to one but much lower on the test set.

So the third regression shows the effect of adding

regularisation via ridge regression on this expanded feature set.

Now, the training and tests r-squared scores are basically the same,

with the test set score of

the regularized polynomial regression

performing the best of all three regression methods.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play