Probabilistic clustering

Loading...

En provenance du cours de National Research University Higher School of Economics

Bayesian Methods for Machine Learning

194 notes

National Research University Higher School of Economics

Bayesian Methods for Machine Learning

194 notes

Cours 3 sur 7 dans Specialization Advanced Machine Learning

Bayesian methods are used in lots of fields: from game development to drug discovery. They give superpowers to many machine learning algorithms: handling missing data, extracting much more information from small datasets. Bayesian methods also allow us to estimate uncertainty in predictions, which is a really desirable feature for fields like medicine. When Bayesian methods are applied to deep learning, it turns out that they allow you to compress your models 100 folds, and automatically tune hyperparametrs, saving your time and money. In six weeks we will discuss the basics of Bayesian methods: from how to define a probabilistic model to how to make predictions from it. We will see how one can fully automate this workflow and how to speed it up using some advanced techniques. We will also see applications of Bayesian methods to deep learning and how to generate new images with it. We will see how new drugs that cure severe diseases be found with Bayesian methods.

À partir de la leçon

Expectation-Maximization algorithm

This week we will about the central topic in probabilistic modeling: the Latent Variable Models and how to train them, namely the Expectation Maximization algorithm. We will see models for clustering and dimensionality reduction where Expectation Maximization algorithm can be applied as is. In the following weeks, we will spend weeks 3, 4, and 5 discussing numerous extensions to this algorithm to make it work for more complicated models and scale to large datasets.

Latent Variable Models11:32

Probabilistic clustering6:32

Gaussian Mixture Model10:13

Training GMM10:41

Example of GMM training10:35

Rencontrer les enseignants

Daniil Polykovskiy
Researcher
HSE Faculty of Computer Science
Alexander Novikov
Researcher
HSE Faculty of Computer Science

[MUSIC]

Discuss a latent variable model for clustering.

So what is clustering?

Imagine that you own a bank, and you have a bunch of customers,

and each of them has some income and some debt.

And you want from this data, so

you can represent each of your customers on two dimensional plane as a point.

And from this data,

you want to decompose your customers into three different clusters.

Why?

Well, for example, you want to find people who spend money on cars and

make some promotions for them, some car related loan or something.

This can be useful for different retail companies and banks, and

companies like that.

So find meaningful subset of customers to work with.

And this is a non supervised problem, so we don't have any labels,

we just have raw data, raw axis.

Usually clustering is done in a hard way, so for

each data point we assign its color.

This data point is orange, so it belongs to the orange cluster and

this one is blue.

Sometimes, people do soft clustering.

So instead of assigning each data point a particular cluster,

we will assign each data point from probability distributions over clusters.

So for example, the orange points on the top of this picture are certainly orange.

And they have a probability distribution like almost 100%

to belong to their orange cluster and almost 0 to belong to the rest.

But the points on the border between orange and blue, they kind of not settled.

They have for example, 40% probability to belong to the blue cluster, and

60% probability to belong to the orange cluster, and 0 to the green.

And we don't know which cluster at this point actually belong to.

So instead of just assigning each data point a particular cluster, we assume that

each data point belongs to every cluster, but with some different probabilities.

And to build a clustering methods with this property,

we will treat everything probabilistically.

Why can we want that?

Well, there are several reasons.

First of all, we may want to again handle missing data naturally.

And another reason, that may want to consider clustering

in probabilistic way, is to tune hyperparamters.

So usually, then you want to tune hyperparameters,you do a plot like this.

You consider a bunch of different values, for

example, for the hyperparameter number of clusters.

So on the previous image, we had free clusters, but

we can try some different amount like 5 or 4.

And for each of these particular values of number of

clusters we may train our clustering model.

Which is called GMM and we'll discuss it later in details.

So we're going to plot the training performance here like on the blue line and

here I'm plotting the log likelihood, so the higher the better.

And we can see that whenever we increase the number of clusters,

the actually performance in the training set improves.

Which kind of the usual thing with hyperparameters.

The more clusters you have, the model thinks its better, but it's actually not.

So, for example, if you put one cluster per each data point, the model loss

will be optimal, but it's not a meaningful solution to the problem at all.

So if you consider the validation performance of your model,

then it increases, then you start to increase a number of clusters,

then it stagnates somehow and then it starts to decrease.

And this is the usual picture for tuning hyperparameters.

You tune a bunch of models and

you chose the one that performs to the best in the validation set.

So this was probability small for clustering, but

it turns out that you can do this thing for hard assignment clustering.

Well, at least it's not always how to do it.

So if you train one of the popular hard clustering algorithmic tenets,

it will think that the more clusters you have, the better.

Both in training and on validation loss.

So it doesn't have any meaningful way to understand which number

of clusters do we want from the performance on the validation set.

The probabilistic way of dealing with clustering is also not ideal.

So for example here, we're not sure whether we want 20 clusters or

60 or 80, but it gives at least something, you have some boundaries.

What is the reasonable variable from this hyperparmeter?

So this was the first reason why we may

want to consider probabilistic approach to clustering.

And the second one is that we may want to build a generative model of our data.

So if we treat everything probabilistically,

we may sample new data points from our model of the data.

And in the case of customers, it will mean simple new points on the traditional grid.

It look like the points we used to have in the training set.

And if you're points are, for example, images of celebrity faces,

then sampling new images from the same probability distribution

means generating fake celebrities and their images from scratch.

And this is kind of a fun application of building probabilistic model for data.

So to summarize, we will want to build a probabilistic model for clustering.

And this may help us in two ways.

First of all, it may allow us to tune hyper parameters.

And it may give us a generative model of the data.

So in the next video, we'll build a latent variable model for clustering.

[MUSIC]

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play