Variational EM & Review

Loading...

En provenance du cours de National Research University Higher School of Economics

Bayesian Methods for Machine Learning

196 notes

National Research University Higher School of Economics

Bayesian Methods for Machine Learning

196 notes

Cours 3 sur 7 dans Specialization Advanced Machine Learning

Bayesian methods are used in lots of fields: from game development to drug discovery. They give superpowers to many machine learning algorithms: handling missing data, extracting much more information from small datasets. Bayesian methods also allow us to estimate uncertainty in predictions, which is a really desirable feature for fields like medicine. When Bayesian methods are applied to deep learning, it turns out that they allow you to compress your models 100 folds, and automatically tune hyperparametrs, saving your time and money. In six weeks we will discuss the basics of Bayesian methods: from how to define a probabilistic model to how to make predictions from it. We will see how one can fully automate this workflow and how to speed it up using some advanced techniques. We will also see applications of Bayesian methods to deep learning and how to generate new images with it. We will see how new drugs that cure severe diseases be found with Bayesian methods.

À partir de la leçon

Variational Inference & Latent Dirichlet Allocation

This week we will move on to approximate inference methods. We will see why we care about approximating distributions and see variational inference — one of the most powerful methods for this task. We will also see mean-field approximation in details. And apply it to text-mining algorithm called Latent Dirichlet Allocation

Why approximate inference?5:15

Mean field approximation13:59

Example: Ising model15:39

Variational EM & Review5:52

Rencontrer les enseignants

Daniil Polykovskiy
Researcher
HSE Faculty of Computer Science
Alexander Novikov
Researcher
HSE Faculty of Computer Science

All right, in this video,

we will see a variational Bayes EM,

and also summarize methods that we have seen until now.

Let me remind you what Expectation Maximization Algorithm means.

We try to maximize the margin of likelihood,

the logarithm of it actually.

It would be a logarithm of P of your data, given the parameters.

We also derive the variation lower bound for it,

and it is actually an expectation over some distribution Q of T times the logarithm

of the ratio between the joint distribution over the latent variable set of data,

given the parameters, and this distribution Q of T,

that is distribution over the lesson variables.

And we try to maximize the variation lower bound,

we do this in an iterative way.

On E-step, we maximize the lower bond with respect to Q and,

on abstract, we'll maximize it with respect to Theta.

We also proved that on E-step,

the maximization of the lower bound with respect to Q is equivalent to

minimizing the curl divergence between the Q and the posterior on the learning variables,

given the parameters and the data.

And on the M-step,

we can maximize the expected value of the logarithm of the joint distribution.

Actually, we do this since the denominator of the variational lower bound,

the Q of T, does not depend on Theta, and so we can drop it.

Let's see an E-step more carefully.

We minimize the curl divergence between the variational distribution Q of T,

and the posterior distribution over the latent variables,

given data and parameters,

and we do this minimization with respect to Q.

In week two, we've shown that the minimum of

this curl divergence actually is the second distribution,

that is, the Q of T equals to probability of T given X and Theta.

However, for many models,

it is not the case that we can compute for posterior exactly.

For example, in the next model,

module C and model called Latent Dirichlet allocation.

And for it, it seems to be impossible to compute the full posterior exactly.

So, we can try to use a variational inference in this case.

The new E-step would be as follows.

We minimize the curl divergence in some variational family.

We can use a meaningful approximation, for example, here.

And this method, where on E-step,

we perform a variational inference is called a variational EM.

All right, we've seen a lot of models.

Now, let's summarize them.

Here's our notation again.

We have data that is known we denoted as X,

we have parameters, Theta,

that are unknown and also,

latent variable is T that are also unknown.

I have two criterions here.

First is accurate and inaccurate.

The methods that are higher are more

accurate and the methods that are lower are more inaccurate.

In the meantime, on the right,

we have another criterion,

the slow and the fast.

The first methods would be quite slow,

and the last ones would be really fast.

You have to compromise between accuracy and speed.

The first algorithm is a full inference,

we'll try to find the full distribution,

the full joint distribution over the latent variables and parameters, given the data.

It is actually very accurate since it is exact inference,

however, for minimums, it is really slow.

Sometimes, we can do Mean field.

For mean fields, we will find

the posterior distribution approximated by the product of two distributions,

the distribution over the latent variables and distribution over parameters.

We can use EM algorithm.

In EM algorithm, we'll find only a point wise estimation of the parameters.

We will still have a distribution over T, however,

the Theta would be equal to the maximum of posterior estimation of it.

All right, but it turns out that for many cases,

it is still not the case that we can use the EM algorithm.

And then we can use a variational EM.

In variational EM, we apply, for example,

a meaningful approximation and factorize the probability over

the latent variables into probabilities for each dimension.

And so, we'll find the factorized distribution of latent variables,

and a point estimate of the parameters.

All right, and finally,

if we can do anything, we can use a Crisp EM.

In Crisp EM, we approximate the latent variables and the parameters with point estimate.

We'll do in an interesting way,

we'll find the maximum probability value for the lesser variables and on the next step,

we'll estimate the maximum probability of estimate all of the parameters.

You should be familiar with such methods,

it is used for k-means clusterization.

So, we've seen the methods,

we've seen the variational inference,

and in the next module,

well see an application of variational inference for the Latent Dirichlet allocation.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play