UL. Clustering and Estimation of Equity Correlation Matrix

Loading...

En provenance du cours de New York University Tandon School of Engineering

Fundamentals of Machine Learning in Finance

65 notes

New York University Tandon School of Engineering

Fundamentals of Machine Learning in Finance

65 notes

Cours 2 sur 4 dans Specialization Machine Learning and Reinforcement Learning in Finance

The course aims at helping students to be able to solve practical ML-amenable problems that they may encounter in real life that include: (1) understanding where the problem one faces lands on a general landscape of available ML methods, (2) understanding which particular ML approach(es) would be most appropriate for resolving the problem, and (3) ability to successfully implement a solution, and assess its performance. A learner with some or no previous knowledge of Machine Learning (ML) will get to know main algorithms of Supervised and Unsupervised Learning, and Reinforcement Learning, and will be able to use ML open source Python packages to design, test, and implement ML algorithms in Finance. Fundamentals of Machine Learning in Finance will provide more at-depth view of supervised, unsupervised, and reinforcement learning, and end up in a project on using unsupervised learning for implementing a simple portfolio trading strategy. The course is designed for three categories of students: Practitioners working at financial institutions such as banks, asset management firms or hedge funds Individuals interested in applications of ML for personal day trading Current full-time students pursuing a degree in Finance, Statistics, Computer Science, Mathematics, Physics, Engineering or other related disciplines who want to learn about practical applications of ML in Finance Experience with Python (including numpy, pandas, and IPython/Jupyter notebooks), linear algebra, basic probability theory and basic calculus is necessary to complete assignments in this course.

À partir de la leçon

Data Visualization & Clustering

UL. Clustering Algorithms4:32

UL. K-clustering8:20

UL. K-means Neural Algorithm7:06

UL. Hierarchical Clustering Algorithms10:54

UL. Clustering and Estimation of Equity Correlation Matrix5:40

UL. Minimum Spanning Trees, Kruskal Algorithm6:40

UL. Probabilistic Clustering6:37

Rencontrer les enseignants

Igor Halperin

Now, after we introduced various clustering techniques,

we are ready to apply them to some interesting problems in finance.

And here, I suggest we take a look at one of

the most fundamental problems for many areas from finance.

We are looking to estimation of Assets Correlation Matrices,

and more specifically, Equity Correlation Matrices.

The problem of estimating equity correlations from data

is very important in such areas of finances,

quantitative trading, asset management,

and systematic risk monitoring.

Now, to explain the problem,

let me set the limitation first.

Assume that we have a set of normalized log-returns at site for some set of stocks

with i between one and N. Using these standardized returns,

we can't compute the empirical correlation matrix, C_ij.

The number of free parameters in such matrix will be of the order of half of N squared.

On the other hand, the total number of observations

for N stocks observed over T steps will be N times

T. This implies that for

a reliable estimation of

the true correlation matrix from an empirical correlation matrix,

we need to have T much larger than N. However,

this condition does not always hold in reality.

For example, if your investment universe counts

3,000 stocks and you have 15 years of daily data,

then you are closer to the case when T and N are of the same order and such case

the empirical and the true correlation matrices

may turn out to be quite different from one another.

Therefore, if you want to use

equity correlation matrices for out-of-sample forecasts or portfolio construction,

we better have to somehow clear the empirical correlation matrix of observational noise.

Such procedures are referred to as the de-noising of equity correlation matrices.

Now, different approaches are available for such de-noising of correlation matrices.

One class of methods is called the random matrix theory,

where the spectrum of the empirical correlation matrix is compared with the spectrum of

a random correlation matrix to identify and subtract the noise component in the matrix.

This method was suggested in 1999 by a group of physicists led by Eugene Stanley.

There are also some other methods that try to do the same.

For example, there are Bayesian Methods and

in particular the so-called shrinkage methods,

where the empirical correlation matrix C is adjusted towards

some pre-defined prior matrix C had that has some simple structure,

for example, for us it's with the constant correlation within each industry group.

And finally, there are clustering-based filtering methods.

With this class of methods,

the equity correlation matrix is viewed as a sort of graph,

which we will de-noise aggregating links between sub-clusters.

Let's now formalize this problem a bit.

First, we formalize the clustering problem as follows.

Given the set of N items with their pair-wise distances, d_ij,

we want to divide it into K subsets in

such a way that the minimum distance between points in different clusters is maximized.

Here's how such a maximum minimum distance algorithm can be implemented.

We maintain clusters as a set of connected components of graph.

We then iteratively combine different clusters

containing the two closest items by adding an edge between them.

And we stop this procedure when we get exactly K clusters.

What I described is known as the Kruskal algorithm.

The Kruskal algorithm is an example of

a single-linkage agglomerative clustering that we discussed in the last video.

Now, the Kruskal algorithm operates on

distances while we are only given the equity correlations, rho_ij.

There exists a convenient way to define pair-wise distance d_ij from

pairwise correlation rho_ij as square root of two,

times one minus rho_ij.

This choice is legitimate as long as it satisfies three axioms of an Euclidean metric.

First, it's not negative and equals zero only when i equals j.

Second, it's symmetric with respect to a placement of i by i and vice versa.

Third, it satisfies the triangle inequality for

distances between arbitrary three points i, j,

and k. In their next video,

will look into how the Kruskal algorithm works

and how it can be used to filter equity correlation matrices.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play