In this module we're going to discuss statistical biases in performance

evaluation. The goal here is not to build realistic

models but rather stylized models, so simpler models, but models that

nonetheless will help explain to us that biases do arise, and that biases can be

significant. When we are evaluating the performance of

fund managers. Let's get started.

Some fund managers claim to have special skills.

What do I mean by special skills? Well for example they might claim to be

very good at picking stocks or they might claim to be very good at timing the

market. And what do I mean by timing the market?

I mean the following. Maybe they know a good time to enter into

the market, in other words when to invest in the S&P 500 or the Euro stocks, or

whatever. And also they might claim that they know a

good time when to get out of the market, when is a good time to sell.

So, that's what timing the market means. This skill is often referred to

generically as alpha. And the term comes from the capital asset

pricing model. If you recall the cap end that you'ld seen

in the previous module. We know that the expected return on an

asset is equal to the risk free rate plus beta times the expected return on the

market minus the risk free rate. I've added in here this additional term

here alpha. So this can be viewed in the context of

the capital asset pricing model as the excess returner, the skill if you like,

that a manager can earn on an investment. I don't want you to think that alpha is

necessarily associated with account fund. It's a generic term used to evaluate

skill, whereby a manager can earn excess. Risk adjusted returns.

Why is this important? Well, a fund manager who has alpha, or who

has skill, can often charge substantial management fees.

And so what we want to know, is whether or not the skill is real.

In general, it's very hard to tell, but you can still do some interesting

analysis, and that's what we're going to do in this module.

We're actually going to look at this question from three different

perspectives. The first two perspectives, will require

the binomial distribution, and so we're going to review a little bit about the

binomial distribution now. If you recall, we say that x is a binomial

distribution. Or we write x tilde Bin n p, if the

probability that x equals r equals n choose r, times p to the r, times one

minus p to the n minus r. And so x might represent for example, the

number of heads in n independent coin tosses, where p is equal to the

probability of a head. The mean and variance of the binomial

distribution are given by these quantities here.

The expected value of x is n times p, and the variance of x is n p times 1 minus p.

You can also see from one that the probability that x is greater than or

equal to r is equal to the sum of the probabilities that x equals r, x equals r

plus 1 up to x equals n. And that's given to us by this summation

here on the right hand side. So, to see our first perspective on this

problem, consider the following situation. Suppose a fund manager has a track record

of ten years and that this fund manager has outperformed the market in nine of the

ten years. The fund manager claims to have great

skill and that his fees should reflect this.

What we want to know is, well how can we assess his claims?

Does he really have great skill? So first analysis might assume the

following, in a given year he outperforms the probability p and under performs with

probability 1 minus p. And by the way, when I say outperform, I

mean outperform relative to the risk the manager is taking on.

So I'm talking about risk adjusted returns here.

Outperformance or underperformance is assumed to be independent across years.

If the manager has skill, then p is greater than a half.

Otherwise p is less than or equal to a half, and the manager has no skill.

So the first question that now comes to mind is the following.

How likely is such a track record if the fund manager had no skill?

Well to answer this, let x be the number of outperforming years.

If the fund manager has no skill, then x is going to be binomial, with n equals 10,

and p equals a half. We said that the fund manager outperformed

in 9 of the 10 years. So, a track record as good as this would

corresponds to x being equal to 9 or indeed x equals 10.

So we want to compute the probability that x is greater than or equal to 9.

Assuming the manager had no skill. In other words, assuming p is equal to a

half. That is given to us by our binomial

probability we saw in the previous slide. We can evaluate this easily in Excel or

some other piece of software, and we find the answer of 0.0107.

So therefore, if the fund manager has no skill, then the probability of having a

track record as good as his, or better, is only 0.0107.

So at this point, you might think it's fair to conclude, that the fund manager

does indeed have skill. After all, you could think of this in a

statistic setting, for those of you who are familiar with the statistics, you'd be

aware of the concept of a p value. And usually a p value that's less than or

equal to 0.05 would be assumed to be significant.

So we've got a p value here if you like, of 0.0107, this seems significant.

It seems unlikely in this case that the fund manager is no skill.

Given the track record of 9 successful years out of 10.

So that's the first perspective we're going to take.

Here is a second perspective, suppose instead that there are M fund managers and

that the manager who claims to have skill has the best track record of these

managers. The questions that now arises is, does

this change anything in our analysis? Should it change something in our

analysis? To answer this suppose we start with the

hypothesis that none of the fund managers have skill and that track records of fund

managers are independent. There are two possible questions now that

arise. The first question is, how likely is the

third manager to have such a track record if all fund managers have no skill.

Well, in this case, if we've identified the third manager in advance, then the

first perspective gives us the answer. It would actually.

Actually be 0.0107, as we calculated beforehand.

The second question or possible question to, to ask, is how likely is the best

manager to have such a track record if all fund managers have no skill?

And now, what is the appropriate question here?

Which of these two questions is more appropriate?

This actually depends on our prior hypothesis.

What do I mean by that? Well let's come back to this slide here.

We've actually identified this manager here, this is the third manager.

And this is the best performing manager of the M managers.

And so what I mean by prior hypothesis is the following.

Did we identify this third manager in advance?

Maybe this third manager was our friend, or a cousin, or somebody else we specified

in advance before we saw any data. If that is the case, then this question is

the more appropriate question. We've seen the manager in advance, we're

interested in his performance, not because he was the best performing manager, but

because he was the third manager or some manager we've seen in advance.

However, if we're interested in the third manager because he was the best performing

manager, and that's why his track record has come to our attention, then the

appropriate question is actually the second one.

How likely is the best manager to have such a record, if all fund managers have

no skill? And this is because the reason we're

interested in the third manager is not because he was number three in the list.

But because he was the best performing out of all m managers.

And that's why he came to our attention. So, it really depends on our hypothesis.

Why are focusing on this manager? Is it because he was the third manager,

we'd pre-selected him in advance? Or is it because he was the best manager,

and that's why the manager's track record has come to our attention.

Depending on which hypothesis is correct, we get a very different answer.

With the first hypothesis, we see it's 0.0107.

With the second hypothesis, for our interest in the manager is because he's

the best performing manager out of M, the second hypothesis is the more appropriate.

So let's answer this second question. We're going to assume none of the managers

have skill. We're going to let z i be the event that

the ith manager out-performs in r years or more out of the n years.

We're going to let v be the event that the best manager outperforms in r years that

event, or more. Then what we're interested in is the

probability of v. The probability of v is equal to 1 minus

the probability of zed 1 bar up to zed m bar.

Now what is zed i bar? Zed i bar is the complement of zed i.

So zed i is the event that the ith manager performs in r years out of n, so zed i bar

is the event that the ith manager outperforms.

In less than r years out of n, so it's the complement of zed i.

So therefore just to make sense of this, if the best manager is to outperform in r

years, the event that the best manager doesn't outperform in r years and more.

Is equivalent to all managers. All m of them outperforming in less than r

years out of n. And that's this quantity over here.

Because we're assumption that these z, zed i's are independent, and IID.

I can write this as just the probability of zed 1 bar.

Times the probability of zed q bar, up to the probability of zed m bar.

Just keeping these separate. And these are all IIDs, so this is,

there's m of them. And so I get the probability of zed one

bar to the power of m. Now, recall, the probability of zed 1 bar.

Zed 1 bar is the event that's the compliment of zed 1.

So therefore, the probability of zed 1 bar is equal to 1 minus the probability of zed

1. And therefore, I can plug in the numbers.

I'm assuming, in this case, I believe, that m is equal to 20.

And I know that the probability of zed 1 is 0.0107.

Because I actually calculated this. Onto our first perspective.

So this is for a fixed single fund manager.

The probability of zed 1, we found to be 0.0107.

So we need the probability of zed 1 bar, here, which is 1 minus that.

Take it to the power of m, we get this number here, 0.1942.

And actually you can easily see what happens.

As m gets bigger, as we increase m, the number of fund managers in the

marketplace. Place then actually probability of v will

actually increase to one as m gets very large.

What are our conclusions. Well our conclusions are as follows.

If there are a lot of fund managers in the market, then the, the fact that one

particular fund manager has a fantastic record, does not, in and of itself

constitute evidence that, that fund manager has skill.

After all, we have shown here, that with just 20 fund managers we would expect the

best manager to have a record of outperforming in 9 years.

Or more out of 10 to recover probability 0.1942.

And that's not a very small number. If we take m equal to 30 or 40, this

number will get bigger. And so even though we've assumed all the

fund managers have no skill here. The fact that we're focusing on the best

manager's track record. Suggests that the best track record, will

be good. So we haven't established evidence to

suggest that a particular fund manager has skill just because he's got a great track

record. It all depends on the hypothesis.

How did we identify this manager? I should also point out that while we have

focused on the best manager here, we could also have focused on the second best

manager or the third best. Manager and so on.

If n gets sufficiently large, we're not going to just expect the best manager to

do well. The second best manager will also have a

very good track record. The third best manager will have a very

good track record. And so on.

As long as m gets sufficiently large. And this is true even if all of the fund

managers have no skill. Here's another perspective.

This is our third perspective. Suppose again that all fun, fund managers

have no skill. At the end of every year, fund managers

who have out performed the market that year, survive.

And fund managers who have under performed in the market that year, get fired.

Here's a question. After one year of this experiment, what

would be the average track record of fund managers in the market?

Well. It's got to be perfect.

Only the fund managers that have a good year survive, so they will have a perfect

track record. The fund managers that have an imperfect

track record, in other words they under performed that first year, well they've

been fired, so they're no longer available, they're no longer in the

marketplace so we won't see them. And so their performance does not enter

into our calculation of the average track record.

And indeed you can look at this after 2 years, or 3 years you'll only see perfect

fund managers in the market. So only perfect fund managers survive, and

they appear to have a perfect track record.

And yet it's clear that in fact, that these fund managers have no skill.

You could also generalize this in fact, as follows: you could imagine that fund

managers who've outperformed in a given year will stay in the market with some

probability p. And that fund managers who've

underperformed in the market will be fired with a different probability.

And we can assume that fund managers who've under-performed get fired more

frequently than fires, than managers who've over performed.

You could do that type of experiment, you could simulate that kind of system and

what you'll still see is that the average track record of fund managers in the

market would actually be greater than a half.

So actually they will appear to be a level of skill in the market, even though that

skill isn't really there. This is an example of what is called

survivorship bias. The reason being that some people have

survived, they're the people who have outperformed in the market, and they

actually make the market look better. They look like the, the collection of fund

managers look better than they really are. They've just survived because the poor and

unlucky fund managers have actually Under performed and lost their, and lost their

jobs. So this is an example called survivor bias

or survivorship bias. Its a very common phenomenon in finance

and beyond and we're actually going to see some more examples of survivor,

survivorship bias very soon. Now some final thoughts here.

I've been assuming up until now that the fund manager or the fund managers have no

skill. And you might ask the question, well, is

that fair? Is it fair to assume to begin with that

fund managers have no skill? After all, in practice there are some

managers with skill. And that's absolutely true.

There are some managers in practice with skill.

I don't there are too many of them and certainly lots of fund managers may think

they have the skill but the reality is quite different.

Nonetheless it is true that there are some fund managers out there with skill.

That having been said, I don't think we're being unfair in assuming to begin with

that fund managers have no skill. And that's because it is the

responsibility of the fund manager to convince us that they have skill.

After all, fund managers with skill try to charge fees to manage our money.

So it's up to them to convince us that they have skill.

It's not up to us to give them the benefit of the doubt and assume they have skill.

So it's perfectly reasonable to start off with the assumption that they've no skill,

and then to see where that takes us. And if we can find such a manager, is the

resulting out-performance sufficient to justify the management fees?

That is not clear as well. If the management fees are too large, then

those fees are going to dominate the out-performance.

That the manager provides and so in that situation we wouldn't want to invest with

the manager anyway.