0:00

So we were talking about instances where the design matrix X was orthonormal,

it had orthonormal columns, so that X transpose x was the identity matrix.

And let's talk about some of the most famous cases where this happens.

Easily the biggest one is the so called Fourier basis.

So in the Fourier basis, the columns of X are trigonometric.

So imagine if your y is a time series.

And then the columns of x are trigonometric terms,

basically sines and cosines of different periods,

from slow, low varying signals

to really high-frequency signals toward the end of the x matrix.

0:50

So what this means is that the coefficients

associated from a least squares fit, from these basis elements and

my observed time series y,

the coefficients are just the inner product of these specific columns.

Let's say xi is column i from x, and y.

And so the transformation that takes y to

the collection of these basis elements, in other words,

that takes y to x transpose y, that transformation

is called the Fourier transformation, or the discrete Fourier transformation.

In fact, it can be done faster then this calculation.

It can be done capitalizing on some of the redundancies

in trigonometric relationships.

It can be done faster to result in the so-called fast discrete Fourier transform.

What you get with the Fourier transform is,

each coefficient is the amount of sort of the signal

that's correlated with a sine or a cosine turn of that frequency.

So as an example, if I had a signal that looked like this.

2:10

So it looks like maybe a sine term and a cosine term of,

a slow variation sine term and a high-frequency term added together,

then those two terms, the coefficients associated with those two terms,

will wind up having big coefficients when we do x transpose y.

2:35

So when we want to reconstruct the signal again, that's just summation,

right, the basis elements times the inner product of the x's and the y's.

So we can do tricks, for example, if we want to reconstruct the signal,

but only omit, only include those terms that are very low frequency.

So for example, if we want this big rolling term but we want to get rid of

the high frequency information, we might do what is called a low pass filter.

So we would let the low-frequency terms through, and

we would filter out the high-frequency terms.

So this would amount to, just in our reconstructed signal here,

only adding those components associated with the low-frequency terms in the basis.

And conversely, we might want to just capture the high-frequency stuff and

filter out the low-frequency stuff.

So in this sum, we would just take the high-frequency terms and omit the others.

And then the reconstructed signal would have filtered out the low-frequency

information.

So again, this all boils down to least squares, like we've been studying.

However, I think it's fair to say that signal processors tend to

think of this in a different way.

But it's nice to put this very important concept, Fourier analysis,

in the domain of least squares, which we've been talking about in this class.

The second basis that I would mention to discuss a little bit is so-called

wavelet bases.

And wavelet bases are similar to Fourier bases.

They're an orthonormal basis that have some nice properties associated with them.

And there is a discrete wavelet transform,

just like there's a discrete Fourier transform.

But it's actually able to get the transform faster than if you were to do

it by calculating each of these inner products by themselves.

4:34

Again, the Fourier basis and the wavelet basis, though, to really discuss them

we'd have to get into the nitty-gritty of the construction of the bases.

So I think I'm going to only then cover, really, in a little bit more detail, only

one of the three cases that's a little bit more statistical, one of the three most

important cases of orthonormal bases that's a little bit more statistical.

And that's so-called principal component bases.

So in the next lecture, we're going to talk about a particularly important

version of a basis where x transpose x = I.