0:00

Welcome back to the course.

[INAUDIBLE] processing for music applications.

In the previous programming class, we talked about the peaks detection,

the fundamental frequency detection algorithm.

So how to identify the fundamental frequency of harmonic sound.

Now we are ready to put together the whole harmonic model analysis and synthesis.

So this is the block diagram of the system that we have and

we're going to explain, in which we start from the standard

blocks that we already have seen in the sinusoidal model.

So that it takes the inputs out in windows then it performs the DFT,

and then it finds the peaks, and these two blocks

are the ones that are specific for the harmonic model.

So the first one is the F0 detection, so

from the peaks, it identifies the best fundamental frequency.

1:29

So let's go to the file harmonic model, the BY.

This is the file that includes most of the functions of

the harmonic model, the actual implementation of the x0

detection is in the utility functions file.

But here, we have the wrapper to the fundamental frequency at zero

detection that we already mentioned in the last lecture.

Then, we have the harmonic detection that we'll be talking about shortly and

then, we have the complete harmonic model, that has both analysis and

synthesis but this is meant to be run in real-time.

So it doesn't allow for cleaning of the tracks, so

we will be talking about today on the analysis and

synthesis separate because this allows us to do cleaning of tracks.

And so, handle little bit of memory of a past few frames and

this will result into better analysis and synthesis.

So let's talk about these function.

Harmonic model and all, so it inputs all the parameters of the analysis,

the input array of the samples, sampling rate.

The window, FFT size, hub size, the threshold, T for the peaks.

The number of harmonics that are allowed, so this is the maximum number.

Then the range of frequencies for the fundamental

frequency attraction, so this mean F0 and the max F0.

Then there is this error of threshold that is also part of the F0 detection

algorithm, so this is a minimum error that we'll be accepting.

And then, for the sort of a track continuation,

harmonic continuation, we have these two parameters.

One is the harmonic deviation slope,

this is sort of a deviation slope from the previous

3:38

series of harmonics and from the ideal harmonic series.

And this is a way to account for that higher harmonics should allow for

bigger deviation than lower harmonics.

And then finally, parameter to control the length of harmonics tracks so that

we don't accept shorter tracks than the one's that specify with this parameter.

And the actual code is again, very similar to the sign of model,

we have a main loop that iterates over the whole sound and

it keeps analyzing the spectrum and in the peaks.

Interpolating those peaks and then here is where the specific code of

the harmonic models tabs, so we call the fundamental frequency algorithm.

Okay, so this is the one from the utility functions that

we talked about in the last lecture.

Then we have these few lines that make sure that this new fundamental

frequency is not that different from the previous one, so that it maintains

sort of continuity and then there is this function called harmonic detection.

This is the one that identifies the harmonics of fundamental

frequencies and it's above this code.

So let's go to here, this is the harmonic detection function.

Okay, so this is the one that given the fundamental frequency,

F0 and the peaks of the spectrum, it identifies which of these

peaks are close to multiples of this fundamental frequency.

We also as inputs, we accept some information from the previous strings so

that we can identify if there is some time continuity.

And then these deviation that is the deviation we allowed from

perfect harmonicity and from perfect continuity from the previous frame.

5:52

Okay, so the code is not that complicated.

The core is basically these three lines In which

we defined the deviation from perfect harmonicity.

So we find the peaks compare or

a particular peak compared with a particular perfect harmonic.

We also compared a peak with respect to the previous harmonic

coming from the previous frame.

And then, we define the threshold base

on these harmonic deviation of slope that is the one that will be accepted.

So if the current peak is within that threshold therefore,

within the kind of vertical deviation and horizontal deviation,

so it has this frequency and time evolution tracking.

So we accept that peak as harmonic and

we keep adding to the harmonic that we create and that's all,

that's basically what the harmonic detection does.

So if we go back to the harmonic model now [COUGH] after this harmonic detection.

Let's say this is a setup some code to create the array of values that

keep adding up to the whole matrix of the output.

And finally, if once the whole process is done, so

once we have gone through the whole sound.

We clean the tracks, so

we delete the tracks that are smaller than a given duration,

and again, this is the same then with it in the sinusoidal analysis.

Okay, so I wrote a script that basically analyzes

a sound with this harmonic model and now function,

so we took this vignette sound and specified all the parameters.

So the number of parameters are starting to get quite a few.

So we have the window, we have the window size, the FFT size, the threshold.

There is no need for this comma here, so minus 90 as a threshold,

then minimum duration in time, the number of maximum number of harmonics.

The minimum fundamental frequency and

maximum fundamental frequency, the errors threshold that we accept and

then this deviation slope that we just talked about.

Then there is these two parameters that are normally fake because they

are more related to synthesis which is the FFT of the synthesis buffer.

And the hop size which are given that the FFT buffer requires for

the synthesis, it requires the hop size to be one-fourth of that.

So we normally put H as being 128, so

we normally don't allow any changes from that.

So from the graphical interface, there is no control over these two parameters.

Okay, and then we read the sound, we get the window and

we call this harmonic model an function.

And here, I just plot the frequencies of these trajectories, so let's run this.

So let's run test three, now let's first save it.

Okay, and now let's run it.

Okay and this is the trajectories we obtained,

okay they are okay, they're pretty good.

If we zoom in into a particular region,

we might be able to see a little more, okay?

Of course, if we zoom in we're starting to see

the problems at the transitions between nodes.

And especially, we see a problem as the harmonics go higher and higher up.

This is because this is a linear scale and

this deviations are kind of logarithmic.

So as the frequency goes up, in fact the changes

get much larger because they are logarithmic changes.

So in fact,

this is the reason why we need this deviation slope that we specified here.

So for example, if we reduce this deviation slope,

let's make instead of 0.01.

Let's make 0.001 and let's run it again.

10:48

It does restrict that the deviation allowed in the higher frequencies.

So it has the increase of deviation is much smaller,

so it has not allowed these larger deviations.

So with that, we have quite a bit of control of how this

tracking process is implemented.

Okay, so in the SMS tools package, there is this harmonic

model function that performs both the analysis and synthesis.

And is in fact, the file that is called from the interface,

so in this file we have a one main function and

it has all these parameters that we already talked about.

But apart from the analysis that we already performed, it does the synthesis.

Okay, and then it saves the file into a synthesis file and

it plots analysis and synthesis of this.

So let's run this harmonic model function.

12:02

Okay, also this would take a little bit longer because it has to do

the analysis and then the synthesis.

And then it just plots to the first 5,000 hertz and

specified to plot on the phase 5,000 hertz.

So here we see all these trajectories that evolved in time and

the synthesis at least the wave of loops are quite similar.

So let's listen to the output sound, so

now we created a file called bignish_harmonicmodel.wav.

So I can play it so, if I put the exclamation

mark play just say type vignesh.

[MUSIC]

Okay, so this is the synthesized version of the original file and it's quite good.

I mean, we will see starting next week that in fact there is some things

that are left that are not included in this resynthesis and

this is basically the residual component that We are missing.

But anyway, that's all for the harmonic model and we go back to the slides.

13:42

definitely encourage you to go through that and run it and abuse it if you want.

So that was all for this programming classes on the harmonicModel.

Hopefully, this has given you sort of a depth,

in-depth view of what it entails to implement this type of models.

And well, we will continue, so next week this will get a little bit

more complicated because we will try to see what is missing out of this model.

And we will try to model this missing part,

this is going to be the concept of the residual or the stochastic components.

So I will see you next week, bye-bye.

Thank you.