Sounds transformations 2

Loading...

En provenance du cours de Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

168 notes

Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

168 notes

In this course you will learn about audio signal processing methodologies that are specific for music and of use in real applications. We focus on the spectral processing techniques of relevance for the description and transformation of sounds, developing the basic theoretical and practical knowledge with which to analyze, synthesize, transform and describe audio signals in the context of music applications. The course is based on open software and content. The demonstrations and programming exercises are done using Python under Ubuntu, and the references and materials for the course come from open online repositories. We are also distributing with open licenses the software and materials developed for the course.

À partir de la leçon

Sound transformations

Filtering and morphing using the short-time Fourier transform; frequency and time scaling using the sinusoidal model; frequency transformations using the harmonic plus residual model; time scaling and morphing using the harmonic plus stochastic model. Demonstrations of the various transformation interfaces of the sms-tools package and of Audacity. Presentation of the stftTransformations, sineTransformations and hpsTransformations functions implemented in the sms-tools package, explaining how to use them.

Sounds transformations 116:43

Sounds transformations 216:06

Rencontrer les enseignants

Xavier Serra
Full Professor
Dept. of Information and Communication Technologies, UPF
Prof Julius O Smith, III
Professor of Music and (by courtesy) Electrical Engineering
CCRMA

Welcome back to the course on audio signal processing for music applications.

In the previous theory lecture, we started talking about sound transformations.

In particular, about how to use the short time free transform and

the model to manipulate sounds.

In this second theory lecture, we want to continue that,

and talk about the harmonic plus residual model and

how it can be used for transforming the frequencies of the harmonics.

And then, the harmonic plus stochastic model, and

how can it be used for time stretching and morphing operations.

These morphing by the way is quite different from the one that

was possible with the short time period transform.

And of course, these are just some example operations, example transformations.

But the number of possible uses of these models is much larger.

Let's start with the harmonic plus residual model.

This is the plot diagram that we have seen before in which from the input sound,

we analyze the spectrum, find the peaks.

And then, find the fundamental frequency, find the harmonics and

we subtract those harmonics from the original signal.

We need to compute the spectrum of the original signal again in order

to develop a spectrum from which we can subtract these harmonics in a proper way.

Okay, and then where we do the transformations is in the frequencies and

amplitudes of the harmonics.

We don't apply any transformation to the residual in order to

apply transformation to the residual it make sense to have a model for it.

And that what we are going to be introducing

in the next model the Harmonic plus sarcastic model.

And here, the transformations on the harmonics, again, like in the sinus solar

model, it's applied only on the frequency and amplitudes.

The phase are forgotten, we don't use them,

we will be re-synthesizing them again after transformation.

And the main issue of that is because that phase is a very sensitive to

transformations and it's very difficult to handle them.

So it's better to just regenerate them after transformations and

it doesn't sound that bad.

Anyway, so after the transformations we can

sum these new harmonics to the residual signal and

these residual signals since it does not include any deterministic.

Any harmonic information, it will merge quite nicely.

And then, it will create a sound that has the same residual as the original and

the harmonics will be modified.

In terms of types of transformation that can be done to these harmonics

to the frequency of these harmonics.

There are many and here I explained three common ones.

The most traditional one is to just transpose

all the frequencies by a given factor.

So, we multiply all the harmonics by a scaling frequency factors so,

that would correspond to a transposition so we multiplied by two.

It would correspond to converting the sound to an octave higher if

we multiplied by 0.5 it would be an octave lower.

Then instead of multiplying, we can just shift all the frequencies.

For example, we can sum a factor to all the frequencies, so

we'll be shifting all the harmonics by an additive component.

And finally,

the last one I want to mention is what is called frequency stretching.

In which, we are applying frequency change that is dependent on the harmonic.

So here, what we are doing is basically dividing every harmonic by it's harmony

number, so basically get a frequency around the fundamental.

And then, we multiple by the harmony number to the power of a scaling factor.

Therefore, the higher harmonics will be modified very

differently from the lower harmonics, it's very much harmonic dependent and

that can create some nice effects that we will see.

So this is a plot of these three transformations that

exemplify these operations quite nicely.

So on the top left, we have the original harmonics all located

at the harmonic number location, one two, three, four, and five.

Then if we transpose by two on the right,

it's just simply every by two of harmonic one in position two,

harmonic two in position four, etc.

We are maintaining a harmonic spectrum with just transposing by a factor or two.

If we shift instead by a given value, in this case since

we are having the frequencies normalize to integer values.

Here by 0.5 will be shifting by half of the frequency

of the first fundamental, of the first harmonic.

And in here, we see that we're shifting everything by a constant value.

From now after this transformation, we do not have any more harmonic series

because the first harmonic is not just the fundamental frequency anymore.

Okay, so we have now a series of values that are equally spaced but

they are multiples of another frequency.

So the sound may be quite a bit different and quite interesting in some cases.

And finally, the frequency stretching operation,

it's a quite different operation.

Now it changes the distance between the harmonics and

it is like an accordion.

So if it is here at one point three means that well the first

value the fundamental will not be touched.

Then first one will be stretched by a factor of 1.3 and

then these are going to be powers of these.

So as we go higher up, it's going to go sort of the distance we'll increase

kind of exponentially and these create again quite very interesting effects.

So let's listen to sound in which we have applied some stretch and

transpose kind of transformation.

So let's listen, so it's a flute sound that we can listen to.

[SOUND] Okay, A4 below that we see the harmonics

that have been identified on top of the residual spectrum of the original signal.

And then, what we're doing is leaving the residual cities so

that the background spectrogram is exactly the same as the top one.

And we're just modifying the harmonics, so

if you look here, the harmonics have been heavily modified.

And if you see they have been stretched and

at the same time transposed a little bit, so let's listen to that modified sound.

[SOUND] Okay, of course quite different.

And if you pay attention the residual is there and since the residual is basically

these breath noise is emerges also quite well with these transpose harmonics.

Okay, let's go to the other model, the harmonic plus stochastic model.

It's very similar to the previous one and now, the transformation is applied both

in the modeling of the receiver signal and of course in the harmonic component.

So we have an stochastic approximation and therefore,

we can transform that quite easily because it's a model that has simplified

that the residual quite a bit and is quite flexible in terms of manipulations.

And then of course, after transformation we obtain the stochastic component and

we obtain the frequency amplitudes of the and faces of the harmonic sinusoids and

we can synthesize the tool and some together.

And so in terms of what is actually done is very similar to what

we presented to the sinusoidal model.

Now, what we are introducing is the transformations

on the stochastic component in a similar way.

So the frequencies of the harmonics can be scaled by a factor and

the time of the reading of these frequency values

also can be changed in order to obtain these time scaling operation.

So we have this scaling function, the amplitude is the same way and

the stochastic component is handled in the same way, okay?

And of course, the phases are generated and

they are generated by starting from an initial phase.

And then adding the frequency of every harmonic that we want

to generate to that phase, so we can generate the instantaneous phase that way.

Okay, let's show an example of that.

This is the sax phrase that we are also we have heard before.

Let's listen to that again.

[SOUND] Okay, and then we have the analysis of it.

Where we have analyzed the harmonics and

the stochastic component and then we can change that.

In this particular case, we have done time scaling, so we have changed the timing and

now with the residual approximating residual stochastic,

it's very easy to do time scaling of the stochastic and the harmonics.

And that's it,

we have just focused on the timescaling transformation in this particular example.

Let's listen to that, a very simple timescaling operation.

[SOUND] So, basically the only thing we have done is we

have compress the first part of the signal by quite a bit.

In fact, it's like half of the duration and

the second half has been extended by twice as much.

So in fact, the overall duration of the sound remains the same and

of course as you can imagine, this is very flexible and

we can do a lot of envelopes that we can play around with.

Okay, now, let's talk about the morphing using this harmonic plus stochastic model.

Okay, here we have simplified the block diagram.

We have two sounds, x1 and x2.

Now they're basically at the same level and basically,

what we're going to do is interpolate the two representations.

So from x1, we obtain the frequencies and amplitudes of

the harmonics and the stochastic approximation of the residual.

And of sound two we do the same thing and then,

what we are doing is interpolating these two sets of functions.

We are interpolating the frequencies, the magnitude of the harmonics and

we are interpolating the stochastic envelopes of the residual.

And then of course, we can synthesize back the output sound by

generating the stochastic component and the sinusoidal component.

Let's look at the particular frame of a sound in which we have applied like

a 50% interpolation, so on top left is X1.

One particular frame the harmonics

visualize as the location is the frequency and

the height is the amplitude and then the blue one is sound one.

X to the red one is a sound too and then, we can interpolate these two.

Basically, harmonic by harmonic we can interpolate and as you see,

the X2, it's a different frequency.

It's a sound that has a higher pitch, so

the interpolated result has a pit which is in between the two,

and has a shape again that is in between the two, okay.

So it's basically, a 50% interpolation between the two set of values and

the suppressing component below does the same idea.

So on X1, this shape is the approximation of the residual,

so the [INAUDIBLE] is where we actually have values has an envelope.

And then X2 is another envelope for X2 and the output is the interpolated envelope,

so it's a shape that is in between these two shapes.

Okay, let's use that in a particular example in which we start from a violin

sound, and we're going to interpolate it progressively towards a soprano sound.

So here, we see on the top the violin sound, and we can listen to.

[SOUND] Okay, and it's analyzed, so we see the harmonics.

Well, we see I think here is the first like 40 harmonics and then,

the stochastic component of that.

And then, we see below the analysis of the soprano sound,

let's listen to this [SOUND] Okay, where we see again the harmonics,

it's a higher pitch and the stochastic component in the background.

And now,

we can choose the interpolation values to go from one set of values to the other.

So it will progressively go from one sound to the other.

Let's listen to the result.

[SOUND] Of course, since the soprano sound is

higher than the violin, we hear this glissandi.

But at the same time, we hear the evolution of the timbre, and

every single parameter is slowly changing from one to another.

Again, the possible type of Functions to control these interpolations

is enormous so we can play around quite a bit with these ideas.

That's all, so again, Wikipedia, there is not much and there is not much that we can

refer to in terms of trying to understand some of these transformation aspects.

And in fact a lot of these things is just by trying, so

I would encourage you to try these things and

develop your own intuition of what it works and what it doesn't.

And that's all, so

this has been the second theory lecture on sound transformations.

Of course, in this week, what is important is the programming and

the demonstration classes, even more than the theory lectures.

Because after all, it's a very applied topic, and it has to be grasped

from an application, and from an intuition and musical perspective.

So thank you very much for your attention and I see you in the next lecture.

Bye-bye.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play