Sound and music description

Loading...

En provenance du cours de Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

168 notes

Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

168 notes

In this course you will learn about audio signal processing methodologies that are specific for music and of use in real applications. We focus on the spectral processing techniques of relevance for the description and transformation of sounds, developing the basic theoretical and practical knowledge with which to analyze, synthesize, transform and describe audio signals in the context of music applications. The course is based on open software and content. The demonstrations and programming exercises are done using Python under Ubuntu, and the references and materials for the course come from open online repositories. We are also distributing with open licenses the software and materials developed for the course.

À partir de la leçon

Sound and music description

Extraction of audio features using spectral analysis methods; describing sounds, sound collections, music recordings and music collections. Clustering and classification of sounds. Demonstration of various plugins from SonicVisualiser to describe sound and music signals and demonstration of some advance features of freesound.org. Presentation of Essentia, a C++ library for sound and music description, explaining how to use it from Python. Programming with the Freesound API in Python to download sound collections and to study them.

Audio features30:21

Sound and music description24:48

Rencontrer les enseignants

Xavier Serra
Full Professor
Dept. of Information and Communication Technologies, UPF
Prof Julius O Smith, III
Professor of Music and (by courtesy) Electrical Engineering
CCRMA

Welcome back to the course on audio signal processing for music applications.

This week we are talking about the application

of audio analysis in the description of sounds.

In the last lecture we talked about audio features,

presenting various spectral wave analysis methods with which

we can obtain features of a sound that may be relevant for describing it.

Now in this lecture we want to go beyond the idea of describing single sounds.

And introduce the concept of describing collections of sounds.

So we'll first present the idea of the music information plane.

And then we will distinguish between sounds and music with the aim of

developing methodologies that are either relevant to the more generic concept of

sound, or are more specific for the characteristics of music.

So we'll first talk about sounds and sound recordings and

then collections of sound recordings.

And then we'll continue with the idea of music recordings and

collections of music recordings.

So this is the music information plane that should help us understand

what do we mean by music description in our particular context.

So we can dither different abstraction levels and

the left column are these different abstraction levels.

And we can go from the physical level,

which is basically the lowest level that we're dealing with.

And we go up to the cognitive level, okay.

So that would be the highest level that we can see there and some steps in-between.

So at the physical level, when we talk about sounds and music,

we can talk about concepts like the frequency or

the duration of the sound, or the spectrum and

some clear characteristics of the spectrum like the centroid.

And also we can talk about intensity of the sound.

If we go a level higher, a sensorial level,

then instead of frequency we can talk about the pitch of the sound.

And then in search of duration, we can talk about times,

the sensorial time of the duration.

And then instead of taking about spectrum we can talk about timbre

which is a sensorial concept.

And finally in terms of

intensity now we talk about loudness which again is a sensorial concept.

And we can go a level higher and talk about

perceptual topics, or perceptual concepts that are more musical,

that are related to musical concepts.

So in here, we talk about successive and

simultaneous intervals of pitches, what will be called nodes.

And then, we talk about time.

We talk about structuring of time, and we talk about things like the beat.

And then the timbre we talk about aspects of the timbre that we can identify and

characterize with some aspect of a musical sound.

And for example, the spectral envelope would be that.

And then finally instead of loudness,

when we talk about musical loudness we normally refer to dynamics.

And we have vocabulary that talks about the dynamics of musical sounds.

And we can still go a level higher and go towards the more

formalized way of talking about musical concepts.

And therefore, when we talk about pitch-related concepts,

we talk about things like melody or key or tonality.

When we talk about timing related concepts, so

we talk about rhythmic patterns, we talk about tempo, we talk about meter.

And when we talk about spectral timbre characteristics, we identify musical

instruments or the voices, entities that have a characteristic timbre.

And then finally when we talk about dynamics or loudness we are interested

in direct articulations of the sounds and how sounds change from one to another.

And finally we can reach the highest level, the cognitive level,

the level that relates to as humans in a very subjective way.

And how we listen to music and

what issues are relevant for us in the interaction with music.

So at this level the columns are not anymore valid.

There is interaction between all these different concepts.

And then we can talk about the emotion or the musical style or

semantic concepts that clearly integrate all these other levels of

descriptions to obtain these concepts.

These ways of describing music that are clearly more generic and

definitely are subjective or cultural.

And that would be a level that definitely would be hard to reach and

in this class we are definitely not going to talk much about that.

So we will focus on the low level descriptions of sounds.

Hopefully reaching a high enough level of description

that is of relevance to user applications.

If you want to describe sounds in a generic way,

sounds like the ones we find in free sound, we can group audio features.

The audio features that we talked about in the last lecture and

we can group them in different categories.

So we can talk about the timbre related features and

we mentioned quite a few of them, like the spectral centroid,

or the MFCCs, or the high-frequency content, etc.

Then we can talk about another group of features that relate with dynamics.

And that's basically the loudness and the level of a particular recording,

and then we can talk about the pitch related features.

And here is where we can talk about the pitch or the pitch salience,

and finally we have to describe also the time varying aspects of it.

Aspects of a sound that relate to the evolution of the sound to the texture

of the sound, and this we can group them under the term morphological features.

And here we can talk about things like the envelope of a sound or the onset rate or

many other type of descriptions that we could include under this.

We already have seen quite a few of these descriptors, so

what is interesting now is from these descriptors from these features

that we can analyze, we can now talk about collections of sounds.

So let's talk about how to describe collections of sounds, and clearly

there are many ways that we can analyze a collection of sounds and describe it.

And we'll focus on three basic concepts.

The first one and the most important concept that we need to develop is

the idea of similarity If you want to talk about collection of sounds,

we have to talk about the similarity between these sounds so

we can form the year of collections and groups them.

Once we can talk about similarity then we can cluster sounds.

We can groups sounds according to some criteria.

And finally, if we know some classes, some existing labels

that we use to describe a particular group of sounds, then we can classify sounds.

We can assign classes to particular sounds.

In our context, sound collection can be represented by a diagram like this one.

We consider a sound as a set of other features,

each feature having a numerical value.

In order to properly describe a sound, we have to use many features.

But for simplicity, we will be taking only, in this case, two features.

So if you consider a sound as represented by two features,

we can display a sound, a set point in a two dimensional space.

And that's what we're seeing here.

Every feature is one dimension.

So here, we're showing two audio features.

The horizontal line is the mean of the spectral centroid.

So we have analyze notes of three instruments, a violin, flute, and trumpet.

We have computed the spectral centroid and we have taken the mean of it.

So this is a multi-frame feature.

And also we have done the mean of one of the MFCC coefficients,

the second coefficient.

So that's the mean of the second coefficient in the vertical line.

And we can see that the violin has a quite high value for

this coefficient, for the MFCC value, and

it has a centroid that it quite covers quite a bit of space.

The trumpet has this MFCC coefficient quite lower.

So these blue dots are more in the lower side.

And the flute sound is kind of in between and

also the MFCCs are in between.

So that we can kind of see that these types of sounds

are distinct according to these two features.

Now, in order to play around with the space, the most fundamental

thing is to measure the distance between sounds, between points.

So we have to find a way in a multi dimensional space,

not just in this simple 2D space, how we compare 2 sounds.

How do we find the similarity between the two?

So Euclidian distance is one of the simplest ways to measure

the distance between two points in a multi dimension of space.

So in this case, p would correspond to one sound,

the collection of features of one sound.

And q would correspond to another sound,

the collection of feature values of the other sound.

And then, for every dimension i, we just take the distance between

those two values on that particular feature.

Then we square it and we sum over all

the features of the dimensions, and then we take the square root.

And that's the Euclidian distance.

In the case of 2D space of just 2 features, that becomes much simpler.

So in this case, the red and the blue are two sounds with two features.

And we can just measure this Euclidian distance, and it's basically the line

that separates these two points, the length of this line.

Now that we know how to measure distance,

we can cluster sound.

K-means is a clustering algorithm.

If we give to the algorithm the desired number of clusters, it will create

the clusters and it will return the mean value of each of these clusters.

K-means clustering aims to partition and observations, so

the observations would correspond to the number of sounds in

two K-clusters, so into K-categories or groups of sounds.

And each observation belongs to the cluster that has the nearest mean.

So this mean serves as the prototype of the clustered that we are creating.

The problem of finding this means or

this clustering is computationally difficult is what is call NB.

However, there are efficient heuristic algorithms that are commonly employed and

that converge quickly to local optimum.

So this equation expresses these mean immunization process that we

have to go through in the K-means algorithm.

So the goal is to find the mu for every cluster, so

we have this K-cluster that minimizes this overall sum,

so we have to do it sort of holistically

of attaining this overall minimization result.

When here in the plot we see the three steps

in this process of obtaining these clusters.

On the left one, we start from a collection of points.

In fact, this is not sound features.

This is just random points in space.

And the goal is to cluster them according to two clusters.

So we're going to find two clusters.

So we initialize the algorithm by putting

two points that will be used as the initial means of two clusters.

So the middle diagram, the red and the blue are the two initial means.

And these two initial means, this collection of samples of

sounds get clustered in the way that we see here with the red cluster and

the cyan cluster.

And now, with K-means, we iterate over this

minimization that, this equation that we have here.

And after a certain iteration, it converges,

and it converges to the clustering that we have on the right.

So it has clustered the red dots in the lower left corner and

the cyan dots in the upper right corner.

And clearly, this is a much better clustering than the initial

random clustering that the algorithm started with.

So now, with that, we can have collections of sounds and

automatically find classes that group sounds

that might have a similar audio features.

The last thing that we talked about for

describing sound collections is the classification of sound.

And that means that we know some classes.

We have identified certain categories of sounds.

And what we want to do is given a new sound,

we want to classify to one of these known classes.

So the K nearest neighbors classifier, KNN,

is an algorithm used for this type of classification.

And the rule of that we implement with KNN,

it classifies a sound by assigning to it the class

that is most frequent in the neighbors.

So we find K neighbors, and whatever is the majority vote of those neighbors

then becomes the class of this query or of this new sound.

So this block diagram exemplifies this process,

these set of rules that are implemented in the KNN algorithms.

So we start from a query, okay so that would correspond to a new sound, and

we are starting with target examples.

So we are starting with collection of samples, of sounds that have a label.

For example, in the diagram below, we have two such collections,

label collections, the blue and the red ones.

And the cyan dots are our query.

So we have to label or assign these query samples to one of these two collections.

So what we do is we measure the distance with the Euclidean distance.

We measure from every query sample to all the neighbors, okay?

And we take the K top results.

So we only look at the K nearest neighbors.

And from those what we do is we take a majority vote

based on the classes they belong to.

So the last box is basically we know the classes that the neighbors belong to.

And we take the majority of the vote and we assign the class that is the majority.

So, on the right diagram, we see the result.

So the cyan dots have been assigned a color.

So some have been assigned the blue class,

and the rest have been assigned in the red class.

So this is a very simple but quite efficient way

to classify sounds or, of course, any other type of data in to classes.

If we now go to musical sounds, recordings of pieces of music.

The features to be analyze should be more specific and

more related to musical in meaningful concepts.

So let's start by defining some categories or features or

descriptors that are relevant musically.

So we can talk about timbre related descriptors.

And things that we mentioned like instrument characterization or

instrumentation characterization, or even the remixing of

musical recordings that is also an important feature of music.

Then another category would be related to melody and harmony.

And that includes things like the phrase, the motive, or

the tonic of the piece of music and even if we talk about non-western music

traditions like Indian music tradition, we talk about raga, or

like in the Turkish music tradition, we talked about makam.

So, these are melodic concepts that can be described and

that are important to characterize particular piece of music.

Then we can talk about rhythm.

And then again we talk about patterns.

Or we can talk about tempo.

Or we can talk about beat.

And in the case of many music traditions,

the concept of metrical cycle is an important way to think about rhythm.

And finally, another way of describing music is

by describing the structure of a piece of music, defining the sections or

the movements that a bigger piece of music might have.

These descriptions cannot be obtained by just performing audio analysis.

We normally start from audio features, but then we have to develop models

from a combination of features that can capture the essence of each concept,

and clearly this is beyond the aim of this class.

And this is very much an open research area, very active and

that hopefully we'll be evolving through the years and

we'll be able to eventually do things like this.

And if we go to music collections, it's even harder.

The description of music collections can be very complex if we want to

do musically relevant tasks.

This is a very active research topic again.

And this is what is referred as music information retrieval.

In which we want to automatically classify pieces of music and be

able to perform tasks such as recommending a piece of music or

finding pieces of music that might be related to another one.

And then, the concept that we talk about sounds, also apply but

they have to be adapted here so similarities of fundamental concepts.

But then we have to divide or we can find different facets of the similarity and

we can talk about rhythmic similarity,

we can talk about similarity of the instrumentation,

of the melodic aspects or the harmonic aspects structural similarity.

And then, of course, we could combine them in order to find similar songs.

And these types of similarity are clearly not Euclidean distances.

We have to develop similarities that are much more sophisticated.

And then we can classify and

cluster these pieces of music according to different criteria.

The classification for example, can be classified according to genre,

or style, or artist, or the school that the music tradition comes from.

Again, this is much beyond what we can cover in this class.

That is a fascinating topic that is a natural continuation of

the kinds of things we talked about.

So given that this is a very open research problem,

the references come from research papers.

And typically a lot of this research is label under what we call

music information retrieval, so

the Wikipedia entry for music information retrieval is a good starting point.

And then for the more specific things that we have talked about,

you can look at the specific entries for Euclidean distance or for

the K-means clustering has a good entry in Wikipedia, or

the concept of classification base on K-nearest neighbors.

Of course, these are just two examples of different clustering and

classification strategies.

There's a lot of different strategies coming from the field of machine learning

that has brought many new possibilities to do these type of tasks.

And that's all, so in this lecture, we have opened the door into

a huge research field that aims at automatically describing and

organizing large collection of sounds and music recordings.

We just introduced some of the basic concepts and

specific methodologies that can be used to start working on this topic.

In the programming lectures,

we will show a little bit examples of how to actually do some of this.

That clearly we cannot make justice to this field of research.

However, I hope you got a taste of it.

And I will see you next class.

We will present some more demonstrations and practical examples of all this.

See you next time, bye bye.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play