Explore
For Enterprise
Join for Free
Log In

Arts et humanités
Business
Informatique
Science des données
Technologies de l'information
santé
Mathématiques et logique
Développement personnel
Sciences physiques et ingénierie
Sciences sociales
Apprentissage des langues
Diplômes
Certificats

Explorez l'intégralité de Coursera

Parcourir
Rechercher
For Enterprise
Connexion
Inscrivez-vous gratuitement

4.1a: Sorting and Filtering Data Using Pandas

Loading...

Foundations of Data Science: K-Means Clustering in Python

Université de Londres

4.7 (15 notes) | 1.8K étudiants inscrits

Organisations all around the world are using data to predict behaviours and extract valuable real-world insights to inform decisions. Managing and analysing big data has become an essential part of modern finance, retail, marketing, social science, development and research, medicine and government. This MOOC, designed by an academic team from Goldsmiths, University of London, will quickly introduce you to the core concepts of Data Science to prepare you for intermediate and advanced Data Science courses. It focuses on the basic mathematics, statistics and programming skills that are necessary for typical data analysis tasks. You will consider these fundamental concepts on an example data clustering task, and you will use this example to learn basic programming skills that are necessary for mastering Data Science techniques. During the course, you will be asked to do a series of mathematical and programming exercises and a small data clustering project for a given dataset.

Visualiser le programme de cours

Compétences que vous apprendrez

K-Means Clustering, Machine Learning, Programming in Python

Avis

4.7 (15 notes)

5 stars
12 ratings
4 stars
2 ratings
3 stars
1 ratings

À partir de la leçon

Week 4: Introducing Pandas and Using K-Means to Analyse Data

4.1: Using the Pandas Library to Read csv Files5:29

4.1a: Sorting and Filtering Data Using Pandas8:07

Enseigné par

Dr Matthew Yee-King
Lecturer
Dr Betty Fyn-Sydney
Lecturer in Mathematics
Dr Jamie A Ward
Lecturer in Computer Science
Dr Larisa Soldatova
Reader in Data Science

Transcription

In this video, we're going to get deeper into

the Pandas library and find out how we can do filtering

and sorting operations and finally how we can do statistics on a Pandas DataFrame.

So, just as by way of introduction,

a Pandas DataFrame is a more sophisticated data structure than we've seen before.

So, previously, we've seen Lists and NumPy arrays.

As we move from Lists to NumPy arrays,

we found out that we gained a bunch of functionality and in the same way,

when we moved from NumPy arrays into Pandas DataFrames,

we again gain a bunch of functionality.

So, let's go in and see if we can write a simple sorting function for our DataFrame here.

So, what we've got here is some code which basically pulls in

some dataset from a CSV file. So, I'm going to run that now.

Then, if I just type in data into my Python notebook and run that,

you'll see that it comes up with this rather nice view of the data as a kind of table.

So, we just examine that a bit.

It's very much like a spreadsheet.

So, we've got a set of columns and with each the different variables in the dataset.

Then we've got a set of rows where each row represents a data point in the dataset.

So, yeah, it's familiar.

It's a tabular data structure.

It's similar to a NumPy array but with the additional element that we've got all types of

different data in there and we've got headings for each of the columns.

So, it's a bit more sophisticated

and I'm going to show you now that we can do more stuff as well.

So, what we're going to do first is see if we can sort all of

these data points by the average income.

So, how do we do it?

I'm going to take the data and I just do data,

sort values, and I need to tell it what I want to sort by.

So, I want to sort by average income,

and this all the tab completion is working quite nicely there,

so, don't forget to use that!

Then, I'm going to do a little trick here, which is,

I want the actual data to be affected.

So, I want my DataFrame to change as a result of sorting it whereas,

otherwise, it will just return the sorted version.

But I want to sort the actual data itself.

So, I'm going to pass this argument here ‘inplace’ and set it to True.

So, that means the data itself is going to be

affected and we're not just going to get a copy of it.

Then let's print out again.

Okay. So we run that, and you can see,

now if we go to average income,

that it's going up from

572 all the way up through to much higher numbers.

So, that's it. I've done my sorting there,

and of course you can sort by any of the columns

and do it in place and it will sort the data itself.

The next thing I want to do is show you how I can do a filter operation.

So, let's say, I wanted to select all of

the countries in the list that have an income greater than 15,000.

So, say, I wanted to do some data processing just

on the higher income group to see if there's anything interesting going on there.

How am I going to do that? I can do this.

I just say, richest and I'm going to say data.

What I do is I now specify a filter.

So, I say, that's specifying here,

I'm looking in the DataFrame and I'm going to specify a filter here.

The filter is that the average income

is greater than 15,000.

Okay? So, there's my filter and I'm applying the filter within this data array here.

So, it's a slightly unusual syntax but if you've used something like R before,

you might be familiar with this style of filter.

But anyway, then at the end we'll print out

richest in that nice table display. So, let's run that.

You can see now that I've only got my higher income elements here, and of course

they're still sorted because I did that sort originally.

Let's say I want to pull something out of that dataset.

So, who is the richest?

Well, so the lowest income in the rich set is going to be richest.iloc[0].

So, this is how we kind of pull out a row, if you like. So, let's just do that.

You can see if I pull up iloc[0],

I get Italy which,

in that set of rich countries,

is the one with the lowest average income.

If I wanted to, say, find the highest,

I can just do a minus one there — minus one as an array index,

always gives me the last thing in the array.

So, I don't have to worry about how long the array is,

I can just say, minus one and it will give me that one.

You can see, I've got here,

Luxembourg with their gigantic average income of 26,000.

So, that's where I'm going to be moving later.

What else can we do with these indexes?

Well, we can actually specify a range.

So, let's say, I wanted the first few.

You can see what I've pulled out there is the first five of

that set by saying [0:5] and I can do various other things with these indexes.

So, that's how we can do sorting and filtering.

What I want to do next is show you how we can actually do

statistics using the NumPy functionality in our DataFrame.

So, let's dive in and do that.

The final thing is statistics.

So, let's say, I wanted to find,

what's the mean of the richest?

So, richest.

I can just do that with NumPy.

First of all I'll import NumPy.

Import NumPy as np and then I'm going to say,

numpy.mean(richest).

But it's not just the mean of the richest,

(not that rich people are mean...)

It's just pulling out the richest value ‘avg_income’ like that.

Okay? So, what I'm saying is,

calculate the mean of the average income in that rich dataset.

We can print that one out.

So, the mean in that set of countries which are richer than 15,000 is

19,000 and I could compare that to the mean,

for example, compare that to the mean of

the whole dataset which would be np.mean(data),

sorry, the full dataset np.mean(data['avg_income']).

Then, I can print those out.

So, I'll print out all_mean,

rich_mean, like that.

Okay. So, there we have our result, so we've got, this is the mean of the whole dataset,

So, 6,000 and this is the mean of the people who are over 15,000 in average income.

So, there we have it. I've just dug in a little bit into

the DataFrame object from Pandas and showing you how you can apply sorting operations,

filtering operations and then also statistical calculations from the NumPy library.

So, you can see we have now moved from variables through

two Lists onto NumPy arrays and finally on to DataFrames.

We now have a really powerful data structure

here to work with when we're doing our data processing.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play

Coursera
À propos
Direction
Carrières
Catalogue
Certificats
Certificats MasterTrack™
Diplômes
For Enterprise
For Government

Communauté
Learners
Partners
Developers
Beta Testers
Translators

Se connecter
Blog
Facebook
LinkedIn
Twitter
Instagram
YouTube
Blog Tech

Plus
Conditions
Confidentialité
Aide
Accessibilité
Presse
Contact
Répertoire
Filiales