This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.
Worked Example: Twitter API (Chapter 13)
Loading...
Compétences que vous apprendrez
Json, Xml, Python Programming, Web Scraping
Avis
4.8 (19,797 notes)
- 5 stars16,457 ratings
- 4 stars2,810 ratings
- 3 stars422 ratings
- 2 stars64 ratings
- 1 star44 ratings
RG
May 31, 2018
This course really helped me in learning how the network and api work. Many things happen in the background which we are not aware of. To understand those things one should really try this course.
XZ
Jan 09, 2017
Very nice intro class to learn how to access web data. Even though the assignment is a little bit easy, i still get a lot of fun exploring the skills i studied to access some websites as exercises
À partir de la leçon
JSON and the REST Architecture (Chapter 13)
In this module, we work with Application Program Interfaces / Web Services using the JavaScript Object Notation (JSON) data format.
Enseigné par
Charles Severance
Professor
Welcome to Python for Everybody.
We're doing some sample code,
playing through with some sample code samples and you can get this by downloading it.
I've got this whole thing downloaded and I've got all the files here,
and these are the files we're going to play with today.
Today, what we're going to do is talk about the Twitter API.
The one thing we got to learn about the Twitter API is we have to authorize ourselves.
So, we have to make sure that we have a Twitter account and then we get some keys.
So, in this particular application if you want to duplicate what I'm doing,
you have to go to apps.twitter.com,
click this create new application button,
and then get some codes, okay,
and the codes show up as soon as you hit this button and then one more button,
which I'm not going to do on screen.
So, what happens is there are four codes that you got to put in this file hidden.py,
consumer key, the consumer secret,
token key, and token secret.
These are just messed up,
so I'll show you how this works and blows
up first and then I'll put my keys in here without showing you.
But basically, this is a little file you got to edit or these Twitter ones don't work,
you'll see what happens.
So, the first one I'm going to do is do the simplest one of all and that is,
I call this thing Twitter test and it just is going to
go ask for the user timeline and we
can take a look at this and we're going to take
the URL and we're going to augment the URL.
This is the base, we found this looking at the Twitter API documentation.
We're going to pass a parameter of screen name Dr. Chuck and account of two,
so this is just a Python dictionary.
An augment comes from this little bit of called code called twURL.
This uses a bit of code called OAuth,
which is built into Python as well, right?
Yeah. That's built into Python as well and it augments the URL.
It takes the the key, the secret,
the token key and does a thing and signs it,
and then makes this big, long,
ugly URL which you will soon see and does this,
it's a signature of the URL.
So, we pass this data back and forth to
Twitter with a signature and then they recheck the signature
and it's a digital signature that knows that this URL came from
a program that knows the key secret and token and token secret.
So, this augment basically,
is something that I wrote,
twURL augment is something I wrote to make it easier to add all these OAuth parameters.
You feed this code by putting your data into hidden.py.
Lots of people get this to work so,
don't worry it's cool when you finally get it to work.
So, let's take a look at what it does just know
that this makes an awesome URL that does all the security,
and we'll see one of those URLs.
So, ignore the certificate errors.
This has to do with the fact that we're using HTTPS and
Python doesn't have enough certificates put into it by default for a lot of reasons,
but our quick and dirty ways they turn them off.
Thank you Python for reducing security by teaching us,
so that this is the best way to do it,
that's a grumpy up moment from on my part.
So, what we're going to do is we're going to do a URL open.
This bit here is to shut off the security checking for the SSL certificate,
and then we're going to read all the data,
and then we're going to want to print it out.
We're also going to ask the connection this url.
Remember, I told you a long time ago
that URL lib eats the headers but you can get them back.
Now, we're going to ask to get a dictionary of the headers back,
and so we'll print those out, okay.
So, this is really just testing
the body and the headers and printing them out in as raw a way we can do.
So, let's go run this.
Now, this is going to fail the first time we do it because we
haven't put the hidden variables in there.
So, if I say python3twtest.py,
it's going to run and blow up and it's going to give you this 401 authorization required.
That's a good sign because that means that you haven't yet
updated your values in hidden.py.
So, this is that augmented URL and you can
see the consumer key and consumer secret and the OAuth token and whatever, okay.
So, these tokens are wrong.
These aren't Control-C, they aren't real.
But you'll notice it doesn't end the key and the secret of
the token key of the token secret and the secret.
That's all actually encoded in the signature.
It turns out that you need to have the key and the secret and
the token secret to generate the signature and where is the signature?
There's the signature, right there's the signature.
So, this signature combined with the notes,
that you can only do this signature as a time and includes all kinds of things.
So, even if you type this in, well,
you'll see these go by and it's
not really breaking my security too much when you see these afterwards.
So, don't get all excited when you say,
"Oh, you revealed your token and your key."
Now, I can reveal my token and key but I'm not going to reveal the secret.
So, this adds all this OAuth stuff,
OAuth nonce, OAuth timestamp.
These timestamps and nonces are made it so that you can't
replay my URL even if you see the exact URL.
Once I hit it, then you can't hit it again and so,
that's what the nonce does.
So, I'm going to close hidden.py here.
I'm going to update hidden.py in another window,
okay. So, I just in another window, I updated hidden.py.
I'm not going to show you that.
But now, I'm going to run python3twtest.py.
So, twURL is going to read hidden and now,
these keys and secrets are my real ones that I haven't shown you.
So, this should work, fingers crossed.
It worked, okay.
So, it worked.
So, I'm calling Twitter.
Here's the URL.
Now, don't worry, the token and the consumer key are not enough
to break into my account and neither is the signature because you can't replay this.
In about five minutes,
you can't replay this anymore, okay.
So, you can't generate the signature.
I've done one, the signature includes the time and date,
so you can't trust me.
Go read up on OAuth,
don't worry, I haven't really revealed anything.
But, so, the first thing we see is this.
So, we see, and we should put it like the line and dashes here.
This is the Json.
It ain't very pretty, it's not very pretty.
Okay, and so that's the Json, from there to there.
It's just what most APIs give us back,
it's really dense Json, right.
So, this is a byte array.
Remember, how you have to do a.decode,
I didn't do a.decode here.
So, this is telling, Python is telling us this is a byte array which it's
a raw set of bytes that came from the Internet,
which probably are UTF-8.
If I input a decode here,
then it would decode.
If say.data.decode there, then it would be fine.
But we don't care,
this was just a dump, do we get anything?
So, then, here let's do this, print.
I'll just make this code different.
Put some equal signs here,
a lot equal signs.
So, we can easily see where the thing starts and stops.
So, we'll run that again if you look at those URLs.
So, that was all of that stuff and then this is the headers,
and so the headers again are not pretty.
You get the headers it's a dictionary,
you got cache-control, no cash comma.
This is the string key value,
you got to find your commas key value,
but the one that's really interesting here is which one is it?
X-rate-limit-remaining, right there, x-rate-limit-remaining remaining.
So, that means that for this particular API in this header
tells me that I've got 898 calls left,
and this is when I will get more calls and so watch.
I'm going to do this again and you will see that I can only do this 897 more times now.
Run it. I can only do this 897.
So, I am being tracked at this point.
I'm being tracked by Twitter.
Twitter knows that it's Dr. Chuck that's doing this and Dr. Chuck is done 900,
he's done 899, 897,
and if I keep running this,
eventually Twitter will tell me,
"You got to wait for a while",
and that's because Twitter doesn't want me under my Dr. Chuck account
pulling out lots and lots of stuff out of Twitter and making my own website.
I do actually have my own twitter website using some cool software
www.doctorchuck.com/Twitter
and this I have to run in rate limits and causes all kinds of you know, whatever.
So, okay the rate limit.
So, I'll save that.
So that's tweet, this is just a test, okay?
Because I want to do something interesting,
so we're not parsing the Json that comes back,
we're not doing anything tricky with this and away we go.
So, let's take a look at some more code.
I think I don't need this anymore.
So, now I am going to parse this.
So, most of this looks the same.
I've got that same user timeline Json,
I'm going to ignore the SSL certificate someone write a loop.
So, I'm going to ask the Twitter I'm going to print,
I'm going to get a Twitter account and quit if it's a blank line or phase to entering.
I want to use the Twitter URL augment the same way that's
going to do all the signing using from hidden.py and
retrieve it and I'm going to retrieve it ignoring
the SSL errors and then I'm going to decode.
This time I'm going to decode it so that I get a real Unicode string,
and I'll print the first 250 characters of it,
I'm going to grab the headers,
and I'm going to print the remaining, the rate limits.
This is sort of a very simple version of this same thing.
It really is decoding the data and only printing the first 250 characters.
So, let's run that, Dr. Chuck.
Boom, and it's got 896.
So that's just a little simpler version that with a little less brutal debugging.
Okay, so now let's do something even more fun.
Let's go to twitter2.py and tear it apart.
So again, we're going to look at my friends list or someone else,
any of his buddies' friends list.
We're going to ask for the friends and ask for the screen name,
ask for the first five friends,
and then look at their statuses,
open it, decode it,
get the headers, print the rate remaining.
All this stuff is the same as in twitter1 but now we're going to parse the JavaScript.
I'm not even putting this in a try and except because,
hey, I'm talking to twitter.
I'm going to guess that twitter is going to give me the right stuff.
You probably want to put a try and except here.
Then I'm going to do a debug print.
I'm going to do a Json pretty print.
Let's make that to be two, so it looks a little better.
And then, I'm going to run it and then you're going to see how we have to parse this.
We're going to see that it's a list.
So, we're done with that and now we're running twitter2.py.
So, I'm going to go to Dr. Chuck and this is going to ask the question,
who Dr. Chuck's friends are?
Okay, let's go to the top.
So it hit this API and it has the screen name Dr. Chuck,
count equals five and all this oauth stuff.
Again, this is not a security breach by showing you all of this because the signature,
the secrets aren't there.
Okay, so if we look at it,
it's an outer object or dictionary and then the outer has a users which is a list.
Then, each user has some stuff in it.
So this one's Stephanie Teasley.
It's got her screen name,
it's got some descriptions.
Keep on going and it's got her status,
her latest status, for my friend her status.
Her source, where she's at.
Man she's got a lot of stuff here.
Okay. There we go, that was the first one and then
the next one that I'm following is LiveEdu etc.
So you'll see that this is an array.
So that outer thing is an array of users.
Now, JS here is a dictionary.
So I can say for you in JS sub-users.
Well JS sub-users is a list.
So the first U is going to be
this Stephanie Teaseley U and the second U is going to be LiveEdu.
So, that's all it took to get through all that stuff and figure that out.
And then I'm going to say, get me the screen name of my person.
So let's go in here.
So that's going to pull Stephanie Teasley as stephteaseley out then.
Then I'm going to go find her status.
Let's find her somewhere in here.
U sub status, sub text. Come on.
Okay, there's sub status,
sub status is all this stuff. More and more.
Right there, that status,
that U sub status is that and then U sub status,
sub text is this step.
So it's going to extract this bit right here, okay.
So, use that as text and I print out the first 50 characters of the screening status,
and I do that for the first five because I told it I only wanted five.
And then of course I get to see that the right limits.
So let's go down to the bottom.
So all of this is the debug print of the Json I got back.
Here is the program starting to print.
Here is the screen name of my first friend and
here's the first 50 characters of her most recent status.
Here is the screen name of my,
and these are in reverse order,
who I've been following.
So I've been playing with this live coding stuff.
So I'm following them.
What!
KeyError status. That didn't work.
Why not?
Well, that's because Livecodingtv somehow doesn't have a status.
So, most of these work.
So now you'll get to see me fix something and when you download it, it will be fixed.
So, it says KeyError status.
So that means that I've got to do a thing that says,
if status not in U print no status found.
Continue. Since sometimes there's no statuses.
Who would have thought. I did not know that.
Okay, so let's run this again.
Did I get to see my remaining?
Oh, actually let me change the order of this.
Let me put this down here.
It'll be wrong from the slides but it'll be prettier now.
Let's put the headers after the dump of the data.
Okay. So let's run it again. Did I save it?
Yes. Dr. Chuck. A whole bunch of stuff.
So I got 13 remaining calls on this one.
So it is not the same as the other one.
I don't get to call this too many more times so hopefully I'll get the debugging to work.
I got a bad space here.
No status found.
And I need to put three spaces there.
No status found. I'll make an asterisk.
So let's run it again.
See I got 13 remaining.
So, it's important you write code that's aware of your remaining.
That's why I make it so obvious about that.
Go retrieve all that.
I got 12 remaining but I now have another space here.
Hang on I got to fix that.
I need yet another space.
Hopefully, I can make this as pretty as I want it to work.
Wait a second, I didn't even do Dr. Chuck, I got that wrong.
Typed my name wrong.
Okay, so now it works.
Oh well. So, now I have my five recent friends are stephteasley,
Liveeduofficial, Livecodingtv, Nancy Gillby and GreggyKrueger.
And so there are there are statuses and I tore all this Json apart using twitter2.py.
Of course, after fixing hidden.py,
which I'm not going to show you because it actually
contains my real consumer key and consumer secret,
you're seeing the consumer key and the token key go by on each of
these URLs but what you're not seeing
is these two things which are the thing I'm protecting,
so that it's not a problem.
Okay. So I will send that up, there you go.
Welcome. I hope you found this useful.
The code will be fixed when you take a look at and download it here from samplecode.zip.
Explorer notre catalogue
Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.
Coursera propose un accès universel à la meilleure formation au monde,
en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.
© 2019 Coursera Inc. Tous droits réservés.
