Worked Example: Twitter API (Chapter 13)

Loading...

En provenance du cours de University of Michigan

Using Python to Access Web Data

16260 notes

University of Michigan

Using Python to Access Web Data

16260 notes

Cours 3 sur 5 dans Specialization Python for Everybody

This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

À partir de la leçon

JSON and the REST Architecture (Chapter 13)

In this module, we work with Application Program Interfaces / Web Services using the JavaScript Object Notation (JSON) data format.

13.5 - JavaScript Object Notation (JSON)9:24

Worked Example: JSON (Chapter 13)5:50

Interview: Douglas Crockford - Discovering JSON7:07

13.6 - Service Oriented Approach2:37

Video: Service Oriented Architectures5:15

13.7 - Using Application Programming Interfaces8:11

Worked Example: GeoJSON API (Chapter 13)7:41

13.8 - Securing API Requests11:01

Worked Example: Twitter API (Chapter 13)19:41

Rencontrer les enseignants

Charles Severance
Associate Professor
School of Information

Welcome to Python for Everybody.

We're doing some sample code,

playing through with some sample code samples and you can get this by downloading it.

I've got this whole thing downloaded and I've got all the files here,

and these are the files we're going to play with today.

Today, what we're going to do is talk about the Twitter API.

The one thing we got to learn about the Twitter API is we have to authorize ourselves.

So, we have to make sure that we have a Twitter account and then we get some keys.

So, in this particular application if you want to duplicate what I'm doing,

you have to go to apps.twitter.com,

click this create new application button,

and then get some codes, okay,

and the codes show up as soon as you hit this button and then one more button,

which I'm not going to do on screen.

So, what happens is there are four codes that you got to put in this file hidden.py,

consumer key, the consumer secret,

token key, and token secret.

These are just messed up,

so I'll show you how this works and blows

up first and then I'll put my keys in here without showing you.

But basically, this is a little file you got to edit or these Twitter ones don't work,

you'll see what happens.

So, the first one I'm going to do is do the simplest one of all and that is,

I call this thing Twitter test and it just is going to

go ask for the user timeline and we

can take a look at this and we're going to take

the URL and we're going to augment the URL.

This is the base, we found this looking at the Twitter API documentation.

We're going to pass a parameter of screen name Dr. Chuck and account of two,

so this is just a Python dictionary.

An augment comes from this little bit of called code called twURL.

This uses a bit of code called OAuth,

which is built into Python as well, right?

Yeah. That's built into Python as well and it augments the URL.

It takes the the key, the secret,

the token key and does a thing and signs it,

and then makes this big, long,

ugly URL which you will soon see and does this,

it's a signature of the URL.

So, we pass this data back and forth to

Twitter with a signature and then they recheck the signature

and it's a digital signature that knows that this URL came from

a program that knows the key secret and token and token secret.

So, this augment basically,

is something that I wrote,

twURL augment is something I wrote to make it easier to add all these OAuth parameters.

You feed this code by putting your data into hidden.py.

Lots of people get this to work so,

don't worry it's cool when you finally get it to work.

So, let's take a look at what it does just know

that this makes an awesome URL that does all the security,

and we'll see one of those URLs.

So, ignore the certificate errors.

This has to do with the fact that we're using HTTPS and

Python doesn't have enough certificates put into it by default for a lot of reasons,

but our quick and dirty ways they turn them off.

Thank you Python for reducing security by teaching us,

so that this is the best way to do it,

that's a grumpy up moment from on my part.

So, what we're going to do is we're going to do a URL open.

This bit here is to shut off the security checking for the SSL certificate,

and then we're going to read all the data,

and then we're going to want to print it out.

We're also going to ask the connection this url.

Remember, I told you a long time ago

that URL lib eats the headers but you can get them back.

Now, we're going to ask to get a dictionary of the headers back,

and so we'll print those out, okay.

So, this is really just testing

the body and the headers and printing them out in as raw a way we can do.

So, let's go run this.

Now, this is going to fail the first time we do it because we

haven't put the hidden variables in there.

So, if I say python3twtest.py,

it's going to run and blow up and it's going to give you this 401 authorization required.

That's a good sign because that means that you haven't yet

updated your values in hidden.py.

So, this is that augmented URL and you can

see the consumer key and consumer secret and the OAuth token and whatever, okay.

So, these tokens are wrong.

These aren't Control-C, they aren't real.

But you'll notice it doesn't end the key and the secret of

the token key of the token secret and the secret.

That's all actually encoded in the signature.

It turns out that you need to have the key and the secret and

the token secret to generate the signature and where is the signature?

There's the signature, right there's the signature.

So, this signature combined with the notes,

that you can only do this signature as a time and includes all kinds of things.

So, even if you type this in, well,

you'll see these go by and it's

not really breaking my security too much when you see these afterwards.

So, don't get all excited when you say,

"Oh, you revealed your token and your key."

Now, I can reveal my token and key but I'm not going to reveal the secret.

So, this adds all this OAuth stuff,

OAuth nonce, OAuth timestamp.

These timestamps and nonces are made it so that you can't

replay my URL even if you see the exact URL.

Once I hit it, then you can't hit it again and so,

that's what the nonce does.

So, I'm going to close hidden.py here.

I'm going to update hidden.py in another window,

okay. So, I just in another window, I updated hidden.py.

I'm not going to show you that.

But now, I'm going to run python3twtest.py.

So, twURL is going to read hidden and now,

these keys and secrets are my real ones that I haven't shown you.

So, this should work, fingers crossed.

It worked, okay.

So, it worked.

So, I'm calling Twitter.

Here's the URL.

Now, don't worry, the token and the consumer key are not enough

to break into my account and neither is the signature because you can't replay this.

In about five minutes,

you can't replay this anymore, okay.

So, you can't generate the signature.

I've done one, the signature includes the time and date,

so you can't trust me.

Go read up on OAuth,

don't worry, I haven't really revealed anything.

But, so, the first thing we see is this.

So, we see, and we should put it like the line and dashes here.

This is the Json.

It ain't very pretty, it's not very pretty.

Okay, and so that's the Json, from there to there.

It's just what most APIs give us back,

it's really dense Json, right.

So, this is a byte array.

Remember, how you have to do a.decode,

I didn't do a.decode here.

So, this is telling, Python is telling us this is a byte array which it's

a raw set of bytes that came from the Internet,

which probably are UTF-8.

If I input a decode here,

then it would decode.

If say.data.decode there, then it would be fine.

But we don't care,

this was just a dump, do we get anything?

So, then, here let's do this, print.

I'll just make this code different.

Put some equal signs here,

a lot equal signs.

So, we can easily see where the thing starts and stops.

So, we'll run that again if you look at those URLs.

So, that was all of that stuff and then this is the headers,

and so the headers again are not pretty.

You get the headers it's a dictionary,

you got cache-control, no cash comma.

This is the string key value,

you got to find your commas key value,

but the one that's really interesting here is which one is it?

X-rate-limit-remaining, right there, x-rate-limit-remaining remaining.

So, that means that for this particular API in this header

tells me that I've got 898 calls left,

and this is when I will get more calls and so watch.

I'm going to do this again and you will see that I can only do this 897 more times now.

Run it. I can only do this 897.

So, I am being tracked at this point.

I'm being tracked by Twitter.

Twitter knows that it's Dr. Chuck that's doing this and Dr. Chuck is done 900,

he's done 899, 897,

and if I keep running this,

eventually Twitter will tell me,

"You got to wait for a while",

and that's because Twitter doesn't want me under my Dr. Chuck account

pulling out lots and lots of stuff out of Twitter and making my own website.

I do actually have my own twitter website using some cool software

www.doctorchuck.com/Twitter

and this I have to run in rate limits and causes all kinds of you know, whatever.

So, okay the rate limit.

So, I'll save that.

So that's tweet, this is just a test, okay?

Because I want to do something interesting,

so we're not parsing the Json that comes back,

we're not doing anything tricky with this and away we go.

So, let's take a look at some more code.

I think I don't need this anymore.

So, now I am going to parse this.

So, most of this looks the same.

I've got that same user timeline Json,

I'm going to ignore the SSL certificate someone write a loop.

So, I'm going to ask the Twitter I'm going to print,

I'm going to get a Twitter account and quit if it's a blank line or phase to entering.

I want to use the Twitter URL augment the same way that's

going to do all the signing using from hidden.py and

retrieve it and I'm going to retrieve it ignoring

the SSL errors and then I'm going to decode.

This time I'm going to decode it so that I get a real Unicode string,

and I'll print the first 250 characters of it,

I'm going to grab the headers,

and I'm going to print the remaining, the rate limits.

This is sort of a very simple version of this same thing.

It really is decoding the data and only printing the first 250 characters.

So, let's run that, Dr. Chuck.

Boom, and it's got 896.

So that's just a little simpler version that with a little less brutal debugging.

Okay, so now let's do something even more fun.

Let's go to twitter2.py and tear it apart.

So again, we're going to look at my friends list or someone else,

any of his buddies' friends list.

We're going to ask for the friends and ask for the screen name,

ask for the first five friends,

and then look at their statuses,

open it, decode it,

get the headers, print the rate remaining.

All this stuff is the same as in twitter1 but now we're going to parse the JavaScript.

I'm not even putting this in a try and except because,

hey, I'm talking to twitter.

I'm going to guess that twitter is going to give me the right stuff.

You probably want to put a try and except here.

Then I'm going to do a debug print.

I'm going to do a Json pretty print.

Let's make that to be two, so it looks a little better.

And then, I'm going to run it and then you're going to see how we have to parse this.

We're going to see that it's a list.

So, we're done with that and now we're running twitter2.py.

So, I'm going to go to Dr. Chuck and this is going to ask the question,

who Dr. Chuck's friends are?

Okay, let's go to the top.

So it hit this API and it has the screen name Dr. Chuck,

count equals five and all this oauth stuff.

Again, this is not a security breach by showing you all of this because the signature,

the secrets aren't there.

Okay, so if we look at it,

it's an outer object or dictionary and then the outer has a users which is a list.

Then, each user has some stuff in it.

So this one's Stephanie Teasley.

It's got her screen name,

it's got some descriptions.

Keep on going and it's got her status,

her latest status, for my friend her status.

Her source, where she's at.

Man she's got a lot of stuff here.

Okay. There we go, that was the first one and then

the next one that I'm following is LiveEdu etc.

So you'll see that this is an array.

So that outer thing is an array of users.

Now, JS here is a dictionary.

So I can say for you in JS sub-users.

Well JS sub-users is a list.

So the first U is going to be

this Stephanie Teaseley U and the second U is going to be LiveEdu.

So, that's all it took to get through all that stuff and figure that out.

And then I'm going to say, get me the screen name of my person.

So let's go in here.

So that's going to pull Stephanie Teasley as stephteaseley out then.

Then I'm going to go find her status.

Let's find her somewhere in here.

U sub status, sub text. Come on.

Okay, there's sub status,

sub status is all this stuff. More and more.

Right there, that status,

that U sub status is that and then U sub status,

sub text is this step.

So it's going to extract this bit right here, okay.

So, use that as text and I print out the first 50 characters of the screening status,

and I do that for the first five because I told it I only wanted five.

And then of course I get to see that the right limits.

So let's go down to the bottom.

So all of this is the debug print of the Json I got back.

Here is the program starting to print.

Here is the screen name of my first friend and

here's the first 50 characters of her most recent status.

Here is the screen name of my,

and these are in reverse order,

who I've been following.

So I've been playing with this live coding stuff.

So I'm following them.

What!

KeyError status. That didn't work.

Why not?

Well, that's because Livecodingtv somehow doesn't have a status.

So, most of these work.

So now you'll get to see me fix something and when you download it, it will be fixed.

So, it says KeyError status.

So that means that I've got to do a thing that says,

if status not in U print no status found.

Continue. Since sometimes there's no statuses.

Who would have thought. I did not know that.

Okay, so let's run this again.

Did I get to see my remaining?

Oh, actually let me change the order of this.

Let me put this down here.

It'll be wrong from the slides but it'll be prettier now.

Let's put the headers after the dump of the data.

Okay. So let's run it again. Did I save it?

Yes. Dr. Chuck. A whole bunch of stuff.

So I got 13 remaining calls on this one.

So it is not the same as the other one.

I don't get to call this too many more times so hopefully I'll get the debugging to work.

I got a bad space here.

No status found.

And I need to put three spaces there.

No status found. I'll make an asterisk.

So let's run it again.

See I got 13 remaining.

So, it's important you write code that's aware of your remaining.

That's why I make it so obvious about that.

Go retrieve all that.

I got 12 remaining but I now have another space here.

Hang on I got to fix that.

I need yet another space.

Hopefully, I can make this as pretty as I want it to work.

Wait a second, I didn't even do Dr. Chuck, I got that wrong.

Typed my name wrong.

Okay, so now it works.

Oh well. So, now I have my five recent friends are stephteasley,

Liveeduofficial, Livecodingtv, Nancy Gillby and GreggyKrueger.

And so there are there are statuses and I tore all this Json apart using twitter2.py.

Of course, after fixing hidden.py,

which I'm not going to show you because it actually

contains my real consumer key and consumer secret,

you're seeing the consumer key and the token key go by on each of

these URLs but what you're not seeing

is these two things which are the thing I'm protecting,

so that it's not a problem.

Okay. So I will send that up, there you go.

Welcome. I hope you found this useful.

The code will be fixed when you take a look at and download it here from samplecode.zip.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play