Worked Example: Twitter API (Chapter 13)

Loading...

En provenance du cours de University of Michigan

Using Python to Access Web Data

14452 notes

University of Michigan

Using Python to Access Web Data

14452 notes

Cours 3 sur 5 dans Specialization Python for Everybody

This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

À partir de la leçon

JSON and the REST Architecture (Chapter 13)

In this module, we work with Application Program Interfaces / Web Services using the JavaScript Object Notation (JSON) data format.

13.5 - JavaScript Object Notation (JSON)9:24

Worked Example: JSON (Chapter 13)5:50

Interview: Douglas Crockford - Discovering JSON7:07

13.6 - Service Oriented Approach2:37

Video: Service Oriented Architectures5:15

13.7 - Using Application Programming Interfaces8:11

Worked Example: GeoJSON API (Chapter 13)7:41

13.8 - Securing API Requests11:01

Worked Example: Twitter API (Chapter 13)19:41

Rencontrer les enseignants

Charles Severance
Associate Professor
School of Information

[MUSIC]

Welcome to Python for Everybody.

We're doing some code, sample code,

playing through it with some sample code samples.

And now you can get this by downloading it.

I've got this whole thing downloaded and I've got all the files here.

And these are the files we're going to play with today.

Today, what we're going to do is talk to you about the Twitter API.

And the one thing we gotta learn about the Twitter API,

is we have to authorize ourselves.

And so, we have to make sure that we have a Twitter account and

then we get some keys.

And so for- in this particular application,

if you want to duplicate what i'm doing, you have to go to apps.twitter.com.

Click this create new application button and then get some codes, okay?

And the codes show up as soon as you hit this button and then one more button,

which I'm not going to do on screen.

And so,

what happens is there are four codes that you gotta put in this file hidden.py,

the consumer_key, consumer_secret, the token_key, and token_secret.

These are just messed up, so I'll show you how this works and

blows up if first, and I'll put my keys in here without showing you.

And but basically there's this little file you've got to edit or

these Twitter ones stuff don't work, you'll see what happens.

So, the first one I'm going to do is, do the simplest one of all.

And that is, I call this thing Twitter test and it is just going to go ask for

the user timeline and we can take a look at this.

And we're going to take the URL, and we're going to augment the URL.

This is the base, we found this looking at the Twitter API documentation.

We're going to pass a parameter of screen_name drchuck and count of 2, so

this is just a Python dictionary.

And augment comes from this little bit of code called TWURL.

And this uses a bit of code called OAuth which is

built into a Python as well right, yeah.

Let's built in a Python as well and it augments the URL and

it takes the key, the secret, the token key, and does a thing and signs it.

And then makes this big long ugly URL,

which you will soon see, and it's the signature of the URL.

So, we pass this data back and forth to Twitter with a signature and and a name,

recheck the signature and it's a digital signature that knows that this URL came

from a program that knows the key secret and token and token secret.

And so this augment basically is something that I wrote TWRL

augment is something I wrote to make it easier to add all these oauth parameters.

And you feed this code by putting your data into hidden.py.

Lots of people get this to work, so

don't worry it's kind of cool when you finally get it to work.

So, let's take a look at what it does, just know that this makes an awesome

URL that does all the security, and we'll see one of those URLs.

So, ignore the certificate errors.

This has to do with the fact that HT, that were using HTTPS and

Python doesn't have enough certificates put into it by default for

a lot of reasons, but our quick and dirty way is they turn them off.

Thank you Python for reducing security by teaching us, so

that this is the best way to do it.

That's a moment from on my part.

So, what we're going to do is we're going to do a urlopen.

This bit here is to shut off the security checking for the ssl certificate.

And we're going to read all the data and then we're going to just print it out and

we're also going to ask the connection does URL.

Remember I told you a long time ago that urlib eats the headers, but

you can get them back.

And now we're going to ask to get a dictionary of the headers back and so

we'll print those out.

Okay, so this is really kind of just testing the body and the headers and

printing them out, sort of in as raw a way we can do.

So, lets go run this.

Now, this is going to fail the first time we do it,

because we haven't put the hidden variables in there.

So, if I say python3 twtest.py it's going to run and blow up and

it's going to give you this 401: Authorization Required.

That's a good sign,

because that means that you haven't yet updated your values and hidden.py.

And so, this is that augmented URL.

And you can see the consumer key and the consumer secret and

the oauth token and whatever.

Okay, so these tokens are like wrong, these aren't, oops.

Control C, are they aren't real?

But you notice that there in key in the secret, are the token keys and the secret,

that's all actually encoded in this signature, it turns out that you need to

have the key and the secret and the token secret to generate the signature.

And where is the signature?

There's the signature, right there's the signature.

And so, this signature combined with a nuance that you can

only do this signature as a time and it includes all kinds of things.

So, even if you type this in, you'll see these go by, and

it's not really breaking my security too much when you see these afterwards.

So, don't get all excited when you say you revealed your token and your key.

Now, I can reveal my token and key, but I'm not going to reveal the secret.

Okay, so this adds all this oauth stuff, oauth_nonce,

oauth_timestamp, and these timestamps and nonces are mated, so

that you can't replay my URL, even if you see the exact URL.

Once I hid it, then you can't hid it again, so that's what the nonce does.

So I'm going to close hidden.py here and

I'm going to update hidden.py in another window.

Okay, so I adjust in another window, I updated hidden.by,

I'm not going to show you that.

But now I'm going to run Python twtest.py, so twrl it's going to read hidden and

now these keys and secrets are my real ones, that I haven't shown you.

So, this should work, fingers crossed.

Yehey, it worked, okay so it worked.

So I'm calling twitter, here's the URL.

Now don't worry, the token and the consumer key, are not enough to

break into my account and neither is the signature because you can't replay this.

In about five minutes, you can't replay this anymore, okay,

so You can't generate the signature.

I've done one, the signature includes the time and date, so you can't, trust me.

Go read up on don't worry.

I haven't really revealed anything, but so the first thing we see is this.

So, we see and we should have put like a line of dashes here, this is the JSON.

It ain't very pretty, it's not very pretty.

Okay and so that's the JSON from there to there.

It's just what most API give us back it's really dense JSON, right?

And so this is a bite array remember, how you have to do a dot decode.

I didn't do a dot decode here and so this is telling.

And Python is telling us this is a byte array, which it's a raw set of bytes

that came from the internet which probably are UTF-8.

And if I put a decode here, then it would decode.

If I say dot, data dot decode there then it would be fine, but

we don't care that was just dump do we get anything and so then, and

let's do this print just make this code different put

some lot equal sign here, a lot of equal signs so we can easily see where the.

where the thing starts and stops, so we'll run that again.

If you'll look at those URLs, so that was all of that stuff and

then this is the headers.

And so the headers, again, are not pretty.

To get the headers it's a dictionary, you got cash control, no cash,

comma this is the string key value.

You have find your commas key value.

But the one that's really interesting here is, which one is it?

x-rate-limit-remaining, right there, x-rate-limit-remaining.

So, that means for this particular API, and

this header tells me that I've got 898 calls left.

And this is when I will get more calls and yeah, so let's see, yeah.

So watch, I'm going to do this again and you will see that I can only

do this 897 more times now, run it I can only do this 897.

So, I am being tracked at this point.

I am being tracked by twitter.

Twitter knows that it's Dr. Chuck that's doing this and, Dr.

Chuck has done 900, he's done 899, 897.

And if I keep running this eventually Twitter will tell me,

you gotta wait for a while.

And that's because Twitter doesn't want me, under my Dr Chuck account,

pulling out like lots and lots of stuff out of Twitter and making my own web site.

I do actually have my own Twitter web site, using some cool software.

www.dr-chuck.com/twitter

And this I have to run and it breaks limits and causes all kinds of,

whatever, so.

Okay so, rate limit so, I'll save that, so

that's tweet, this is just to test it.

Okay, because we're doing, I want to do something interesting.

So, we're not parsing the JSON that comes back.

We're not doing anything tricky with this and away we go.

So, let's take a look at some more code,I think I don't need this anymore.

So, now I am going to parse this.

So, most of this looks the same, I've got that same user timeline in JSON.

I'm going to ignore the SSL certificates, I'm going to write a loop.

So I'm going to ask the Twitter.

I'm going to print, I'm

going to get a Twitter account and quit if it's a blank line or [INAUDIBLE].

I'm going to use the Twitter URL augment the same way.

That's going to do all the signing using from hidden.py or retrieve it.

And I'm going to retrieve it, ignoring the SSL errors and

then I'm going to decode it.

This time I'm going to decode it so I get a real Unicode string.

And I'm going to print the first 250 characters of it.

I'm going to grab the headers and

I'm going to print the remaining, the rate limits.

So, this is sort of a very simple version of this same thing.

It really is decoding the data and only print the first 250 characters,

so let's run that.

Dr chunk boom and has got 896, so let's just a little

simple version a little less a brog to debugging.

Okay,so now let do something even more fun we going to go to twitter2.pi and

tear it apart, and so again we going to look at my friends list.

Or someones else anybody's friends list we're going to ask for the friends and

ask for the screen name, ask for the first five friends.

And then look at their statuses, open it, decode it,

get the headers, print the remaining.

All this stuff is the same as in Twitter one.

But now, we're going to parse the JavaScript.

I'm not even putting this in to try and accept because,

hey, I'm talking to Twitter.

I'm going to guess that Twitter's going to give me the right stuff.

You'll probably want to put a try and accept here.

Then, I'm going to do a Debug Print.

I'm going to do a JSON Pretty print, let's make that be 2 so it looks better.

And then, we'll I'm going to run it, and

then you're going to see how we have to parse this.

So we're going to see that it's a list.

So, we're done with that and now we're running twitter2.py.

So, I'm going to go to drchuck, and

this is going to ask the question who drchuck's friends are?

Okay, let's go to the top.

So, it hit this API and it has the screen

name drchuck count equals five and all this oauth stuff.

Again, this is not a security breach by showing you all of this

because the signature, the secrets aren't there okay, so

if we look at it, it's an outer object or

dictionary and then the outer has a users which is a list.

And then each user has some stuff in it.

So, this one's Stephanie Teasley, it's got her screen name.

It's got some descriptions, keep on going.

It's got her status, her latest status.

For my friend, her status, her source.

Where she's at, man man, she's got a lot of stuff here.

Okay, there we go, that was the first one.

Okay, and then the next one that I am following is Live EDU, etcetera.

And so you]'ll see that this an array.

So, that outer thing is an array of users.

Now, js here is a dictionary.

So, I can for u in js sub-users, js sub-users is a list.

So, the first u is going to be this Stephanie Teasley u, and

the second u is going to be live EDU, so

that is all it took to get through all that stuff and figure that out.

And then I am going to say, get me the screen name of my person, so

let's go in here.

So that is going to pull Stephanie Teasley as "stephteasley" out.

Then I am going to go find her status, lets find her somewhere in here.

The use of status of text, come on.

Okay, there is sub status, sub status is all of this stuff.

More, more, more, more, more, right there, that's status,

that's use of status is that, and then use of sub status of text,

is this stuff so it's going to extract this bit right here, okay.

And so u status text and

I print out the first 50 characters of the screen name status.

And I do that for the first five because I told it I only wanted five.

And then of course I get to see the rate limit.

So, let's go down to the bottom.

So, all of this is the debug print of the JSON I got back.

Here is the program starting to print.

Here is the screen name of my first friend, and

here is the first 50 characters of her most recent status.

Here is the screen name and these are in reverse order of who I've been following.

So I've been playing with this live coding stuff so I'm following them, what.

Key error s that didn't work.

Why not?

That's because live coding tv, somehow doesn't have a status.

So most of these work.

So,now you'll get to see me fix something.

And when you download it, it'll be fixed.

And so it says KeyError status, so

that means that I've gotta do a thing

that says, if status not in uprint.

No status found.

Continue, [LAUGH] Since sometimes there's no statuses.

Who would have thought, I did not know that.

Yep, so you Okay,

so let's run this again.

Do I get to see my remaining?

Actually let me change the order of this, let me put this down there.

That will be wrong from the slides but it'll be prettier now.

Let's put the headers after the dump of the data.

Okay, so let's run it again.

Did I save it?

Yeah, drchuck, blah, a whole bunch of stuff, so I've got

13 remaining calls on this one, so it's not the same as the other one, I don't get

to call this too many more times, so hopefully I'll get the debugging to work.

[LAUGH] Sort of,

I got a bad space here, no, not status found, no status found.

And I need to put three spaces there, no status found, I'm making asterisk.

So let's run it again, see I got 13 remaining.

So, it's important you write code that's aware of your remaining.

That's why I make so obvious about that.

But retrieve all that, I got 12 remaining but my code starts to look.

Dang it, I had another space here, hang on got to fix that.

Leave yet another space.

Hopefully, I can make this as pretty as I want it to work.

Wait a sec [LAUGH] I didn't even do drchuck, are you that wrong.

Type my name wrong okay, so now it works.

(Laughing) Well so, now I have my first

most five recent friends are stephteasley, liveeduofficial, Livecodingtv,

nabgilby And GreggyCrueger, so there are their statuses and

I tore all this JSON apart using twitter2.py.

Of course, after fixing hidden.py, which I'm not going

to show you because it actually contains my real consumer_key and consumer_secret.

You're seeing the consumer_key and the token_key go by on each of these URLs, but

what you're not seeing is these two things,

which are the thing I'm protecting, so that it's not a problem.

Okay, so I will send that up, but there you go.

Welcome, thank, I hope you found this useful.

The code will be fixed when you take a look at it, and download it here,

from samplecode.zip.

[MUSIC]

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play