Worked Example: BeautifulSoup (Chapter 12)

Loading...

En provenance du cours de University of Michigan

Using Python to Access Web Data

17635 notes

Explore Related Videos

At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you

University of Michigan

Using Python to Access Web Data

17635 notes

Cours 3 sur 5 dans Specialization Python for Everybody

This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

À partir de la leçon

Programs that Surf the Web (Chapter 12)

In this section we learn to use Python to retrieve data from web sites and APIs over the Internet.

12.3 - Unicode Characters and Strings17:17

12.4 - Retrieving Web Pages6:36

Worked Example: Using Urllib (Chapter 12)3:39

12.5 - Parsing Web Pages9:28

Worked Example: BeautifulSoup (Chapter 12)9:20

Rencontrer les enseignants

Charles Severance
Professor
School of Information

[MUSIC]

Hello, everybody.

Welcome to Python for Everybody.

We're going to do a little bit of sample code.

If you're interested in getting the sample code,

you can download this ZIP here at https://www.py4e.com/materials.php.

And you will download and you will get all the files.

And all the files that I'm looking at here.

And so, the one I'm going to play with today is the file called urllinks.py.

So the first thing you gotta do before urllinks.py works is,

you have got to install BeautifulSoup.

And I've got some simple instructions at the beginning of the file.

And so one way to do it is install it using Python installed

process to install this BeautifulSoup for all Python applications.

And if you are the owner of your computer and

you're going to use BeautifulSoup a lot, it's a fine idea to do that.

But I want to show you a simpler way that if you don't own your own computer and

you just want to make it so that BeautifulSoup works.

You can download this file, this file right here,

BeautifulSoup4.zip.

Unzip it and put it in the same folder as here.

And so if you look in this folder, I have a subfolder called bs4.

And that's the unzipped version of this, and it has these things.

I didn't write this code, so I'm sorry if the name is bad,

but this is the code to BS4, and this is what's in bs4.zip.

And it's in the same folder as.

Urllinks.py.

And so what happens is, is when you do this from bs4 import BeautifulSoup,

that either can go to sort of this global magic place that Python installs stuff and

pulls in the BeautifulSoup object.

Or, it can go to the folder bs4 and pull it in, okay?

And so that's how that works, so you have to do one of these two things.

I prefer to keep it simple.

Download and unzip this file and

put it in the same folder as this code anywhere you go.

So from the previous example we're going to use urllib of course.

And then we're going to pull in the BeautifulSoup,

from the BeautifulSoup4 library we're going to get the BeautifulSoup object.

Now, if you do this with SSL, if these websites were going to play with SSL,

you pretty much have to do this little hack.

And these three lines, don't worry too much about it.

The whole idea, you could do Google on Stack overflow and figure this out.

But this is the way you ignore errors when you have SSL certificate errors.

And so

we have to add this parimeter, context=ctx which is this variable that we create.

So this part and this part sort of just do them.

If you don't you can take them out actually,

but otherwise you won't be able to do HTTPS site.

So let's take a look at what we're doing other than dealing with the HTTPS problem.

We're going to ask the user for a URL.

We are going to retrieve all the HTML.

We're going to do a URL open just like we did before.

Now this would return us something we could loop through line by line

with a for loop.

But instead, we're going to say, hey read the whole thing.

And that basically returns us the entire document at that web page.

In a single big string with new lines at the end of each line.

And this is not in Unicode but it's probably UTF-8 string.

But it turns out BeautifulSoup knows how to deal with UTF-8 and

it also knows how to deal with Unicode strings.

So what we're saying is BeautifulSoup read through and

deal with all the nasty bits, right?

So HTML is very very flexible.

So dr-chuck.com/page1.htm.

And so if we take a look at the source of this, view page source, make this bigger.

You might be able to do regular expressions but

it does things like breaks stuff across lines.

There could be a line break here, there could be all kinds of things, right?

And so writing a regular expressions or splits or

whatever is really hard for HTML.

And so what we do is someone has written this, it's called BeautifulSoup.

And it's basically, this is the code and

it's based on a joke from a children's story.

It basically someone has just went through and figured all the bad things that could

possibly happen when you're reading and parsing HTML.

So either you use it or you'll slowly but

surely derive all the things that it doesn't work.

And so when we look at this line right here, this line at a high level

is saying we're giving you ugly, nasty HTML that could make no sense whatsoever.

Please read it and have all the brains that you have and

all the weird stuff, figure that out for us and give us back an object.

I happen to call it soup, you don't have to call it soup.

An object, and that is a proxy for that HTML.

But this soup object is clean.

And so what we can do is we can sort of retrieve all the anchor tags.

So we can talk to this object and say ask it, give me the anchor tag.

What's an anchor tag?

Well, if we take a look at this source, the anchor tag is the a through the /a.

That is the tag, it is the tag.

It is attributes that are on the tag,

it is the text within the tag, and everything.

So that's what we're going to get.

Now, I called it tags plural, not because plural matters at all, but

because we're going to get a list of tags.

Because even though this webpage has lots and

lots of tags, if we look at say dr-chuck.com.

And, view source, woah, that's kind of small, view page source, right?

And we go look for anchor tags.

We got 45 of them and they all kind of have weird stuff in them, right?

So this line will give us back a list of tags.

It will give us all the tags in this document.

So it goes the tag goes from there to there.

And than we're going to do is we're going to write a loop to loop through

all the tags.

So that's basically hoping,

like it's hopping through the documents sort of like this, that's what it's doing.

Hop, hop, hop, hop, hop.

And it's pulling the text of the href attribute so

it's going to pull out this bit right here.

Woops, darn that was so cool cause that's a flaw.

Look at that.

This is my own page.

There is no closing quote here but it's going to work because HTML soup is like,

I know what to do about that, I can deal with that.

So let's check to see if that one works because that's like a mistake.

But that's one of the things we like about BeautifulSoup.

So we're going to read through and then we're going to pull out all the href.

So [LAUGH] this is probably thousands of lines of

code that you really don't want to run.

So Python 3 urllinks.py.

And so well let's start with a simple one,

http:// www.dr-chuck.com.

And it reads it.

No, that's actually the hard one because we've got a whole bunch.

So let's see if the tsugi one worked.

It found that one, it's right after sakaiproject.org.

Where is that?

Is there another Tsugi?

No, it didn't find that one, that's kind of funky.

Look, it found it wrong but that's okay.

So you see, it found all these and did a lot of nice stuff for us.

If we do it, python3 urllinks.py,

and do the easy one,

http://www.dr-chuck.com/page1.htm we

will only see one.

And there we go.

Now, the SSL is if you are looking at a page that has SSL.

Python, URL links too, so I'll go to

https://www.si.umich.edu, and that will get a bunch of links.

And so you'll see if it wasn't for that, so all kinds of stuff coming back.

And if it wasn't for this bit right here and

this bit right here this HTTPS wouldn't have worked.

And, it's not that that website had a bad URL,

it has a certificate that's not in Python's official list.

And so the URL is okay.

So that gives you a quick summary of using the BeautifulSoup

library in Python, along with the urllib.

[MUSIC]

[MUSIC]

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play