Challenges

Loading...

En provenance du cours de University of Maryland, College Park

Combining and Analyzing Complex Data

26 notes

University of Maryland, College Park

Combining and Analyzing Complex Data

26 notes

Cours 6 sur 7 dans Specialization Survey Data Collection and Analytics

In this course you will learn how to use survey weights to estimate descriptive statistics, like means and totals, and more complicated quantities like model parameters for linear and logistic regressions. Software capabilities will be covered with R® receiving particular emphasis. The course will also cover the basics of record linkage and statistical matching—both of which are becoming more important as ways of combining data from different sources. Combining of datasets raises ethical issues which the course reviews. Informed consent may have to be obtained from persons to allow their data to be linked. You will learn about differences in the legal requirements in different countries.

À partir de la leçon

Record Linkage

Module starts with the current debate on using more (linked) administrative records in the U.S. Federal Statistical System, and a general motivation for linking records. Several examples will be given on why it is useful to link data. Challenges of record linkage will be discussed. A brief overview over key linkage techniques is included as well.

Gentle Introduction9:45

Rencontrer les enseignants

Richard Valliant, Ph.D.
Research Professor
Joint Program in Survey Methodology

Welcome, so

let's talk a little about a bit about the challenges in record linkage.

[COUGH] No matter how big the size of your records is, we have

some traditional challenges when linking records from multiple data sources.

One big one is the lack of unique identifiers.

Let's take for example, Julie Lane.

A dear colleague of mine professor at the Wagner School in New York at NYU.

And if I searched for her on the Internet using Google image search,

immediately, two other pictures appear.

They've share a feature with her,

all three are females, but clearly different people up here.

Not surprisingly, because Julia Lane is not that of a unique name.

The same can happen with firms.

Summit Consulting is a statistical consulting company here in DC.

But if we look for

Summit Consulting on the Internet, we'll find another company in Texas.

So the lack of unique identifiers across different files can

be a real challenge for

any one who wants to make sure that the same entities are matched to each other.

Another challenge, what's often referred to as dirty data.

Data can be quite messy and dirty due to typographical errors,

variations in the record.

So let's assume that, for example, marital status changed over time, or

someone moved and the address will be different in different records.

Even so, it's the same entity.

Values can be out of date in a similar way.

And of course, values can be missing.

In which case, we hope that we can insert them from one database to the next.

Different coding schemes often appear when we deal with dates.

That can be a big problem, because dates are recorded in one format,

in one data source, and in a different format in the next.

On the right hand side, you see some examples here.

John Smith's being spelled differently, capitalized and not,

Charles abbreviated to Chuck, same with William and

Bill, often the same person in data sets, and the list goes on.

The street names too, can have very different ways they are recorded.

Wrote being abbreviated drive, and so on.

And you see, in this example, a couple of different ways to write down dates as they

might appear in different databases.

These examples on the right are taken from a NIH resource.

The full reference is given on the bottom of the slide and

you can look at this online book yourself.

Another challenge that we aware of and often happens is,

the issue of privacy and the nature of the data that want to be linked.

So on the right hand side you see a chart from the Eurobarometer.

Where people were asked,

how sensitive they would consider the following information to be.

And you can see that, financial information,

medical records are considered to be

sensitive by a large fraction of the respondents in this survey.

In many countries, you have to give informed consent and

we have a separate manual on that.

And of course, the use of the data will affect of all these issues.

In the big data context, a couple of new challenges arise.

One important one is scalability.

So the naive compares in a one record to another, will explode.

You have a lot of different records in all of these databases.

And if you think doing pair wise comparison to see if they are the same or

not, that will take a long time.

So we need to move to an approach that's much more efficient.

And research needs to be done too, into those techniques.

Also, sometimes we have networks and complicated relationships that you

might need to take into account when matching these records.

And mostly a big challenge is that, we often do not know what is the true entity.

So there aren't really good training set's available for

the algorithms to learn what is the correct match and what is not.

So this is why there is a lot of new research done in this field currently.

The extended record linkage process, as depicted by Bender and

his colleagues, is therefore not one away, just compare data filing in B.

But where there's a lot of preprocessing going on in

both files before you compare them.

So the preprocessing could include the passing of individual names,

correction of typos, anything of that nature.

Ideally, you also reduce the search space.

So if you know that someone is Chicago, maybe you can limit

the search just to the area of Chicago and this is referred to as blocking.

And then you do your comparison within those smaller sets before you classify

them as true links or no links.

Now to put this in perspective, it is the case,

just like with a lot of analysis that we do that, that 75% of the work,

according to Gill in 2001, amounts to this cleaning and passing step.

20% is roughly what the matching, checking,

the matching process is correct, and only 5% goes to the actual linkage effort.

The importance of preprocessing can't be overstated.

I'd like you to read these two quotes by Winkler, by yourself.

So one last note on identifiers.

Typically, they have first and last name, address, birth date and the like.

And you might have across these different records, different identifiers.

As I said, if they are from multiple sources, you might see changes,

different spellings and so on.

It's a good thing to not throw them away,

even if you decide that there's sort of a core piece that belongs to all of them.

Because, If you add additional data sources,

it could be helpful to have these original identifiers to then

later on link it to one of these different varieties in your data set.

So variations within a given unit is often possible,

can arise almost anywhere.

And as I said, it's helpful to keep those.

In summary, Christen for

that reason makes this process even more complicated, I would say.

Where you allow for a couple of feedback loops looping at the matches,the

non-matches, and potential matches and a clinical review and

then see if you can improve the process going forward.

How you get to matches and non-matches and potential matches,

we'll address in the next segment.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play