Explore
For Enterprise
Join for Free
Log In

Parcourir
Rechercher
For Enterprise
Connexion
Inscrivez-vous gratuitement

Data-Intensive Science Examples

Loading...

Data Manipulation at Scale: Systems and Algorithms

Université de Washington

4.3 (698 notes) | 49K étudiants inscrits

Cours 1 de 4 dans Science des données à grande échelle Spécialisation

Data analysis has replaced data acquisition as the bottleneck to evidence-based decision making --- we are drowning in it. Extracting knowledge from large, heterogeneous, and noisy datasets requires not only powerful computing resources, but the programming abstractions to use them effectively. The abstractions that emerged in the last decade blend ideas from parallel databases, distributed systems, and programming languages to create a new class of scalable data analytics platforms that form the foundation for data science at realistic scales. In this course, you will learn the landscape of relevant systems, the principles on which they rely, their tradeoffs, and how to evaluate their utility against your requirements. You will learn how practical systems were derived from the frontier of research in computer science and what systems are coming on the horizon. Cloud computing, SQL and NoSQL databases, MapReduce and the ecosystem it spawned, Spark and its contemporaries, and specialized systems for graphs and arrays will be covered. You will also learn the history and context of data science, the skills, challenges, and methodologies the term implies, and how to structure a data science project. At the end of this course, you will be able to: Learning Goals: 1. Describe common patterns, challenges, and approaches associated with data science projects, and what makes them different from projects in related fields. 2. Identify and use the programming models associated with scalable data manipulation, including relational algebra, mapreduce, and other data flow models. 3. Use database technology adapted for large-scale analytics, including the concepts driving parallel databases, parallel query processing, and in-database analytics 4. Evaluate key-value stores and NoSQL systems, describe their tradeoffs with comparable systems, the details of important examples in the space, and future trends. 5. “Think” in MapReduce to effectively write algorithms for systems including Hadoop and Spark. You will understand their limitations, design details, their relationship to databases, and their associated ecosystem of algorithms, extensions, and languages. write programs in Spark 6. Describe the landscape of specialized Big Data systems for graphs, arrays, and streams

Visualiser le programme de cours

Compétences que vous apprendrez

Relational Algebra, Python Programming, Mapreduce, SQL

Avis

4.3 (698 notes)

5 stars
397 ratings
4 stars
183 ratings
3 stars
67 ratings
2 stars
33 ratings
1 star
18 ratings

DK

Jan 24, 2016

Good! I like the final (optional) project on running on a large dataset through EC2. The lectures aren't as polished and compact as they could be but certainly a very valuable course.

LG

Jun 20, 2017

Comprehensive and clear explanation of theory and interlinks of the up-to-date tools, languages, tendencies. Kudos and thanks to Bill Howe.\n\nHighly recommended.

À partir de la leçon

Data Science Context and Concepts

Understand the terminology and recurring principles associated with data science, and understand the structure of data science projects and emerging methodologies to approach them. Why does this emerging field exist? How does it relate to other fields? How does this course distinguish itself? What do data science projects look like, and how should they be approached? What are some examples of data science projects?

A Fourth Paradigm of Science3:53

Data-Intensive Science Examples6:00

Big Data and the 3 Vs5:16

Big Data Definitions4:44

Big Data Sources6:59

Enseigné par

Bill Howe
Director of Research

Transcription

[MUSIC] Now, the problem is that the same technology stack and to some extent even the same approach is difficult to apply in the case of the Large Synoptic Survey Telescope. The reason why this guy is producing so much more data, is not just that it's much higher resolution and can perceive a much deeper field in the sky, but also because it's returning to the same point in the sky, frequently, every three days. And so, this allows you to look at things that change over time. So asteroids, comets, a supernova, and so forth. Okay, and by comparing these images in the time series, there's all sorts of new questions you can ask. So both because of the science they are gonna do, and because of the shear scale, and because some of the complexity of the details of how the data is acquired. The previous solution wont work, and so this is motivated in a whole new area of research, to study data management techniques and data analysis techniques to support this project. So in life sciences, these high through put sequencers are capable of producing terabytes per day when running continuously. And major labs that do this work such as the McDonnell Genome Institute have 25 to 100 of these machines running all the time. So this is spitting out an enormous amount of data for I was gonna say variety of samples. So it may be individual organisms or it could even be samples from the environment where there's no one particular organism in there, but there's entire population. So for a variety of uses, these guys are able to spit out the data. In oceanography, the regional scale modes of the NSF Ocean Observatories Initiative is a project led here at U-Dub. Ocean Observatories Initiative is a multi-institutional partnership. The Regional Scale Nodes is run at the University of Washington. So this project is involving 1,000 kilometers of fiber optic cable on the seafloor connecting thousands of instruments in chemical, physical and biological. Thousands of chemical, biological, and physical sensors, including live video from the sea floor to monitor volcanic activity. Okay, so again, the database [INAUDIBLE] the data sets and data infrastructure required for this effort is significant. It has motivated a lot of new research. In the information space, there's a lot of science to be done directly on the web itself. And so just the web, a single computer can read 30 to 35 megabytes per second from one disk. And so it would take about four months just to read the entire Web. So summing it up a little bit, eScience is about the analysis of data, so the automated or semi-automated extraction of knowledge from massive volumes of data, and so your main instrument for looking for answers are the algorithms and the technology as opposed to direct inspection. There's just too much of it to look at yourself, but it's not just a matter of volume, as we'll talk about in the next segment. This is another link back to what's going on in business. There's this concept of big data and there's the three V's of Big Data that we'll talk about a little bit more next time, but let me just mention them here. The three V's are volume, variety, and velocity that you'll read about. And I'll give you where those terms came from in the next segment. The volume refers to just simply the number of rows or the number of bytes, the sheer scale. Variety is perhaps the number of columns or dimensions, but in science, for example, a lot of, in the life sciences in particular, you'll have experiments that involve accessing multiple public databases, as well as multiple sensors, your own data that you've collected and that of your colleagues, and the integration task of putting all this data together is a significant bottleneck. Even with the when the actual scale of the data is not necessarily all that bad. The complexity of the data. And then, the velocity. We start with a large optic survey telescope that, although the scale itself is enormous the fact that 40 terabytes are being collected every two days means that the infrastructure needs to keep up with that pace. And just transferring that data from the telescope facility to the data analysis facility is an engineering challenge. And you'll also see other V's here with things like veracity, can we actually trust this data. So a bit more of that next time. So to summarize here, science is in the midst of a generational shift from a data poor enterprise where there's never enough data to a data rich enterprise where there's so much of it, you don't know what to do with it. And as a result, data analysis replaced data acquisition as the new bottleneck to discovery. So it's not that the cost of going out and getting the data, it's the cost of actually analyzing the data you might already have. So this fine, but what does this have to do with business, which is probably where a lot of you are coming from, and where your interests lie. Well, what we see is that business is beginning to look a lot like what's always been happening in science. So businesses are acquiring data aggressively and keeping it around indefinitely in case it becomes useful. They're beginning to hire people that have training and skill sets that look a lot like what's been important in science for a long time especially in mathematical depth. And they're beginning to make decisions with this data that are very empirical. So always wanting to sort of backup every decision with a ClearCase based on data. And so for these reasons, I think that you can take the lessons learned in science and apply them in business, and actually, vice versa, as well, because one thing where science is lagging behind business is in the adoption of technology. There's been proportionally a lot less spent on IT infrastructure in science than there has in business, and so it's a great time for this. There's this cross-pollination of ideas between both fields. Okay, and so going back to the first slide that I gave, eScience and Data Science have essentially everything in common. So we might use examples interchangeably between the two. [MUSIC]

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2019 Coursera Inc. Tous droits réservés.

Disponible sur Google Play

Coursera
À propos
Direction
Carrières
Catalogue
Certificats
Certificats MasterTrack™
Diplômes
For Enterprise
For Government
Pour le campus

Communauté
Learners
Partners
Developers
Beta Testers
Translators

Se connecter
Blog
Facebook
LinkedIn
Twitter
Instagram
YouTube
Blog Tech

Plus
Conditions
Confidentialité
Aide
Accessibilité
Presse
Contact
Répertoire
Filiales