After taking this course, you will be able to describe two different approaches to converting raw data into analytics-ready data. One approach is the Extract, Transform, Load (ETL) process. The other contrasting approach is the Extract, Load, and Transform (ELT) process. ETL processes apply to data warehouses and data marts. ELT processes apply to data lakes, where the data is transformed on demand by the requesting/calling application.
Offert par


À propos de ce cours
Computer and IT literacy.
Ce que vous allez apprendre
Describe and contrast Extract, Transform, Load (ETL) processes and Extract, Load, Transform (ELT) processes.
Explain batch vs concurrent modes of execution.
Implement an ETL pipelinethrough shell scripting.
Describe data pipeline components, processes, tools, and technologies.
Compétences que vous acquerrez
- Extraction, Transformation And Loading (ETL)
- Apache Kafka
- Apache Airflow
- Data Pipelines
Computer and IT literacy.
Offert par

Réseau de compétences IBM
IBM is the global leader in business transformation through an open hybrid cloud platform and AI, serving clients in more than 170 countries around the world. Today 47 of the Fortune 50 Companies rely on the IBM Cloud to run their business, and IBM Watson enterprise AI is hard at work in more than 30,000 engagements. IBM is also one of the world’s most vital corporate research organizations, with 28 consecutive years of patent leadership. Above all, guided by principles for trust and transparency and support for a more inclusive society, IBM is committed to being a responsible technology innovator and a force for good in the world.
Programme de cours : ce que vous apprendrez dans ce cours
Data Processing Techniques
ETL or Extract, Transform, and Load processes are used for cases where flexibility, speed, and scalability of data are important. You will explore some key differences been similar processes, ETL and ELT, which include the place of transformation, flexibility, Big Data support, and time-to-insight.
ETL & Data Pipelines: Tools and Techniques
Extract, transform and load (ETL) pipelines are created with Bash scripts that can be run on a schedule using cron. Data pipelines move data from one place, or form, to another. Data pipeline processes include scheduling or triggering, monitoring, maintenance, and optimization. Furthermore, Batch pipelines extract and operate on batches of data. Whereas streaming data pipelines ingest data packets one-by-one in rapid succession. In this module, you will learn that streaming pipelines apply when the most current data is needed. You will explore that parallelization and I/O buffers help mitigate bottlenecks. You will also learn how to describe data pipeline performance in terms of latency and throughput.
Building Data Pipelines using Airflow
The key advantage of Apache Airflow's approach to representing data pipelines as DAGs is that they are expressed as code, which makes your data pipelines more maintainable, testable, and collaborative. Tasks, the nodes in a DAG, are created by implementing Airflow's built-in operators.
Building Streaming Pipelines using Kafka
Apache Kafka is a very popular open source event streaming pipeline. An event is a type of data that describes the entity’s observable state updates over time. Popular Kafka service providers include Confluent Cloud, IBM Event Stream, and Amazon MSK. Additionally, Kafka Streams API is a client library supporting you with data processing in event streaming pipelines.
Avis
- 5 stars66,36 %
- 4 stars22,72 %
- 3 stars4,54 %
- 2 stars4,54 %
- 1 star1,81 %
Meilleurs avis pour ETL AND DATA PIPELINES WITH SHELL, AIRFLOW AND KAFKA
It's great introduction for airflow and kafka but still an introduction it is shallow doesn't offer much but at the end you will understand what you need to continue further in both technologies.
Perfect environment to make experiments! Very easy and powerful in use.
Excellent introduction to this topics. Labs contain all you need to know how to start using this type of technologies. Highly recommended.
Good introduction to Airflow and Kafka however only one airflow operator is explored
Foire Aux Questions
Quand aurai-je accès aux vidéos de cours et aux devoirs ?
À quoi ai-je droit si je m'abonne à ce Certificat ?
D'autres questions ? Visitez le Centre d'Aide pour les Étudiants.