Data Lake, Firehose, Glue, Athena, S3 And Aws Sdk For .Net

0

1

2

3

4

5

Posted on 30 Sep 08:15 | by mitsumi | 20 views

Last updated 3/2020
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 746.03 MB | Duration: 1h 55m

Leverage AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK to build a Data Lake.

What you'll learn
How to build a Data Lake using Firehose API for .NET, S3, AWS Glue and Athena
Requirements
You must have an active AWS Account to utilized the included code.
Prior knowledge of AWS Web Services would be helpful, but not needed.
Description
The purpose of this class is to demonstrate a proof of concept using a series of lab exercise's (in the AWS Console using AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK, with C# code using the AWS SDK) of building a Data Lake in the AWS ecosystem.In this class, we will be sending data from a local SQL Server database to AWS RDS securely, and automatically.I am utilizing the .NET SDK for AWS, however, this could easily be migrated to your language of choice once you understand the concepts that I am teaching. I will be providing the full working source code of this proof of concept.I will also provide a working example of creating Parquet files and sending to S3 (without using Firehose).
Overview
Section 1: Introduction
Lecture 1 Introduction
Section 2: Overview of AWS Web Services in a Data Lake
Lecture 2 AWS Serverless pieces to our Data Lake jigsaw puzzle
Section 3: AWS RDS and S3
Lecture 3 S3 Bucket creation
Lecture 4 RDS creation
Section 4: AWS Kinesis Firehose, C# and the AWS SDK
Lecture 5 Creating Kinesis Data Firehose delivery streams
Lecture 6 Console .NET core C# entity-framework project
Lecture 7 Kinesis Firehose C# API class
Lecture 8 Kinesis Firehose C# API send data example
Section 5: AWS Glue, Athena, ETL jobs, Triggers, and using Hangfire
Lecture 9 Creating Glue Crawlers
Lecture 10 Query Data in Athena using SQL
Lecture 11 Creating AWS ETL jobs
Lecture 12 Hangfire - Automation
Section 6: AWS Lake Formation, Cognito and Parquet .NET
Lecture 13 AWS Lake Formation
Lecture 14 AWS Cognito
Lecture 15 Parquet .NET - Alternative to using Firehose
Section 7: Final thoughts and advice
Lecture 16 Conclusion
AWS developers interested in learning about building a data lake using AWS services and the AWS SDK.