Hey Cloud Gurus, welcome
to Section 3: Data Storage.
You have just come out of the Data Engineering crash course.
Now that Brian has thoroughly prepared you
for the concepts behind data engineering,
we're going to get started with a foundational concept
of data storage.
In this section, we'll be covering the introduction,
which we're doing right now, using Azure Data Lakes,
where we'll get more familiar with what those are
and how they fit into data engineering,
getting the folder structure right,
talking about the different zones,
how we design our folders, understanding file types,
learning about things like Parquet files
and the different options available to us there,
partitioning data, where we divide up
a single database instance into multiple chunks.
From there, we'll go over some best practices
regarding partitioning and continue that on in part 2.
It's a big topic, and so we want
to make sure we give it the appropriate coverage.
After that, we'll continue on to distributing data,
talking about Azure Synapse Analytics
and how it handles its underlying distribution;
archiving data; pruning data to pull back
only the subset we want and eliminate the rest;
compressing data to reduce the size of our database
and gain performance; sharding data,
where we split it out across multiple computers;
implementing data redundancy;
looking at our options for copying data
within our primary and secondary regions;
and finally, wrapping everything up with a review
in the section recap.
In this section, you'll get a chance
to apply your newfound skills in a practical way
using hands-on labs.
Specifically for this section, you'll be provisioning
and configuring Azure Data Lake storage, Gen2.
As you go throughout this section,
I want you to keep 1 theme in mind.
Properly designed storage lays a foundation
for achieving maximum query performance
and data availability.
Even though we'll get a bit into the weeds here and there
talking about infrastructure and formats,
it all comes back to this ultimate goal.
We're trying to increase performance and availability.
And that's it, I'm excited to get started
on this section with you,
learning how to lay that foundation for making things fast.
When you're ready, let's jump into the first video
on using Azure Data Lakes.
I look forward to seeing you there.