So, let's just pause on this screen for a minute
because the lake looks amazing
and I would love to be sitting in that rowboat
just kind of rowing around the mountains.
(Brian sighing)
Feeling refreshed?
Me too.
Let's talk about Introduction to Data Lakes.
In this lesson,
we are going to be talking about the terminology.
What is a Data Lake? We're
going to talk about structured versus unstructured data
and what that looks like.
We're going to talk about why Data Lakes are used
and we're going to take a look
at some Data Lake architecture.
To start off with,
let's talk about structured versus unstructured.
So really that's a discussion of SQL versus NoSQL.
Now, when we talk about structured data, or SQL,
we're talking about relational data.
Now, relational versus non-relational,
when you think relational, think Excel spreadsheet, right?
You have rows and columns.
When you think non-SQL or non-relational,
what you're thinking about there is a variety.
Things that change in shape or size.
You might have a row,
but inside of a cell within that row, you have JSON.
So, non-relational is different shapes and sizes.
Relational is Excel spreadsheet. Fixed shape, fixed size,
fixed schema versus dynamic. Same discussion, right?
Fixed schema is the same rows and columns.
Dynamic means those rows and columns might change depending
upon the data source or even change
within the same data source over time.
When we look at SQL,
SQL is well-designed for complex queries because it's fixed.
It's easier to do complex queries on fixed data.
When you look at NoSQL,
that's not very great for complex queries
because it takes a lot longer to search for the data,
because again, there's no fixed schema.
So with that, in a SQL database or a SQL,
you're going to see more vertical scaling
as you look at how we grow.
So vertical scaling means we're going to add more RAM
and CPU power, right?
That's how we're going to solve the problems that we have.
In NoSQL, we're going to have more horizontal scaling.
So we're going to add multiple devices together
and you'll see that as we go through the DP-203
and look more in depth,
especially at Azure Synapse, for example.
So with that, let's reintroduce Blob storage.
Wait, what?
What is a Blob, exactly?
A Blob is just a general-purpose object store
and it's really for virtually any storage scenario.
So when we look at, specifically, Blobs,
we're going to be thinking about serving images,
serving video or audio,
storing lots of log files, or disaster recovery.
It's also fantastic as a backup source.
So when you look at backup for regulatory,
you'll see Blob as a very common scenario
or common storage choice.
Now, if you remember from the last lesson,
we also talked about Data Lakes being a part
of Blob storage.
So let's talk about Data Lakes with Blob storage.
The big thing that separates Data Lakes
when we look at Blobs versus Data Lake
in Blob storage is the use
or enabling of hierarchical namespace.
So what that big word really means is just taking our files
and we're going to organize those into directories
or subdirectories,
much like you would see in Finder for Macs
or File Explorer for PC, right?
That directory/subdirectory,
that is an example of hierarchical namespace.
And so when you're standing up Blob storage,
if you turn on hierarchical namespace,
you can create a Data Lake.
One more important piece to think about
when we create an hierarchical namespace or a Blob storage,
once you create that hierarchical namespace,
which is done upon creation of the Blob storage,
once you enable that, you cannot change it on the fly.
So if I've enabled hierarchical namespace,
I can't go back and un-enable it later down the road.
So why we want hierarchical namespace?
The main 2 reasons are, 1;
it decreases our processing time
because the files are more structured,
and 2; file systems are more familiar and easier.
This is really important
if you want to have non-IT people
in your Blob storage.
Because, with those files and directories,
it looks a lot more like a general desktop
to be able to search through folders
and find individual files.
So those are the 2 main reasons
that we would want to enable a Data Lake.
So let's talk about Data Lake architecture now.
So, to start off with, we have our data source.
This could be a laptop or a desktop
or it could be another database.
It really doesn't matter.
We just have some sort of data source
that is pulling into the cloud.
So we're going to ingest that with Data Factory
and then we have our Data Lake. Now,
when the data first comes in, we're going to create zones.
This is a very common practice.
The first zone is typically raw
which just means it's just the raw data.
So we might have multiple sources coming in
and all of those are getting dumped
into this raw data source.
They're not processed.
They're not curated.
There might be errors in it.
We have no idea.
It's just a direct dump into our Data Lake.
From there, we're going to use Data Factory
to move data from raw into processed,
and it moves data into a process zone typically
by moving it through a transformation service
like Databricks.
And so we're going to take that data
and we're going to clean it up a little bit.
And maybe we remove the null columns that are in there,
or maybe we make sure that all of the columns match
or we map things.
We would do that, combine those sources together,
and move that into a processed zone
which is generally a cleaner area of a Data Lake.
Once that's done, we would repeat the same process
and we would move data into a curated zone.
Now a curated zone is usually going to be a different look.
So finance might want data slightly different than HR.
And so the curated section is generally going
to be more specifically cleaned up
with a focus on a different look
for a silo-like or business group-like HR,
or finance, or marketing, or something like that.
Now keep in mind when we talk about this,
this is a very basic example.
Your business might do things slightly different
but this is pretty commonly what happens
and what you want to think
about most importantly is those zones.
Because if you don't have these zones
and everything is just raw,
you wind up with what's known as a data dump.
Basically, you have massive amounts of data
and it turns into so much data that it becomes very hard
to pull insights out of to create business reports
or other things that the business needs
to make decisions on.
So keep that in mind as we talk about Data Lakes
and what you should have as you move forward.
Finally, after all that's done,
like I said, you're going to pull those data insights out
of the curated folder and you're going to create reports
and those reports are used to make business decisions.
Should we change the price?
Should we buy this new thing?
Should we restructure, or whatever?
So now let's take a look at this lesson in review.
To start off with, we talked about terminology.
We talked about the differences
between Blobs and Data Lakes
with the introduction of hierarchical namespace.
That is very important.
We talked about structured and unstructured data.
So, SQL versus NoSQL.
Now this one is really important.
Make sure that you understand the differences
between structured and unstructured
and why you would choose one over the other,
because this is going to drive
not only a very important data engineering concept,
but it'll also drive what solutions you choose in Azure.
So it's a critical component to understand.
We talked about why Data Lakes are used.
Again, it's that hierarchical namespace.
So think about that.
And we talked about Data Lake architecture.
So proper planning and zone creation
to help avoid those data dumps.
So make sure you have a good handle on that
and I will see you in the next lesson.