Hello, and welcome to Mars,
or at least the view from Mars.
In this lesson, we are going to be talking about
data engineering at a glance.
And so what does that mean, exactly?
So, today we're going to talk about
the 4 pieces of data, right?
So, the first thing we have is data movement.
Then we're going to talk a little bit
about data ingestion and what that is. We're
going to talk about data storage and data transformation.
So, this lesson is going to be a lot about data.
Let's get started.
In the beginning of data we have a shopper.
Let's call her Sally.
Now, Sally likes to buy lots of stuff.
So, she is going to go to the store and purchase
all of these things.
Now, when she does that, she generates data.
Now, this data can't just live
at the register where it's generated, it has to move.
And so the very first piece of data engineering
is data movement.
We have all these data points being created somewhere
and then they have to move.
And as that data moves,
we have to be able to do stuff with it, right?
So, what we'll have is we'll have--
jumping into a business sense now--
we'll have, on the left, our ingestion sources.
So, this is the data that's being generated.
It could be media, it could be logs,
it could be business data that's a database
coming from some sort of shopper like Sally, right?
And that data needs to be pulled into our cloud.
Now, to do that we're going to have what's called
an ingestion layer.
The ingestion layer is where we take all
of those different data sources--
could be a bunch of different cash registers--
it could be, like I said, log files. Whatever you want.
And we're going to ingest that data
into something like Data Factory.
Now, we're going to talk a lot about these services later.
So, don't get bogged down in what Data Factory is.
Just know that Azure Data Factory can be a fantastic
ingestion source to pull data into the cloud.
Once we have that data in Data Factory,
we got to do something with it, right?
So, we've moved it, but where are we moving it to?
That's where our storage layer comes in
and we'll take something
like Azure Data Lake Storage Gen2
and we will move our data from Data Factory
into the Data Lake.
Now, once we have it in that Data Lake
we're going to have all kinds of different data in there
and it's not going to be particularly clean.
This is where data transformation comes in.
And this is a great example.
So, we have our antiques sign
or anitques, I don't know.
However you want to say that.
So, we have a misspelling here, right?
And when we look at data, that's what's going to happen.
You're going to have all sorts
of differences between different data types.
It could be duplicate data.
It could be data that's misspelled.
It could even be something like a country.
So, for instance, if we have the country,
like United States, we could have some places,
some data sources listing that as US
some listing it as USA, some United States,
some may even have a 2-digit country code.
All of those sources reference the United States
but they're all different.
And when we try and use that data
it becomes very difficult with duplicates
or missing data, or misspelled data, or different types.
And so we do data transformation.
So you could use a service like Azure Databricks
or Azure Synapse Analytics to do data transformation.
So, we could pull data from that Data Lake
and we could start to manipulate
and work all of that data together
so that it's all similar.
This is data transformation, and you'll also hear it called,
like, a cleaning step or
a process data layer.
That's what we're talking about.
We're talking about making all of the data the same
so that it can be used for business decisions.
So, that's our data transformation.
And so then the question comes up, why do we care?
Right?
The real goal of data engineering is to give you accessible,
clean data that's in a usable format.
And when we do that, we empower these things on the right.
We empower actionable business decisions.
So, we give reports that you could actually look at and say,
oh, I can see trends here.
Or I can see a step that we need to do in our business.
We can also generate machine learning
through that and machine learning insights.
So, we can give information to customers,
such as recommended products, or things like that.
And by doing all of this, we lower those customer barriers.
Just like the example I just gave,
if I'm looking to purchase something,
my barrier is going to be lower
if it's a 1-click purchase.
If the system knows my information
and can pull it back up easily,
if the system can recommend new products to me
that I might find interesting
based upon my previous purchasing habits, right?
All of those things--if you can send me a coupon, even--
because you know that I purchased something in the past.
All of these solutions that are generated off of the data
that's coming through our system
and driven through data engineering
can help to lower customer barriers.
So, in this lesson, we've done a very high-level
look at data movement.
Hey, where's it coming from?
Where's it going?
We looked at data ingestion.
So, how do we ingest data into the cloud system?
And what does that look like?
Data storage. We talked about the management of data storage
and why it's important to store data in the cloud.
And then we talked quite a bit about data transformation.
So, the cleaning, the manipulating,
transforming data into a usable format.
Now, keep in mind through the entire rest of this course,
we are going to be diving way down deep
into all of these concepts, but data engineering--
generally, anything you're going to do--
is going to fall into one of these 4 concepts.
So, keep that in mind and we will see you
in the next lesson as we dive a little further
into data engineering.