OK so in the previous video we understood that a data engineer essentially creates this data pipeline
for us for people like data scientists to have available data to work with.
But I want to use a different example let's pretend that Keiko Corp. has a need and that need is for
water.
The employees need water to work hard right so let's think of this as a problem that a data engineer
solves.
Now bear with me we're going to be talking about water for a few minutes but I promise it relates to
data.
You see the very first thing for us to get water is rain right.
We need rain and this is similar to the data sources that we had.
That is the Web sites the mobile phones the cameras the cars the drones all of these are collecting
data.
That's what the raindrops are.
So in order for us to get water to our employees we have to get water from somewhere and that water
comes from rain.
Next that water gets collected into different streams and rivers from all over the place right from
all over the world.
Just like data from all different places these streams of water start for me.
And what happens next.
Well these streams of water turn into lakes.
They all flow into some sort of a lake and that lake kind of sit still.
But again we as a Keiko Corp just want water from employees.
Well we obviously have a dam on a lake and that dam on a lake or a system to use that lake.
Well we transport that water to something like a filtration sanitary area so that we can clean the water
sanitize it so it's safe for people to drink.
So we process that water that has been collected and then finally after all of that we create all the
plumbing and pipes for us to deliver water to Keiko corp.
That's how we satisfy the needs of Keiko corp to provide water for their employees.
And this is how data works.
And this is what a data engineer built a data engineer starts off with what we call data ingestion that
is acquiring data from various sources and we acquire all these different sources of data and ingested
into what we call a data lake a data lake is a collection.
Well all this data into one location from there we could just leave the lake as it is.
But there's a problem.
We don't want the lake to overflow dry up.
We need to manage this lake somehow.
We don't want to flood the village.
We need dams.
We need pipes to the houses.
We need to filter and clean the water and we need piping to the houses.
So after we ingest this data from various sources we collected in a data lake.
We then perform something called data transformation that is converting data from one format to another
usually into something we call a data warehouse a data warehouse is a place that stores accessible data
that is useful for the business that is although the lake collects so much water so much data that was
kind of all over the place.
A data engineer looks at this data and forms or uses the parts of the data that are useful and puts
it in a data warehouse so that other parts of the business can use it.
Again a good way to think about it is that a data lake is a pool of raw data.
The purpose of which we still don't know what to do with in a data warehouse is a location for structured
filtered data that has been processed and has a specific purpose.
That is we have data in here that is useful for the business.
So that means that data lakes are usually less organized and has less filtration of data than something
like a data warehouse.
Now why would businesses want to do that.
Well for one it's a lot easier to analyze data when it's organized well we also might have data in here
that we don't really need.
We also save on storage space right.
Because we don't have to store all of this data and just store structure data only.
That is only useful to us.
We also save ourselves some money so to review a data engineer simply built this pipeline of taking
the data creation and data capture using data engineering practices to build this pipeline.
So that data can now be analyzed by data scientists and data analysts but we still have a little bit
more to go so I'll see you in the next video.