Welcome back.
I want to take a quick section to cover a topic that is often forgot about in data science and machine
learning courses.
However in order to truly understand a topic we need to understand the high level principles and the
landscape of that topic.
In this case we want to work as a data scientist or a machine learning engineer.
So we need to at least understand the topic of data engineering but what does that mean.
What's with all these titles.
Data scientist data analysts data visualization.
This is madness.
This is so confusing.
Well let's talk about data engineering and how it fits into this whole ecosystem that we've been talking
about up until now.
Now we learned that data science is all about using data to make business decisions right.
One of the ways that we can make business decisions is to use a technique like machine learning that
allows a computer to learn and figure out the solution to a problem that may be a little too complicated
for a human to solve or maybe too tedious and takes too long of a time so we want to automate it so
machine learning is a technique and data science is the idea of using data and converting it into something
useful for a product or business and data analysis is a subset of data science that allows us to analyze
the data that we have.
But the missing piece is that of data engineering and not all companies have data engineers.
But as companies get bigger and bigger they need to hire for this role.
Let's take a look at why.
Who brings all this data to us in a nice little file.
Where's this data coming from.
How do we manage it.
Do we throw it out after we're done.
Who labeled this data.
You see a data scientists or a machine learning expert doesn't need to concern themselves.
Most of the time with these topics you ideally have the data accessible to you because this part of
data is handled by a data engineer.
So who is this mythical figure.
Well a data engineer would take all of the data points that are incoming let's say for a company.
Well let's say this company has all these products and all these datas are coming from their users from
their security cameras from their Web site from IO T devices and a data engineer takes all this information
and then produces it and maintains it in databases or a certain type of computers so that the business
has access to this data in an organized fashion so they're kind of like the librarians where they collect
all this information and they organize it for us so that people like machine learning or data science
experts can use this.
But this is a simplified version of what a data engineer does in order for us to really understand this.
We need to go back to the beginning and ask the question what is data I'll see in the next one.
By.