Welcome back.
So we just went through this whole flow of how data is used by a company but there's a few more things
I want to talk about because right now we just talked about water and how does this really apply.
What kind of tools do data engineers actually use.
Well you may have heard of something like Kafka and we'll get to Kafka later on.
But something like collecting these data streams and putting them into a data lake might be something
that Kafka or Apache Kafka might do for data lakes for example.
These are simply programs that companies can use.
For example you may have heard of Hadoop or Amazon S3 or the Azure Data Lake.
These are programs that have been built by engineers to hold large amounts of data like a data like
and then for a data warehouse.
We have tools like Google big query Amazon rest shift even Amazon Athena these are data warehouses that
allow engineers to make queries or analyze this structure data.
Now I know what you're thinking.
Who does what.
In this whole system we've learned that the data engineer creates this entire system for us.
They use different tools and programs to ingest data and then put it into a data lake or a data warehouse.
So as a data scientist and a machine learning expert which data do you use.
Pause the video and think about it ready for the answer well most of the time you'd be working with
a data lake because remember if you're doing machine learning the more data you have the better.
Right.
And with machine learning you can use structured or unstructured data so you can go into a data lake
and actually grab a bunch of data to use for your models whether they're in CSC forms or any other forms
and usually data warehouses are used by B AIS or business intelligent people or business analyst or
data analysts to make visualization or analyze data because the data warehouse has usually more structured
data that has been cleaned out than it was when it was a data lake.
It's a lot easier to understand and use the data for information.
Now as a data scientist you can use the data from a data warehouse.
And this isn't just a rule it's usually you use whatever data is usable to you.
But this is a good way to think about things that a data scientist uses as much data that they can as
much useful data as they can while somebody like a business intelligence person or a data analyst has
already the data cleaned processed by a data engineer and use something like a data warehouse to analyze
data and Google's big query is exactly that.
It allows somebody with not too much engineering experience or programming experience to analyze this
data in a data warehouse.
Another way to think about it is like this usually a software engineer a software developer app developer
mobile developer they build programs and apps that users and customers use and that releases data then
a data engineer would build this piping and pipeline for us to ingest data and store it in different
services like Hadoop like Google big query so that that data can be accessed by the rest of the business.
Next we have data scientists that use the data lake as well as the data scientists to extract information
and deliver some sort of business value.
Finally we have data analysts or business intelligence to use something like a data warehouse or structured
data to again derive business value.
And I hope that clears the picture a little bit.
When you hear all these titles out there in the industry now the industry is fast evolving and there's
definitely some overlap.
And sometimes what one job description might say might be different than the other but these are general
simplified rules that you can use to understand how each role plays into the part of a company.