Hey it's Andre here with a quick little tip for you now.
Over the next couple of videos Daniel is going to show you his machine learning workflow and one of
the things that he's going to do is well clean the data transform it and do all sort of manipulation
to it.
Now I want us to take a high level view right now because when we're working with Jupiter notebooks
it can get really confusing.
It almost seems like we're doing a bunch of steps without understanding the why.
But I want to take a step back here and understand why we do the things we're going to do over the next
couple of videos you see as a data scientist.
More data doesn't necessarily mean it's always good.
I mean yes generally the more data we have the more data we have to work with.
But at the end of the day we can have completely useless data.
We only want useful data so that we can get real results real actionable steps that we can take and
what we're going to see.
Daniel do is do this step before we even create our machine learning model.
You see we want to run this step cleaning data transforming data and reducing data so let's talk about
that.
Why do we need to clean data.
Well we want to remove and replace data because sometimes data is missing.
Right.
We might have tables with missing values missing labels.
That's not going to help us.
And it's not going to allow us to build the right machine learning model so we usually remove a row
or a column that's empty or has missing fields or we might calculate some sort of average if we're doing
house prices will fill an empty price list to maybe an average house.
You might even notice some outliers in your data and just remove them next is data transformation.
You see computers don't really understand concepts right.
Most of the time computers just understand ones and zeros just numbers really in order for us to teach
computers most of the time.
We want to transform our data into some sort of form that a computer can understand.
So what we're going to see is Daniel actually convert some of our information into numbers.
You might have things like the price of a house or a salary that are numbers and easy for us to create
models with because computers understand numbers fairly easily but it might find it hard to know what
a color is.
What is green what is blue.
A computer doesn't understand that.
Instead we convert colors into numbers like our G.B. collars so that a computer understands a good way
we transform data is usually between zeros and ones such as Do you have heart disease or do you not
know heart disease is zero heart disease as one we have to make sure that data across the board uses
the same units maybe one data uses meters another data uses yards.
This idea of transforming data into a useful form is really really difficult and it just comes with
practice.
But you're going to see Daniel how he transforms data so it's more and more workable.
So once we've cleaned the data once we've transformed the data.
Next is to reduce our data why reduce our data don't we want as much data as possible.
Well theoretically yes but remember everything costs money.
The more data we have the more CPR you.
The more energy it takes for us to run our computation.
So if we able to get the same result on less data that actually saves us money it saves companies money
especially big companies like Google that have a lot of data and run a lot of spew.
Sometimes the idea of data reduction can also be called dimensionality reduction or column reduction.
You can have many columns on your table but it makes things really really slow so you can actually remove
some columns that maybe you find irrelevant and won't be needed for your machine learning model.
Again these are things that Daniel is going to go through but I want you to keep this in mind.
This idea that as a data scientist you can't just assume all data you have is automatically going to
be perfect.
You have to clean it.
You have to transform it.
You have to reduce it.
This idea of wrangling data of moving around data into a form that is useful to you is part of the skill.
So watch Daniel do that and learn along the way.
I'll see you in the next one.
Bye bye.