Welcome back.
In the last video we saw how to convert non numerical data into numerical data.
That way we could feed a machine learning model or at least get one to try and figure out some patterns
in our data.
In this one we're gonna figure out what we can do when we've got missing value creating a little heading
here.
What if there were missing values.
There's two main ways to deal with missing data.
Let's put that here actually.
So we'll go One it's fill them with some value.
Also known as imputation.
So this is when you have say some samples and they're missing information you could fill it with some
value.
That's called imputing data.
The other one is to remove the samples with missing data altogether.
Now it should be known that there's no real good way to fully deal with missing data.
If you do fill missing data with some value you're going to be kind of introducing some real data to
your dataset.
And if you do remove samples with missing values well then you kind of going to be working with less
sample.
So in the real world you're not always going to have complete data sets.
But when you do deal with missing data it's kind of still a big area of research of how to do it.
But we're going to go through basically these two ways of figuring it out so we can at least get a machine
learning model working with data that once had missing values.
So in saying that we do need a dataset that has missing values.
So luckily here's something I've prepared earlier my beautiful Google Sheet which is our faithful car
sales data set.
It's missing data here.
We keep going through far out.
What are we done.
Someone has we've failed at recording with this data set.
And this is kind of often what you'll see a lot of times in the real world when you're working in data
sets.
You'll have these samples so like a row one no one is missing the color of the car.
But it has everything else.
Well that's all right.
Let's see how he'd work with it.
So the first thing we do is we're going to import car sales missing data we're going to go call it something
nice and easy.
Car sales missing equals PD read see as V and remember everything's in the Data folder might not be
on your machine but in my case it is car sales extended mine's called missing data dot CSI V.
Wonderful.
I'm gonna go.
Car sales missing we're going to look at the head of it.
Beautiful.
Excellent.
So we've got it here.
It doesn't look like it's missing any data but a real quick way to calculate to see if it is we go car
sales missing the convenience of having Independence Day to frame is that we can call pandas functions
is an A for is non or his name.
I think in pandas it's called man.
If we do import this it will fill it with Nan so it's something like N and I'll keep that blank there.
So we'll go here is in a dot some and this will show us how many missing values there are.
So let's go shift into so in the make column there's forty nine missing values in the color call on
this fifty and then the other columns is about 50 odd missing values in each column.
All right let's try and convert it to numbers.
Let's try and convert now just like we did in the previous video data to numbers what we might do is
cheat a little here we'll move up we'll come up here to the code that we've got before we're going to
just copy this and this is one of the only times I'll allow you to copy code just to save a little bit
of time here go here but first I think what we might need to do is transform this make it into x and
y because otherwise we're going to be using it on the wrong X so let's do that we need a new X and Y
create X and Y we want X and goes.
Car sales missing don't drop price we're gonna do the same again we're going to use the example of trying
to use these four columns to predict the price column it's a price we've got access equals one Y equals
car sales missing and we're going to get this the price column wonderful I want to try to do this and
see what happens value era why is this so we've tried to do the exact same code as before you saw me
copy and paste that we've got X why do we think that's not working input contains NAND.
Of course that's why car sales are missing let's check it out.
Will this show us and then yes it will.
So we've got at least one man that we can see here it's in the make column is about forty nine of these
nans in the color column the odometer doors and price there's 50 of them.
So what we have to do is before we can even convert our data into numbers is we have to do something
with these missing values.
So let's see how we might do that.
There's a couple of ways.
The first one is that we can fill missing data with pandas so let's check this out we'll make another
heading fill missing might put in this option one fill missing data with pandas
let's say what we might do.
So we might go fill the make column we'll do this for each column actually fill the make column and
then we'll go.
Car sales missing what we might do is any categorical column we'll fill it with a missing string any
of the others like numerical we might just fill it with the mean of the column.
Remember how I said feeling data is never a perfect practice.
So if there's missing values and you fill it out with something it's never going to be perfect.
Like if you actually had the real value there.
But there's a few things that are kind of just like Level 1 ways of filling data like a categorical
you might fill it with missing a numerical column you might fill it with the mean or like the average
of that column like any missing values will have the average of that column so they're not too far away
from the rest of the values and you're not filling it with some massively outlier value.
Let's see what that will look like.
Make dot fill in a mailing to go missing and then we'll go in place Eagle shrew because we want it to
happen straight away and we go fill the column we're gonna do the same thing.
Car sales missing we're going to get the color column in there fill in and we can just do the exact
same as we've got above because they're both categorically in place equals true.
Wonderful.
Now we're going to fill the odometer column.
Now remember this one's we're doing it in order here.
So we've got make color odometer fill the odometer column we might do the mean here.
So car sales missing.
How would we do this.
You want to fill it.
I don't want a came.
We have to put that little spice that same way fill in.
But this time we want the main scratches head.
Car sales missing.
Don't much care.
This might turn out to be a fairly long.
It's actually pretty easy.
Doug main overthinking it.
Classic implies ego's true.
Oh right.
And then finally we want to fill the doors column and all our doors is a different one as well because
we said before that it can kind of be a category for what we might do is because majority of the doors.
Like say if we go here we check it out.
Car sales how many doors would you say the average car has like if you had to guess how many doors a
random car how many doors does it have.
I'm going to say for Madonna kind of agrees with me there.
So we're just going to fill the missing cars the missing all values in our data frame with 4.
So fill in a nice and simple 4 in place equals true wonderful.
Now let's check it again check our data frame again.
So we want car sales what's it look like car sales missing is and I don't see how any missing values
do we have now.
Wonderful.
So all of the feature columns so make color odometer.
Doors are all full of values except for the price column.
Now we kind of left this one out on purpose because price is special because it's the one we're trying
to predict what we might do is just remove the rows from our data frame that have missing price values.
So we are going to lose some data but it's hard to really predict something when it doesn't have a label.
So let's do that.
Remove rows with missing price value so we can do this.
Car sales missing that drop and a in-place equals true and that should just remove the price rose or
the values in the data frame the rows in the data frame which don't have a price value.
Well let's check it one more time.
Car sales missing that is in a dot some beautiful.
So we have no missing values in our data frame anymore.
So what this means is that we've kind of lost we did have a thousand rise at the start.
Now we have 950.
We have lost 50 samples at the sacrifice of filling up all the data and removing any samples that don't
have a label a.k.a. the price column.
So now let's bring back our code from before the one that error it and we'll get that disgusting error.
So let's bring this down here.
We're going to fill this up.
Actually we need to re split our data into x and y so x equals car sales missing don't drop price access
equals one.
Now this is really good practice right.
It seems that we're we're just re typing a lot of the same code again and we are usually where you want
to kind of avoid this but we're seeing what it's like just continually filling up data or massaging
data into a way that we need it.
So here continual practice you'll always need to do this is split data into x and y or data and labels
and what's saying here or target I'm thinking of are our categorical problem our classification problem
that we worked on before now let's copy and paste this code from before we should have filled up all
the values.
So fingers crossed this should work similar to how it did before.
Wonderful.
Now what it's kind of done is given this output instead of just outputting numbers.
So what we might do is we don't really need X in car sales there we go.
So we can see that now we have converted again our car sales missing data into numbers and we fill it.
So that is just one option.
You might be thinking where inside psychic loan why do we use panders.
Well panders is very versatile and kind of fun to use I really like using panders but we can actually
do what we've just done.
So filling up these values here we can replicate this or what we've done with panders with pure psychic
line.
So why don't we look at that.
What we'll do is we'll wait to the next video for that so we'll get this set up.
So option 2 fill missing values with socket line.
All right.
So have a little review of what we've done there.
Go back up and see what we've done.
You might want to try are playing around with the car sales missing data frame.
See if you can fill the missing values play around with these functions here with pandas and then see
if you can convert it into numbers.
Otherwise I'll see in the next video we'll learn how to fill missing values in a data center with pure
so I learn.