Now we filled up all the missing numerical values in our data frame.
Let's do the same.
But for the rest of the missing values which are these ones here with numbers.
So we found this out using the F temp dot is in a dot some.
So we got a fair few that are still missing and I have a hunch that they're all categorical over how
we found columns before that with the string data type.
Let's do that again but this time because we've filled up all the numeric data let's find all the columns
that aren't numeric data and that should tell us all of these columns are still have missing values.
Does that make sense.
Because we've filled up all of the columns with numeric data.
We want to find the columns that aren't numeric data but make a little headings here.
So filling and turning categorical variables into numbers.
We're working on categorical variables now because we've already worked on our numerical variables so
let's code check check for columns which aren't numeric.
We'll go for label content India.
Tim dot items and then we'll go if not paid a note API dot types dot is numeric.
So this is make sense if not so if they're not numeric because we filled up all the variables which
are numeric now we want to find the ones which aren't numeric content and then we want to go print label
beautiful.
Now this is gonna tell us all of the values all of the columns that aren't numeric but some of them.
OK.
Usage band that has a fair few missing value so that has three hundred thirty nine thousand missing
values.
So now we know all of the columns that aren't numeric.
What can we do.
How do we fill these up when we turned a string column into categories.
We had access to an attribute called codes which gave us a numerical value for all of the variables
in that column.
Let's have a look an example before we do it.
So PD categorical.
So what this is going to do is go Panda's categorical type and then we're going to pass it the f 10
state we'll look at a state code because that's the one we use an example for and we'll see what this
does categories fifty three objects in this is categories.
Okay can we do D Here categorical D type.
Okay yeah.
And what it might do is dot codes.
Okay so this is how we turn all of this date the variables in the state column into numbers.
So how would we do this for everything all of the categorical columns because that's what we need to
do right we need to fill in the missing categorical variables and turn them into numbers at the same
time.
So we might use this.
Let's keep that they'll make another sell here.
We might go turn categorical variables into numbers and fill missing.
So let's say let's loop through for content in DFT camp.
Got items and then if not so if it's not just like we've done up here.
So if it's not a numerical type paid a lot API Times Dot is numeric type.
But remember we've got the not here so it's checking if it's not numeric daytime content.
Wonderful because we know it's not numeric we might check to see if it's missing again and we're going
to add a binary column just like we did in the previous video with numerical data.
So I had a binary column to indicate whether a sample had missing value wonderful to DFT HAMP.
Here we go.
Label.
Plus we'll do a little string.
Here is underscore missing equals payday is now content so that's just going to return true or false.
If the value was missing too.
If the categorical value is missing it will return true.
It will make a column or assign that to true.
If it's not missing it will just have false a.k.a. true as one false is 0 turn categories into numbers
and add plus 1.
Then you might be wondering Daniel why are we adding plus 1.
Well actually let's write the line of code first because let's see the code PD categorical content codes
plus 1.
Now be wondering why we're doing plus 1.
All these codes.
So we did this plus 1.
See how it goes up.
Keep that there goes up OK.
Why would we do this.
Well the way it panders categories works is it automatically if there was some missing value in here
it is signs that a code of negative 1.
But we don't want that.
We want it to be not negative when we want it to be zero.
So maybe if we find a column that has missing values we might be out to see this.
If not we go usage.
No.
Is it using banned.
That one has a lot of missing values doesn't it.
Usage ban.
Okay let's have a look at this column.
Codes are negative one negative one negative one beautiful.
We don't want this to be negative one so we're going to add one there.
So all of our numbers are positive in our data frame.
That's the reason why we add plus 1 here.
So if we were to do this born that way we know when we turn our data frame and America into all numbers
or categories that way we know zero is actually missing all the other categories are just plus 1 on
what their category code is.
So let's run this cell.
This might take a little while a fair few missing value so we have to fill all of these.
So what we're doing we're turning all of these columns that are non numeric.
Remember that's what we found here.
Not numeric.
We're taking those missing values all actually first we're turning them into categories.
If they have a missing value that row will be assigned negative 1 but we're adding plus 1 to the code
so all the missing values are going to be 0 and all of the values that are still there.
The categories are gonna be some number pertaining to whatever Panda's decided that their categories
are beautiful.
It's worked OK.
So now we can check to see what we've done by going DFT temp dot info so we say here.
Okay so now we have one hundred and three different columns.
Wow.
Well let's have a look.
So if we go.
Def temp don't head T Okay so see these are all now categorical is missing columns that have been adjusted
that have been added on the end.
Now this is from this line of code here to see what I mean add a binary column to indicate whether a
sample had missing value.
So this means that for sample 0 it had a differential type.
It was missing.
So if we went back and found that column for sample 0 it would be filled with a value 0 instead of just
missing
one last check is to see if there are any more missing values cause what we've done filling up the numeric
and categorical values of our data frame and turning them into numbers.
There shouldn't be any more missing values.
Beautiful.
So they should be all zero.
So maybe I can a bit cut off 10.
See what I mean is getting truncated 20 wonderful so we've got no more missing values in our data frame
and our data should be all numeric.
What does that telling you.
What do you know about machine learning models.
What are those two points.
Hint to you.
When thinking about machine learning models will way to the next video so we can find out.