We saw in the last section how to fill missing values in a data set and then convert it to numbers while
we filled it with pandas.
And then we converted it to numbers using psychic learn.
But it'd be nice to do missing values with cyclone and convert it to numbers of psychic loans so we're
kind of sticking to the one library.
Let's see how we do then.
So first things first we'll need to reinstall and create our car sales missing data because we've kind
of ah filled the missing values in place up here using pander.
So the one we have right now already has its missing values field so we'll reread it back in.
Read CSB data.
I'm going to go.
Car sales.
I've got a few missing extended Where is it.
Car sales.
Missing data.
Extended missing data does see a swing.
Beautiful.
That's what we want and we'll just check.
Why am I doing car missing missing cars missing dot head.
Wonderful.
Now we'll just clarify whether we have some missing data in this one again.
How do we do that is and I know some wonderful.
So we have some missing value is not so wonderful if you're actually working on this data set because
ideally your data set doesn't add missing values but that's all right.
We're going to figure out how to fill them up.
So how do we do that.
The first thing first we're gonna split it into a subset.
Well first we're gonna get rid of rows which don't have price values and then we'll split it into x
and y because we don't want to deal with data that doesn't have labels car sales missing dot drop in
a subset equals.
We're going to give it the price column in place equals true.
So what we're saying is basically with this line here get the car sales missing data frame drop the
end and values that are in the subset of the price column.
These 50 values in the price column that don't have values remove them from the data frame and then
we're going to recalculate how many missing values we have might put here.
Little comment dropped the rows with no labels.
Beautiful.
We've lost some from the make color a dominant doors columns because they may have been overlapping
with the samples in the price column that had missing values.
So what's next.
Well we want to split into x and y.
Plenty of practice doing this this is what a lot of lot of machine learning problems end up being right
is it's turning your data into features and labels and then getting a machine learning model to hopefully
learn some patterns in those features and predict labels.
That's what we're going to try and do here.
Access equals one beautiful why we've seen this before.
Car sales missing price.
So again we're just using these four columns here to predict the price column.
Well that's our goal anyway.
So how would we fill the missing data and take care of these missing loans with psychic loan.
Well let's see.
We've got two little handy dandy.
Well one we've seen before but one we haven't import simple computer.
Well it might do is fill missing values with psychic line.
Now you might remember and I recall right up there when we started this part too well so I could learn
or Part 1 which is getting the data ready feeling missing values is also called imputation which is
what this does.
And just simple impute up from socket line.
And that's going to help us fill missing values and then we've seen this one before which is the sign
hit learn not compose import column transformer which allows us to define some kind of transformer and
then apply it to whichever columns that we want to use it on.
We want to fill want to just do it the exact same We did it with pandas but this time just reproduce
it with socket lines.
So fill categorical values with missing and numerical values with main every kind of writing these notes
like so we can remind ourselves and communicate through our code make sure we're kind of talking it
through doing the right stamps because otherwise you can get a bit lost with everything going on.
So simple computer we're calling this class that we just imported here we're going to tell it this strategy
equals constants so the strategy we want it to fills.
It's basically going hey run over the categorical values I've shortened categorical to cat here it's
gonna go over it saying for every value keep your strategy constant and the fill value
is going to be missing.
OK that's starting to make sense and then we're going to go we're gonna have a special one for our door
feature simple impute up as door is again that little funny one that is numerical but is also categorical
we're going to keep this as constant we're going to say Phil value equals four nice and simple and then
we'll have numerical computer which we could have really called now and Putin actually let's do that
because when keeping these these fairly short now computer simple computer strategy Eagles main so we'll
see what these mean and in a second we're just kind of as I say we're working through it and letting
go define columns so cat columns or cat features Eagles make and color and then we've got a double feature
because again member Dawes is special case and then we've also got numb features which is odometer
Cayenne wonderful.
And then what we're going to do is create and impute up something that fills missing data because that's
what imputation is right.
If you hear that term imputation kind of referring to find a missing value and fill out with something
or calculate something to fill it with.
So this is where we're going to leverage our column transformer.
I love that word column transformer always makes me think of Optimus Prime for some reason.
What's your favorite Transformer.
Mine's definitely Optimus Prime or bumblebee on the hype train they're getting distracted.
Daniel we're talking about machine learning well actually transformers are machines that can learn so
we're not really getting distracted.
So what we're doing here is we're setting up our computer and kind of passing it a few things to get
it ready.
So just bear with me a second I'm kind of coding and trying to talk at the same time I'm getting distracted
so I adore computer.
We're gonna go door computer and then we're going to go door feature.
And then finally one more and we got num computer and then we're gonna go numb computer and then we
may go numb features beautiful.
We've got a fair bit of code here and we're gonna go transform one more transform the data so going
to go field X because remember X has some missing values at the moment.
If I go up here we want to go X dot is and I dealt some which is just the same as up here except we've
got no price price category there.
So we come down here filled X equals we're going to take our little impute here impute her not fit transform
X and then we're gonna have a look at the field X.
Will this work.
Fingers crossed we've got a lot of code here.
I've got one two twenty 24 lines of code plus comments it worked.
No errors.
What has actually just happened.
Well let's step through this what we might do is zoom out so we can just a little bit so we can see
it all in one hit.
Beautiful.
So what we've done is we've imported from psychic learn simple computer class as well as a column transform
the class and then we've defined some impurities and remember impute are just filling missing data using
the simple computer class which takes strategy and a fill value.
If the strategy is constant we have to pass it a fill value saying hey go to the categorical columns
constantly fill them.
If you find a missing value you just fill them with missing the string missing the same thing for outdoor
computer it's going to say keep the strategy constant.
So for every missing cell do the same thing and fill it with four.
Yeah that makes sense.
And for our numerical columns in this case which is the odometer column fill it with the mean.
So keeping that one nice and simple then we've defined which columns are which.
So our categorical columns are making color realistically doors is also that but we've kind of made
it its own because it's again halfway between the number halfway between the category.
So we've got the door feature and then we've defined the numbers feature or the numerical features which
is our odometer and then we've used the column transform a class which is what we imported when we created
and computer passing it the imputations we wanted to do all the transformations we wanted to do.
So these are the names.
So that's what column transformer takes it takes a list of multiple different Transformers.
So see him we've got a list and now within the list we have troubles of the name the computer we want
to use.
This is just the name of the computer right.
So if we had to access this imputation later we can use this as its name.
I've just kept it simple and called it the exact same thing as what the variable computer is.
And these are the features that we want this specific computer to change.
So this one Kadam pewter is going to use cat in pewter on the categorical features door computer on
the door feature and then numerical computer on the numerical features.
That was a mouthful.
And then once our transformer is defined we can call this field X because we're going to use our computer
and fit transform on our X data to fill up the values of X well but don't take my word for it let's
say it in code.
So car sales fell let's create another data frame because we want to use the same checking method that
we've used before.
We'll pass it filled X and then we'll pass it the column names.
This is just the names of what these respective columns where you can see cars and make colour doors
and then odometer.
So that's all we're going to do here.
Make Carla doors DOMA to posit Kayyem as well and beautiful.
That should work.
Now we're gonna have a look at the head of this data frame car sales filled.
Go ahead wonderful that looks familiar to what we've seen before.
But the icing on the cake here the real test is seeing is an a what does is output use of all.
We now have no missing values.
Thanks to this bunch of code that we've written here.
So we've used simple computer plus column transformer to fill the missing values so these missing values
here with some preset defined ones that we've created here and now car sales filled which is what we've
made out of our transformed data now has no missing values.
Beautiful.
So now what should we be able to do.
We have no missing values should be able to convert these into numbers.
So when we come up here and we have some code that has done this before this is in so we're going to
copy this come back down.
This is the only time I'll let you copy and paste code.
I know I've said that before but again to just save us a little bit of time what we might do is again
split.
Car sales filled into x and y so we want X equals car sales field don't drop I know that already is
that already is our x value.
Beautiful so we can make this into him car sales filled
wonderful.
So this is telling us we've got a sparse matrix here now.
What can we do now.
We've got our data as numbers and filled no missing values.
Let's put a model that lets fit our model MPD at random just to make sure everything's working right.
Always trust the code.
That's what we're gonna do.
We're gonna reemployment what are we seeing this.
Nothing outperforms at random forest regress are going to have heaps of practice doing this from this
K model selection.
They want something to split our data into train and test and then we also actually want to split our
data into try and tense by going ex train x test y train y test equals train test split wonderful and
we're gonna pass it this time transformed x c this variable here because what we've done is we've passed
it.
Car sales field and basically just turned all the categorical features we've won hot encoded them so
we'll go here transformed X also pass at Y because y still saved in memory why does it have to change
because that's just labels there already numbers and the test size is zero point.
Wonderful.
And so we'll set up our model which is a random forest regress.
And then we're going to model not fit.
So this is telling our model to find Hey random forest regress are find the patterns between X train
and Y train and then we'll get to a model dot score on our test data sets it's going to say hey evaluate
I know you've found some patterns in this training dataset but now evaluate those patterns on this test
dataset and moment of truth shift into beautiful we're getting a warning here because there's some changes
happening in a future version of socket line you might not have this warning if you're using socket
line version point 2 to all it's saying is that an estimate is will be changed from the default 10 inversion
zero point two to one hundred.
So we can basically get rid of this warning by going here equals 100 wonderful.
So even though this model has more estimates than the previous one we've used on our car sales missing
data it performs worse.
So this one gets a score of zero point 2 1 9 and if we come up here where is it.
We did trying to model in the previous section.
This has got 3 point 3 0 4.
The maximum score is 1.
They're both not doing as ideally as we might like.
But why do you think of this one performed worse.
Well it's because it's only got nine hundred and fifty samples.
So if we go land car sales filled as well as land car sales missing now we want maybe the original car
sales.
There we go.
So our previous model was built on this one and the model we've just built was built on this one.
So one has a thousand samples and the other only has 950 samples which is why it's done slightly worse.
And so that's a big paradigm of machine learning right.
Like most of the time if you have more data usually a machine learning model is able to find better
patents and so that's kind of what's happened here is that because we've dropped the samples that don't
have labels our model hasn't been out of Find as many patents even though when they removed 50 out of
a thousand samples we have covered a lot here already so far.
The most important key takeaways from this are most datasets you come across won't be in a form ready
and immediately to start using with machine learning models and some take a bit more preparation than
others.
And so most of the time what you'll have to be doing is your data will have to be numerical and it can't
have missing values.
And so the process of filling missing values is called imputation and the process of turning your non
numerical values into numerical values is referred to as feature engineering or feature encoding.
So with that being said we've covered part 1 of how to deal with you Dana.
Let's get into part two and figure out where the hell did I get this idea for choosing a random forest
regress.
For our problem why did I pick this machine learning.
Well we'll have to wait and see.