Now I know at the end of the last video we said we'd make sure all the data was filled up like there
were no missing values.
But after reviewing it I think it makes a little bit more sense to make sure it's all numerical first.
So let's make a little heading.
We'll call this one point one.
Make sure it's all numerical.
Wonderful.
And now to begin with we kind of need another dataset.
Now why is that.
Well because our heart disease data center this one here is already all numerical.
So that been said let's import another dataset which is our trusty car sales data.
Now if you remember this right at the start of kind of revealed a little bit here.
But this one only has 10 rows.
And what I've done is I've gone and manufactured a little extended version of this which has the same
headings make color odometer doors and price.
But this time we've got a thousand rows.
I'm not going to scroll through because that would take too long.
You're just gonna have to trust me on that.
Anyway I can prove it.
Car sales that's important PDA reads here's me.
Remember I've moved all of our CSA folders into a data file which is kind of best practice.
Keep your working directory tidy car sales extended.
Dot CSI v wonderful and we're going to have a look at it.
Car sales dot head wonderful.
And then we'll figure out how many there are a thousand beautiful you so.
And then we'll look at the data types make as an object color as an object odometer doors and price
are all integers.
And now these are objects because they're strings and they are categories whereas these are numerical
values.
Now what we have to do.
Because this section is make sure it's only miracle is before we can run a machine learning model.
We have to convert these to numbers but just to prove it let's try anyway and see what happens.
So first of all what we're going to do is split the data into x and y so we'll use these four columns
make color odometer doors we're not price but these four to try and predict the price of a car.
Now have a think to yourself Does this actually make sense.
Could you do that in real life.
Do we expect our machine learning model to do well on this kind of problem.
If you were given only the make.
So Honda the color a white Honda with 35000 431 kilometers on the odometer and four doors.
Could you predict the price was fifteen thousand three hundred twenty three dollars.
Maybe maybe not but you probably need a little bit more information.
The same probably goes with our machine learning model.
But nonetheless let's try and make one split into X Y which is we want the feature matrix.
So we're going to go X equals.
Car sales don't drop.
Now we don't want the price column or what have I done actually press shift into getting very trigger
happy in this neighborhood.
Price access equals one and we go into do Y equals car sales.
Now this is going to be just the price column because we want to use the X to predict the Y and then
we'll also split into training and testing split into training and test.
So the X train x test y train y test equals.
We saw this before train test split we'll pass it X we'll pass it y and we'll pass it test size equals.
We use 20 percent of the rise of the test data.
Wonderful.
Well let's try to build a machine learning model that will learn on the training data.
So the X train and the Y train and then predict on the the test data all right build machine learning
model.
Now this time we're going to use still random forest but I want you to guess what this random forest
can be used for so we're going from SBA loan ensemble.
Import random forest regressive.
If you're remember back in our workflow we were predicting a classification problem.
Now this random forest aggressor is the same as a classifier random forest.
But this time it can predict a number which is what we're trying to do right.
We're trying to predict the price of a car given some attributes about it.
So we're going to import a regression model.
We're going to settle up this time instead of calling it CnF.
We'll call it model regress are beautiful.
And then we'll go model dot fit.
We want to train it on the training data so it's going to learn the patterns the relationships between
X variables and the price the label.
And then we're gonna score it evaluated on the test data why test now How beautiful is that.
This is the power of S.K. loan right.
We can train create a model trainer and run it evaluate it in four lines of code.
We could really do it in less but four lines of code is pretty damn good to build a full blown machine
learning model.
So let's do it.
Oh no.
We're going to error.
What's happened value error.
We kind of expected this though right.
Value error could not convert string to float Nissan.
Now I see what's happened is that our machine learning model can't deal with strings.
So as we kind of knew we have to convert these into numbers.
So let's see how we do that with SDK loan and we're going to kind of go through this little chunk of
turning the categories making color into numbers with SDK loan and it might seem like a bit to take
in to begin with but we'll ride it out we'll see the code in action and then we'll go back through it
and see what it's actually doing.
So from SBA loan Doc pre processing SBA loan has a fair few modules that at each do a number of different
things.
One hot encoder wants and we haven't seen him before.
We'll put a little comment on here what we're going to do turn the categories into numbers.
Wonderful from SBA loan dot compose import column transformer.
I wonder what that does you can probably kind of guess what column transformer does but we'll see it
in action first then we're going define our categorical features so in our case our categorical features
are make color and a tricky one is doors.
Now you might be thinking why doors.
Let's go back up have a look at it.
Doors 4 5 4 4 3.
Why is this.
Well let's see it.
Car sales doors.
Value counts.
It's safe to assume that doors is numerical right because you've got 4 5 and 3 as a values.
But it's also categorical because you can go cars with four doors fit into this category.
Cars with five doors fit into this category.
So the seventy nine of them and cars with three doors fit into this category.
So this 65 of those.
So we're going to in our case treat doors as categorical.
And I'm just continually spamming command s.
The Notebook saves it's like I've ingrained it into habit just like pressing shift and enter to run
itself.
So we've got three categorical features beautiful and then we're going to make this little variable
called One heart.
We're going to take advantage of the one hot encoder class that we've imported.
So we'll instantiate that there and we're going to make a little variable called transformer and then
this one is going to use the column trains former.
This is going to accept a list of couples with a name.
So in this case we'll call it one heart and then we'll pass the actual transformer that we want to use
which is the one hot variable that we've just instantiated.
We're going to format this to make sure it's nice spacing I'm going to pass it the list of features
we'd like to transform.
And then finally we don't need that.
Let's bring that back up stuff all over the place here.
Bring that back up.
Wonderful.
And then here we're going to pass one final variable parameter to column transformer called remainder
Eagles pass through beautiful and then we're going to have a little variable here called transformed
X which is basically going to be the version of our X data.
Let's have a look at what X is basically transformed x.
We don't want all of those is going to be the version of this except converted into numbers.
So let's see that.
So we've got our transformer here.
This is what we're going to go to transform that fit transform on X and then we're going to have a look
at transformed x.
Well we've got some invalid syntax over a comma here.
Name error transform is not defined.
Classic transformer.
There we go.
Wonderful.
So this is spitting out an array.
Maybe if we put this into a data frame that would make sense.
And this guy paid a data frame.
See if we can actually transformed X wonderful so you might be looking at this going what the hell is
this.
But if we bring back X what we're going to see dot head work and a bit zoomed in so we're going to have
to kind of keep bouncing up and down a little.
But what has happened is that because we've told psychic line that hey make color and doors are categories
what this code here has done has transformed it.
Specifically one hot encoded it.
So now we've converted the make color indoors columns into one hot encoded variables but the odometer
column so which is here you can see the numbers are actually the same still.
So 35 4 3 1 Yep 1 9 2 7 1 4.
Yep Correct correct.
So the odometer column hasn't been changed.
So if we look back up here the reason why make color indoors have been changed because we define them
here.
We instantiated a 1 Hot encoder.
We'll see what that is in a moment.
And we've created a transformer using column transformer.
So basically this is saying hey column transformer.
Take the one hot encoder and apply it to the categorical features.
And for the remainder of the columns that you find pass through.
Don't do anything to those.
And then what we've done is created transformed X and fit our transformer.
So fit.
Transform to our X data.
And that's a kind of a big of a mouthful there.
And you might be thinking what does one hot encoding even mean.
Well let's have a look.
We'll go to our keynote because this is one art encoding in a nutshell.
If we have four cars so car 0 1 2 3 and that colors are red green blue and red.
If we wanted to one hot and code these so basically turn the categories into numbers it would look something
like this is what one hot encoding does.
We've got cars 0 and because the color of cars 0 is red.
This little arrow here could be the code that we've used to transform our data frame.
So to transform our categories into numbers.
So this is one hot encoding.
So because cars 0 the color is red it gets a 1 and it gets a zero for the other two categories.
And the same for green.
But this time because it's green and gets a zero for red and a 1 for green and a zero for Blue Blue.
Exactly the same zero for Red zero for green and a one for blue because it's a blue car.
And then finally because car 3 is also red it has a 1 just like car 0 and there's 0 and 0 4 green and
blue.
Now that's what we've done here except not only with the color column.
We've done it with the make and the doors column.
So that's why we have 0 1 2 all the way up to 11 features with zeros and ones.
Now what we've done is we've encoded the different parameters of each sample so Honda white and number
of doors into zeros or ones.
So now the beautiful thing is our data is all numerical.
So what can we do.
Well we should be out of fit a model but just in case that wasn't clear.
We've got one more way that we can do this now so we can go dummies equals there's a function in panders
called Get dummies which is kind of like one hot encoding but I'm not sure I'm actually not sure why
it's called dummies I just know that the function is get dummies go dummies PD don't get dummies and
we want to transform car sales and pass a list of the columns that we want to transform or turn into
dummies.
So here doors now let's have a look at dummies.
Here we go.
So I think because doors Dawes is numerical.
It hasn't worked on here but this is this is what we can see what's happening so we've got zero make
BMW is zero so it's a Honda it's got zero for Nissan zero for Toyota and it's got zeros for black blue
green red and white.
So if we went through this all it's done is turn the make and the color into zeros and ones.
Beautiful.
So now that our data is all in zeros and ones let's try and refit the model
so we want to go N.P. not random seed.
We've got 42 just so our results are reproducible.
We're going to use x transformed up here.
So this one here instead of X so let's do that X train we'll set up a new new training and test data
equals train test split we're gonna pass it transformed X and Y can stay the same because Y is already
numerical and then we're going to have tests.
Size equals zero point two.
Excellent.
And then we can go model that fit X train y train beautiful.
It worked.
And now let's evaluate it model school X test.
Why test
wonderful.
Now it worked right.
So but because we use transformed X and because up here we did this little bit of code here to to transform
our data from being categorical into numerical and previously our model could not convert floats to
strings to floats for Nissan.
So when we used our first x variable up here our data was still in categorical form so still using strings
to to represent a sample.
And what happened it air it out got a value error but now right down here when we run the same code
model not fit X train y train and score we don't get any errors.
Why.
Because our data is all numerical.
Now as for this score here we imagine the best score you can get is one point zero the model probably
hasn't done as well as it could possibly do.
But when we talked about that if you were to look at this data here the X data and you were trying to
trying to use this to predict a car's price it will probably be pretty hard.
So maybe the model has found some patterns but they just weren't that great because there wasn't too
much information about each sample.
But that's not the point.
We can look into a valuation metrics later.
The point here is that we've converted our model our data sorry from being non numerical to completely
numerical and that has allowed us to fit a machine learning model on it.
Now we've covered that.
We've converted all our data numbers.
What happens if there was missing values.
Let's check it out in the next video.