We've seen how to make predictions with our machine learning models once it's loans and patterns from
the data a.k.a. using those patterns.
Now how do we figure out whether those predictions are valid such as Could we use them in production.
Or is our model just making things up or those predictions do they actually hold water.
So what we're going to cover in this section is step for evaluating a model.
So we'll get rid of this but we'll put in a little heading here evaluating machine learning model beautiful.
Now the first place we're going to have a look at is up here and this is a socket loan documentation.
We can actually find this by going socket learned evaluate a model that should come up three point three
metrics and scoring.
That's what we're after.
So as you can see here there are three different API is for evaluating the quality of a model's prediction.
We're going to have a look at each of these.
So we're going to estimate a score method.
We've got the scoring parameter and we've got metric functions
we could read through this but I prefer as you probably do preferred to is getting hands on with the
codes.
Let's see it in action all right now to do so we're going to bring back our heart disease classification
problem.
We could scroll up and copy it.
But again I want you to to get some practice writing it out right.
Because this is what we're here for We're here to practice writing machine learning code.
So we're going to import is a markdown sales.
I'm going to change that to code.
We're going to import the random forest classifier.
We've seen this before and then we're going to set up a random seed.
Wonderful.
And then we're going to create our x and y our feature variables.
Heart disease don't drop.
We've already imported the data from heart disease and access equals one.
Beautiful.
And we'll create our labels.
Heart disease.
This is target.
Excellent.
Then we'll split it into train and test.
So x test y train y in test Eagles train test split x y test size equals zero point two.
Wonderful.
Then we'll instantiate our random forest classifier random forest we can probably breast have here.
We certainly can.
And then we're going to fit it.
That wasn't too hard right.
By now we're becoming experts at writing this little section of code and I'm being realistic here.
That's a full blown machine learning pipeline right there.
As long as the data's in the right format and we've got the target column we can do this pretty quickly.
So now we run this.
We can see that our model fits itself to the data.
So basically it's finding the patterns in X train and Y train or between those two.
And so now we can use the scoring parameter what we might do actually is copy this.
We go three ways to evaluate psychic loan models slash estimates.
Now this is just from the documentation.
So we want one is estimate a score method.
This is what we'll have to look at first and then two is the scoring parameter scoring parameter.
We'll have a look at that shortly and then three is problem specific metric functions.
Beautiful.
Now I've got a little heading we know what we're working with.
So the first things first we're going to check out the score methods may you put another heading in
here.
One two three four point one evaluating a model with a score method.
Now we've already seen this one right because this is basically the default it's a way to get a quick
sniff a quick understanding of how our is doing.
So if we call CSF dot score right because that's the score method every estimate in psychic loan has
this little score method.
So once you've instantiated machine learning model here and you fit it to some sort of data you can
get its score.
So look we could even get its score on the training data how does it go on here 1 so it fits the training
data perfectly.
And then if we go here.
Score on the test data 85 percent.
So we've seen this figures before right now what is happening here.
Well let's have a look.
Let's press shift tab.
Remember you can press shift have within any method to see what it does or see its Doc string returns
the mean accuracy on the given test data and labels.
And so what's happening here is that a model that would predict perfectly would get 100 percent here
and actually I would say you should be skeptical of any model that gets always 100 percent because no
model is perfect right a huge machine learning model is always getting its predictions right.
I'd say there's some sort of error in your data or some sort of error in the way you've trained it.
So our model doesn't get everything correct but at 85 percent it's still far better than just guessing
right.
Because remember we've got two labels heart disease or not.
And so guessing would be just getting about 50 per cent now.
Let's do the same as above except with some regression code and this time I'll let you off of this video.
We've already typed out a little machine learning pipeline will come up here and we'll copy our regression
code.
We know it's regression because we've got the random forest aggressor and so we write it in a little
come in here.
Let's do the same but for regression
beautiful what we want do is just fit it all.
We've already got the fit function they're saying confusing myself.
Beautiful.
So the model is now fit.
And now we'll do the score because we've run this sell our x test data has been replaced with the Boston
data frame rather in the heart disease.
So now we can just call it on x test and then Y test wonderful.
And so you might be thinking well these numbers are quite similar here.
Point a five point eighty seven and you're right they are pretty close but in fact our regression model
is actually when we call score.
It's actually using a different metric returns the coefficient of determination or r squared of the
prediction.
Now we'll drive a little bit deeper into some specific metrics per problem.
But the thing to remember here is that the score function on every machine learning model has some kind
of default evaluation metric built into it.
So if we call the random forest regress.
Chances are it will use the coefficient of determination as the default score metric.
Now if we call any regress.
If we go back to our machine learning map we call any one of these estimates here in the green boxes
the default metric will likely be the coefficient of determination because they will all be regression
models and the same goes for classification here.
Right.
Returns the mean accuracy on the given test data and labels.
So for all of these classification models in the green squares here the default evaluation metric is
accuracy.
And so what happens when the score method gets called the model makes predictions on X test creates
y predictions like we've seen up here before y reds and then it compares those predictions to the test
to the actual labels and then returns back some sort of metric to compare how well our model actually
did.
Alright.
That's the score parameter in a nutshell make some predictions compares them to the actual real labels
and then give us an idea of how well our models are doing.
So this is probably the first one that you'll call when you first train and fit a model.
You call the score parameter.
That's why it's listed as the first one here in three point three The Psychic loan documentation for
metrics and scoring.
So now we've seen score let's check out the scoring parameter.
So what we'll do here we'll create another heading ready for the next video.
For point two evaluating a model using the scoring parameter.
So before we get into that one I would say press shift tab here and have a read of the doctoring here
and see if you can figure out what the coefficient of determination is and the same thing goes for the
accuracy here or the classification default score metric.
Press shift tab and have a read through here and see if you can understand what's going on.
If the doctoring doesn't really help you try.
Check out the documentation here.
Model evaluation.
But otherwise I'll see in the next video.