Now we've seen cross validation in action.
Let's see some other important classification model evaluation metrics.
And to do so we're going to start a new heading here.
It's so important to evaluate our models of valuation metrics and the four main ones we're going to
cover for classification models accuracy the next one is area under ROIC curve.
Don't worry if you're not sure these are we gonna go through them.
Third one is a confusion matrix sounds fancy run.
And the final one is a classification report.
So to start off we'll keep it nice and simple we'll do accuracy.
So I will remind ourselves of how we can train a machine learning model and evaluate it by importing
cross value score from psychic loans model selection and then we'll import our random forest classifier
one of Reagan press tab.
Yeah.
There we go.
Beautiful.
We'll set up a random seed and then we'll create our X data
and we'll create our y data
beautiful.
Now we'll set up a test and try and split.
So we'll go here x train x test y train y test.
Well actually we don't need to because we can just use the cross Val school.
So let's do that.
We might put a little heading up here to make sure we know that in this case we're doing accuracy F
equals random forest classifier and then we'll print out the cross value score by going passing it out
classifier L X data and what do you fivefold.
Cross Validation.
Wonderful.
So here we go.
We've got a number of estimates will change getting this classic warning.
So this is just a reminder we can set no estimate as to one hundred and that'll get rid of that beautiful.
So what this is going to give back is because the default score parameter of our classifier is the main
accuracy.
This is what the cross Val score is measuring here.
Actually we might save this to a variable cross Val score equals that.
Run it again.
And then if we take the MP main of cross Val score we've seen this one in the cross validation video.
This is going to give us the main accuracy of our model and basically this comes out as a decimal but
we can easily type this out to be print I actually f heart disease classifier accuracy.
So whether or not our model can classify whether someone has heart disease or not given their parameters
we want to put in here.
NDP don't mean cross Val score we're gonna times this by 100 and we want to decimal places we maybe
need a percentage sign on the end here beautiful this is gonna be cross validated accuracy actually
so that's important to note cross validated accuracy so that's not too bad right.
And what accuracy is actually saying is given a random sample given the sample that a model hasn't seen
before.
How likely is it to predict the right label.
So if we have a look at our heart disease data we might do it under this cell.
So it doesn't interfere with ours.
So like a lone cone disease don't head so given a sample that looks like this has age sex C.P. dressed
TB.
P.S. Cole FBA has all these features given a sample like that to our train model.
How likely is it to predict the right target and in our case our models cross validated accuracy is
eighty two point four eight per cent so around about eighty two point five.
So that means 82 or about just over eight times out of ten our model will predict the right label given
a sample something like this based on the original training data.
So that's accuracy in a nutshell.
What we're going to dive into next is area under the ROIC curve and if that sounds confusing it's a
bit of a mouthful but we'll go through it.
So that's kind of how you would present your models accuracy in print out something like this again
it returns by default a decimal in communication stalwarts it's easier if you are represented as a percentage.