Now we've chewed the hyper parameters of our logistic regression model and it's getting pretty good
score here.
But remember this score.
The default score for a classifier is accuracy.
What we want to do now is create some evaluation metrics around our model that go a little bit beyond
accuracy.
More specifically let's create our tuned Shane learning model.
Actually machine learning classifies for a better one because these evaluation metrics are specific
to classification.
So we're going to go beyond accuracy.
So here and more specifically we want an ROIC curve and a UC school.
We also want the confusion matrix putting flaws in there that actually looked like it was from The Matrix
with lots of numbers.
You know that net flowing green screen they want a classification report.
What else do we want we want precision recall and f1 score and it would be great and it would be great
if cross validation was used where possible.
So let's turn that into markdown so that's what we're gonna be working on we'll use our churned grid
search logistic regression model as well as the best type of parameters for it and we'll see where they
come into play in a second.
So first of all when we evaluate a model it's always comparing how a trained model's predictions compare
to the truth labels.
So what we have to do is make some predictions first so we can compare them to the truth labels a.k.a.
the labels in the y test dataset.
So let's do that or write a note here to make comparisons and evaluate our trained model.
First we need to make predictions.
So what we'll do is make predictions.
We choose and model and there's a beautiful function called predict that we can use for that so we'll
save it to y parades.
We use G.S. log rig which is just our trained version of our grid search model.
So G.S. log rig don't predict and we're gonna predict on the test data why.
Wonderful and let's say them just to make sure we're not going crazy beautiful and now we need to compare
them to the test dataset.
So we have a look at this here.
OK so if we wanted to go 0 0 we can know it's got this one wrong because that's supposed to be a zero
there because this is the truth labels here and then we can keep going through there but we're not going
to do that.
We want to use some code to do that.
The first thing first we want of rock curve because remember that we've got this little list up here
so rock curve and AUC school.
What is a rock curve.
Well we looked that up.
So what is a rock curve.
We went through this process of understanding AC rock curves so if we were to read through this what
we would get is the rock curve is created by plotting the true positive rate against the false positive
rate at various threshold settings.
Okay.
Beautiful so let's see how we do that the rock curve is a way of understanding how your model is performing
by comparing the true positive right to the false positive right.
And if we want to figure out what a true positive and a false positive is we can have a look at our
confusion matrix for that.
So a true positive model predicts one when the truth is 1 and a false positive is just the model predicts
one when the truth is supposed to be zero.
And a perfect model is going to get an R U C score.
What we see in a second of 1.0.
So let's see how we do it.
So import ROIC curve function from SBA loan dot matrix.
That's where it's from.
That's where it lives.
But we've actually already done this.
As with all the other SBA loan functions we've been using right back up the top right back up here.
So this is Model evaluations right now in this next section.
This next video we're going to tackle everything from here.
So these are we can see here we've got randomized search of a grid search of a confusion matrix plot
rock curve that's we're about to use precision school recall score F one school.
So these are for our classification model.
So let's go damn we can save ourselves a line of code.
We don't actually have to import that come right back down here.
We can just go plot rock curve.
Now this is a relatively new addition to this socket line library which I'm very very happy with because
usually you had to write these on your own.
This function this plot rock curve and calculate it actually calculates the AEC metric for us so the
area under the curve metric.
Let's see it in action go plot rock curve G.S. log rig.
And if we look at this in the documentation and tell us what it does pass it and estimate which is our
machine learning model pass it X pass it y plot receive a operating characteristic a.k.a. rock curve
that's that's what that's gonna do for us beautiful engineers log rank x test we wanted to do it on
the test data set always evaluate this should be calculate always evaluate in machine learning models
in the test dataset and let's see all beautiful.
Okay so remember a perfect rock curve we've seen in previous videos we go up to this corner and then
across like that.
But as he is up pretty well then we're going to area under the curve.
So if we calculated all this area under here of zero point nine three of course a perfect model will
achieve one point zero.
So our model is not perfect but it's an AUC score of zero point nine three when the average of just
tossing a coin essentially would be zero point five.
So we're edging close to a perfect model not bad for a model that just came out of the box.
All right.
The next thing we want to do is a confusion matrix.
So let's say that.
So confusion Matrix.
How could we do that.
So I think circuit line has a function and I don't think I know socket line has a function we can go
confusion matrix and want to compare the ground truth labels with the predicted labels.
OK.
I could just leave it as that it's a bit bland.
We can improve the visualization of this confusion matrix using seaborne.
That's what you want.
So we want to go.
I believe we already have Seabourn here you're doing.
Yes and yes yes we do.
So we've you see bomb before in this project.
So we can go what I might do is I know ahead of time that we need a bigger font size font size equals
one point five.
That's just s an S dot set.
Now we're going to create a little function here in case we wanted to make another confusion matrix
because socket lines confusion matrix function isn't up to scratch as is making this video could always
maybe that's an opportunity for a pull request.
So we go here nice and simple.
So we're just gonna pass it this function our test libels and Al predicted libels.
And then we're gonna create a nice looking plots a nice looking confusion matrix using C bonds.
We've seen this before.
Hate Matt wonderful.
So we go fig X equals pale T dot sub plots fig size equals 3 3 0 shift in top trigger happy again X
it S.A. dot heat map.
Wonderful.
We're gonna pass it a socket line function side confusion matrix y test y Fred's and then we want to
add here and not so annotate.
Yes please annotate Eagles true.
Now we're gonna go see bar because we don't want to Calabar because I've seen the Calabar it doesn't
look too great on the confusion matrix penalty don't ex label true liable and then we're gonna add peyote
don't why liable gonna guy predicted libel and let's see it.
So we're gonna plot comes from that confusion matrix y test y parades o f what have we done.
Font Size that sack of font size wrong what is this tab or scale.
There we go tab when I complete for Lent are classic and it's giving us this little cutoff thing here
so we can fix this I believe by going bottom top because acts dot get Y limb Yeah and then we want to
go ax dot set y limb make a little bit pretty set y lamb and it's going to be bottom last point five
top minus zero point five.
Now of course if your confusion Matrix came out looking well you might not need these lines but doesn't
matter.
There we go.
It might look a little bit better so you can see that the model gets confused so predicts the wrong
label relatively the same across both classes.
So in essence there's four occasions here where the model predicted zero the predicted label so predicted
someone didn't have heart disease when they should have been predicted as one.
So that's a false negative.
So it's predicting zero instead of one a false negative remember is we come back here false negative
model predict zero when the truth is 1 and over here is the false positives.
So okay you've got three instances here where a model predicts one someone does have heart disease when
they actually don't.
And you can see these are both things that we want to avoid right especially when I'm thinking about
something as severe or as serious as heart disease even predicting when it's not present so here if
we predict zero so no disease when it is present.
Okay that's bad but also predicting that it is there when it's not actually there that's also bad.
So that's something that you want to have to consider when you're building these type of models is a
false negative worse or is a false positive worse.
And again a perfect model would have none of these but in reality you're going to probably end up having
some sort of confusion here with your model you're not going to ideally these would be perfect across
here across the line.
Now we've done a confusion matrix and got that there we could share that with our boss we could share
that with our boss a rock curve there.
What's next.
Classification report.
Okay.
And it would be great if cross validation was used where possible.
All right so that's what we might tackle in the next video.