Now we've just covered accuracy as a method of evaluating our classification models.
And you might be thinking why can't we just leave it at that.
Well as we start to go through these other metrics here you'll start to understand why it might be important
to get a few more different evaluation metrics on the board rather than just accuracy.
In my case when I was first learning how to build classification models I would always think yeah just
higher accuracy is better right.
And then slowly after looking at different examples I start to realize okay now I see the value of using
other metrics like these but let's not talk about it let's see it in action the next one we're going
to cover is the area under the receiver operating so area under the receiver operating characteristic
curve.
The beautiful thing is it's also known as a U.S. So area under a curve or R O C so receive a operating
characteristic.
So let's do this ROIC curve.
So these are the two things you'll look out for area under curve or rock curve.
We put it here.
You see or rock curve beautiful.
So if you hear someone talking about you say you were ROIC or rock or something like that they're probably
talking about this metric here.
Now what does a rock curve measure.
Well by formal definition a rock curve is a comparison of a model's true positive rate a.k.a. TPR versus
a model's false positive rate.
Let's write that down.
Here are a comparison of the models.
True positive right which is also known as TPR vs. a model's false positive rate which is also known
as FBA.
Now you might be wondering okay what is a true positive and what is a false positive.
Well let's have a look here.
A true positive equals model predicts one when truth is one that makes sense.
So in our case if we have a look up here if we our targets are 1 0 0 a true positive is when our model
predicts a 1.
The real label is a 1.
Okay.
Yeah that makes sense.
So if we go here a false positive is model predicts 1 when truth is 0.
So why is it called false positive.
Well it's because it's predicting 1.
So.
Okay.
The positive class has heart disease.
When the truth is actually 0.
So it's giving us a false positive sense that that a person may have heart disease.
In our particular case for our heart disease classification problem.
And then if we go here and through negative
ego's model predicts zero when truth is zero.
So that makes sense.
So that's predicting the model is getting the right prediction there is predicting someone doesn't have
heart disease when they actually don't.
That's great.
And then a false negative equals model predicts.
What do you think this would be.
False Negative.
We look at false positive a false negative is when model predicts zero.
When truth is one so that's because it's predicting not heart disease when it's actually is heart disease.
Right.
So now we know these we can see that a rock curve is a comparison of a model's true positive rate TPR
vs. typos everywhere.
Daniel come on.
Versus a model's false positive rate or FPL.
So now we know this.
Let's see it in action so we can do this using psychic loans metrics library planning going to import
ROIC curve so see our sign rock of ROIC curve and then we're going to make predictions with probabilities.
And how can we make predictions with probabilities.
We saw this in our making predictions.
We can do that with Y probs equals CnF dot predict Kroeber for probability we're gonna make some predictions
on the X test data.
Now I just want to make sure maybe we do that we create create X test just to make sure it's the right
x test data x test
etc. So we want to go here x train x test y train y test equals train test split x y test size equals
zero point two wonderful.
And so now we can do that there because a rock curve is a comparison of a model's true positive rate
versus a model's false positive rate.
We want to only keep the positive classes.
So actually let's see what y probs looks like.
Why proms.
Why hasn't this worked.
What do we got here.
This random Florence classified is not fitted yet code fit that would make sense.
So we want
the classifier.
Of course you can't make predictions without without fitting them without the model learning any patterns
so we're gonna fit it here.
Then we'll make some predictions and we'll have a look at this maybe only the first 10 so we're not
taking our paper space.
Okay.
Beautiful.
And so because ROIC curve only is a comparison of the models true positive rate versus a false positive
rate we only want probabilities that the model has predicted for the positive class.
So if you imagine here our models trying to predict 0 or 1 This is the probability that the label is
a zero and this is the probability that the label is one and so on for.
For all of these samples until until it finishes up so there's gonna be however many are in the test
set here.
Sixty ones is gonna be sixty one of these.
So essentially what we want is why probs positive
so this is the probabilities that it's the positive class a.k.a. zero is the negative class and one
is the positive class.
Why is we're gonna use some slicing here but only column 1 of every row so that we can do that just
so you can know what's going on.
Positive and we'll only look at the first 10 again.
So does this make sense.
We're getting zero point four three.
Yep zero point seven seven Yep zero point four eight so on so on and so on for the entire list of Y
problems.
And now we can calculate FBR TPR and thresholds gave FBR TPR thresholds.
Why do I know this.
We'll see this in a second why do I know that this is gonna be FBI common TPR common threshold rock
curve.
I'm going to pass it.
Why test and why.
Probs positive.
Now how would you figure out what rock club returns in this case.
Well what I would do is I'd press shift tab and this is how I know.
So we've got rock curve it takes y truth so that's our test labels.
It takes y score so that's out.
Y probs positive.
And if we come down here compute the receiver operating characteristic rock.
Yeah.
Beautiful now parameters y true y score target scores can be either probability estimates of the positive
class confidence values or non threshold measures of the decision.
So that's the probability estimates of the positive class.
That's where we got this slice from right.
And then if we come down here it's gonna tell us what it returns FBR increasing false positive rates
such that element IE is a false positive rate of predictions with score above thresholds TPR.
So that's a true positive rate and thresholds.
So that's how we can figure out what a function returns.
And again that's just viewing the doc string in energy and notebook.
You can always look at the documentation for this if you wanted to do so you might look up something
like how to calculate rock curve for psychic learn so then we're gonna check check the false positive
rates FBR beautiful so that's giving us a big array but looking at these on its own doesn't really make
much sense it's much easier to see it visually.
And since I get loan doesn't really have a built in function to plot a rock curve what we might have
to do is and quite often you come across this right as you'll have to find a function or write your
own that'll do it for you.
So that's what we're gonna have a look at in the next video or we'll plot a rock curve and this function
here rock curve will start to make a bit more sense rather than just be an array of numbers.