So the final classification model evaluation metric.
We're gonna have a look at is a classification report and now really a classification report is also
a collection of different evaluation metrics rather than a single one.
So that's where the report comes from.
It's going to report back of a number of different parameters evaluating our classification will.
Let's see one in action.
So from S.K. learned up metrics import classification report to do it.
We just pass it some true labels.
Okay.
Why test and some predictions.
So once again this evaluation metric is just comparing the true labels or our data versus the predictions
that our model has made.
So let's see what happens here.
All right.
There's a fair few things going on.
We got precision recall have f1 score support zero.
Maybe that's for the class zero for not heart disease one maybe that's for heart disease.
If we come down with good accuracy point seven five macro average weighted average.
Mm hmm.
What's going on here.
Well let's go to our keynote and have a look at our classification report anatomy.
So over here we see we've got the same thing to just import classification report.
We've done one on the test labels and the predictions seem slightly different numbers here what's in
our notebook.
But that's okay so what's precision or precision indicates the proportion of positive identifications.
So a.k.a. the model pretty class one which were actually correct.
Now a model which produces no false positives as a precision of one.
So that's what we can see our precision scores if the model produced no false positives and we know
that our model hasn't because we've looked at the classification or the confusion matrix and we've seen
it confused on some examples and it's predictive them as false positives.
So our precision scores are a little bit lower than one.
Now the next one is recall.
So recall indicates the proportion of actual positives which were correctly classified.
Now a model which produces no false negatives has a recall of 1.0.
And again we know from our confusion matrix that this isn't the case.
So our recall is a little bit lower now f1 score a combination of precision and recall a perfect model
achieves an f1 score of one point zero.
Okay.
So we see our f1 score is not 1.0 but it's still relatively high.
Then if we go support the number of samples each metric was calculated on.
So we see here.
So this means we look at this twenty nine.
So we look at that.
That's 4 0 0 and down here at 61.
So that means that these metrics here will calculate on 61 different samples and these metrics along
here were calculated on twenty nine samples which had the class label zero and that this one is 32 with
the class label 1.
Okay.
Has heart disease.
And so if you total these up 29 plus 32 you get 61 accuracy we've seen that before.
This is just the accuracy of the model in decimal form.
Perfect accuracy is 1.0 a.k.a. our model is making predictions right 100 percent of the time if its
accuracy is one point zero then we have a look at macro average.
Now what is that short for macro average as you may have guessed.
So it's the average precision recall and F1 scores between classes.
Now the key here is macro average doesn't take class in balance into this should be effect into account.
So pretend this is account.
So if you do have class imbalances pay attention to this metric.
Okay.
So in our case what does it class imbalance.
Well if we look at our example we don't have class imbalances here.
Why is that.
Well because we have relatively the same amount of samples with class 0 and Class 1 meaning that about
50/50 split here it's not exactly 50/50 but it's not like we have 60 examples of class one.
And one example of class zero.
So we've got about 30 in each.
So if you did have class imbalances you really want to check out your macro average now because our
example the class is a relatively balanced.
That's why the macro average and the weighted average shortfall weighted average AVP the weighted average
is a precision recall in f1 score between classes weighted means each metric is calculated with respect
to how many samples are in each class.
So this metric will favour the majority class.
For example we'll give a high value when one class outperforms another due to having more samples.
So this is why that our values here are quite similar because we have balanced classes.
Now you might be thinking as what kind of situation would it be that these values would be out of whack
right.
Like where would this happen.
You might also be thinking we've learned a whole bunch of different metrics here we come back to our
notebook we've gone overclassification reports we've done confusion matrix we've done accuracy we've
done area under rock curve.
So when should I use each of these.
So now we've kind of gone through them might be tempting to think well why don't I just ditch all these
other ones that I really haven't kind of trouble grasping and just use accuracy.
Well let's have an example let's see when other metrics come into play and maybe you shouldn't just
use accuracy because this is a trap that I got caught in right when I first started building classification
models I was like Yes my model is getting 90 plus percent accuracy it must be a great more let's say
a scenario.
So for example let's say there were 10000 people and one of them had a disease and you're asked to build
a model to predict who has it.
All right.
So let's actually do that let's cut this up.
So this is where precision and recall become valuable and in fact all the metrics in our classification
report become valuable here so disease.
True.
So there was ten thousand people and one of them had a disease.
So what we might do is create an umpire Ray of 10000 people and then we might change one of these I
can spell disease so disease zero index equals 1.
So there's only one positive case
then if we come down here and say our model predicted or disease spreads equals and P zeros ten thousand.
Now this means model predicts every case as zero.
All right and then what we're going to do is create the PD data frame so we can visualize this classification
report.
What if that's what our model did right.
This is what's happening here.
Disease true so there's only one positive case one in 10000.
And we build a model and our model predicts that every single case is zero.
So it misses the one prediction what would happen if we build a classification important so we go disease
to ruin over here disease spreads want to pass this little parameter so it fits nicely into a data frame
so output dict equals through all right.
So what's happening here.
We look at this this is just another format of this classification report up here.
If we didn't do this it would air out.
So this is just to visualize what's going on.
True so this is a prime example right of where you want to use another metric other than accuracy is
when you have a very large class imbalance.
So in our case we have a massive class imbalance because in our original dataset disease equals true
there's only one example so only one example of where the Labor would be one and everything else is
zero.
We've built a model that just predicts zero for every case.
Right.
Because it just it's only one sample so it's really hard to learn that there is a pattern there for
this one particular case.
And so what happens is if we were to measure just accuracy on our model that is predicted zero for everything
it comes out with an accuracy of point nine nine or in other words 99 per.
And so ask yourself although the model achieves ninety nine point nine nine percent accuracy is the
model still useful.
Right.
That's why we look at these other metrics such as the macro average.
That's where we can see okay our models falling down here on the precision it's actually getting zero
precision for class 1 which makes sense because it didn't even predict Class 1 and here is getting basically
100 percent on class 0 because it has only predicted class 0.
So this is a kind of use case scenario of where you want to make sure that you're using a wide spectrum
of evaluation metrics for your classification models and not just accuracy.
And now we've only just touched on some of the metrics that you can use for evaluating a classification
algorithm.
But there are a few more so if you want to have a look at more.
So we go here.
Model evaluation psychic learn.
We'll look at the documentation if you come in here.
This is a little bit of extra curricular is to have a look at this section here.
So all I've done is gone into the socket learn documentation for model evaluation.
We come to classification.
We can see a few here the ones that we've covered.
And so the ones that we've covered are probably the bare minimum that you want to use right.
So accuracy area under the rock curve or a you see confusion matrix and classification report.
So keep those in your tool bag and if you need more be sure to refer to the documentation.
So that's going to wrap up our classification specific evaluation metrics.
Let's dive into the next section where we'll have a look at some more specific regression model evaluation
metrics.