1
00:00:00,360 --> 00:00:05,910
OK so we've got a confusion my drinks and we've got a rock curve as well as an A you see Metrolink.

2
00:00:06,000 --> 00:00:09,660
Let's write this down actually lets out communicate with ourselves.

3
00:00:09,660 --> 00:00:17,760
So now we've got a rock curve and you see metric and a confusion matrix so we've got a rock curve an

4
00:00:17,760 --> 00:00:20,820
asymmetric and a confusion matrix.

5
00:00:20,940 --> 00:00:29,900
Let's get a classification report as well as cross validated.

6
00:00:29,900 --> 00:00:36,170
This is a key one here cross validated precision recall and if one school

7
00:00:38,550 --> 00:00:44,010
beautiful song if you want a classification report we've already imported the function and so we can

8
00:00:44,010 --> 00:00:51,000
do it and go print classification report in our classification report is going to give us the precision

9
00:00:51,000 --> 00:00:53,660
recall in F one school but let's see it first.

10
00:00:53,690 --> 00:00:57,720
Classification report we can find it by passing it wide test.

11
00:00:57,720 --> 00:01:02,940
And why parades boom this is given us everything that we need right.

12
00:01:03,040 --> 00:01:06,330
Because validated precision recall that school.

13
00:01:06,690 --> 00:01:08,400
Well not really.

14
00:01:08,760 --> 00:01:15,070
The reason being is because this classification report it's been on why spreads.

15
00:01:15,070 --> 00:01:23,090
It's only being done on one set of test data so we've got a classification report.

16
00:01:23,120 --> 00:01:29,290
It's been found by this and we can tell that by coming up here back to where we made predictions.

17
00:01:29,330 --> 00:01:33,270
So we only made predictions on the test dataset.

18
00:01:34,100 --> 00:01:37,340
So if we come back we only make predictions on here.

19
00:01:37,460 --> 00:01:48,580
So what we need to do is a cross validation score of precision recall and F1 which is the values that

20
00:01:48,580 --> 00:01:49,360
we've got here.

21
00:01:49,360 --> 00:01:52,060
Precision recall in F1.

22
00:01:52,180 --> 00:01:55,390
Now you might be looking at this going.

23
00:01:55,880 --> 00:02:00,010
Daniel's timing we need to cross validate precision recall in F1 still not exactly sure.

24
00:02:00,010 --> 00:02:01,950
I kind of forgot what's actually going on here.

25
00:02:01,960 --> 00:02:03,020
But that's okay.

26
00:02:03,160 --> 00:02:04,600
Let's get a refresh of what's happening.

27
00:02:04,600 --> 00:02:11,770
We've got precision so precision indicates the proportion of positive identification so a.k.a. the model

28
00:02:11,770 --> 00:02:17,350
predicts Class 1 which is has heart disease which were actually correct.

29
00:02:17,350 --> 00:02:20,530
So a model which produces no false positives right.

30
00:02:20,530 --> 00:02:25,920
So false positive up here is false positive predicted one when it's supposed to be zero.

31
00:02:26,110 --> 00:02:30,260
A model which predicts none of these has a precision of one point zero.

32
00:02:30,270 --> 00:02:30,660
All right.

33
00:02:30,670 --> 00:02:39,430
So that's precision now recall indicates the proportion of actual positives which were correctly classified

34
00:02:40,500 --> 00:02:45,280
so a model which produces no false negatives has a recall of one.

35
00:02:45,300 --> 00:02:51,420
So model which produces no false negatives so remember false negatives where the model predicts zero

36
00:02:51,630 --> 00:02:55,680
in a binary classification predicts zero where it should have predicted one.

37
00:02:55,710 --> 00:03:01,410
So model with zero in this box will have a recall of one and a model with a precision of 1 will have

38
00:03:01,500 --> 00:03:03,280
a zero in this box.

39
00:03:03,300 --> 00:03:10,080
So if our model was perfect these are both big zeros and our precision and recall would both be one

40
00:03:10,080 --> 00:03:15,750
point zero and now F1 is a combination of precision and recall.

41
00:03:15,750 --> 00:03:22,980
So if our model was perfect and had precision and recall scores of one point zero F one would also be

42
00:03:22,980 --> 00:03:26,190
one point zero now support.

43
00:03:26,420 --> 00:03:31,890
Support is the number of samples each metric was calculated on in our test dataset.

44
00:03:32,090 --> 00:03:38,020
There are 29 samples with a value is 0 and 32 samples of the value of 1.

45
00:03:38,030 --> 00:03:44,510
So has heart disease and now accuracy we've seen that before and macro average.

46
00:03:44,510 --> 00:03:51,410
This is show for average as the average precision recall an f1 score between classes.

47
00:03:51,410 --> 00:03:58,670
So between these two to the macro average doesn't take class imbalance into account.

48
00:03:58,670 --> 00:04:05,160
So if you have class imbalances in our case we don't because we have relatively similar numbers for

49
00:04:05,170 --> 00:04:05,710
values here.

50
00:04:05,720 --> 00:04:12,210
So say for example this was 2 and this was fifty nine then you would have class imbalances.

51
00:04:12,290 --> 00:04:14,150
If you do have class imbalances.

52
00:04:14,210 --> 00:04:22,750
Pay attention to the macro average but for the weighted average is the weighted average precision recall

53
00:04:22,750 --> 00:04:30,580
an f1 score so weighted it a key here between classes so weighted means each metric is calculated with

54
00:04:30,580 --> 00:04:39,300
respect to how many samples there are in each class so a.k.a. this metric will favour the majority class

55
00:04:39,360 --> 00:04:42,580
e.g. this will be a high value.

56
00:04:42,720 --> 00:04:49,500
When one class outperforms another due to having more samples because that class has more samples and

57
00:04:49,500 --> 00:04:55,410
it will be given a higher White who was a bit of a mouthful but you can look up this information in

58
00:04:55,410 --> 00:04:57,420
the classification report documentation.

59
00:04:57,460 --> 00:05:00,530
The classification report SBA loan.

60
00:05:00,720 --> 00:05:05,300
You can also look up each of those metrics that we just covered individually right.

61
00:05:05,310 --> 00:05:14,370
So if you wanted to find out more actually on want to use a guide Derek is going to tell you some information

62
00:05:14,490 --> 00:05:16,050
about the classification report.

63
00:05:16,080 --> 00:05:22,020
If you want to search for more go precision recall F1 and you could look these up macro average weighted

64
00:05:22,020 --> 00:05:29,300
average in the socket loan documentation but remember we just said before that these metrics here are

65
00:05:29,330 --> 00:05:35,900
only calculated if we look at our cross validation diagram are only calculated on one test split.

66
00:05:36,020 --> 00:05:42,230
So what we want to do is recalculate these metrics but with using cross validation which makes him a

67
00:05:42,230 --> 00:05:44,030
little bit more robust.

68
00:05:44,240 --> 00:05:46,120
So how can we do that.

69
00:05:46,120 --> 00:05:46,870
Mm hmm.

70
00:05:47,030 --> 00:05:51,470
Well we might do is wait till the next video so spoiler alert in the next video we're gonna calculate

71
00:05:51,470 --> 00:05:54,020
these metrics here using cross validation.