1 00:00:00,580 --> 00:00:05,260 In the last video we talked about how we can convert our model's prediction outputs okay. 2 00:00:05,270 --> 00:00:09,110 This long list of numbers that are really hard to understand if you're just looking at them for the 3 00:00:09,110 --> 00:00:15,050 first time into something that we can understand a predicted label in the form of a string or what we 4 00:00:15,050 --> 00:00:20,750 want to do is to make it a little bit more visual because this is pretty tedious just comparing it like 5 00:00:20,750 --> 00:00:21,100 this. 6 00:00:21,120 --> 00:00:28,670 We'll create some functions that are gonna help us basically do this step as well as visualize the actual 7 00:00:28,670 --> 00:00:34,940 image that a model is making a prediction on and then compare it to that image is actual labor or the 8 00:00:34,940 --> 00:00:37,650 truth label and just a little tidbit. 9 00:00:37,700 --> 00:00:40,640 These are also known as confidence intervals. 10 00:00:40,700 --> 00:00:43,360 If you hear them this is a prediction probability. 11 00:00:43,360 --> 00:00:45,140 There is also confidence interval. 12 00:00:45,140 --> 00:00:48,580 So one methodology of evaluating a model is to go okay. 13 00:00:48,650 --> 00:00:54,800 Only give me the predictions that have a confidence value over point seventy five. 14 00:00:54,860 --> 00:00:59,610 Every prediction that's max value is under point seven five gets pushed to the wayside. 15 00:00:59,630 --> 00:01:04,310 Those are things you might want to pass to a human classifier but everything over point seven five you 16 00:01:04,310 --> 00:01:07,040 might want to let the model predict itself. 17 00:01:07,100 --> 00:01:08,580 Let's write a little note to ourselves. 18 00:01:08,690 --> 00:01:13,430 Having the above functionality 19 00:01:15,990 --> 00:01:30,420 is great but we want to be able to do it at scale and it would be even better if we could see the image. 20 00:01:30,720 --> 00:01:39,600 The prediction is being made on part of being a data scientist or machine learning engineer is not just 21 00:01:39,600 --> 00:01:41,200 being able to write this sort of code. 22 00:01:41,220 --> 00:01:44,630 A more important step than building a machine learning model is evaluating it. 23 00:01:44,670 --> 00:01:50,670 So this is what we're kind of doing we're communicating to ourselves what's going on because it's one 24 00:01:50,670 --> 00:01:54,090 thing to just show someone and accuracy score. 25 00:01:54,090 --> 00:01:56,250 We could just go yep there you go. 26 00:01:56,250 --> 00:02:00,160 Our model's accuracy is 100 percent on the training set and that might go well. 27 00:02:00,270 --> 00:02:01,030 Okay. 28 00:02:01,290 --> 00:02:02,620 Let me see what's going on. 29 00:02:02,640 --> 00:02:04,770 So that's what we're doing here. 30 00:02:04,770 --> 00:02:09,810 So I've yet to report it to someone else or if you had to show a colleague or maybe your partner with 31 00:02:09,810 --> 00:02:14,430 a business who doesn't really know what you're talking about when you tell them the probability of prediction 32 00:02:14,430 --> 00:02:16,650 is zero point seven five. 33 00:02:16,650 --> 00:02:19,610 It's much easier when you can show them something visual. 34 00:02:19,710 --> 00:02:20,750 So we go here. 35 00:02:20,760 --> 00:02:30,850 Note prediction probabilities are also known as confidence levels markdown. 36 00:02:30,990 --> 00:02:32,030 Wonderful. 37 00:02:32,040 --> 00:02:32,430 All right. 38 00:02:32,460 --> 00:02:33,780 So a little function. 39 00:02:34,020 --> 00:02:38,180 Let's go turn prediction probabilities which is what we have up here. 40 00:02:38,190 --> 00:02:43,980 These are all probabilities so if you hear me say prediction probabilities I'm referring to a long array 41 00:02:44,550 --> 00:02:48,060 of numbers like this that are probabilities. 42 00:02:48,290 --> 00:02:54,810 And we go here turn probabilities into their respective label. 43 00:02:54,810 --> 00:02:57,030 Easier to understand. 44 00:02:57,030 --> 00:02:58,590 So we'll go def get. 45 00:02:58,740 --> 00:03:03,910 We might do it of a single one so maybe get pred label will pass it. 46 00:03:03,940 --> 00:03:10,500 Prediction probabilities array which remember is just this prediction probabilities 47 00:03:13,160 --> 00:03:14,650 because what we're doing here. 48 00:03:14,660 --> 00:03:20,420 Remember our goal is to just function eyes this ugly Prince damn well it's not really that ugly it's 49 00:03:20,420 --> 00:03:25,270 actually some pretty neat code but we want to make this available to us at scale. 50 00:03:26,500 --> 00:03:28,700 Prediction probabilities yes. 51 00:03:29,230 --> 00:03:40,930 And then we want to label our doctoring for ourselves turns an array of prediction probabilities into 52 00:03:40,930 --> 00:03:46,110 a label we can just do a simple return statement here. 53 00:03:46,120 --> 00:03:54,010 Unique braids or we want to do is index a unique breeds for the ARG max of prediction probabilities 54 00:03:55,630 --> 00:04:01,270 again we're just replicating this one here all we're doing is we just want the predicted label from 55 00:04:01,270 --> 00:04:09,160 this list of numbers let's see what that looks like if we want to let's give a little example get a 56 00:04:09,160 --> 00:04:26,090 predicted label based on an array of prediction probabilities typos galore Fred label get pred label 57 00:04:26,300 --> 00:04:31,760 predictions let's go on the first index so all we're doing we're just calling our function our beautiful 58 00:04:31,760 --> 00:04:38,870 little function that we've just written and hopefully this returns us the parade label of 0 59 00:04:42,600 --> 00:04:46,260 proletariat that's what we're after 9. 60 00:04:46,500 --> 00:04:51,810 What's your favorite number mine used to be eighty one because nine times nine is eighty one one plus 61 00:04:51,810 --> 00:04:57,990 eight is eighty one eight minus one is seven I have a dog called Seven and I had a small lacks level 62 00:04:57,990 --> 00:05:03,360 eighty one once I traded it to a little American boy who has made a friends with at a resort his name 63 00:05:03,360 --> 00:05:10,130 with Gareth and he left the resort without ever trading back my know that that was the end of that story. 64 00:05:10,230 --> 00:05:11,470 All right wonderful. 65 00:05:11,550 --> 00:05:16,110 We've got a list of different predictions that our model has made and we've got some functionality to 66 00:05:16,110 --> 00:05:21,060 turn those prediction probabilities into a label. 67 00:05:21,060 --> 00:05:26,250 So we're making our predictions more easier to evaluate for our eyeballs. 68 00:05:26,250 --> 00:05:28,800 We've already got some metrics on them with tensor board. 69 00:05:28,920 --> 00:05:37,200 Now what we'll probably do is sense our validation data is in a batch form it's a batch dataset. 70 00:05:37,340 --> 00:05:38,960 We want to get a list. 71 00:05:38,960 --> 00:05:43,220 We want to UNbacked it basically because we want to rip out those images. 72 00:05:43,250 --> 00:05:48,810 So we want to get these images out of here and these labels because these are the truth labels. 73 00:05:48,950 --> 00:05:55,340 That way we can make predictions on the validation images and then get our prediction labels using our 74 00:05:55,340 --> 00:06:05,060 function and then compare those prediction labels to the actual validation labels so let's make a small 75 00:06:05,060 --> 00:06:07,310 function which does that. 76 00:06:07,330 --> 00:06:10,140 So what we'll do go here. 77 00:06:12,050 --> 00:06:21,590 Now since our validation data is still in a batch dataset remember it's in a batch dataset because that's 78 00:06:21,590 --> 00:06:24,740 very efficient tend to fly to make calculations on. 79 00:06:24,920 --> 00:06:36,320 We'll have to on batch ify it and just made that word up to make predictions on the validation 80 00:06:39,020 --> 00:06:46,630 images and then compare those predictions to the validation labels. 81 00:06:46,850 --> 00:06:48,770 Truth labels. 82 00:06:48,770 --> 00:06:51,990 This is gonna be really cool once it is up and running. 83 00:06:52,340 --> 00:06:53,810 So let's right here. 84 00:06:53,810 --> 00:06:58,470 Create a function to UN batch batch data set. 85 00:06:58,520 --> 00:07:02,870 The people who make tens of flow have actually thought about this problem well not even a problem it's 86 00:07:02,870 --> 00:07:07,910 just a little roadblock because if you batch the data set surely there's a way to unmatched and there 87 00:07:07,910 --> 00:07:12,950 is if we go tensor flow 2.0 on batch 88 00:07:16,780 --> 00:07:19,830 whereas in data set on batch 89 00:07:22,700 --> 00:07:27,920 and batch here we go splits elements of a dataset into multiple elements. 90 00:07:27,990 --> 00:07:34,400 For example if the dataset elements a shape b something something something where B may very fit I'll 91 00:07:34,410 --> 00:07:40,890 let you read that if you want but basically you've got elements here have data set from generator data 92 00:07:40,900 --> 00:07:44,580 set down on batch list data set as Num by iterator. 93 00:07:44,580 --> 00:07:51,240 So once we have unmatched it we can call as Num pi iterator and it's going to turn it into an intolerable 94 00:07:51,690 --> 00:07:52,290 list. 95 00:07:53,270 --> 00:07:59,810 So again if this code doesn't make sense when you first rate it the best way to figure it out is to 96 00:07:59,810 --> 00:08:00,360 write it. 97 00:08:00,410 --> 00:08:05,540 So let's do that let's write it in a non function first and then we'll function is it. 98 00:08:05,720 --> 00:08:11,870 So images because we want to get the validation images or we want to on batch these we want to get these 99 00:08:12,320 --> 00:08:22,300 and these into separate entities not one labels will make an empty list oh maybe these I don't want 100 00:08:22,300 --> 00:08:24,040 to override a variable I had before. 101 00:08:24,160 --> 00:08:27,220 So like images will just underscore them. 102 00:08:27,280 --> 00:08:34,100 There we go then we want to loop through unmatched data. 103 00:08:34,860 --> 00:08:37,530 So we go image label right. 104 00:08:37,590 --> 00:08:46,190 We're going to topple here of image label in Val data dot unmatched. 105 00:08:47,430 --> 00:08:52,560 So all I'm doing calling on batch and I'm going to do these in one hit. 106 00:08:52,560 --> 00:09:02,190 So I'm going to call on batch and dot as Num pi iterator in one hit so let's see what happens if we 107 00:09:02,190 --> 00:09:05,620 go print image. 108 00:09:05,850 --> 00:09:12,140 This might actually bus our computer but we'll see it anyway. 109 00:09:12,230 --> 00:09:19,270 If in doubt run the code what's it going to return boom so there we go it's gonna go that's gonna go 110 00:09:19,270 --> 00:09:24,400 for two hundred or so because that's how many samples are in our batch now validation batch but all 111 00:09:24,400 --> 00:09:32,880 it's doing now is printing out the image and the label image and the label image and the label but the 112 00:09:32,880 --> 00:09:38,940 image is remember in the form of a processed image a pre processed image which is that pre processed 113 00:09:39,060 --> 00:09:41,400 image function we wrote right at the top. 114 00:09:41,730 --> 00:09:48,860 We don't want them just like that we want to apprehend them to our list of images so append image and 115 00:09:48,920 --> 00:09:53,540 labels append label. 116 00:09:53,610 --> 00:10:00,070 So now what we should have if we go image is zero. 117 00:10:00,190 --> 00:10:01,020 Image is zero. 118 00:10:01,200 --> 00:10:08,230 So there we go so there's one image and its counterpart label should have just done that the same time 119 00:10:09,860 --> 00:10:10,580 label zero 120 00:10:14,870 --> 00:10:18,100 why in that comment. 121 00:10:18,170 --> 00:10:20,270 That's why labels 122 00:10:23,500 --> 00:10:26,320 see confused myself with the variables that we're saving. 123 00:10:26,350 --> 00:10:30,400 This is why it's helpful to function I things because if you create different variables and you lose 124 00:10:30,400 --> 00:10:36,030 track of of what variables they are you confuse yourself there we go so there's a true label. 125 00:10:36,100 --> 00:10:43,270 So this is image validation zero and it's ground truth label so our goal is to visualize this image 126 00:10:43,870 --> 00:10:48,520 and compare this ground truth label to the prediction label that it's made through here. 127 00:10:48,550 --> 00:10:49,500 Go get pred. 128 00:10:49,540 --> 00:10:53,170 Label labels 0 129 00:10:58,190 --> 00:11:06,120 wonderful Khan with the prediction label was so we could go get Red Label predictions. 130 00:11:06,260 --> 00:11:06,660 0 131 00:11:10,350 --> 00:11:11,040 Border Terrier. 132 00:11:11,070 --> 00:11:13,270 So that's a prediction that our model got wrong. 133 00:11:13,350 --> 00:11:15,170 So this is what we're trying to visualize right now. 134 00:11:15,180 --> 00:11:15,960 Let's do this. 135 00:11:15,960 --> 00:11:21,620 That's not really that fun and we want to make things fun so let's function eyes this and then we'll 136 00:11:21,630 --> 00:11:22,960 finish the video there. 137 00:11:23,130 --> 00:11:28,860 Actually that's a challenge before I go off and start functioning using this little sequence I want 138 00:11:28,860 --> 00:11:29,760 you to give it a try. 139 00:11:29,790 --> 00:11:38,510 Call the function something cool like UNbacked Fi and it should take an input of a data set a batch 140 00:11:38,560 --> 00:11:43,310 dataset and it should do this functionality and return these to list. 141 00:11:43,310 --> 00:11:44,650 So that's your little challenge. 142 00:11:44,690 --> 00:11:49,380 I'll give you a couple of seconds all right. 143 00:11:49,670 --> 00:11:50,530 I'm back. 144 00:11:50,820 --> 00:11:52,860 Didn't really leave I was here the whole time. 145 00:11:52,860 --> 00:11:58,380 Hopefully you had a little turn of a little try of a function izing this beautiful and batch of five 146 00:11:58,380 --> 00:12:11,020 function that we're going to write so we'll write some dark string takes a batch dataset of image label 147 00:12:11,830 --> 00:12:27,990 tenses and returns separate rise of images and labels tick tick tick and we want to create images because 148 00:12:27,990 --> 00:12:36,030 it's within a function we can use whatever variable name we want and then we're gonna go loop through. 149 00:12:36,040 --> 00:12:43,170 So basically all I'm doing is just rewriting this up here instead of copy posturing I'm just running 150 00:12:43,180 --> 00:12:46,020 it out again because I want to practice want to practice. 151 00:12:46,030 --> 00:12:53,250 So for image label in all we only have to use data so this is because I've set up a data parameter so 152 00:12:53,250 --> 00:12:55,750 I can just pass it any batch data set I want. 153 00:12:55,840 --> 00:12:58,250 So data on batch 154 00:13:01,370 --> 00:13:19,900 dot as Num pi iterator beautiful images dot append image labels dot append label and then we can return 155 00:13:20,710 --> 00:13:27,850 images labels we test add a little functions on batch of fi let's see what your abilities are like on 156 00:13:27,850 --> 00:13:36,370 batch if I do you stand up to the challenge of batch of filing now a validation dataset equals on batch 157 00:13:36,370 --> 00:13:44,680 of Fi and we're gonna pass it allow data batch exactly the same as what we've done here we just function 158 00:13:44,680 --> 00:13:51,230 I used it and we'll check out Val images zero and Val labels zero 159 00:13:57,360 --> 00:14:04,630 boom suggests the exact same thing they're that's what we've got here. 160 00:14:04,640 --> 00:14:17,310 One final check is we can get pride label what was it OK can maybe I'm saying that wrong figure here. 161 00:14:17,540 --> 00:14:28,010 So get Red Label Val images 0 0 9 needs to be labels 0 did that okay. 162 00:14:29,710 --> 00:14:30,810 That worked all night. 163 00:14:31,750 --> 00:14:33,810 So that's gonna work because that's numbers as well. 164 00:14:34,150 --> 00:14:40,600 So we actually want Val labels right card. 165 00:14:40,910 --> 00:14:41,620 Wonderful. 166 00:14:41,660 --> 00:14:41,910 Okay. 167 00:14:41,930 --> 00:14:48,200 So now we have a function for unmatched filing our data we can get our validation images which is great. 168 00:14:48,200 --> 00:14:54,110 So we can now work out some little functionality to make predictions on those images we can get the 169 00:14:54,110 --> 00:15:01,490 ground truth label and then compare that to the prediction label Let's delete this code here and we'll 170 00:15:01,490 --> 00:15:06,260 finish up the video and in the next one what should we do. 171 00:15:06,260 --> 00:15:12,990 We should make some functions to plot an image so visualize an image of a beautiful little dog. 172 00:15:13,040 --> 00:15:18,170 We'll also compare the truth label to the predicted label that sounds like a plan to me. 173 00:15:18,170 --> 00:15:19,100 I'll see you in the next video.