1
00:00:00,520 --> 00:00:04,230
So we've really started to evaluate how our DOD vision model is going.

2
00:00:04,240 --> 00:00:10,360
By visualizing our models predictions should I give you a little challenge before we get into this video

3
00:00:10,360 --> 00:00:12,790
which is saving and reloading a model.

4
00:00:12,790 --> 00:00:18,340
What an interesting thing might be is now that we've visualized the images the predicted label the predicted

5
00:00:18,340 --> 00:00:27,180
probability the true label and the top 10 predictions how could you make a confusion matrix to figure

6
00:00:27,180 --> 00:00:30,450
out where your model is confused with what we've got.

7
00:00:30,540 --> 00:00:36,510
So remember a confusion matrix is comparing the predicted labels to the true labels and then seeing

8
00:00:36,510 --> 00:00:41,520
which ones the model gets confused because that would probably be really helpful with this sort of problem

9
00:00:41,970 --> 00:00:45,600
and helpful with any kind of multi class classification.

10
00:00:45,750 --> 00:00:48,920
So that's a little challenge to you a little bit of extra curricular.

11
00:00:49,050 --> 00:00:52,500
I'll put it here before we save and load a model.

12
00:00:52,500 --> 00:00:54,020
Let's go here.

13
00:00:54,210 --> 00:01:08,740
Challenge how would you create a confusion matrix with our models predictions and true labels.

14
00:01:09,440 --> 00:01:17,210
So if you do figure it out make sure you share it in the discord chart you can share your work and help

15
00:01:17,210 --> 00:01:19,730
others if they're trying to figure it out as well.

16
00:01:19,810 --> 00:01:22,090
With that being said good luck with the challenge.

17
00:01:22,100 --> 00:01:26,030
Let's create a little title for saving and reloading a model.

18
00:01:26,030 --> 00:01:27,610
Now this is helpful.

19
00:01:27,620 --> 00:01:33,860
Of course it's taken us a while to get here so it makes sense to want to save our progress after we've

20
00:01:33,860 --> 00:01:37,570
tried to model let's figure out how we would do that.

21
00:01:37,630 --> 00:01:38,930
A trained model.

22
00:01:39,340 --> 00:01:42,140
And it also helps if you wanted to share your model.

23
00:01:42,150 --> 00:01:47,850
So say we wanted to export our dog vision prediction capability to someone else.

24
00:01:47,860 --> 00:01:49,950
We wanted to put it in a web application.

25
00:01:50,090 --> 00:01:53,650
We're not might need to get it out of our car lab notebook.

26
00:01:53,650 --> 00:01:55,380
So that's what I'm gonna do here.

27
00:01:55,530 --> 00:01:56,560
TENSOR flow.

28
00:01:56,950 --> 00:02:02,930
How to Save a model so this is what I would do.

29
00:02:03,170 --> 00:02:08,040
Oh that guy's really excited just like me just like us.

30
00:02:08,070 --> 00:02:11,060
Let's save and load models.

31
00:02:11,060 --> 00:02:13,190
Again we could read through here.

32
00:02:13,960 --> 00:02:17,510
But what we're gonna do is we're just going to write the code first and then if you want to read up

33
00:02:17,510 --> 00:02:20,890
on a little bit more you can read through the documentation here.

34
00:02:21,020 --> 00:02:24,920
So we'll create a function to do it so we can call it on any model.

35
00:02:24,920 --> 00:02:34,110
So create a function to save a model and do it nice and simple.

36
00:02:34,110 --> 00:02:41,340
Def save model pass on a model and suffix equals now and you'll see what suffix means in a second.

37
00:02:41,340 --> 00:02:51,480
So just like how log directory in dog vision we might create a new folder called models.

38
00:02:52,870 --> 00:02:56,270
And what we're going to do is as you might have guessed is save.

39
00:02:56,270 --> 00:03:02,850
We've got our training logs here and save our models to dog vision so we keep it all within that one

40
00:03:02,850 --> 00:03:03,930
project folder.

41
00:03:04,050 --> 00:03:11,820
Sticking with what we've got in our projects set up create a project folder.

42
00:03:11,950 --> 00:03:13,340
So this is going to go.

43
00:03:13,390 --> 00:03:26,680
Saves a given model given model in a model's directory and appends a suffix string.

44
00:03:26,680 --> 00:03:28,150
Nice and simple.

45
00:03:28,150 --> 00:03:36,750
So if we go here let's create a model directory path name with current time.

46
00:03:37,030 --> 00:03:41,200
So just like our tend to flow logs callback that we created a few videos ago.

47
00:03:41,260 --> 00:03:46,460
We want to first of all save our model with the current time so we can track our experiments.

48
00:03:46,810 --> 00:03:51,110
And we're gonna give it a little suffix so we also know a little bit about our model.

49
00:03:51,130 --> 00:03:54,800
So we're not just opening our models folder and it's just timestamps.

50
00:03:54,820 --> 00:03:58,860
We'd also like to know how many images was it trained on and stuff like that.

51
00:03:58,870 --> 00:04:00,100
So that's what we'll do.

52
00:04:00,320 --> 00:04:09,760
We'll go model day equals US path the joint just like our tensor board my drive slash dog vision slash

53
00:04:09,820 --> 00:04:16,720
models and then we'll append will join it up with the current time which we can use date time for that.

54
00:04:16,890 --> 00:04:24,950
So daytime daytime now that string from time.

55
00:04:25,100 --> 00:04:40,690
And we're going to go year month to month day our month on minute second.

56
00:04:41,810 --> 00:04:43,610
OK beautiful.

57
00:04:43,920 --> 00:04:50,790
But now we also want model path where we want to combine the model directory with our suffix so model

58
00:04:50,860 --> 00:05:01,150
the plus hyphen plus suffix plus dot height 5.

59
00:05:01,160 --> 00:05:05,060
Now if you're wondering what height 5 is that's the same format of model.

60
00:05:05,120 --> 00:05:09,740
It's just a file specification just like JPEG and if you read through the documentation it's somewhere

61
00:05:09,740 --> 00:05:10,270
here.

62
00:05:10,590 --> 00:05:11,600
It's got a huge five

63
00:05:14,300 --> 00:05:21,130
there we go save whites with a page five ext.

64
00:05:21,190 --> 00:05:28,110
There we go so that's gonna be the extension and we just want to print a little update for ourselves.

65
00:05:28,340 --> 00:05:37,960
If saving model to we'll go and model path Dr. Dot just to let us know that it's in progress and we'll

66
00:05:37,960 --> 00:05:39,040
go to model dot save.

67
00:05:39,040 --> 00:05:46,400
So there's a method called Save that you can call on a model just like model not predict and you posit

68
00:05:46,420 --> 00:05:47,500
a file path.

69
00:05:47,500 --> 00:05:49,350
Let's see if the documentation will come up.

70
00:05:50,580 --> 00:05:51,450
There we go.

71
00:05:51,480 --> 00:05:55,920
Saves the model to tend to flow save model or single HD F5 file.

72
00:05:55,920 --> 00:05:58,230
So that's what I meant by page 5.

73
00:05:58,230 --> 00:05:59,550
So that's what we're doing.

74
00:05:59,730 --> 00:06:07,860
And then return model path because we want to see the model path we can use that later if we want okay.

75
00:06:08,110 --> 00:06:12,540
Now we've got a function to save a model while we're here.

76
00:06:12,550 --> 00:06:17,740
We'll build a function to load a model because if we save the model we want to potentially load that

77
00:06:17,740 --> 00:06:18,220
back in.

78
00:06:18,250 --> 00:06:24,160
So it makes sense to load our trained model and then we'll test them both at the same time create a

79
00:06:24,160 --> 00:06:31,130
function to load a trained model so this one is gonna be very similar to what we do.

80
00:06:31,140 --> 00:06:31,770
So we go.

81
00:06:32,680 --> 00:06:36,120
Def load model I want to pass in a model path.

82
00:06:36,150 --> 00:06:41,430
So it's just like hey we've saved a model this low return the model path we've saved a model to this

83
00:06:41,430 --> 00:06:49,140
file path which is going to be our models folder somewhere here and so to load model we want to go.

84
00:06:49,200 --> 00:07:04,480
Okay what can we call we can go loads a saved model from specified path and then we go print f loading

85
00:07:04,990 --> 00:07:08,800
saved model from where is it loading it from.

86
00:07:08,800 --> 00:07:16,440
So just this is just for our own sanity to make sure that we're loading the right model model not t

87
00:07:16,440 --> 00:07:24,190
after carers models dot load model again saving and loading models you would find this load model here.

88
00:07:24,310 --> 00:07:25,690
So load model.

89
00:07:25,690 --> 00:07:29,910
There we go recreate the exact same model including its white and the optimizer.

90
00:07:30,580 --> 00:07:36,270
So because we've used tens of fly hub we actually need to do a little special thing here.

91
00:07:36,910 --> 00:07:46,140
So model path custom objects so this documentation a lot of carriers intensify.

92
00:07:46,140 --> 00:07:51,910
Documentation is of the simplest use case but since we're not using the simplest use case we're using

93
00:07:51,910 --> 00:08:00,550
a test flight hub model we need to pass it this little custom objects dictionary so carer's layer we

94
00:08:00,550 --> 00:08:07,000
just need to tell it hey we've got an irregular layer so a layer you wouldn't have seen before in our

95
00:08:07,000 --> 00:08:12,130
model and that's all we have to do so all we're doing is loading a model from model path and saying

96
00:08:12,130 --> 00:08:20,600
hey we've got a custom little piece of equipment in our model in particular where is it right back up

97
00:08:20,600 --> 00:08:25,040
here look how much we've done you should be really proud of yourself you're watching this hub dot carer's

98
00:08:25,040 --> 00:08:26,540
layer that's all we have to do.

99
00:08:27,020 --> 00:08:31,490
So instead of just using these kind of layers which are the traditional carer's layers tend to flow

100
00:08:32,270 --> 00:08:33,610
we're using this special one.

101
00:08:33,650 --> 00:08:35,690
So that's what we're telling this load model function

102
00:08:39,420 --> 00:08:43,920
and then we want to return the model.

103
00:08:43,990 --> 00:08:47,110
So now the ultimate test let's put it here.

104
00:08:47,680 --> 00:08:59,470
Now we've got functions to save and load a trained model let's make sure they work so what we'll do

105
00:08:59,650 --> 00:09:00,820
is we'll save our model

106
00:09:03,630 --> 00:09:05,340
trained on a thousand images.

107
00:09:05,340 --> 00:09:08,010
So this is where the suffix comes in.

108
00:09:08,040 --> 00:09:12,690
So because we traded on a thousand images we want people to know that this is only a thousand image.

109
00:09:12,700 --> 00:09:20,880
We want ourselves to know that this is only a thousand image model suffix equals a thousand images and

110
00:09:20,880 --> 00:09:24,160
we'll probably put Adam there for Adam optimizer in case you wanted to change.

111
00:09:24,180 --> 00:09:26,340
You could really put whatever suffix you want here.

112
00:09:26,340 --> 00:09:27,850
Thousand images.

113
00:09:27,860 --> 00:09:31,090
The mobile Net V2.

114
00:09:31,340 --> 00:09:31,980
Adam.

115
00:09:32,370 --> 00:09:34,260
That's probably nice and simple.

116
00:09:34,270 --> 00:09:36,520
So let's see what this does.

117
00:09:36,570 --> 00:09:43,080
And again this is just a communicative part not 100 percent necessary but I found in practice it's very

118
00:09:43,080 --> 00:09:48,180
good to communicate just even if it's to yourself as well as if you wanted to share with someone else

119
00:09:48,210 --> 00:09:53,520
they go they wouldn't just look in your models director and go hey which model is which they could go

120
00:09:53,630 --> 00:09:57,450
Oh there's a suffix here and then maybe you could tell them about that suffix

121
00:10:00,660 --> 00:10:03,040
saving up unable to create file.

122
00:10:03,060 --> 00:10:04,250
What's happened here.

123
00:10:04,440 --> 00:10:05,830
You know what.

124
00:10:06,030 --> 00:10:07,810
That's exactly what I thought.

125
00:10:07,830 --> 00:10:08,330
Look at that.

126
00:10:08,340 --> 00:10:10,690
What's wrong with that classic.

127
00:10:10,710 --> 00:10:15,660
So these trust me you're going to run into these errors these and reshape errors getting your data into

128
00:10:15,660 --> 00:10:16,440
the right shape.

129
00:10:16,490 --> 00:10:22,120
Going to plague you for your entire machine learning career.

130
00:10:22,280 --> 00:10:25,100
Typos are a program is worst nightmare.

131
00:10:25,280 --> 00:10:26,810
So we go here up there we go.

132
00:10:26,810 --> 00:10:27,400
Saved.

133
00:10:27,470 --> 00:10:29,780
So there's the file path to the model.

134
00:10:29,790 --> 00:10:34,070
And now if we come up into the models directory there we go.

135
00:10:34,090 --> 00:10:35,490
There's our saved model.

136
00:10:35,490 --> 00:10:36,810
So dot Hage five.

137
00:10:36,870 --> 00:10:38,310
Wonderful.

138
00:10:38,310 --> 00:10:49,200
So if we X out of that let's now test our load model so load try and model so all we have to do is we're

139
00:10:49,200 --> 00:10:55,160
going to go loaded thousand image model long nine there.

140
00:10:55,170 --> 00:11:00,280
But just for clarity and we go load model.

141
00:11:00,540 --> 00:11:08,290
This is why we return that so we can just copy that they're will like that shouldn't take too long.

142
00:11:11,950 --> 00:11:12,220
Now.

143
00:11:12,220 --> 00:11:13,640
Probably gonna have some warnings here.

144
00:11:13,720 --> 00:11:14,350
Yes.

145
00:11:14,440 --> 00:11:17,010
So these warnings.

146
00:11:17,110 --> 00:11:18,090
What is it saying.

147
00:11:18,280 --> 00:11:22,720
Sequential models without the input shape past the first line cannot reload the optimizer state as a

148
00:11:22,720 --> 00:11:26,510
result your model is instantiating that's a fairly long warning.

149
00:11:26,710 --> 00:11:34,430
But if you come up to here and check warning and I think it's one of the top ones.

150
00:11:34,860 --> 00:11:35,550
There we go.

151
00:11:35,550 --> 00:11:40,920
This may generate warnings if you load a model related to the saving State of the optimizer these warnings

152
00:11:40,980 --> 00:11:46,890
and similar warnings throughout this notebook are in place to discourage outdated usage and can be ignored

153
00:11:48,990 --> 00:11:54,390
so these warnings can be ignored according to the documentation.

154
00:11:54,570 --> 00:11:58,230
So be helpful if they just remove them rather than just have them printed out.

155
00:11:58,230 --> 00:12:06,140
But aside from that let's just see if it works if it works then that's where all gravy.

156
00:12:06,240 --> 00:12:11,620
So evaluate the pre saved model.

157
00:12:11,700 --> 00:12:16,870
Now another function you can call on your train models is evaluate and pass it the data set value data.

158
00:12:18,030 --> 00:12:21,580
And so we're going to run that.

159
00:12:21,600 --> 00:12:23,780
So this is our model before saving it.

160
00:12:23,910 --> 00:12:26,720
And this is our model after loading it.

161
00:12:26,940 --> 00:12:35,400
So evaluate the loaded model the model dot evaluate Oh no I only nearly messed up.

162
00:12:35,420 --> 00:12:43,160
We're on autopilot now and started typing about the cell above image model dot evaluate Val data.

163
00:12:43,710 --> 00:12:51,290
So if all is swell they should return the exact same thing.

164
00:12:51,310 --> 00:12:51,890
Wonderful.

165
00:12:52,360 --> 00:12:57,250
So this is just an evaluate function to quickly evaluate your models performance on the validation set

166
00:12:58,120 --> 00:13:04,660
and we can say here the loss is one point 2 6 0 7.

167
00:13:04,660 --> 00:13:05,420
Same as this.

168
00:13:05,430 --> 00:13:07,960
And the accuracy is the exact same.

169
00:13:07,960 --> 00:13:15,480
So there we go how to save and load a model now is saved and loaded a model.

170
00:13:15,480 --> 00:13:16,910
I think it's time.

171
00:13:17,250 --> 00:13:20,670
I think it is time to step it up and train a model.

172
00:13:20,670 --> 00:13:22,200
We've got it all working.

173
00:13:22,440 --> 00:13:26,120
Let's train one on the full dataset.

174
00:13:26,550 --> 00:13:29,950
So get yourself ready get yourself excited.

175
00:13:30,150 --> 00:13:31,140
I'll see you in the next video.

176
00:13:31,140 --> 00:13:33,750
We're going to train a model on the full 10000 images.