1
00:00:00,420 --> 00:00:04,860
In this lesson we're going to set up tents aboard in our Jupiter notebook.

2
00:00:04,860 --> 00:00:10,740
And during this process we're going to get started working with summaries on tensor board and the file

3
00:00:10,740 --> 00:00:11,730
writer.

4
00:00:11,730 --> 00:00:17,720
The first thing that we're gonna do is under the constants let's set up a logging path.

5
00:00:18,030 --> 00:00:27,490
So call this one logging underscore path and we'll set that equal to a string called tensor board underscore

6
00:00:27,850 --> 00:00:36,030
and missed on a score digit underscore logs and we'll have a forward slash at the end.

7
00:00:36,030 --> 00:00:42,690
Hit shift enter on that cell and then scroll down to the section where we've had loss optimization and

8
00:00:42,690 --> 00:00:43,860
metrics.

9
00:00:43,920 --> 00:00:53,250
Let's add another markdown cell here just above and we'll call this one tensor board setup.

10
00:00:53,460 --> 00:01:02,280
And here what we'll do is we'll create the folders for tensor board so I'll add a common to folder for

11
00:01:03,060 --> 00:01:10,740
tensor board and we'll do him as we'll pick a folder name so the folder on the screen name is gonna

12
00:01:10,740 --> 00:01:19,720
be equal to maybe an F string and that string will read model 1 and then we'll add maybe the time right

13
00:01:19,740 --> 00:01:23,190
the current time that we have just like we did in the last module.

14
00:01:23,370 --> 00:01:27,690
If we scroll back up to our input statements we're going to add a single line here.

15
00:01:27,920 --> 00:01:37,260
That's gonna read from time import as TR f time hit shift enter on that cell scroll back down to where

16
00:01:37,260 --> 00:01:41,210
we were adding our folder name and let's call that function here.

17
00:01:41,250 --> 00:01:45,200
S t r f time open parentheses.

18
00:01:45,300 --> 00:01:46,320
Double quotes.

19
00:01:46,320 --> 00:01:57,690
Percent H colon percent M double quotes what we'll do then is we'll join the root logging path with

20
00:01:57,690 --> 00:02:09,850
the folder name so we'll call this directory is equal to who stop path dot join parentheses logging

21
00:02:09,920 --> 00:02:17,150
underscore the path comma folder name and kind that we've got our directory that we want to create.

22
00:02:17,190 --> 00:02:26,880
We'll try to create the folder with O S dot make dears parentheses directory.

23
00:02:26,880 --> 00:02:36,050
This is a review from last module and we'll catch an exception in case we have an OS error that is except

24
00:02:36,320 --> 00:02:39,780
OS error as exception.

25
00:02:39,980 --> 00:02:48,830
Colon next line we'll just print it out print out the exception exception don't SDR error so we'll get

26
00:02:48,830 --> 00:02:50,440
the error as a string.

27
00:02:50,440 --> 00:02:52,920
Suppose of some sort of code or no.

28
00:02:53,240 --> 00:03:03,800
Finally will I had an LS block where we just say print successfully created directory use exclamation

29
00:03:03,800 --> 00:03:07,470
mark add some attitude to our print statements.

30
00:03:07,550 --> 00:03:08,380
Fantastic.

31
00:03:08,480 --> 00:03:13,050
Let's hit shift enter on the cell and at that point it should have credit to folders for you.

32
00:03:13,100 --> 00:03:18,350
This root folder write him and this sub folder right here for this model.

33
00:03:18,350 --> 00:03:24,260
Now we've talked a lot about how when tensor flow runs at session it can do a lot of calculations the

34
00:03:24,260 --> 00:03:32,600
way that 10 support gets hold of these calculations is through something called a summary the summaries

35
00:03:32,660 --> 00:03:37,700
are essentially what allows tensor board to generate these pretty charts and he's pretty graphs that

36
00:03:37,700 --> 00:03:39,380
we saw in the previous module.

37
00:03:39,380 --> 00:03:45,020
The thing that you'll also remember is that tensor board was looking towards these files to read this

38
00:03:45,020 --> 00:03:48,050
information and create these charts in the first place.

39
00:03:48,110 --> 00:03:52,670
Writing these files requires a component called a file writer.

40
00:03:53,270 --> 00:03:58,490
So the file writer will take the summaries and write them to these files.

41
00:03:58,490 --> 00:04:04,610
So let's create a first summary and that summary is gonna be our accuracy metric so that a line of code

42
00:04:04,610 --> 00:04:16,040
here that's going to read T F dot summary dot scalar because accuracy is just a number and in the parentheses

43
00:04:16,550 --> 00:04:19,400
I'm going to say what the name is of the summary.

44
00:04:19,400 --> 00:04:23,670
So I'm going to call this accuracy and then what the value is.

45
00:04:23,720 --> 00:04:26,570
And this will be this accuracy metric here.

46
00:04:26,990 --> 00:04:32,020
So now that I've created my summary I can hit shift enter on the cell and I can come down here.

47
00:04:32,030 --> 00:04:39,050
We've created my session and here I'm going to add a markdown cell and that markdown cell is going to

48
00:04:39,050 --> 00:04:44,360
read set up file write up and merge summaries.

49
00:04:44,360 --> 00:04:51,200
I'm going to add a cell here and in that cell I'm going to do a couple of things I'm going to merge

50
00:04:51,350 --> 00:04:54,560
all my summaries so I might have more than one laying around.

51
00:04:54,560 --> 00:05:01,870
I've just got one at the moment but we're gonna add a few more in a second so I'm going to see merged.

52
00:05:02,400 --> 00:05:12,170
Summary it's gonna hold on to TAF dot summary dot merge all so all my summaries all the things that

53
00:05:12,170 --> 00:05:16,520
I want to read all the calculations that I want to keep hold of.

54
00:05:16,520 --> 00:05:18,600
We're going to combine them going to merge them.

55
00:05:18,740 --> 00:05:22,680
I want to store them in this thing called merged summary.

56
00:05:22,700 --> 00:05:32,810
The next thing that I'll do is I'll create my file writer so my file to which I'll call train on a school

57
00:05:32,810 --> 00:05:33,440
writer.

58
00:05:33,890 --> 00:05:43,760
It's gonna be equal to TMF dot summary dot file writer and this file writer will need a directory to

59
00:05:43,760 --> 00:05:44,260
work in.

60
00:05:44,300 --> 00:05:47,690
It's gonna need to know where to save these files.

61
00:05:47,690 --> 00:05:54,170
So I want these files to go in this directory right here that we're creating and I wanted to create

62
00:05:54,170 --> 00:05:56,900
a sub directory here called train.

63
00:05:56,960 --> 00:06:04,430
So I'll say directory plus single quotes forward slash train this directory is the same one that we've

64
00:06:04,430 --> 00:06:05,350
used here.

65
00:06:05,390 --> 00:06:12,050
Now that I've got my file writer I need to tell my file right home where to look for the calculations

66
00:06:12,050 --> 00:06:20,780
to write down and I can do that with file write a dot add on a school graph open parentheses session

67
00:06:21,590 --> 00:06:28,250
dot graph so what have I done here all of tensor flows calculations are in this thing called the graph

68
00:06:28,460 --> 00:06:35,720
and the calculations are evaluated during a session for the file right to know which calculations to

69
00:06:35,720 --> 00:06:42,470
save and write down to my disk I have to tell it which graph to use and I'm telling it here to use my

70
00:06:42,470 --> 00:06:43,670
sessions graph.

71
00:06:43,700 --> 00:06:51,590
So let me hit shift enter on the cell and I'm going to move this comment here maybe to a markdown cell.

72
00:06:51,620 --> 00:06:58,970
So now I can come down to my training loop and make use of my file writer because my file writer will

73
00:06:58,970 --> 00:07:01,910
write my summaries during the training.

74
00:07:01,910 --> 00:07:05,160
What I'd like you to do is that every epoch.

75
00:07:05,330 --> 00:07:12,830
I'd like it to create a summary for me and I'm going to do that by moving this line out of this loop

76
00:07:12,830 --> 00:07:20,750
here and to my fetches my fetches remember when we're asking our session to return some calculations

77
00:07:20,750 --> 00:07:28,700
for us I'm going to add my merged underscore summary so all the summaries that are part of this notebook

78
00:07:29,090 --> 00:07:35,710
are gonna be in our merged summary at the moment it includes my accuracy calculation so we're kind of

79
00:07:35,710 --> 00:07:36,550
doubling up him.

80
00:07:36,610 --> 00:07:39,690
But this one we're using here.

81
00:07:39,700 --> 00:07:41,840
The other one we're actually going to use intensive boat.

82
00:07:42,220 --> 00:07:47,930
So for the sake of argument Brooke good to have two things that our session will return to us.

83
00:07:47,930 --> 00:07:49,190
They're both the same thing.

84
00:07:49,210 --> 00:07:53,410
So one of them is the merch summary at the moment which will get a lot more involved.

85
00:07:53,530 --> 00:07:55,190
And the other one is the accuracy.

86
00:07:55,210 --> 00:08:00,650
The point is you can provide a list of things that you want the session to return to you.

87
00:08:00,790 --> 00:08:07,560
And on the other side we can store those things that we're returning in two separate variables called

88
00:08:07,600 --> 00:08:09,820
the first one s for summary.

89
00:08:09,820 --> 00:08:15,700
The second one for batch accuracy which will be this one right here and then our file writer will spring

90
00:08:15,700 --> 00:08:16,520
into action.

91
00:08:16,810 --> 00:08:23,100
So train underscore right to don't add under school summary.

92
00:08:23,140 --> 00:08:29,890
We'll take our summary s which we're retrieving from the session and then write it to our desk.

93
00:08:29,980 --> 00:08:34,960
And what I'd like you to do is also track the epoch in which it generated the summary.

94
00:08:34,960 --> 00:08:40,660
So every epoch every time we pass the entire data set through the neural network we're gonna ask our

95
00:08:40,660 --> 00:08:47,660
session to calculate all our summaries for us which at the moment only include our accuracy.

96
00:08:47,980 --> 00:08:50,620
And we're then going to write this to the disk.

97
00:08:50,740 --> 00:08:55,620
So now that we've added this code let's rerun our entire notebook.

98
00:08:55,690 --> 00:09:01,300
But in order to do that and not confuse tensor flow we're going to have to add a little bit of code

99
00:09:01,300 --> 00:09:06,490
at the end which will reset for the next run.

100
00:09:06,580 --> 00:09:11,260
And what I'd like to do here is I'm going to close everything right.

101
00:09:11,260 --> 00:09:13,560
I might take my file writer.

102
00:09:13,570 --> 00:09:18,820
I'm going to close it on I take my session and I'm going to close that.

103
00:09:18,940 --> 00:09:26,440
And then for good measure I'm going to reset the graph so TAF dot reset default graph parentheses will

104
00:09:26,440 --> 00:09:28,720
reset all the calculations.

105
00:09:28,750 --> 00:09:35,200
So when execute the cell FNM I come back up to where I've got set up tensor flow graph and was a run

106
00:09:35,200 --> 00:09:36,190
all below.

107
00:09:36,190 --> 00:09:41,330
So I'll take a little bit of time we're rerunning the training session and now we can fire up tensor

108
00:09:41,330 --> 00:09:41,800
board.

109
00:09:41,830 --> 00:09:49,420
So go to your command prompt or your terminal and open up a new window and their type tensor board space

110
00:09:49,870 --> 00:09:57,940
hyphen hyphen log directory log dear is equal to and then paste in the path of where you're storing

111
00:09:58,360 --> 00:09:59,590
all your logs.

112
00:09:59,590 --> 00:10:04,210
Now we've got a couple of sub directories the path that I'd actually like you to paste into the shell

113
00:10:04,480 --> 00:10:06,410
is this one right here.

114
00:10:06,430 --> 00:10:09,970
That way you're getting all the logs for all the models that you're going to run.

115
00:10:10,000 --> 00:10:17,330
If you hit enter then tentacle it will take a minute or so but after it's done I'll give you this year

116
00:10:17,340 --> 00:10:23,560
earlier which you can take and then paste into your browser when that's all done you should see your

117
00:10:23,680 --> 00:10:27,790
accuracy summary come up under the scalar section.

118
00:10:27,790 --> 00:10:28,270
Brilliant.

119
00:10:28,420 --> 00:10:29,740
So we've got that working.

120
00:10:29,740 --> 00:10:33,370
Let's add a chart for our loss for good measure.

121
00:10:33,370 --> 00:10:40,330
So back in Jupiter we can come back to where we've added our summary for accuracy scalar and we can

122
00:10:40,330 --> 00:10:45,630
add another summary for the loss and we can add another summary for the loss.

123
00:10:45,880 --> 00:10:48,390
And we do this in a very similar way.

124
00:10:48,610 --> 00:10:58,650
TMF summary scalar because our loss is also just a number and open parentheses call it loss comma and

125
00:10:58,650 --> 00:11:01,540
then loss loss is gonna be the name for this summary.

126
00:11:01,810 --> 00:11:06,520
Maybe we should just call it cost to differentiate it a little bit and the value that I'd actually like

127
00:11:06,520 --> 00:11:10,900
to plant is the result of this calculation right here.

128
00:11:10,900 --> 00:11:16,720
So if I come back up here click on the cell here and say run all below our training well recommends

129
00:11:16,900 --> 00:11:18,010
from zero.

130
00:11:18,010 --> 00:11:23,540
And if we refresh tensor a board then we should see that a new sub folder has been created here.

131
00:11:23,590 --> 00:11:29,780
Model one at whatever time you're running it slash train that's the sub folder right here.

132
00:11:30,250 --> 00:11:32,890
And and here we have another file.

133
00:11:33,010 --> 00:11:40,810
But in contrast to the first one this one has two summaries it has both the accuracy and it has the

134
00:11:40,810 --> 00:11:41,440
cost.

135
00:11:41,440 --> 00:11:47,320
So the name that shows up here is the name that we've given the summary and the values that show up

136
00:11:47,320 --> 00:11:50,670
here are the values that come out of the session.

137
00:11:50,680 --> 00:11:59,080
The reason that we get both the cost and the accuracy is because we're fetching the merged summary from

138
00:11:59,080 --> 00:11:59,820
our session.

139
00:11:59,890 --> 00:12:05,230
The merged summary if you remember includes all the summaries in the notebook.

140
00:12:05,260 --> 00:12:07,550
Thanks to this merge all function here.

141
00:12:07,570 --> 00:12:12,970
The next thing we're gonna do is we're going to add the validation loss and the validation accuracy

142
00:12:13,300 --> 00:12:15,770
and we're gonna plot these on the same chart.

143
00:12:15,790 --> 00:12:20,070
So side by side to do that create another file write to him.

144
00:12:20,360 --> 00:12:24,390
I'll call this one validation underscore right.

145
00:12:25,090 --> 00:12:31,180
And this will be equal to TMF dot summary dot file right.

146
00:12:31,720 --> 00:12:40,620
And it's going to create its own sub directory so three plus single quotes forward slash validation.

147
00:12:40,630 --> 00:12:45,470
Now in theory I should add the graph to our validation writer as well.

148
00:12:45,580 --> 00:12:51,250
But the training writer the validation writer actually using the same graph so I can leave this line

149
00:12:51,250 --> 00:12:51,970
of code out.

150
00:12:52,000 --> 00:12:57,370
I don't actually need it so I'll just add shift enter on the cell and then it might come down to where

151
00:12:57,370 --> 00:12:59,030
I'm doing my training.

152
00:12:59,050 --> 00:13:06,790
This bit him is actually going to pertain to the training data set right for our validation data said

153
00:13:07,190 --> 00:13:09,000
I'm going to take a little bit of a shortcut.

154
00:13:09,040 --> 00:13:15,160
We've got ten thousand examples and the validation data set and I'm actually not going to bother patching

155
00:13:15,160 --> 00:13:15,810
them.

156
00:13:15,820 --> 00:13:22,570
So for brevity I'll add the validation bit right here in our loop.

157
00:13:22,570 --> 00:13:28,680
It's still gonna be evaluated every epoch but it's gonna be outside of this loop.

158
00:13:28,830 --> 00:13:34,290
What we're going to do here is we're gonna request a summary let's call it summary.

159
00:13:34,370 --> 00:13:42,700
I'm going to request that summary from our session again with session says dog run and then we'll need

160
00:13:42,700 --> 00:13:45,330
those fetches right fetches.

161
00:13:45,370 --> 00:13:53,110
It's gonna be equal to the merged summary and then we have to supply the data that it should be calculated

162
00:13:53,110 --> 00:13:58,780
over and that's going to be once again our feed dictionary the feed dictionary this time around it's

163
00:13:58,780 --> 00:14:05,470
going to be made up of the validation data set though so capital X is gonna correspond to X on a square

164
00:14:05,490 --> 00:14:10,910
vowel and Capital Y is going to correspond to y underscore Val.

165
00:14:11,170 --> 00:14:17,860
This is how we're gonna calculate the cost and the accuracy from the validation features and the validation

166
00:14:18,220 --> 00:14:19,540
labels.

167
00:14:19,540 --> 00:14:26,710
Once we've got our summary our validation rider our file writer will spring into action with validation

168
00:14:26,710 --> 00:14:35,650
on a score writer don't add on the score summary and here we're gonna supply the summary and the epoch

169
00:14:35,920 --> 00:14:37,560
that the summary was generated under.

170
00:14:37,570 --> 00:14:38,820
So let's try this out.

171
00:14:38,870 --> 00:14:40,710
I think going to scroll back up him.

172
00:14:40,720 --> 00:14:42,860
I've got set up tensor flow graph.

173
00:14:43,090 --> 00:14:49,210
I'm actually going to delete these two sub folders and then I'm going to run all below we.

174
00:14:49,300 --> 00:14:56,530
We start training once again five epochs and then when we refresh tensor board or alternatively wait

175
00:14:56,590 --> 00:14:58,490
I think it does refresh by itself.

176
00:14:58,660 --> 00:15:05,430
Just very very impatient we can see that there are now two new items here.

177
00:15:05,620 --> 00:15:08,220
We've got this red thing and this blue one here.

178
00:15:08,230 --> 00:15:15,160
Now if you're intensive board and you're seeing what I'm seeing right here namely a single dot for our

179
00:15:15,160 --> 00:15:21,670
validation data set then simply refresh the page in your browser and tens a board will refresh and it

180
00:15:21,670 --> 00:15:23,900
should then pull this in correctly.

181
00:15:23,910 --> 00:15:26,970
I have definitely seen tense it would get confused sometimes.

182
00:15:27,100 --> 00:15:30,280
But a quick refresh will usually solve the problem for you.

183
00:15:30,370 --> 00:15:37,420
What you should see then is you should be able to track the accuracy of the training data and the validation

184
00:15:37,420 --> 00:15:39,490
data on the same chart.

185
00:15:39,490 --> 00:15:44,270
Typically you'll see the accuracy on the validation data set lower than the training data set.

186
00:15:44,390 --> 00:15:45,970
And that's perfectly normal.

187
00:15:45,970 --> 00:15:51,610
If you scroll down you'll also see the loss on the validation data set and the training data set on

188
00:15:51,610 --> 00:15:52,860
the same chart.

189
00:15:52,870 --> 00:15:59,620
Now back in Jupiter at the very bottom we've closed out our file writer and we've done this for our

190
00:15:59,620 --> 00:16:00,860
train underscore right.

191
00:16:00,880 --> 00:16:04,750
So let's not forget to do that for our validation a school right.

192
00:16:04,810 --> 00:16:10,540
We're going to close this one as well that we will release all the resources for the next time we run

193
00:16:10,540 --> 00:16:11,100
our notebooks.

194
00:16:11,110 --> 00:16:13,720
So I know there's a lot going on in this lesson.

195
00:16:13,750 --> 00:16:15,320
So what are the key learning points.

196
00:16:15,370 --> 00:16:18,280
What are the key takeaways from our discussions so far.

197
00:16:18,370 --> 00:16:25,670
Well any value that you want to visualize intensive board is tracked by this thing called a summary.

198
00:16:25,690 --> 00:16:29,530
We've created two different types of summaries in this lesson.

199
00:16:29,530 --> 00:16:34,900
The first one was for the accuracy and the second one was for the loss when we used our file writer

200
00:16:35,140 --> 00:16:36,670
to add these summaries.

201
00:16:36,730 --> 00:16:40,210
Then we wrote the information from the summaries toward disk.

202
00:16:40,240 --> 00:16:44,060
That's how we ended up with these files in our logging directory.

203
00:16:44,080 --> 00:16:50,340
The component that does the actual writing to the disk is something called a file writer.

204
00:16:50,410 --> 00:16:56,260
We've created two of these are trained writer and our validation writer the main thing that we needed

205
00:16:56,260 --> 00:17:00,880
to tell a file writer when we create one is where to dump the files.

206
00:17:00,880 --> 00:17:04,280
In this case it was our logging directory for our model.

207
00:17:04,330 --> 00:17:09,700
So setting up the file writer just requires a path to the location of where to dump the data.

208
00:17:09,700 --> 00:17:15,370
The only other thing that we had to do to set up our file writer was to point it to our graph and we

209
00:17:15,370 --> 00:17:21,580
did all of that using the add graph method and the graph that we used was our sessions graph.

210
00:17:21,640 --> 00:17:28,300
So far in the kind of summaries that our file writer was writing to our disk was a scalar type of summary

211
00:17:28,500 --> 00:17:34,060
the scalar summaries are the ones that generate this two dimensional chart since we had more than one

212
00:17:34,060 --> 00:17:40,420
summary that we did we use this merge all function that way when the file writer was writing our summaries

213
00:17:40,450 --> 00:17:41,350
to our disk.

214
00:17:41,350 --> 00:17:44,180
It could write all the summaries at the same time.

215
00:17:44,200 --> 00:17:49,500
In other words the file writer only had to write the merged summary in the coming lessons.

216
00:17:49,510 --> 00:17:54,250
We're going to unlock a whole lot more power from tense board and we're going to do that by using some

217
00:17:54,250 --> 00:17:56,250
of the other summaries that we have available.

218
00:17:56,260 --> 00:17:57,930
I'm already looking forward to it.

219
00:17:57,970 --> 00:17:58,960
See in the next lesson.