1
00:00:00,400 --> 00:00:02,700
Hello and welcome
back to the course on Deep Learning.

2
00:00:02,700 --> 00:00:04,566
Today we're finally at step number four.

3
00:00:04,566 --> 00:00:06,166
Full connection.

4
00:00:06,166 --> 00:00:08,300
So what is this step all about?

5
00:00:08,300 --> 00:00:11,933
Well, in this step, we're adding a whole

6
00:00:12,333 --> 00:00:16,900
artificial neural network
to our convolutional neural network.

7
00:00:16,900 --> 00:00:21,000
So to all of the things that we've done
so far which are convolution

8
00:00:21,000 --> 00:00:24,966
pooling and flattening,
now we're adding a whole new a.

9
00:00:24,966 --> 00:00:27,233
And then on the back of that.

10
00:00:27,233 --> 00:00:28,866
How intense is that.

11
00:00:28,866 --> 00:00:32,100
That is just that is something
that is definitely something.

12
00:00:32,433 --> 00:00:35,433
And so here we've got the input layer.

13
00:00:35,466 --> 00:00:37,133
We've got a fully connected layer
an output layer.

14
00:00:37,133 --> 00:00:40,566
And by the way the fully connected layer

15
00:00:40,566 --> 00:00:43,633
in artificial neural networks
we used to call them hidden layers.

16
00:00:43,933 --> 00:00:45,866
And here we're calling them
fully connected layers

17
00:00:45,866 --> 00:00:47,300
because they are hidden layers.

18
00:00:47,300 --> 00:00:51,100
But at the same time, they're a
more specific type of hidden layers.

19
00:00:51,100 --> 00:00:54,100
They're a fully connected layer
in artificial neural networks,

20
00:00:54,300 --> 00:00:57,433
hidden layers
don't have to be fully connected.

21
00:00:57,433 --> 00:01:01,400
Whereas in convolutional neural networks
we're going to be using fully connected

22
00:01:01,566 --> 00:01:01,866
layers.

23
00:01:01,866 --> 00:01:05,166
And that's why, they're generally called
fully connected layers.

24
00:01:05,633 --> 00:01:09,066
And so basically that's whole column
or vector of outputs

25
00:01:09,066 --> 00:01:12,566
that we have after the flattening
we're passing it into the input layer.

26
00:01:12,566 --> 00:01:15,200
And here
we've got a very simplified example.

27
00:01:15,200 --> 00:01:18,000
just for illustration purposes.

28
00:01:18,000 --> 00:01:22,600
And what the main purpose
of the artificial neural network is, is to

29
00:01:23,033 --> 00:01:28,866
combine our features into more attributes
that predict the classes, even better.

30
00:01:28,866 --> 00:01:33,000
So we already in our vector of outputs,

31
00:01:33,000 --> 00:01:36,866
in the flattened
and often the flattened result.

32
00:01:36,866 --> 00:01:37,800
From what we've already done,

33
00:01:37,800 --> 00:01:41,333
we have some features encoded
in the numbers in that vector,

34
00:01:41,600 --> 00:01:45,300
and they can already do
probably a pretty good job at predicting,

35
00:01:46,066 --> 00:01:51,000
what's what's, class we're looking at,
whether it's a dog or a cat

36
00:01:51,000 --> 00:01:53,766
or whether it's a tumor or not, a tumor
and so on.

37
00:01:53,766 --> 00:01:56,666
But at the same time,
we know that we have this,

38
00:01:56,666 --> 00:02:00,766
structure called artificial
neural network, which is designed which,

39
00:02:00,900 --> 00:02:05,000
which has a purpose
of dealing with attributes

40
00:02:05,400 --> 00:02:09,400
and coming up or dealing with features
and coming up with new attributes

41
00:02:09,400 --> 00:02:13,500
and combining attributes
together to even better predict,

42
00:02:14,133 --> 00:02:16,666
things that we're trying to predict.

43
00:02:16,666 --> 00:02:18,600
And we know that from the previous part.

44
00:02:18,600 --> 00:02:20,300
So why not leverage that?

45
00:02:20,300 --> 00:02:22,600
And that's exactly what the plan here is.

46
00:02:22,600 --> 00:02:26,600
So how about we pass on those values
into an artificial neural network

47
00:02:26,600 --> 00:02:30,000
and let it even further 
optimize everything that we're doing.

48
00:02:30,500 --> 00:02:31,800
And so that's what
we're going to be doing.

49
00:02:31,800 --> 00:02:34,666
But let's look
at a more realistic example.

50
00:02:34,666 --> 00:02:36,433
Because this one is a bit too simple.

51
00:02:36,433 --> 00:02:41,166
So here we've got a better looking
artificial neural network

52
00:02:41,166 --> 00:02:44,600
where we have five
attributes on the inputs than we have.

53
00:02:45,033 --> 00:02:47,366
in the first hidden layer
we have six neurons.

54
00:02:47,366 --> 00:02:52,166
In the second, or in the second fully
connected layer we have eight neurons.

55
00:02:52,166 --> 00:02:55,166
And then we have two outputs,
one for dog and one for cat.

56
00:02:55,466 --> 00:02:58,833
And so an important thing to,

57
00:02:58,833 --> 00:03:02,100
talk of for us to talk about here is that
why do we have two outputs?

58
00:03:02,100 --> 00:03:02,800
We are kind of

59
00:03:02,800 --> 00:03:06,800
used to having only one output,
in our artificial neural networks.

60
00:03:06,800 --> 00:03:12,600
Well, one output is for kind of
when you're predicting a numerical value,

61
00:03:12,600 --> 00:03:15,600
when you're when you're running
a regression type of problem.

62
00:03:15,600 --> 00:03:20,333
But, when you're doing classification,
you need an output per class,

63
00:03:20,333 --> 00:03:23,666
except for the exception is
when you have just two classes,

64
00:03:23,666 --> 00:03:25,533
like we have two classes
here, dog and cat.

65
00:03:25,533 --> 00:03:29,300
And we could have just done one output
and made it a binary output

66
00:03:29,300 --> 00:03:32,300
and said one is a dog and zero is a cat.

67
00:03:32,433 --> 00:03:33,966
And that would have worked totally fine.

68
00:03:33,966 --> 00:03:36,166
And actually, in fact,
you'll see had learned

69
00:03:36,166 --> 00:03:39,166
do that in the practical tutorials
and that's how they'll be structured.

70
00:03:39,166 --> 00:03:43,166
But at the same time,
if you have more than, two categories,

71
00:03:43,733 --> 00:03:45,700
for instance, dogs, cats and birds,

72
00:03:45,700 --> 00:03:49,200
then you have to have a neuron
for every category.

73
00:03:49,533 --> 00:03:52,333
And that's why we're going to practice
with two categories.

74
00:03:52,333 --> 00:03:53,233
In this example,

75
00:03:53,233 --> 00:03:57,600
so that we know what to expect
if we ever have more than two categories.

76
00:03:58,400 --> 00:03:59,933
And so what's going to be happening here.

77
00:03:59,933 --> 00:04:02,033
So we've already done all the groundwork.

78
00:04:02,033 --> 00:04:03,200
We've done the convolution.

79
00:04:03,200 --> 00:04:05,466
We've done the pooling and the flattening.

80
00:04:05,466 --> 00:04:09,433
And now the information is going to go
through the artificial neural network.

81
00:04:09,433 --> 00:04:12,166
So let's have a look
at how all that all happens.

82
00:04:12,166 --> 00:04:16,400
There's the information going through
from the very start from the moment

83
00:04:16,400 --> 00:04:19,800
when the image is processed
and kind of a loop convolved,

84
00:04:19,900 --> 00:04:23,466
then pooled flattened and then through
the artificial neural network,

85
00:04:23,466 --> 00:04:25,066
all four steps

86
00:04:25,066 --> 00:04:29,566
and then a prediction is made and
we'll see how this happens in a moment.

87
00:04:29,566 --> 00:04:30,633
It'll be very, very interesting.

88
00:04:30,633 --> 00:04:32,800
But for now
let's just say a prediction is made.

89
00:04:32,800 --> 00:04:37,366
And for instance, 80% that it's a dog,
but it turns out to be a cat.

90
00:04:37,833 --> 00:04:41,433
And then an error is calculated
like, well,

91
00:04:41,433 --> 00:04:45,433
what we used to call a cost function
in a artificial neural network,

92
00:04:46,100 --> 00:04:49,600
and we used a mean squared error
there, or, in common,

93
00:04:49,633 --> 00:04:52,800
delusional neural networks,
it's called a loss function.

94
00:04:52,800 --> 00:04:57,533
And we use a, cross
entropy function for that.

95
00:04:57,533 --> 00:05:00,000
And we'll talk about cross entropy
and mean squared errors

96
00:05:00,000 --> 00:05:01,466
in a separate tutorial.

97
00:05:01,466 --> 00:05:02,700
And how all that happens.

98
00:05:02,700 --> 00:05:07,433
But for now, let's just say we have a,
lost type of function which tells us how

99
00:05:07,433 --> 00:05:10,466
well our network is performing
and we're trying to optimize, optimize it

100
00:05:10,466 --> 00:05:13,466
or minimize
that function to optimize our network.

101
00:05:13,533 --> 00:05:17,566
So we the error is calculated and then
it's back propagated through the network,

102
00:05:17,566 --> 00:05:21,466
just like we had in artificial neural
networks, is back propagated.

103
00:05:21,466 --> 00:05:24,733
And the some things are adjusted
in the network

104
00:05:24,733 --> 00:05:27,733
to help optimize the performance.

105
00:05:27,900 --> 00:05:29,766
And the things that are adjusted
are, as usual,

106
00:05:29,766 --> 00:05:31,900
the weights in the artificial neural
network parts.

107
00:05:31,900 --> 00:05:34,900
So the the blue lines that you see here,
the synapses.

108
00:05:35,200 --> 00:05:37,733
Then also another thing that is adjusted

109
00:05:37,733 --> 00:05:41,566
is the feature detectors.

110
00:05:41,566 --> 00:05:44,400
So we know that
we're looking for features,

111
00:05:44,400 --> 00:05:46,066
but what if we're looking
for the wrong features.

112
00:05:46,066 --> 00:05:49,366
What if this didn't work out
because the features are incorrect.

113
00:05:49,366 --> 00:05:52,500
And so the feature detectors, those
remember those little matrices

114
00:05:52,500 --> 00:05:56,566
that we had, 
that's the three by three matrices.

115
00:05:57,233 --> 00:06:00,300
they are adjusted so that maybe

116
00:06:00,300 --> 00:06:03,800
next time it'll be better
and let's see what happens type of thing.

117
00:06:03,800 --> 00:06:06,833
And but of course, it's all done,
with a lot of,

118
00:06:07,266 --> 00:06:10,033
science in the background of a lot of math

119
00:06:10,033 --> 00:06:14,466
and it's all done through a gradient,
a gradient descent with, backpropagation.

120
00:06:14,466 --> 00:06:17,866
So it's all it's all not just random
perturbations.

121
00:06:17,866 --> 00:06:21,366
It's actually very, thought through
how it's done, but,

122
00:06:21,933 --> 00:06:25,800
nevertheless, the,
feature detectors are adjusted,

123
00:06:25,800 --> 00:06:28,700
the weights are adjusted,
and this whole process happens again.

124
00:06:28,700 --> 00:06:30,633
And then again, the errors back propagate.

125
00:06:30,633 --> 00:06:32,600
And this keeps going on and on and on.

126
00:06:32,600 --> 00:06:35,066
And that's how our network is optimized.

127
00:06:35,066 --> 00:06:37,866
That's how our network trains on the data.

128
00:06:37,866 --> 00:06:40,333
So the and the important
important thing here is that

129
00:06:40,333 --> 00:06:41,733
the data goes through the whole network

130
00:06:41,733 --> 00:06:45,733
from the very start
to the very end, then the errors compared.

131
00:06:47,233 --> 00:06:49,866
so the error is
calculated and then it's back propagated.

132
00:06:49,866 --> 00:06:50,966
So same story

133
00:06:50,966 --> 00:06:55,566
as with artificial neural networks,
just a a bit longer because of that whole

134
00:06:55,700 --> 00:06:58,700
for the first three steps
that we already had.

135
00:06:58,900 --> 00:07:01,500
And now let's have a look at the
the interesting part,

136
00:07:01,500 --> 00:07:02,400
the really interesting part.

137
00:07:02,400 --> 00:07:04,766
How do these two classes work.

138
00:07:04,766 --> 00:07:07,066
Because or
how do these two output neurons work.

139
00:07:07,066 --> 00:07:10,300
Because before we've always
kind of had one output neuron.

140
00:07:10,466 --> 00:07:11,733
What happens when we have two.

141
00:07:11,733 --> 00:07:17,200
How does how does this situation
of classification of images, play out?

142
00:07:17,500 --> 00:07:18,966
Well, let's start with the top neuron.

143
00:07:18,966 --> 00:07:21,633
First we're going to start with the dog.

144
00:07:21,633 --> 00:07:24,500
how do we the main purpose,
what we need to do

145
00:07:24,500 --> 00:07:28,833
first is we need to understand
what weights to assign

146
00:07:28,833 --> 00:07:33,000
to all of these synapses
that connect to the dog, so that we know

147
00:07:33,266 --> 00:07:37,766
which of the previous neurons
are actually important for the dog.

148
00:07:37,766 --> 00:07:38,833
And let's see how that is done.

149
00:07:38,833 --> 00:07:43,966
So let's say hypothetically,
we've got these numbers in our, previous,

150
00:07:44,833 --> 00:07:47,833
layer of previous fully connected
in the final fully connected layer.

151
00:07:47,966 --> 00:07:50,866
And again,
these numbers can be absolutely anything.

152
00:07:50,866 --> 00:07:54,000
They don't have to be that they
they can be any numbers, but

153
00:07:54,000 --> 00:07:57,000
just for argument's
sake, we're going to agree that

154
00:07:57,466 --> 00:08:01,500
we are looking specifically at numbers
between 0 and 1.

155
00:08:02,200 --> 00:08:05,200
so it's easier for us
to argue these things and understand.

156
00:08:05,566 --> 00:08:09,766
And one means that that neuron
was very confident

157
00:08:09,766 --> 00:08:11,466
that it found a certain feature.

158
00:08:11,466 --> 00:08:13,800
And zero is going to mean that,

159
00:08:13,800 --> 00:08:15,966
that neuron didn't
find the features looking for.

160
00:08:15,966 --> 00:08:18,600
So because at the end
of the day, these neurons,

161
00:08:19,900 --> 00:08:21,366
like anything

162
00:08:21,366 --> 00:08:25,166
else on this from on this left side
is just looking at features at an image.

163
00:08:25,333 --> 00:08:29,700
This is already very, very processed,
but still it's detecting a certain feature

164
00:08:29,700 --> 00:08:34,500
or combination of features on the image
right before we in the convolve step,

165
00:08:34,500 --> 00:08:36,866
we had kind of recognizable features
in the pool set.

166
00:08:36,866 --> 00:08:38,200
They're less recognizable.

167
00:08:38,200 --> 00:08:40,100
Then they become even less recognizable,
visible

168
00:08:40,100 --> 00:08:42,433
in the flattened image,
and then they get combined and so on.

169
00:08:42,433 --> 00:08:45,433
But nevertheless,
this we're talking about here,

170
00:08:45,600 --> 00:08:48,633
certain features that are present image
or their combination.

171
00:08:48,633 --> 00:08:52,800
So a one which is being passed
and this is important is being passed to

172
00:08:52,800 --> 00:08:56,966
both the dog and the cat at the same time
to both the output neurons.

173
00:08:56,966 --> 00:09:00,500
So one means that for us, for our argument

174
00:09:00,633 --> 00:09:05,033
it means that, this,
this neuron has is firing up.

175
00:09:05,033 --> 00:09:08,700
It's, it's really rapidly
detecting that feature that,

176
00:09:08,900 --> 00:09:11,766
you know, might be an eyebrow
it might be detecting this eyebrow

177
00:09:11,766 --> 00:09:15,100
for again, for simplicity's
sake, is detecting this eyebrow

178
00:09:15,100 --> 00:09:18,100
and is communicating that to the dog
neuron to the cartoon

179
00:09:18,100 --> 00:09:20,233
neuron saying I can see my eyebrow,
I can see my eyebrow.

180
00:09:20,233 --> 00:09:21,333
And then it's up to the dog

181
00:09:21,333 --> 00:09:24,966
and the cat neuron to,
understand what that means for them.

182
00:09:25,166 --> 00:09:25,733
Right.

183
00:09:25,733 --> 00:09:28,466
And so in this case,
which neurons are firing up,

184
00:09:28,466 --> 00:09:31,733
these three neurons are firing up
the eyebrow and let's say the nose

185
00:09:32,466 --> 00:09:35,900
is saying, I can see, I can see a big nose
and I can see floppy ears.

186
00:09:36,100 --> 00:09:40,466
So it and it's saying that to the dog
and to the cat and then what, the dog.

187
00:09:40,466 --> 00:09:43,300
And then what happens is
we know that this is a dog.

188
00:09:43,300 --> 00:09:47,100
So the dog neuron knows that the answer is
it is actually a dog.

189
00:09:47,733 --> 00:09:50,466
because at the end
we're comparing to the picture

190
00:09:50,466 --> 00:09:53,566
or to the label on the picture,
and it knows the dog.

191
00:09:53,566 --> 00:09:56,733
So basically the dog
neuron is going to say So

192
00:09:56,733 --> 00:09:58,633
I should be triggered in this case.

193
00:09:58,633 --> 00:10:00,333
So these are my neurons.

194
00:10:00,333 --> 00:10:04,433
They're telling the signal that they're
sending to both to me to the dog

195
00:10:04,433 --> 00:10:08,900
and to the cat is actually a indication
for me that it is a dog.

196
00:10:08,900 --> 00:10:10,566
And throughout these lots and lots

197
00:10:10,566 --> 00:10:14,100
and lots of iterations, if this happens
many times, the dog will learn that

198
00:10:14,166 --> 00:10:19,333
these neurons do indeed fire up
when the feature belongs to a dog.

199
00:10:19,500 --> 00:10:20,100
On the other hand,

200
00:10:20,100 --> 00:10:23,300
the cat neuron will know
that it's not a cat, and it will know that

201
00:10:23,533 --> 00:10:26,200
this feature is firing up
and this neuron is telling me it can see

202
00:10:26,200 --> 00:10:30,933
floppy ears, floppy ears, floppy ears,
but at the same time, it's not a cat.

203
00:10:30,933 --> 00:10:35,700
So basically, to me, that's a signal
that I should ignore this neuron as like.

204
00:10:35,700 --> 00:10:36,766
And the more that happens,

205
00:10:36,766 --> 00:10:41,266
the more the cat neuron is going to ignore
this neuron about the floppy ears.

206
00:10:42,300 --> 00:10:45,333
And so basically, that that's

207
00:10:45,333 --> 00:10:49,000
how through lots and lots of iterations
if this happens often.

208
00:10:49,000 --> 00:10:50,033
So this is just one example.

209
00:10:50,033 --> 00:10:51,000
But if this happens often

210
00:10:51,000 --> 00:10:54,333
maybe a one, maybe 0.8, 0.9,
maybe sometimes it won't fire.

211
00:10:54,333 --> 00:10:58,100
But overall on average
this neuron is lighting up

212
00:10:58,266 --> 00:11:01,100
very often when it is indeed a dog.

213
00:11:01,100 --> 00:11:05,833
The dog neuron will start attributing
higher importance to this neuron.

214
00:11:05,833 --> 00:11:08,333
And so there we go. That's that's
how we're going to signify it.

215
00:11:08,333 --> 00:11:09,300
We're going to say that

216
00:11:09,300 --> 00:11:12,800
these three neurons
through this iterative process

217
00:11:12,800 --> 00:11:16,800
with matter, with many, many, many, many
samples and many, many epochs, remember.

218
00:11:16,800 --> 00:11:19,066
So a sample is a row in your data set.

219
00:11:19,066 --> 00:11:23,066
An epoch is when you go through your whole
data set again and again and again.

220
00:11:23,166 --> 00:11:25,033
There are lots and lots of iterations.

221
00:11:25,033 --> 00:11:29,533
This dog
neuron learned that this, eyebrow neuron

222
00:11:29,533 --> 00:11:33,900
and this big nose neuron and this, 
floppy ear neuron,

223
00:11:34,200 --> 00:11:37,033
they all seem to really,

224
00:11:37,033 --> 00:11:40,033
contribute very well to the,

225
00:11:40,066 --> 00:11:43,500
classification of what it's looking for
and which is a dog.

226
00:11:44,366 --> 00:11:45,600
So that's how it works.

227
00:11:45,600 --> 00:11:49,566
And again, these ears
and nose and eyebrows, those are very,

228
00:11:49,566 --> 00:11:55,600
very, approximate
or like very far fetched examples

229
00:11:55,600 --> 00:12:00,133
because by this stage, in this whole
convolution convolutional neural network,

230
00:12:00,533 --> 00:12:03,466
it is completely unrecognizable
what they're looking for.

231
00:12:03,466 --> 00:12:07,300
But at the same time,
it is something in the features of dogs

232
00:12:07,300 --> 00:12:09,300
or cats or whatever you're classifying.

233
00:12:09,300 --> 00:12:11,100
And then so let's move on to the next one.

234
00:12:11,100 --> 00:12:12,433
Now we're going to look at the cat neuron.

235
00:12:12,433 --> 00:12:16,633
But these we're going to remember that
these weights are you know they have

236
00:12:16,666 --> 00:12:17,800
that we've sorted out the dog.

237
00:12:17,800 --> 00:12:19,000
So the dog is kind of like

238
00:12:19,000 --> 00:12:22,500
pretty much ignoring all these other
neurons one, two, three, 4 or 5.

239
00:12:22,666 --> 00:12:25,933
But it's really paying attention
to what these three neurons are saying.

240
00:12:26,400 --> 00:12:28,300
Now, what is the cat listening to?

241
00:12:28,300 --> 00:12:31,300
Well,
whenever it is actually a cat, right.

242
00:12:31,733 --> 00:12:35,500
the this is this is an example
of a situation when it's actually a cat.

243
00:12:35,500 --> 00:12:40,800
So, you'll see that this,
these three neurons, 0.9, 0.9 and one,

244
00:12:41,100 --> 00:12:44,533
they're saying something, they're saying
something to both the dog and the cat.

245
00:12:44,533 --> 00:12:45,700
And this is again important to remember.

246
00:12:45,700 --> 00:12:48,533
So this output signal goes both ways.

247
00:12:48,533 --> 00:12:49,466
It's the same. Right.

248
00:12:49,466 --> 00:12:52,566
It's it's saying one to
the dog is saying one to the cat,

249
00:12:52,566 --> 00:12:55,766
but then it's up to the dog, to the cat
to decide to whether to,

250
00:12:56,166 --> 00:12:59,833
take into account
that signal and learn from it or not.

251
00:13:00,366 --> 00:13:03,766
And both the dog
and the cat can see that this is a photo.

252
00:13:03,966 --> 00:13:06,966
I should have put a photo of a cat here,
but basically imagine a photo of a cat.

253
00:13:06,966 --> 00:13:09,966
Both the dog and the cat can see that
this is actually a cat.

254
00:13:10,033 --> 00:13:12,966
So basically the dog is like, oh, okay.

255
00:13:12,966 --> 00:13:16,900
So these whiskers
and these pointy triangle ears

256
00:13:17,133 --> 00:13:22,033
and this small size,
I guess, or I don't know, oh,

257
00:13:22,033 --> 00:13:26,333
maybe these, this type, you know,
how cats have, these things in the eyes?

258
00:13:26,333 --> 00:13:28,200
Their eyes are like little.

259
00:13:28,200 --> 00:13:29,733
They're not circles. They're,

260
00:13:30,800 --> 00:13:31,900
lines or something.

261
00:13:31,900 --> 00:13:34,833
I got, like, cat eyes. Basically.
These cat eyes.

262
00:13:34,833 --> 00:13:37,366
They're definitely not working for me.

263
00:13:37,366 --> 00:13:39,233
They're not helping me out. Predict.

264
00:13:39,233 --> 00:13:41,766
Because every time these neurons light up,

265
00:13:41,766 --> 00:13:44,100
the prediction is not
what I'm looking for.

266
00:13:44,100 --> 00:13:46,800
On the other hand, the cat is like,
that's interesting.

267
00:13:46,800 --> 00:13:50,633
Every time these this one lights up, it's
or most of the time

268
00:13:50,633 --> 00:13:53,800
it lights up, it matches my expectation.

269
00:13:53,800 --> 00:13:55,200
It matches what I'm looking for.

270
00:13:55,200 --> 00:13:57,966
Okay, I'm going to listen to this guy
more than this one.

271
00:13:57,966 --> 00:13:58,466
This one.

272
00:13:58,466 --> 00:14:00,000
Same thing every time it lights up.

273
00:14:00,000 --> 00:14:02,700
Or most of the times it lights up.

274
00:14:02,700 --> 00:14:04,700
I happen to get a good

275
00:14:04,700 --> 00:14:09,000
I happen to be rewarded for my, prediction
because I get it right.

276
00:14:09,000 --> 00:14:11,300
It's a cat. Okay,
so I'm going to listen to him more.

277
00:14:11,300 --> 00:14:15,000
You know, this one useless to me,
because he's not actually,

278
00:14:15,000 --> 00:14:18,000
you know, like, he's,
he's not even lighting up.

279
00:14:18,000 --> 00:14:20,933
It's a cat, but it's he's not lighting up.
So the opposite is happening.

280
00:14:20,933 --> 00:14:21,966
And this one as well.

281
00:14:21,966 --> 00:14:24,300
It's a cat, but he's not lighting up,
so I'm not going to listen to him.

282
00:14:24,300 --> 00:14:25,900
But this one, when he.

283
00:14:25,900 --> 00:14:27,066
When, What?

284
00:14:27,066 --> 00:14:29,700
What is this? The eyes.
The cat eyes light up.

285
00:14:29,700 --> 00:14:31,733
We can see. I can see that it's a cat.

286
00:14:31,733 --> 00:14:34,466
It matches most of the time.
So I'm going to learn from that.

287
00:14:34,466 --> 00:14:38,666
And I'm going to listen
to these three guys, more often than not.

288
00:14:38,666 --> 00:14:41,166
And so basically the cat
is listening to these three,

289
00:14:41,166 --> 00:14:43,100
and it's ignoring the other five.

290
00:14:43,100 --> 00:14:46,933
And that is how these final neurons

291
00:14:46,933 --> 00:14:49,933
learn which neurons in the,

292
00:14:50,066 --> 00:14:53,600
final fully connected layer to listen to.

293
00:14:53,600 --> 00:14:56,633
So the output neurons learn
which of the fully or

294
00:14:56,666 --> 00:14:59,866
which of the final fully connected
layer neurons to listen to.

295
00:15:00,000 --> 00:15:02,633
And that's how they understand.

296
00:15:02,633 --> 00:15:05,100
And basically that's
how the features are propagated

297
00:15:05,100 --> 00:15:08,566
through the network
and conveyed to the output.

298
00:15:08,833 --> 00:15:12,566
And so even though these, features,
of course, don't have that much meaning

299
00:15:12,600 --> 00:15:15,066
to them, like floppy ears or whiskers,

300
00:15:15,066 --> 00:15:18,200
at the same time,
they do have some distinctive

301
00:15:18,600 --> 00:15:21,766
they are a distinctive feature
of that specific class.

302
00:15:21,766 --> 00:15:25,200
And that's how the network is trained,
because we also during remember,

303
00:15:25,200 --> 00:15:29,666
during the backpropagation process,
we also adjust the feature detectors.

304
00:15:29,666 --> 00:15:33,666
So if a feature is useless to the output,
it's going to it's going

305
00:15:33,666 --> 00:15:38,233
to probably be disregarded because
this doesn't happen over 1 or 2 days.

306
00:15:38,233 --> 00:15:40,866
This happens through thousands
and thousands of iterations.

307
00:15:40,866 --> 00:15:43,966
So, with time,
a feature that is useless to

308
00:15:43,966 --> 00:15:47,100
the network is going to be disregarded
and replacement feature is useful.

309
00:15:47,100 --> 00:15:50,966
And so at the end of the day,
in this final layer of neurons,

310
00:15:50,966 --> 00:15:55,366
you are likely to have lots of features
or combinations of features from the image

311
00:15:55,633 --> 00:16:00,433
that are indeed representative,
or descriptive of dogs and cats.

312
00:16:01,566 --> 00:16:02,400
And so then

313
00:16:02,400 --> 00:16:06,566
once your network is trained up,
then we this is how it's applied.

314
00:16:06,566 --> 00:16:07,400
So this is the next step.

315
00:16:07,400 --> 00:16:09,266
Like we've already trained up our network
when this happened.

316
00:16:09,266 --> 00:16:12,933
This is what happens when the, 
this network is applied.

317
00:16:12,933 --> 00:16:15,533
So let's say we pass on an image of a dog,

318
00:16:15,533 --> 00:16:18,900
the values are propagated
through a network.

319
00:16:18,900 --> 00:16:20,466
We get certain values.

320
00:16:20,466 --> 00:16:24,866
And so this time the dog
and the cat neurons don't know.

321
00:16:24,866 --> 00:16:26,666
They don't have the image of the dog here.

322
00:16:26,666 --> 00:16:28,366
They don't know that it's a dog or a cat.

323
00:16:28,366 --> 00:16:31,733
They have no idea what it is,
but they have learned

324
00:16:31,733 --> 00:16:35,266
to listen to what is being shown here.

325
00:16:35,266 --> 00:16:35,600
Right?

326
00:16:35,600 --> 00:16:39,000
They have learned to listen to the dog ear
and listen to these three neurons.

327
00:16:39,000 --> 00:16:40,800
The cat neuron listens to these three.

328
00:16:40,800 --> 00:16:44,566
And so the dog neuron looks at one, two,
three and says, oh, these are pretty high.

329
00:16:44,766 --> 00:16:46,733
So my probability is going to be high.

330
00:16:46,733 --> 00:16:47,700
That is a dog.

331
00:16:47,700 --> 00:16:50,700
The cat neuron looks at these three
and says, okay these

332
00:16:50,866 --> 00:16:53,500
this one is pretty high
but these are pretty low.

333
00:16:53,500 --> 00:16:54,233
Interesting.

334
00:16:54,233 --> 00:16:57,033
So my probability is going to be 0.05.

335
00:16:57,033 --> 00:17:00,000
And and then and that's and that's
where you get your prediction.

336
00:17:00,000 --> 00:17:04,400
So then your first choice for this
neural network is dog.

337
00:17:04,433 --> 00:17:06,833
Second choice is cat.
And that's pretty much it.

338
00:17:06,833 --> 00:17:08,300
So the answer is dog.

339
00:17:08,300 --> 00:17:11,300
And same thing happens
when you pass an image of a cat.

340
00:17:11,433 --> 00:17:15,500
you get new values and you can see that
even though this one's higher,

341
00:17:15,500 --> 00:17:16,600
these ones are low.

342
00:17:16,600 --> 00:17:20,333
And for the cat, this one's higher,
this one's high, and this one's a bit low.

343
00:17:20,500 --> 00:17:23,866
So the probability here
might not be as great as previously,

344
00:17:23,866 --> 00:17:26,500
but still
you can see that it's a cat of 79%.

345
00:17:26,500 --> 00:17:30,000
And so therefore the neural network
is going to vote that it's a cat.

346
00:17:30,133 --> 00:17:33,133
And so basically or the neural network
is going to conclude that it's a cat

347
00:17:33,233 --> 00:17:36,233
voting is a term
that is used for these guys.

348
00:17:36,233 --> 00:17:41,433
So these neurons in the final fully
connected layer, they get to vote.

349
00:17:41,433 --> 00:17:42,700
And these are their votes.

350
00:17:42,700 --> 00:17:47,066
And again we are just for argument's
sake putting values between 0 and 1 here.

351
00:17:47,066 --> 00:17:48,300
These could be any values.

352
00:17:48,300 --> 00:17:49,500
But they get to vote.

353
00:17:49,500 --> 00:17:54,400
And then these weights
are the importance of their votes.

354
00:17:54,400 --> 00:18:00,466
So this is these these purple weights
are how the dog neuron views their votes,

355
00:18:00,466 --> 00:18:04,733
how much importance it assigns
to these neurons and to those votes.

356
00:18:04,733 --> 00:18:09,600
And this is how much importance
the cat, neuron assigns to these,

357
00:18:10,200 --> 00:18:12,633
those to the votes of these neurons.

358
00:18:12,633 --> 00:18:16,566
And so these neurons vote the dog in
the cats based on their learned weights.

359
00:18:16,566 --> 00:18:20,300
They decide who to listen to,
and then they make their predictions.

360
00:18:20,300 --> 00:18:23,400
And then the whole neural network
concludes that this is, in this

361
00:18:23,400 --> 00:18:24,500
case, a cat.

362
00:18:24,500 --> 00:18:27,200
And then that's, and then that's
your conclusion, and that's

363
00:18:27,200 --> 00:18:31,466
how you get images like this
where you have, a cheetah

364
00:18:31,466 --> 00:18:36,733
and then you have a, cheetah class with,
you know, like a high, high probability.

365
00:18:36,733 --> 00:18:39,733
So this is, you know, the probability
that the network has predicted

366
00:18:39,900 --> 00:18:40,700
and these are lost,

367
00:18:40,700 --> 00:18:43,766
but these still exist because there's
still kind of like a small chance

368
00:18:43,933 --> 00:18:48,000
that other neurons are also listening
to their voters and they're saying,

369
00:18:48,000 --> 00:18:51,333
oh, maybe it's actually a leopard
and the bullet train very, very probably.

370
00:18:51,333 --> 00:18:52,300
Here. Scissors.

371
00:18:52,300 --> 00:18:53,166
You know, this one one.

372
00:18:53,166 --> 00:18:55,366
But hand gloss was very close. Second.

373
00:18:55,366 --> 00:18:59,066
And then finally, best, stethoscope
because you could see, like these guys,

374
00:18:59,100 --> 00:19:03,300
this, this neuron, the scissors neuron,
the output services, this neuron listen to

375
00:19:03,600 --> 00:19:04,466
its voters.

376
00:19:04,466 --> 00:19:07,000
And it had the predominant vote overall.

377
00:19:07,000 --> 00:19:10,000
But then the hand class
had a good outcome as well.

378
00:19:10,066 --> 00:19:10,766
So there we go.

379
00:19:10,766 --> 00:19:15,066
That's how the full connection works
and how this is all this

380
00:19:15,066 --> 00:19:16,533
all plays out together.

381
00:19:16,533 --> 00:19:18,600
I hope you enjoyed today's tutorial.

382
00:19:18,600 --> 00:19:21,266
We're going to summarize all of this
in the summary as well,

383
00:19:21,266 --> 00:19:22,733
and I'll see you next time.

384
00:19:22,733 --> 00:19:24,633
Until then, enjoy deep learning.