1
00:00:00,633 --> 00:00:00,866
Hello

2
00:00:00,866 --> 00:00:03,900
and welcome back to the course on Deep
Natural Language Processing.

3
00:00:03,900 --> 00:00:07,933
And today we're talking about
classical versus deep learning models.

4
00:00:08,400 --> 00:00:09,766
So let's have a look.

5
00:00:09,766 --> 00:00:11,900
as we discussed we've got this diagram.

6
00:00:11,900 --> 00:00:15,966
And on the left we've got the natural
language processing models overall.

7
00:00:16,466 --> 00:00:20,800
And on the right
we have the deep learning models overall.

8
00:00:20,800 --> 00:00:23,966
So there overlap is a deep
natural language processing.

9
00:00:23,966 --> 00:00:27,733
Those are the very advanced types
of natural language processing models

10
00:00:27,733 --> 00:00:29,033
which use deep learning.

11
00:00:29,033 --> 00:00:30,133
And then finally at the bottom here

12
00:00:30,133 --> 00:00:33,533
we've got the sequence to sequence models
which will be interested in at the end.

13
00:00:33,966 --> 00:00:36,533
And so I thought that
this tutorial would be fun.

14
00:00:36,533 --> 00:00:40,833
if we had a look at the comparison
between a couple.

15
00:00:40,833 --> 00:00:43,833
So we'll look at a couple of examples
from here.

16
00:00:44,100 --> 00:00:45,566
we'll look at an example from here.

17
00:00:45,566 --> 00:00:49,433
And then finally we'll mention
the sequence to sequence just to kind of,

18
00:00:49,933 --> 00:00:53,966
lay out the playing field
so we know what we're dealing with.

19
00:00:54,200 --> 00:00:55,800
and what types exist.

20
00:00:55,800 --> 00:00:56,933
Because we could definitely

21
00:00:56,933 --> 00:01:00,600
just jump into sequence to sequence
and look at the most powerful things,

22
00:01:00,866 --> 00:01:04,833
but then it would still be a mystery
what our natural language processing

23
00:01:04,833 --> 00:01:07,766
models, how are they different to deep
natural language processing models

24
00:01:07,766 --> 00:01:08,500
and how are they different

25
00:01:08,500 --> 00:01:11,600
to sequence, to sequence,
what are the applications and so on.

26
00:01:12,000 --> 00:01:15,000
So let's have a look
at a couple of examples.

27
00:01:15,600 --> 00:01:16,733
All right.

28
00:01:16,733 --> 00:01:17,766
Some examples.

29
00:01:17,766 --> 00:01:19,766
We're going to have our diagram
on the right.

30
00:01:19,766 --> 00:01:22,500
And we're going to start going
through these examples one by one.

31
00:01:22,500 --> 00:01:25,666
So number one is the if else rules.

32
00:01:25,800 --> 00:01:28,733
and this is a way

33
00:01:28,733 --> 00:01:32,700
that we used
to create chatbots back in the day.

34
00:01:32,700 --> 00:01:38,766
So if else rules, are located over here
on our diagram in the, just the NLP part.

35
00:01:39,833 --> 00:01:42,333
And what they entail is

36
00:01:42,333 --> 00:01:48,000
a huge list of possible questions
and answers to those questions.

37
00:01:48,000 --> 00:01:53,100
And so once somebody in the chat
asks a question or we can identify

38
00:01:53,100 --> 00:01:56,933
that part of the sentence
is the question that we have prerecorded,

39
00:01:57,466 --> 00:01:58,900
then we will give them

40
00:01:58,900 --> 00:02:02,300
the correct answer, the answer
that is associated with that question.

41
00:02:02,733 --> 00:02:07,800
But as you can imagine, such mechanical,
such a mechanical approach

42
00:02:08,166 --> 00:02:12,133
to answering questions
or chatting with people

43
00:02:12,366 --> 00:02:15,266
does not result in, anything,

44
00:02:15,266 --> 00:02:18,000
human like anything realistic.

45
00:02:18,000 --> 00:02:21,000
And the results in something like this
rather,

46
00:02:21,000 --> 00:02:24,233
where you have questions and then you have
the predetermined answers,

47
00:02:24,233 --> 00:02:28,000
but people want something
tailored to them, something specific, like

48
00:02:28,000 --> 00:02:31,000
they're asking about something else
but doesn't fit into that,

49
00:02:31,300 --> 00:02:32,800
list of questions and answers.

50
00:02:32,800 --> 00:02:35,800
And, it just very quickly becomes a mess.

51
00:02:36,166 --> 00:02:39,600
But that's how,
we used to create chatbots,

52
00:02:40,433 --> 00:02:45,833
the next, type or the next,
natural language processing model

53
00:02:45,833 --> 00:02:48,900
we will look at is called, an audio

54
00:02:48,900 --> 00:02:51,900
frequency components analysis.

55
00:02:51,933 --> 00:02:55,733
And it's used for speech recognition
and sits over here.

56
00:02:55,733 --> 00:02:59,566
So, we're specifically looking 
like we're purposefully

57
00:02:59,566 --> 00:03:01,000
looking at different examples.

58
00:03:01,000 --> 00:03:02,166
So not just chat bots.

59
00:03:02,166 --> 00:03:04,066
We're looking at different applications.

60
00:03:04,066 --> 00:03:07,433
So you can see that
NLP can be applied to speech recognition.

61
00:03:07,766 --> 00:03:10,966
And there used to be
or still are algorithms

62
00:03:10,966 --> 00:03:15,333
that use non deep learning methods
for speech recognition.

63
00:03:15,333 --> 00:03:18,700
And one of them is audio
frequency components analysis.

64
00:03:18,966 --> 00:03:23,133
So in essence what happens is
we look at the sound wave of you know,

65
00:03:23,133 --> 00:03:26,266
somebody talking some prerecorded
or real time

66
00:03:26,566 --> 00:03:29,566
audio of a human speech.

67
00:03:29,700 --> 00:03:34,300
And then we try to identify
what waveforms exist in there.

68
00:03:34,500 --> 00:03:39,866
So in a very kind of simple,
way of explaining it,

69
00:03:39,866 --> 00:03:44,800
let's have a look at, for example, here
we've got a Fourier transformation.

70
00:03:45,666 --> 00:03:48,133
and we've got the
there's our frequency over there.

71
00:03:48,133 --> 00:03:50,566
So it's like one of
these it's going over there.

72
00:03:51,600 --> 00:03:53,066
of course this is, this doesn't

73
00:03:53,066 --> 00:03:57,666
even look like human speech to be, frank,
but the concept stage.

74
00:03:57,666 --> 00:04:02,333
So, like, we look at what frequency is,
can it consists of

75
00:04:02,600 --> 00:04:05,500
and then we compare them to

76
00:04:05,500 --> 00:04:09,000
like the pre recorded frequencies.

77
00:04:09,000 --> 00:04:13,000
So we know that, for instance,
certain certain combination of frequencies

78
00:04:13,000 --> 00:04:16,000
means, this type of word
or this type of work.

79
00:04:16,000 --> 00:04:17,633
And it's not just about the frequencies,
of course.

80
00:04:17,633 --> 00:04:19,600
It's it's much more complex than that.

81
00:04:19,600 --> 00:04:23,333
But this is a very, 
general overview of what happens.

82
00:04:23,333 --> 00:04:26,566
We look at the frequencies,
certain mathematical operations.

83
00:04:26,566 --> 00:04:32,066
This is a key point that we're not doing
any, neural, computations.

84
00:04:32,066 --> 00:04:36,000
We're not creating neural networks,
which just doing mathematical,

85
00:04:36,000 --> 00:04:40,500
calculations around the frequencies
that we can observe of comparing them

86
00:04:40,500 --> 00:04:43,833
to the mathematical calculations
we have in a library of,

87
00:04:44,366 --> 00:04:47,000
and pre analyzed frequencies.

88
00:04:47,000 --> 00:04:50,700
And then we're matching it up with finding
what word a person is saying,

89
00:04:50,766 --> 00:04:54,500
what question they're asking, or,
what the sentence is meaning.

90
00:04:54,866 --> 00:04:57,866
And then that is how we recognize speech.

91
00:04:58,600 --> 00:05:01,766
So there's another version
of natural language processing

92
00:05:02,100 --> 00:05:03,633
which is in the green area.

93
00:05:04,666 --> 00:05:06,366
Next one is

94
00:05:06,366 --> 00:05:09,833
the bag of Words model,
which is used for classification, a very,

95
00:05:09,833 --> 00:05:13,300
very popular approach for,

96
00:05:13,700 --> 00:05:16,833
text analysis
or natural language processing.

97
00:05:17,200 --> 00:05:19,700
And what it does is

98
00:05:19,700 --> 00:05:24,400
basically a while it also sits
in the NLP, part and just the green area.

99
00:05:24,733 --> 00:05:28,333
although as we will see further
on, it can sit

100
00:05:28,333 --> 00:05:31,333
both in the green area
and in the purple area over here.

101
00:05:31,600 --> 00:05:34,666
But that's, something for a later
discussion.

102
00:05:34,666 --> 00:05:37,833
What's the bag of words
model does is there's our bag of words,

103
00:05:38,666 --> 00:05:42,400
and the way it works is a
you might have, like, lots of, text.

104
00:05:42,400 --> 00:05:45,400
For instance, here we've got,

105
00:05:46,000 --> 00:05:49,133
tutors or lecturers or examiners

106
00:05:49,700 --> 00:05:53,500
who are marked, grades.

107
00:05:53,633 --> 00:05:57,600
So who gave a pass or fail
grade to, different,

108
00:05:57,900 --> 00:06:01,600
papers or different assignments
that were submitted.

109
00:06:01,600 --> 00:06:03,400
And then they also left a comment.

110
00:06:03,400 --> 00:06:07,966
So somebody said great job
and left a comment of one meaning a pass.

111
00:06:07,966 --> 00:06:08,500
Somebody said,

112
00:06:08,500 --> 00:06:12,400
amazing work left comment of one
well done, one, very well written one.

113
00:06:12,400 --> 00:06:13,300
And then poor effort

114
00:06:13,300 --> 00:06:16,300
zero could have that better, zero
try harder than exam zero and so on.

115
00:06:16,700 --> 00:06:19,666
And so from this what the bag of words
model will do is it

116
00:06:19,666 --> 00:06:24,800
will put all these words, into a bag
and it will remember.

117
00:06:24,800 --> 00:06:28,400
So how often does the word
grace come up with a one.

118
00:06:28,666 --> 00:06:30,800
And how often does
the word grade come up with a zero?

119
00:06:30,800 --> 00:06:32,366
This is again a very,

120
00:06:32,366 --> 00:06:35,766
general explanation, very high level
explanation of what happens.

121
00:06:36,066 --> 00:06:40,800
But in essence, this is, 
the concept that it will look at the words

122
00:06:40,800 --> 00:06:43,800
and it'll look
it will try to classify these words.

123
00:06:43,866 --> 00:06:48,566
It will
I will try to, associate these words

124
00:06:48,566 --> 00:06:52,100
with either positive results
or negative results in our case.

125
00:06:52,400 --> 00:06:56,100
And so in this case like amazing would be
most likely associated with positive.

126
00:06:56,100 --> 00:06:59,400
You don't often see amazing
when you're trying to say

127
00:06:59,400 --> 00:07:02,400
when when the sentiment is negative
when you try it,

128
00:07:02,400 --> 00:07:05,733
when the person was trying to say
something like the work wasn't

129
00:07:05,733 --> 00:07:08,800
well done
or well would be associated with one.

130
00:07:08,800 --> 00:07:11,800
But like the word work
could be associated with both.

131
00:07:11,800 --> 00:07:17,133
But then these other words, they would be
mostly associated past, some like words

132
00:07:17,133 --> 00:07:23,100
like poor or harder or, could have
maybe it would be associated with zeros.

133
00:07:23,400 --> 00:07:25,800
And so then it'll remember
that will keep these words in a bag.

134
00:07:25,800 --> 00:07:29,866
And next time something comes up,
for instance, somebody says,

135
00:07:31,833 --> 00:07:34,766
good, good job.

136
00:07:34,766 --> 00:07:36,600
keep keep it up or something like that.

137
00:07:36,600 --> 00:07:40,300
It will analyze the words
that are in that new sentence,

138
00:07:40,833 --> 00:07:44,100
by pulling them out of the bag
and looking at them and understanding,

139
00:07:44,100 --> 00:07:47,966
are they mostly associated ones or zeros
and then will be able to predict,

140
00:07:48,533 --> 00:07:52,466
it'll be able to classify
the new, new comment,

141
00:07:52,466 --> 00:07:56,033
even without knowing what the mark was
passive will be able to say

142
00:07:56,033 --> 00:07:59,066
based on the comment, will be able to say,
is it a pass or fail?

143
00:07:59,933 --> 00:08:03,600
so that's like a very simplistic
application of the bag of words model.

144
00:08:03,833 --> 00:08:06,833
But there's another one, an NLP model.

145
00:08:07,200 --> 00:08:08,433
okay.

146
00:08:08,433 --> 00:08:11,200
So let's look at a deep
natural language processing model.

147
00:08:11,200 --> 00:08:14,666
and this one is going to be able
to get over here.

148
00:08:15,000 --> 00:08:18,966
It's called convolutional neural networks
for text recognition.

149
00:08:19,233 --> 00:08:23,000
So they like the model
we're going to be looking at

150
00:08:23,000 --> 00:08:26,300
further down is indeed a deep
natural language processing model.

151
00:08:26,400 --> 00:08:29,900
But I was I wanted to give an example
of a different model.

152
00:08:29,900 --> 00:08:33,666
So not not just the one that we look at,
but like also a deep

153
00:08:33,666 --> 00:08:36,133
natural language processing model,
but a different one and one.

154
00:08:36,133 --> 00:08:40,033
I came across is,
when we use convolutional neural networks

155
00:08:40,033 --> 00:08:43,966
for, text recognition
and then for the, for classification.

156
00:08:44,800 --> 00:08:49,066
And so if, if you're familiar
with convolutional networks

157
00:08:49,066 --> 00:08:52,233
or even if you're not, what they're used
for is it's a neural network

158
00:08:52,233 --> 00:08:56,133
that is used for mostly image recognition
for like videos,

159
00:08:56,566 --> 00:09:01,400
self-driving cars use them to detect
obstacles on roads, people and so on.

160
00:09:01,433 --> 00:09:04,866
So mostly it's used for image
processing or video processing.

161
00:09:05,400 --> 00:09:08,400
And so it is very interesting
to see how convolutional

162
00:09:08,700 --> 00:09:11,200
neural networks
can actually be used for text processing.

163
00:09:11,200 --> 00:09:14,400
And the way it works is,

164
00:09:14,400 --> 00:09:19,566
it's these words are,
transformed into a matrix,

165
00:09:20,266 --> 00:09:24,266
and, that's done through an operation
called embedding of words.

166
00:09:24,633 --> 00:09:27,400
And then once they're in a matrix,

167
00:09:27,400 --> 00:09:30,400
the same principles as convolution,

168
00:09:30,666 --> 00:09:33,766
as we would apply for images in
convolutional neural networks are applied.

169
00:09:33,766 --> 00:09:36,733
So there's a convolution operation
going through these images.

170
00:09:36,733 --> 00:09:39,766
then they're,

171
00:09:39,766 --> 00:09:43,400
they're pulled Max
pull a min pulled or sampled.

172
00:09:43,766 --> 00:09:46,766
and then they're flattened.

173
00:09:47,000 --> 00:09:50,000
and then, you know,
then we have the prediction.

174
00:09:50,200 --> 00:09:53,400
So we won't go into detail right now
of how convolution

175
00:09:53,966 --> 00:09:56,433
convolutional, operations work.

176
00:09:56,433 --> 00:10:00,666
there's lots, of, tutorials out there
and so on, and it's

177
00:10:00,766 --> 00:10:03,200
just not in the scope of this course,
because in this course, we'll be looking

178
00:10:03,200 --> 00:10:06,666
at mostly recurrent neural networks,
which is RNN and a different type

179
00:10:06,666 --> 00:10:07,600
of neural networks.

180
00:10:07,600 --> 00:10:11,366
But I just wanted to put it out there
that you could technically use

181
00:10:11,366 --> 00:10:14,466
convolutional neural networks
for text recognition,

182
00:10:14,466 --> 00:10:18,200
just as you would do, with images,
a very interesting concept.

183
00:10:18,200 --> 00:10:21,300
That was one of the earlier ones
that was explored.

184
00:10:21,300 --> 00:10:26,533
And then we moved on to, recurrent neural
networks, as we'll see down the track.

185
00:10:27,266 --> 00:10:28,933
So there we go. That's another one.

186
00:10:28,933 --> 00:10:32,400
And then finally, the main model
that we'll be looking for,

187
00:10:32,400 --> 00:10:36,166
working with sequence to sequence has
many, many, many different applications.

188
00:10:36,166 --> 00:10:37,766
As we'll see further down.

189
00:10:37,766 --> 00:10:39,300
And that's the one that's over there.

190
00:10:39,300 --> 00:10:42,433
So I'll give you a short
preview of what's coming up.

191
00:10:42,466 --> 00:10:43,600
Looks something like this.

192
00:10:43,600 --> 00:10:47,600
And don't worry,
it might look very complex right now, but

193
00:10:48,133 --> 00:10:52,766
in a couple of tutorials from here
will be, very,

194
00:10:52,766 --> 00:10:57,433
very comfortable with what's going on here
and how this whole construct is working

195
00:10:57,433 --> 00:11:01,800
and why it's called sequence to sequence
and what exactly it allows us to do.

196
00:11:02,400 --> 00:11:03,033
So there we go.

197
00:11:03,033 --> 00:11:07,000
I hope you enjoyed this quick overview
and comparison

198
00:11:07,300 --> 00:11:10,800
of, the different types of models
and different,

199
00:11:11,300 --> 00:11:14,633
applications that they have,
and we'll continue next time.

200
00:11:14,800 --> 00:11:16,033
I look forward to seeing you there.

201
00:11:16,033 --> 00:11:19,233
And until then, enjoy deep NLP.