1
00:00:00,233 --> 00:00:00,600
Hello and

2
00:00:00,600 --> 00:00:03,600
welcome back to your Machine
Learning A-To-Z course.

3
00:00:03,833 --> 00:00:05,366
Are you super excited?

4
00:00:05,366 --> 00:00:08,033
Because I am super excited about this
tutorial.

5
00:00:08,033 --> 00:00:11,033
We're talking about nonlinear support
vector regression.

6
00:00:11,100 --> 00:00:14,666
This is going to be quite
an advanced tutorial, but don't worry,

7
00:00:14,833 --> 00:00:16,400
we will break it down.

8
00:00:16,400 --> 00:00:19,833
And as per
our mission will make the complex simple.

9
00:00:20,133 --> 00:00:23,166
So and before we start, I just wanted to
make sure we're on the same page.

10
00:00:23,166 --> 00:00:24,666
So this is one of the most advanced

11
00:00:24,666 --> 00:00:28,500
tutorials in the Support Vector
series of tutorials.

12
00:00:28,566 --> 00:00:32,166
Please make sure that you've completed
all the previous tutorials,

13
00:00:32,400 --> 00:00:36,700
on SVR and SVM and kernels
before proceeding with this tutorial,

14
00:00:36,700 --> 00:00:39,700
because you'll need all that knowledge
to be able to tackle this one.

15
00:00:40,200 --> 00:00:41,666
All right. And off we go.

16
00:00:41,666 --> 00:00:43,600
Here it is.

17
00:00:43,600 --> 00:00:46,100
So imagine you have a data set like this.

18
00:00:47,133 --> 00:00:49,300
and you're trying to fit

19
00:00:49,300 --> 00:00:53,400
a support support
vector regression onto this, plot.

20
00:00:53,400 --> 00:00:56,966
You want to see what kind of, trend line
is there

21
00:00:56,966 --> 00:01:00,266
or what kind of, equation is behind this?

22
00:01:00,300 --> 00:01:01,300
How do you model?

23
00:01:01,300 --> 00:01:03,033
What kind of model can we fit to this?

24
00:01:03,033 --> 00:01:04,733
A support vector regression model

25
00:01:04,733 --> 00:01:08,066
so that you can predict
what the next value is going to be for.

26
00:01:08,733 --> 00:01:10,566
and you point that is added to the plot.

27
00:01:10,566 --> 00:01:12,300
So if you know x, y is going to be y,

28
00:01:12,300 --> 00:01:16,633
well you might want to try and fit
a linear support vector regression.

29
00:01:16,633 --> 00:01:19,533
And it'll look something like this.
You'll have these support vectors.

30
00:01:19,533 --> 00:01:23,800
But as you can see like intuitively
you can tell it doesn't fit right.

31
00:01:23,800 --> 00:01:27,000
Like you can tell that
something is going on here beyond linear.

32
00:01:27,000 --> 00:01:32,733
And like while this it might model may be
okay there somewhere at some points.

33
00:01:32,866 --> 00:01:35,733
Clearly
if you get a point with X over here,

34
00:01:35,733 --> 00:01:37,000
you're going to get the incorrect value.

35
00:01:37,000 --> 00:01:41,133
Or if X is over here you're going to get
an incorrect value which doesn't fit.

36
00:01:41,400 --> 00:01:43,466
Looks like something else is going on.

37
00:01:43,466 --> 00:01:46,566
You might try something like that,
then that won't fit as well.

38
00:01:47,266 --> 00:01:48,033
That doesn't fit.

39
00:01:48,033 --> 00:01:51,000
So it doesn't look like a linear
model fits here.

40
00:01:51,000 --> 00:01:54,300
And it looks like something
more complex is going on.

41
00:01:54,300 --> 00:01:57,733
So how do we build a support
vector regression

42
00:01:58,100 --> 00:02:01,100
that will fit, our data here.

43
00:02:01,533 --> 00:02:04,200
Well,
this is where nonlinear SVR comes in.

44
00:02:04,200 --> 00:02:07,733
It's going to be, quite interesting
because we need to go to a new dimension.

45
00:02:08,133 --> 00:02:10,933
And that's what that's
why we're going to add

46
00:02:10,933 --> 00:02:13,133
this box is just for visualization
purposes.

47
00:02:13,133 --> 00:02:15,600
Like our data hasn't changed.
Our axes are the same.

48
00:02:15,600 --> 00:02:18,600
But just for us, once
we're transitioning to a third dimension,

49
00:02:19,100 --> 00:02:20,833
to keep track of things,
we're going to add this box

50
00:02:20,833 --> 00:02:24,900
and we're going to add these diagonals
as well as a point, a red,

51
00:02:25,033 --> 00:02:25,900
a circle in the middle.

52
00:02:25,900 --> 00:02:28,900
Just so visually we can see
what's going on.

53
00:02:29,400 --> 00:02:30,133
it'll help us.

54
00:02:30,133 --> 00:02:33,466
It's just like a visual aid has nothing
to do with the algorithm itself.

55
00:02:33,466 --> 00:02:35,400
It's for illustrative purposes.

56
00:02:35,400 --> 00:02:37,733
So now we're going
to actually take this and,

57
00:02:39,366 --> 00:02:42,000
look at it from a different perspective
at an angles.

58
00:02:42,000 --> 00:02:44,633
So at an angle it looks like this.

59
00:02:44,633 --> 00:02:47,866
and that will allow us
to build that dimension Z.

60
00:02:48,300 --> 00:02:51,800
Now let's get rid of the, previous view.

61
00:02:52,000 --> 00:02:55,766
And we're going to make a copy of it,
because there's

62
00:02:55,766 --> 00:02:57,333
going to be quite a lot of things
going on.

63
00:02:57,333 --> 00:02:59,166
And in order to keep track of everything,

64
00:02:59,166 --> 00:03:01,933
we'll have two charts
that are developing in parallel.

65
00:03:01,933 --> 00:03:05,600
So now on the right,
what we're going to do is, just as we did

66
00:03:05,600 --> 00:03:07,866
previously with supposed
to support Vector Machine,

67
00:03:07,866 --> 00:03:10,600
and that's why it's important to have seen
those tutorials.

68
00:03:10,600 --> 00:03:13,066
we're going to add a kernel.

69
00:03:13,066 --> 00:03:16,933
And in this case
we're going to add a new function

70
00:03:16,933 --> 00:03:18,133
to take us into a third dimension.

71
00:03:18,133 --> 00:03:21,433
We're going to add the,
a radial basis function or the RBF.

72
00:03:22,200 --> 00:03:23,800
So here we go. That's what it looks like.

73
00:03:23,800 --> 00:03:24,900
That's with the RBF.

74
00:03:24,900 --> 00:03:26,833
You can see our data vaguely underneath.

75
00:03:26,833 --> 00:03:29,266
But here you can see it more clearly.

76
00:03:29,266 --> 00:03:32,266
And just as a reminder, the, formula

77
00:03:32,266 --> 00:03:35,266
for the RBA function is over there.

78
00:03:35,600 --> 00:03:38,200
Now, if we project our data

79
00:03:38,200 --> 00:03:42,333
into, onto this RBF,
you'll see how it works.

80
00:03:42,333 --> 00:03:46,933
So first of all,
the center is zero or this .0000 on x,

81
00:03:47,000 --> 00:03:51,433
the y axis, projects
straight into the very top.

82
00:03:51,600 --> 00:03:54,866
And if you, we,
we discuss this before in the, tutorials,

83
00:03:55,066 --> 00:03:58,066
if you put 000L
is the distance from the center,

84
00:03:58,533 --> 00:04:01,666
you'll get,
at the very top, you'll get a one.

85
00:04:02,733 --> 00:04:04,466
Now let's see how these other points,

86
00:04:04,466 --> 00:04:08,266
the other points in our visualization
will be projected onto the view.

87
00:04:08,266 --> 00:04:10,166
So we're going to start
with this point over here.

88
00:04:10,166 --> 00:04:15,666
So that point will be projected into this
point in the z axis somewhere here.

89
00:04:15,866 --> 00:04:19,200
By the way, some of these things
might not be absolutely 100% accurate.

90
00:04:19,500 --> 00:04:23,333
as I'm, drawing them here,
but they deliver.

91
00:04:23,333 --> 00:04:26,333
Hopefully
they deliver the point home. Okay.

92
00:04:26,366 --> 00:04:29,566
So now this point would be projected
to over here somewhere

93
00:04:29,766 --> 00:04:33,300
next point, over there
and this point, this point.

94
00:04:33,633 --> 00:04:36,166
And then the following points
would be almost at zero.

95
00:04:36,166 --> 00:04:40,200
So all this blue stuff over here,
the bottom is very, very close to zero.

96
00:04:40,200 --> 00:04:41,700
So it would be very close to zero.

97
00:04:41,700 --> 00:04:44,900
So you can see
we've been keeping track on on the left.

98
00:04:44,900 --> 00:04:47,100
We've done all of these points over here.

99
00:04:47,100 --> 00:04:51,133
Now the other points
the remaining points these ones

100
00:04:51,133 --> 00:04:54,266
they're actually we can't see them
are not in order not to clutter things.

101
00:04:54,466 --> 00:04:57,133
Anything in this quadrant,
in this triangle

102
00:04:57,133 --> 00:05:00,133
here, we're not going to plot it
because it's on the back.

103
00:05:00,166 --> 00:05:05,300
It's on the opposite side of this
radial basis function or this, view.

104
00:05:05,633 --> 00:05:09,133
And we just are going to plot it,
but just we can just imagine

105
00:05:09,133 --> 00:05:12,900
that they are also projecting
onto this mountain on the other side.

106
00:05:13,566 --> 00:05:14,933
Now, what happens next?

107
00:05:14,933 --> 00:05:18,466
Well, next we need to in,

108
00:05:19,500 --> 00:05:21,033
these three dimensions, we need

109
00:05:21,033 --> 00:05:24,900
to, create run a linear model.

110
00:05:24,900 --> 00:05:26,700
So now we're in three dimensions.

111
00:05:26,700 --> 00:05:31,300
Now we can run a linear model and a linear
model in three dimensions is not a line.

112
00:05:31,300 --> 00:05:34,300
So in two dimensions
it's a straight line in a three them.

113
00:05:34,300 --> 00:05:37,566
In three dimensions
a linear model is a hyperplane.

114
00:05:37,566 --> 00:05:41,266
So now if we run a linear model
you'll see that it will.

115
00:05:41,300 --> 00:05:42,866
How does it fit
our data would fit our data.

116
00:05:42,866 --> 00:05:45,233
Something like this.

117
00:05:45,233 --> 00:05:48,466
and what do we want from here.

118
00:05:48,466 --> 00:05:50,233
So once we fit the hyperplane

119
00:05:50,233 --> 00:05:53,233
to our data in these three dimensions,
what do we want from here?

120
00:05:53,500 --> 00:05:56,433
we want to see
where does it actually cross,

121
00:05:56,433 --> 00:06:00,933
where does it intersect our RBF,
our radial basis function.

122
00:06:00,933 --> 00:06:04,933
So basically all the surfaces we see here,
including the floor, this whole surface

123
00:06:04,933 --> 00:06:08,000
where does it intersect will intersect it
over across this yellow line.

124
00:06:08,000 --> 00:06:11,000
So that's the point of intersect
or the line of intersect.

125
00:06:11,166 --> 00:06:14,533
Now if we get rid of the plane,
we're left of this line.

126
00:06:14,533 --> 00:06:19,833
And if we projected back down
onto the two dimensions x and y.

127
00:06:20,066 --> 00:06:21,033
Look watch what happens.

128
00:06:21,033 --> 00:06:23,466
So we're not going to draw it
on this plot. We can draw it on this plot.

129
00:06:23,466 --> 00:06:24,666
So watch what happens.

130
00:06:24,666 --> 00:06:26,666
There
we go. That's our two dimensional line.

131
00:06:26,666 --> 00:06:30,733
So so that's our intersect line
projected on the two dimensional plot.

132
00:06:31,033 --> 00:06:32,933
And that effectively is our model.

133
00:06:32,933 --> 00:06:37,366
So if we now look at it in the perspective
that we started with, you'll see that

134
00:06:37,366 --> 00:06:38,900
that is what it looks like.

135
00:06:38,900 --> 00:06:41,900
That's the, non-linear SVR.

136
00:06:41,933 --> 00:06:44,133
Now, I know
you probably have a few questions.

137
00:06:44,133 --> 00:06:47,366
Let's, try to dig into some of the,

138
00:06:47,433 --> 00:06:51,600
comments
or some of the concepts behind that.

139
00:06:51,600 --> 00:06:55,066
So first of all, here
we have this two dimensional plot.

140
00:06:55,066 --> 00:06:58,266
Here is our,
three dimensional three dimensions.

141
00:06:58,300 --> 00:07:00,966
So why is this SVR right.

142
00:07:00,966 --> 00:07:03,900
So where are the support vectors? Well,
you know, three dimensions.

143
00:07:03,900 --> 00:07:04,933
You know, three dimensional space.

144
00:07:04,933 --> 00:07:09,300
SVR would run very similarly
to how it runs in a two dimensional space

145
00:07:09,600 --> 00:07:14,000
instead of having but instead of having
like a, epsilon instead sort of cube,

146
00:07:14,266 --> 00:07:18,200
we would have an epsilon insensitive space
between hyperplanes.

147
00:07:18,200 --> 00:07:21,000
So we would add these two new hyperplanes.

148
00:07:21,000 --> 00:07:24,600
One would be epsilon above,
one would be epsilon below.

149
00:07:24,833 --> 00:07:26,666
Remember how we had for linear SVR.

150
00:07:26,666 --> 00:07:28,200
We had just a tube.

151
00:07:28,200 --> 00:07:31,133
but now here
we have this space in between them.

152
00:07:31,133 --> 00:07:35,833
And so basically what that means is
any, points

153
00:07:36,166 --> 00:07:42,166
that are in between, 
these two x like the most outmost,

154
00:07:42,200 --> 00:07:47,366
hyperplanes, any points in between them,
they won't be considered towards,

155
00:07:47,900 --> 00:07:52,100
like, the error for those points
will not be considered.

156
00:07:52,100 --> 00:07:56,266
What we want to do is we want to minimize
the error, from this epsilon

157
00:07:56,266 --> 00:07:59,633
insensitive space
to the remaining points that are outside.

158
00:07:59,933 --> 00:08:04,766
And what this looks like over here
is that's our bottom, hyperplane.

159
00:08:04,766 --> 00:08:08,266
That's its projection away,
or that's a projection of, the part

160
00:08:08,266 --> 00:08:11,400
where it intersects with our,

161
00:08:11,400 --> 00:08:15,333
radial basis function,
and that's a top, hyperplane.

162
00:08:15,366 --> 00:08:18,666
That's its intersection
or the projection of its intersection.

163
00:08:19,133 --> 00:08:22,133
So this line over here is projected
into that one.

164
00:08:22,400 --> 00:08:24,900
And, basically anything

165
00:08:24,900 --> 00:08:27,900
in between was, the epsilon

166
00:08:27,900 --> 00:08:30,933
insensitive space that has become
this epsilon sensitive tube.

167
00:08:31,166 --> 00:08:33,766
And these are the support vectors.

168
00:08:33,766 --> 00:08:36,333
The crucial point here is that
this is the outcome originally.

169
00:08:36,333 --> 00:08:39,633
This is these hyperplanes are identified

170
00:08:39,633 --> 00:08:42,766
or built from this view
from the three dimensional view.

171
00:08:42,766 --> 00:08:45,366
So that's
where these points come in place.

172
00:08:45,366 --> 00:08:47,266
So this point is actually underneath.

173
00:08:47,266 --> 00:08:48,400
We can't see it. It's over there.

174
00:08:48,400 --> 00:08:50,666
And there's also these two points
that are on the left.

175
00:08:50,666 --> 00:08:52,466
So from here they're built.

176
00:08:52,466 --> 00:08:56,466
And then this is just a projection
just to put it into a perspective.

177
00:08:57,333 --> 00:08:58,766
So that's why it's still a support

178
00:08:58,766 --> 00:09:02,166
vector regression
because we have these points now.

179
00:09:03,000 --> 00:09:05,700
hopefully that explains
why it's a support vector regression.

180
00:09:05,700 --> 00:09:08,700
There's just one last thing
I wanted to mention.

181
00:09:09,066 --> 00:09:11,733
The final thing is that everything

182
00:09:11,733 --> 00:09:15,400
we've been talking about here
is for illustrative purposes.

183
00:09:15,400 --> 00:09:19,300
In reality,
the way this works is very similar

184
00:09:19,466 --> 00:09:22,766
to the how the support vector
machine works,

185
00:09:22,766 --> 00:09:25,900
how what we discussed about
the kernel trick that in reality

186
00:09:25,900 --> 00:09:28,900
we don't have to go into a third
dimension.

187
00:09:29,066 --> 00:09:34,200
Do all these projections and then, 
find our hyperplanes,

188
00:09:34,200 --> 00:09:36,433
find this epsilon sensitive space,

189
00:09:36,433 --> 00:09:37,000
and then predict

190
00:09:37,000 --> 00:09:40,000
everything back and get this,
it would be too computationally expensive.

191
00:09:40,000 --> 00:09:43,500
So what we actually discussed here
is, a more complex approach.

192
00:09:43,500 --> 00:09:47,466
This would be like the full fledged
approach with going into a third

193
00:09:47,466 --> 00:09:51,300
dimension, doing the hyperplanes
coming back, getting our end result.

194
00:09:51,566 --> 00:09:55,466
That would be, like the
The Full Monty in a way.

195
00:09:55,833 --> 00:10:00,733
But effectively what happens or in reality
is that we use the kernel trick.

196
00:10:00,733 --> 00:10:03,733
So just like with the support
vector machine,

197
00:10:03,766 --> 00:10:05,866
we don't have to go into a third
dimension.

198
00:10:05,866 --> 00:10:09,400
Everything happens in the same space
simply by how we use the kernel trick.

199
00:10:09,600 --> 00:10:11,766
Now, we won't dive
into more detail on that.

200
00:10:11,766 --> 00:10:17,266
I'll let you, think about that,
in your own time on how the kernel trick

201
00:10:17,266 --> 00:10:21,300
would be used here, because simply it's
a it's an easier concept than what

202
00:10:21,300 --> 00:10:24,900
we discussed by looking at the example
of the kernel trick in the SVM,

203
00:10:25,300 --> 00:10:29,233
we can extrapolate how that would work
or how that would benefit us here as well.

204
00:10:29,833 --> 00:10:33,066
But in a nutshell,
this is what the non-linear support vector

205
00:10:33,066 --> 00:10:34,200
regression is all about.

206
00:10:34,200 --> 00:10:38,366
And, the best part is, of course,
the result that it allows us to model,

207
00:10:38,633 --> 00:10:41,633
non-linear relationships like this

208
00:10:41,766 --> 00:10:44,766
and get, insights from our data.

209
00:10:45,000 --> 00:10:46,233
I hope you enjoyed this tutorial.

210
00:10:46,233 --> 00:10:49,800
This was quite an advanced concept, so
congratulations on making it to the end,

211
00:10:50,233 --> 00:10:53,100
and I look forward to seeing you
on the future tutorials in the course.

212
00:10:53,100 --> 00:10:55,166
And until then, happy analyzing!