Okay so we've generated the data and we've plotted a 3D chart. 1
Now before we run our gradient descent algorithm, we're going to have to calculate the slope or the gradient 2
on all the different points on our chart. 3
We're going to have that down, because remember the gradient was what told us how far away we were from 4
the minimum, the steeper the gradient the further away we were and the higher our cost in our wonderful 5
three step machine learning process. Now previously our cost functions were a bit more simple, right? 6
All we had to do was apply our power rule and work out our derivative function. 7
But for our cost function in example 4m that's not going to be enough. 8
And the reason for this is is that this function as a little bit more complex. Before we only had a slope 9
in one dimension - the x dimension. 10
But this time we've got a slope in two dimensions because we've got two parameters - 11
we've got the X and the Y. And not only that, our functional form is a little bit more complex as well. 12
We've got a fraction, we've got exponents, it's getting a little bit more complex to do that derivative 13
here, it's 14
not such a simple process. 15
And this is a little bit of a preparation to the kind of pretty hairy cost functions that you might 16
encounter 17
going forward. A lot of those come from statistical theory and there are definitely a lot more involved. 18
So what I want to show you in this lesson is how you can tackle quite complex cost functions without 19
having to work everything out by hand. 20
So as I said before, this time around we've got a Y and we've got an X, and we're optimizing our cost 21
across two parameters. Just because the Y is low doesn't mean that the cost is low. 22
Right? 23
And this makes sense if you think about it. Looking at this graph just because we've got a low Y doesn't 24
mean that the X is low as well. 25
We have to pick both an optimal X and an optimal Y. 26
And this means working out two slopes - a slope for each parameter. And to calculate the slope we are going to calculate 27
the partial derivative. 28
Now, what's a partial derivative? 29
It's taking a derivative of a function with respect to one of the variables and holding the other 30
one constant. 31
So if we're taking the partial derivative with respect to X then we're holding Y constant. 32
That way we get the gradient in the X direction and the way we're going to go about this is I'm going 33
to show you how to calculate a partial derivative using symbolic computation in Python. 34
So back in our Python notebook at the bottom let's add a couple of cells and then the very first one 35
we're going to change to Markdown and we're gonna add a section heading here. 36
This section heading is going to read "Partial Derivatives & Symbolic Computation". 37
Now the question is what does a partial derivative actually look like and how do you work it out? Well, 38
if you remember your high school maths then this isn't gonna be a problem for you. 39
You're gonna be looking at this function right here and you're already going to be writing out the partial 40
derivatives for it. 41
But on the other hand if you're a bit rusty with your calculus or you simply want to check your work, 42
let me show you some alternative ways that you can solve this mathematical problem. 43
The first approach is being a little sneaky and typing the words "derivative calculator" into Google to 44
find an online derivative calculator that will solve this for you. I have to say this is actually a pretty 45
good idea. 46
So there's a couple of really good ones out there. 47
You know, you'll get a lot of results. 48
I checked a couple of them out, the one I quite liked was symbol lab. 49
So this is a pretty decent one. 50
So on every single one of these Web sites, you're gonna get some text box here where you have to enter your 51
equation. 52
So ours was 1 divided by 3 and then it was to the power of "-x^2 - 53
y^2" 54
So you're gonna be typing this in and it's going to be "1/(3^(-x^2 55
-y^2))" and then you got to make sure on this particular website that you go back down 56
outside of the exponent and then put "+1" here. 57
Otherwise you'll get a very, very different answer than if you have it up top. When you hit "Go", it calculates 58
the derivative for you. 59
So we've got our solution. 60
So right here you're looking at the partial derivative with respect to x. And the really cool thing about 61
a lot of these Web sites is that you can actually see the steps on how they got the solution. 62
So you can see them applying the exponent rule, applying the chain rule and then down here 63
they're applying our old friend the power rule and they're also showing you how to simplify the solution 64
in the end. Now checking out a Web site like this, 65
and you know kind of, checking the solution or revising your calculus is all very well and good but you 66
know, you and I we're doing a Python course together. 67
So how do you do this in Python and in Jupyter notebook? 68
And this is what I'm going to show you right now, because we're going to be doing symbolic mathematics 69
directly in Python using a module called "SymPy". 70
Sympy actually has their own Web site as well so you can check out the documentation here if you like. 71
You can see who's developing it. 72
You can even donate money to them. 73
So, 74
so you know these guys are doing good work, it's a good project. 75
I think the design is a little bit dated, needs a bit of a facelift but hey they're doing symbolic mathematics, 76
not designing for Apple or what have you. 77
Now as always when we want to use an external module in our Jupyter notebook, then we're going to have 78
to import it and we're going to import it at the top where we're importing all the rest of our modules. 79
So keeping all our imports together nice and tidy. 80
So from SymPy we're interested in two things, 81
we're gonna say "from sympy", all lowercase, "import symbols, diff". 82
So we're interested in importing two things. 83
One of them is the symbols functionality which will help us write mathematical notation. 84
And the other is diff, which allows us to differentiate a mathematical function. 85
All right. 86
So let me scroll back down and show you how SymPy works. 87
Also let me fix this typo here. 88
First off, we need to tell Python which mathematical symbols we're gonna use. 89
So instead of X and Y I'm going to use a and b to show the difference between our Python code a bit more 90
clearly. 91
So I'm going to say "a, b = symbols( 92
'x', 'y')" 93
By writing this line of code or telling Python that a stands for x and b should stand for y in the code 94
that we're gonna be writing next. 95
Mm hmm. 96
Now I realize this sounds a little bit confusing. 97
So I'm actually going to print out our function as it is below our cell using this notation. So I'm gonna 98
say "f(a,b)" and hit Shift+Enter. Now if you see the error "symbols is not defined", 99
then you've got to hit Shift+Enter on our import cell. What you should see instead is this, 100
you should see that Python now recognizes our function as a mathematical function and it's substituted 101
x and y for our a and b 102
arguments. So we can see here that thanks to SymPy, a and b are now treated as the x and y in our function 103
and Jupyter notebook now understands how to work with this function as we would do with a mathematical 104
equation. 105
So let's work out the partial derivatives using the diff function. So let me wrap this in a print statement 106
I'm going to say print and then single quotes 107
and in those single quotes I'm gonna say "Our cost function 108
f(x, y) is: ", and then that's gonna be the end of the string comma, "f(a,b)". 109
So I'm going to this in a print statement and I'm going to add some more print statements below because 110
we can now work out the partial derivatives with this diff functionality from SymPy. Because we've imported 111
it above, it's very, very simple to call. 112
It's "diff(f(a,b))" and then you have to provide a second argument to this. 113
You have to say what you're differentiating with respect to. 114
So if we wanted to differentiate with respect to our first parameter, then we would differentiate with 115
respect to a. Hitting Shift+Enter, 116
we see that this here is the same partial derivative that we got from using one of the online derivative 117
calculators. 118
Let me put that in a print statement as well. 119
So I'm going to say print, single quotes, comma and in those single quotes I'm going to say "Partial derivative with 120
respect to x is: " and then have the output that we just had here and then close the parentheses at the 121
end and hit Shift+Enter. 122
So now we know how to work with our cost function and how to work with our partial derivatives and this 123
brings us to our next point. 124
How do we calculate the cost at a particular point in our function? For that we need to evaluate our 125
function. 126
So we need to substitute the particular x and y values, right, like 1 and 1.5 or what have 127
you into our function. 128
The SymPy module gives us this ability through another function, namely evalf - evaluate function. 129
So here's how you would use it on our cost function. 130
I would have "f(a,b).evalf", and then parentheses and in those parentheses I can substitute 131
the values at which point I want to evaluate the cost. 132
So I would say "subs = {a: 133
1.8, b:1.0}". 134
Then close the curly braces, close the parentheses for the evaluation function and then hit Shift+Enter. 135
And what you should see is that the cost at this particular point is approximately 0.99. 136
If I take a look at our graph, I take a look at where x=1.8 and y=1 is, it's around here 137
and going up I'm at around 0.99 on the z axis. 138
Okay so that's pretty useful. 139
The only thing that's a little bit new is this notation right here. 140
This is Python code that we haven't quite seen before, because we've got these curly braces 141
and then inside we have two values - a and b, separated by a comma and then we set the values with colons. 142
So a gets the value 1.8 and b gets the value 1.0. 143
So this notation with the curly braces 144
together with the colons is the Python notation for giving both a and b a value at the same time. 145
The specific term for this is a Python dictionary. 146
We've got a key value pair. 147
A is the key and 1.8 is the value. 148
And then we've got another key value pair, b is the key and 1.0 is the value. 149
Now if you're coming from another programming language, a dictionary might have been referred to as an 150
associative array and it's very similar to an array really but instead of kind of grabbing a particular 151
value by an index, by a number, as we have done with our array, say getting the first one or getting the 152
second one, 153
in this case we are working with a key, 154
so it's like our index has a name. 155
And the reason they're called dictionaries is because they pretty much work like real dictionaries. 156
If you have a word say "bug", then that's the key. 157
And if you have the definition of this word - "unintended behavior or defect in a program that causes 158
it to crash or malfunction", then that's the value and that's it. 159
That's the dictionary data structure in a nutshell. 160
You can always spot it with the curly braces and the keys separated from the values by a colon. 161
All right. 162
So now that I've printed out the cost for our cost function at a particular point, I'm going to wrap this in 163
a print statement again, I'm going to say "print('',)", and in the single quotes I'm going 164
to write "Value of f(x,y) at x=1.8, y=1.0 is: " and then I'll have 165
the rest of our print statement like this and I'll finish it all off with a parentheses at the end. I'm going to 166
split it out over two lines like this, 167
so it's a bit more easy to read. 168
I'm going to hit Shift+Enter and evaluate this cell again. Now as a really, really quick practice, 169
I want to give you a challenge, namely can you evaluate the value of the slope with respect to x at this 170
very, very same point, at 1.8 and 1? 171
And can you print out this value, the value of the partial derivative with respect to a at 1.8 172
and 1 in the cell? 173
I'll give you a few seconds to pause the video and figure this out. 174
All right. 175
So here's the solution. 176
Once again you're going to be working with the evalf functionality, but in this case we're not gonna 177
be evaluating the cost function, we're gonna be evaluating the partial derivative, so we can say 178
"diff(f(a,b),)" and then we have to say if we want to evaluate this function 179
with respect to a or to b. I'm going to say a and then I'm going to close the parentheses for our diff. 180
So this is our partial derivative. 181
We'll put a dot after it and then again we'll say "evalf(subs=)" and now we'll provide 182
our dictionary, open curly braces "a:1.8, b:1.0" and let's print 183
this out with Shift+Enter and here we can see that the slope at this particular point in the x direction 184
is 0.037. Once again, I'm going to wrap this in a print statement and I'm going to say "Value of partial 185
derivative with respect to x: ", and I'm going to close the parentheses at the end and hit Shift+Enter.