Hello everyone and welcome to part two of this multi-layer percept Tron.
Let's go ahead and jump right back to the DUPERE notebook where we left off last time.
OK.
Here I am at the Jupiter notebook.
We've gone ahead already.
Important our data explored what the data actually looks like its format and define some parameters
which you're going to use later on.
Now let's go ahead and create a function for a multilayer percept Tron.
I'm going to say if multi-layer underscore percept Tron and this function is going to take in three
arguments x the weights and the biases and what we're going to do centrally create creator model.
We'll start with two hitting layers and we're going to use what's known as the r e l activation function
which is the very simple rectifier function which essentially either returns X or zero whichever is
greater.
And then for our final output layer we'll just going to go ahead and use a linear activation with matrix
multiplication and I'll show you what that go with that actually looks like in a little bit.
But I'm going to go ahead and create a simple docstring for this function in case I want to reference
it later x.
That's just the placeholder for the data input.
The weights that I expect to get out here is just going to be a dictionary of weights.
And then finally what I want are my biases which is also going to be just a dictionary of biased values
.
Ok so first we want to do our hidden layer.
So this is our first hidden layer and we're going to use what's known as r l you.
It's a simple rectifier activation function.
So remember these neurons are going to need activation functions.
And in this case who's going to use a very simple one.
So I'm going to say layer underscore one is equal to T.F. thought and T.F. dot map mole for matrix multiplication
and it's going to take in the X and taken the weights dictionary and there's going to be our first set
of weights and we're going to actually define this dictionary in the next step as well as the biased
dictionary.
But for right now we'll pass it in and then we'll go ahead and add that to the bias or remember we go
ahead and add the bias to the X times the weight.
So if you remember back from the theory lecture you multiply the weights by the actual data that's coming
in.
So in this case you're doing a matrix multiplication because doing this all at once.
And then we add the bias to it and we're doing this all with tensor flow tensor objects and then also
in Layer 1 we'll go ahead and just say that's now equal to T.F. thought and an r e l you and pasan layer
1.
So essentially what we're doing here is two steps we say X times the weights Plus the bias value.
And here we're taking that whole thing.
So x time is way plus that bias value.
And we're going to pass it into some sort of function.
So some function in this case to function is equal to r e l u and you can actually check out the link
in the Jupiter notebook for this lecture.
But that's a very basic function.
You can check out.
OK.
So this rectifier neural network if you go ahead and check out that link.
It's essentially just I go ahead and put it here.
It looks the function looks like this f of x is equal to max of zero or x.
So that's what the r e l u function is.
So I'm gonna go out and just post that in here.
So are you all right.
And that's what our first hidden layer looks like.
And now we're going to go ahead and do the same thing for the second hidden layer.
So second hidden layer is going to have the same function and layer the underscore too.
I'm going to go ahead and say map multiplication.
And in this case we're taking in layer once it's forty one layer deep and we'll go ahead and take the
second set of weights and then I'm going to say T.F. add and add a bias to that.
So we'll say bias loops bias key beat to close that parentheses and then we'll do the same activation
.
So we're 2 is now equal to T.F. thought and thought and you can actually press tab here and check out
various activation functions that are built in to tensor flow.
But the one we're using is just a very simple one RDL you and it will work fine for our use case layer
underscore too.
OK.
And then finally we have our last output layer go it and scroll down a little bit sooner or last out
player or going to go ahead and do it say underscore layer is equal to T.F..
Matt Moel so matrix multiplication of letter to the incoming Leyre times waits and this case will have
an out key for those weights and then we'll just go ahead and say plus biases and we'll have out.
And then finally we will return the out layer.
OK.
So that's basically the multilevel Perceptor on here member we take in our data.
Multiply it by some way add a bias to it and then we pass that into some sort of activation of function
.
In this case reasoning a very simple one that just looks something like this.
And then finally we have that last output layer.
We're going to go ahead and do now is explain a little bit more about what this dictionary of weights
is and what this dictionary of biases is.
So in order for a tensor flow model to work we need to create two dictionaries containing our weights
in our biased objects for the model.
And we're going to do is introduce a new object type from tensor form called T.F. variable and this
is a little different from a constant because essentially tensor flow has what's known as a graph object
that can become aware of the states of all the variables and a variable is a modifiable tensor that
lives in tents or flows graph of interacting operations and it can be used and even modified by the
computation and it will generally have the model parameters be variables.
So we're going to go in and say weights to that dictionary is equal to.
And I'm going to have three Chirs keys in my weights.
I'll say H-1.
So let's dance for hidden layer one member that's the one we're calling here.
And that's going to be T.F. dot variable.
And remember weights are random to start off with some going to say T.F. dot random normal.
So this is essentially just if we go shift tab and check of the docstring here.
This is just a way of let me grab it from T.F. thought rather than normal.
But go ahead and check this out.
And.
Let me comment these out so we can explore this.
OK.
So TAF ran normal it just outputs random values from a normal distribution.
That's exactly what we want because we want weights to be randomly initialized.
So A.S. T.F. variable and this is the variable format.
So a variable again just maintains state in the graph across all the calls.
So we'll say T.F. thought variable CEF random underscore normal and then I'm going to pass in an underscore
input comma and underscore hit in one.
So essentially what we're asking here for is a matrix of normally distributed random values which is
going to be an input rows by and hidden.
Underscore one columns.
So if we scroll back up here that's 784 rows by 256 columns which makes sense because that's we're going
to be multiplying our matrix by that data itself x.
OK.
So that's hidden layer 1 for the weights and then do the pretty much the exact same thing for it and
layer 2 with a few changes.
So again I want it to be random normal but the shape here shouldn't be an input and a hidden one.
It's the second hidden layer should be the size of the first hidden layer by the second hidden layer
.
So a little different but same idea here and hidden under too.
And then finally we'll have the out and that's going to be a variable and this will be T.F. random normal
and this will be and hidden too.
And then we want the output to actually be number of classes and before we actually run this I'm going
to go ahead and put commas here.
So we actually separate these values.
Go ahead and run it.
Make sure we check out weights and it looks like we're good to go.
Okay and then just a quick note on this end of classes.
Essentially we're going to have the output look like is something like this is going to be an array
with a bunch of zeros with it with their index location indicating the actual number that we predict
and then we'll have one of them encoded to be hot.
So if we look at this example for instance you have 0 1 2 3 4 5 6 and 6 has a 1 in it meaning we believe
that the number output is 6 and this would actually have one more here 6 7 8 9 10.
So it would look something like this as far as what we actually expect the output to look like.
OK so we have her weights dictionary for our multi-layer percept Tron model function and then we're
going to do a very similar thing for the bias.
So you can start off with biases all just being one but you can also randomly initialised them.
So in order to save some time here we'll go ahead and just copy and paste this and change what we need
change some just copy and paste that in there.
And in this case we're dealing with bias at the first level bias to the second hidden layer and then
out.
And in this case we don't need this matrix we just need this to be an item of one list.
Is going to be and classes this will be and hit into and this will be an hidden one.
All right.
That's basically all going to look the same.
So this is just the values we're adding to the weights multiplied by the data.
Okay and finally the next thing we're going to do is just set to placeholders for x and y.
So the placeholder for X is going to be Tiff's placeholder we want float and then we're going to go
out and say none by an underscore input because that's the shape of this matrix I want I want it to
be floats and the shape is going to be done by an underscore input member.
This is just a placeholder that takes in a data type.
Previously we had done it as T.F. dot and Herty left up flow.
But it can also except just the basic string.
And the second argument here is the shape.
In this case I just want the general notation the shape to be none by the number of inputs and then
for the output is going to be a placeholder here with floats and then as the second argument to say
none comma and underscore classes.
OK so then now that we've created are x and y placeholders are biases and are wait's dictionaries in
our actual multi-layer percept Traum function.
Well I can go ahead and do is set this all and construct the model so I say pred is equal to multiday
layer underscore percept Tron that accepts X the waits dictionary and the biases.
And it looks like I got an error here that says bias is not defined.
That should be bias's something to scroll back up here and just fix that.
Here it looks like I'm mess it up here.
Biases biases run that again.
And let's run these cells again.
OK.
Looks like we're good to go now.
Next what we're going to go ahead and do is define our cost optimization functions.
We're going to go ahead and just use tensor Flo's built in functions for this part in order to make
this run in a lot simpler.
So I will say cost is equal to T.F. to reduce mean and I will say T.F. to an end for neural network
and we're going to use what's known as soft Max underscore cross entropy with logic for logistic regression
.
I will say it takes an pred and why.
And we're also going to go ahead and say optimizer is equal to T.F. train.
The atom optimizer can see there's a bunch of itemization of functions available to you and we're going
to set the learning rate of this optimization function equal to our previously defined learning rates
.
And then we're going to say minimize minimize the cost and we'll go ahead and run that.
And the next thing we're going to do is actually just train our model.
But let's go ahead and review what we did in this lecture and the next lecture will kind of put this
all together and actually train this particular model for many POCs or 15 as we find earlier.
So coming back up from where we started in the previous lecture we defined these parameters and that
in this lecture we started off by building out our multi-layer percept Tron and this multilayer perception
model takes an X the weights and the bias's remember the X is the placeholder for the data input and
then when we went ahead and said is that this first hit and layer has l activation.
And you can just either check out the notebook or check out the Wikipedia page.
But this is just a simple record her function that looks something like this.
Take the max of either 0 or X and then we're going to go ahead and then the first hidden layer will
multiply these two matrices the X matrix by the weights and then we'll add the bias to it and then we'll
pass that whole thing in to this function here.
This activation function for the second hidden layer will do the exact same thing except we've taken
the first hit and layer result.
Multiply it by the weights add the bias's to it and we're using these tensor flow operations and we
pass that whole thing into the activation function.
And then finally we have this final activation function or the final output.
I should say we're going to matrix multiply the results of the second hit layer by the weights and then
add the bias's to it.
And if you wanted to you could also say T.F. add for this last output layer step and then we go ahead
and return our underscore layer.
Now we went ahead and defined the weights and the biases and weights and biases are going to start off
as just randomly initialized values or at say T.F. but ran them normal and pasand the shape we wanted
the values for the first hidden layer for the weights.
We wanted it to be an input by unhidden to second hit and layer is going to be the first hidden layer
by the second hidden layer in the output.
The matrix form is Second-Hand layer by the number of classes and then the biases.
So we're just adding stuff is all just essentially one dimensional so the first thin layer second hidden
layer and the number of classes we want as the output.
We set up our placeholder values for x and y.
We set up this function and then we have this chaos and optimizer.
And you can basically just copy and paste these from the notebook.
But these are essentially just built in cost functions and optimization functions from tensor flow.
All right.
That's it for this lecture.
Coming up the next lecture we'll go ahead and train this model.
Thanks everyone and I'll see you there.