In this lesson we're going to talk about the tensor flow graph in detail.
This will give us a chance to review some of the code that we've written so far.
The goal for this lesson is really to connect all the puzzle pieces that we've come across and come
away with an understanding how everything fits together in tensor flow.
I know that's quite ambitious but we can do it.
So far we've seen a couple of different elements crop up in our code for example tensor flow placeholders
tensor flow constants tensor flow variables the tensor flow session and something called a feed dictionary.
These are a lot of little components to get your head around but all of these things link back to the
central idea of the tensor flow graph head over to tensor board and click graphs at the very top.
What you should then see is something like this.
This monstrosity here is art tensor flow graph.
That's pretty intimidating the very first time you see it hand.
I know that I was super confused when I first looked at it.
But let's demystify what the graph is how it works and how are all these little components that we have
in our code.
Link back to it.
So what does the graph.
The best way to understand what the graph is is to think about how we were working with Tensor Fund
the first place namely this idea of a two step process.
Step 1 was defining all the calculations and all the variables.
And Step 2 was running and evaluating our calculations.
The reason we have Step 1 is because tensor flow takes all your code and compiles this thing it compiles
the graph.
In other words it lays down the pipes before it allows us to pump any water through them.
And if I zoom in here what we see is that this graph is composed of nodes and of edges so a node is
something like this or something like this and an edge is the arrow connecting to nodes.
So what does a node represent in this graph a node represents a mathematical operation.
So addition would be an example of a node but also subtraction and multiplication and even some of the
fancy operations that we've done like calculating the activation values through our Rela activation
function.
This is why you can find a real node on our graph as well.
A lot of the other calculations you'll find as well.
For example we can go to our graph and find our soft Max calculation and we can also go to our graph
and find our cross entropy calculation.
If I check intensive board I can see my soft Max here and my cross entropy calculation here.
So every time we do a calculation with tensor flow it shows up on the graph.
When we first set up our neural network we drew some random numbers from a distribution going down all
the way on our graph.
We can find this calculation right here as a node truncated underscore normal.
Now if we look closely we can see that all the nodes are connected by these lines.
They're all connected by these little arrows these arrows are called the edges of the graph and the
edges are the things that carry the data meaning the data flows along these edges.
Say we have this ad operation here in our Python code.
This addition corresponds to this line right here in our tensor flow graph.
We can see that the two edges that go into this edition are the inputs to the addition and the arrow
that comes out of the addition is the result of this addition.
So we've got the result of the matrix multiplication going into our addition and then we've got the
result of the addition going into our reload activation function.
So in our Python code that corresponds to the result of this being added to this.
And then this result feeding into our reload activation function.
This is why you can interpret the edges as the inputs and outputs for these operations.
Now since we're adding our biases to our weights here after the multiplication we can take a closer
look at these numbers here.
This one here is upside down which makes it a little bit hard to read but it does read 512.
Same as this one hit the streets question mark.
Times five hundred and twelve.
So what do these numbers represent.
These numbers represent the shape of these two tenses.
So not only can you think of these little arrows as inputs to the add function.
This little arrow this edge actually represents a tensor and the tensor has a shape in the case of our
biases.
We have five hundred and twelve biases for that first layer.
Right.
Because we've got five hundred and 12 neurons.
And when we created our biases we created five hundred and twelve of them one for each and every single
neuron in the layer.
And this is why there is the number 512 here.
On the other side we also have five hundred and 12 different weights.
The reason we have a question mark here is because we don't know yet how many samples we have in this
tensor because we left this open for later and we did that in this hill here where we created the place
holder and we left one part of the shape blank.
So if the numbers on these edges represent the shapes of our tenses and we have data flowing between
the nodes and now we know where the flow part of the name in tensor flow comes from.
But how do we add things to our graph in the first place.
So for example what did this variable come from and also what is a variable in tensor flow the way you
can think of a variable is something that maintains the state of the graph.
I know that sounds really abstract but variables can be updated and can change in our case the variables
that we care about are the weights and the biases that weren't gonna be updating.
As we're training our neural network so let's look at where we were creating our variables.
We created our first tense of flow variables with this line of code here.
The key thing that we had to specify when creating these variables was their shape.
That's one of the reasons why this argument on the line above was so important.
This is where we specified the shape of the variable that we're creating on the line below.
Now if variables are the things that can change over time then the things that don't change over time
are called constants.
If you wanted to add a constant to the graph then we do so with T F dot constant pain you hear the constant
values are only for the initialization is the biases themselves are actually variables.
It's only their initial starting values which we've created as a constant.
Now all of this was still part of the setup process but at some point we actually wanted to do some
calculations.
We wanted to launch this graph and as a prerequisite we had to initialize all the variables that we
created.
Where did we do that.
We did it right here on this line of code.
This is the line of code that evaluated all the initialization operations that we had above and then
allowed us to start our session.
The reason we have to run this line of code is because prior to this line being executed none of the
variables actually hold any value.
The tensor flow variables only get their values after the initial analyzer is evaluated and that happens
on this line here.
It's from this point onwards when we're inside a session that we can actually evaluate a particular
tensor and look at its values be three.
Remember it was one of our biases.
These were the biases in the output layer which all had a starting value of zero.
And this is the part of the discussion where we can start talking about this thing called a session
a tensor flow session is when our place holders can start getting their values.
Remember our place holders we credit those at the very top and we had a place holder for our data for
our features and we had a place holder for our labels.
We created these two tenses and the very beginning.
So let's find them on the tensor flow graph.
This here is the place holder for our X's.
And the reason I know this is because after this creation operation we end up with a little tensor here
that has a shape of question mark by seven hundred and eighty four and that corresponds to the shape
that we've got right here.
None for the question mark and seven hundred and eighty four for the total number of inputs and how.
The thing about place holders is that place holders must be fed and they must be fed during the session.
If you want tensor flow to do some work.
This is my mental bridge the place holders are hungry and you have to feed them place holders will do
your work for food.
How do we feed them.
Well we feed them with a feed dictionary.
The feed dictionary is what we supply.
When we run a session.
You can see that we do this right here.
The reason why the feed dictionary is important is because it maps the place holder are X and it maps
are other place holder are y to the actual data.
In this case it's our batch of training features and on this line it's the features that are part of
our evaluation dataset.
The important thing is that the shapes match right.
This is the kind of check the tensor flow will actually do.
Does the place holder shape match what we're mapping it to.
So if you recall the shape of this batch underscore X was 1000 because we had 1000 samples in our batch
by seven hundred and eighty four features down here.
We had a different shape.
We had 10000 samples instead of 1000.
But we had the same number of features.
Seven hundred and eighty four.
The reason we can map both of these to the same place holder is because our place holder is willing
to accept a different number of samples as long as the number of features are consistent.
So by leaving this part of the shape blank we're able to feed it a different number of samples.
As long as we're consistent on the other dimension.
So this is something to bear in mind when you're creating your feed dictionary tensor flow will actually
check that the shape of the data matches with the shape that you specified for the place holder.
Once we're done running our session the last important thing that we did was release our resources.
So we started our session.
We ran our session did a bunch of calculations.
So at the end we close our session closing our session frees up all the resources that we used.
So I know that we've written a lot of code and we didn't really get to see it in action until we ran
our session at the very end.
So I hope this quick review tie things together and it allowed us to see the connection between the
graph and our Python code.
What I'd like to do now is actually clean this graph up because one thing that you'll notice is that
the names in it variable variable underscore one variable underscore two.
They're not terribly helpful right.
So we can actually clean our graph up and give these parts of the graph a better name that will make
it a lot more clear as to what's going on.
Let's come to the very top where we're creating our place holders and I'll hit shift tab on my keyboard
and I'll bring up the quick documentation.
If you take a look here you see that there is an additional parameter that we can supply and that is
the name the description for this name parameter is the name for the operation and they say this is
optional which is why we haven't used it so far.
But I do think naming the different operations is quite helpful because it makes the graph at the end
intensive board a lot more clear.
So a lot of come here and then I'll type name as equal to and I'll give it a capital X as the name for
the wine.
I'm going to add a name as well.
And in this case uh I pick something else I might pick labels as the name.
But let's not stop that.
Let's come down where we're setting up our layers our weights and our biases and if we hit shift tab
here again and bring up the quick documentation for the tensor flow variable we also see that it is
able to accept this name parameter.
So let's give it one name is equal to single quotes w 1 and this bias will stick with the same variable
names.
So name is equal to single quotes B one we can do the same thing for all our other layers.
So this one will call.
Name is equal to W2.
This one will call names equal to single quotes.
B 2 and we've got name is equal to w 3 and here name is equal to be three.
Now what I'll do is I'll delete all the sub directories that I've got here.
Intensive board amnesty digital logs.
And I'm going to come back into Jupiter and I'm going to run all the cells below my setup markdown cell.
At this point I should have a brand new sub folder here and I can go into my tensor board.
Let's check our graph and see if we can see our new names.
The first thing I see is a message about no data set being found but I'm going to ignore that and I'm
going to zoom in here and I can see now that my place holder was created here and here I've created
the weights for the first hidden layer here I've created the biases for my first hidden layer and they're
flowing to my add function.
This is my second hidden layer.
This here is my output layer.
My output layer flows to my soft Max activation function.
And then I calculate my cross entropy loss using the labels that I've created here.
So these are the actual labels and those alongside with whatever comes out of my output layer gets used.
When we calculate the loss with our cross entropy loss function.
Fantastic.
So that already makes things a lot more clear than they were before.
But I think we can do better than that still.
One of the things that we can do is we can start grouping some of these calculations to make our graph
even easier to understand and the things that when a group are actually all the calculations that belong
to the same layer.
So for example I want to group the creation of my weights and the matrix multiplication and this addition
of the biases and this activation.
I want to group all of this into a single layer and after that I want to do the same thing for Layer
number two and my output layer.
And that's exactly what we're going to talk about in the next lesson.
I'll see you there.