In this lesson we're going to talk about named scopes and the reason we're gonna talk about name scopes
is so that we can start grouping different types of operations together in our tensor flow graph.
And while we're doing that we'll also get a chance to talk about some Python programming concepts in
particular.
We're going to make use of something called the context manager.
Wouldn't it be nice to group certain operations together in our graph for example this set of operations
here all happens in the first hidden layer.
We've got our weights that are multiplied with our inputs and then we've got our biases that are added
to our weights after the multiplication and then we have all of that feed into our activation function.
All of this could be represented as a single unit.
If we do that for every single layer we can see how the different layers are connected on our tensor
flow graph the way we can accomplish this intense flow is with something called a name scope the name
scope will help us organize our calculations.
And it does this using something called the context manager.
So if you think about it all the different calculations that are part of say the first hidden layer
they should really belong to the same context.
So with T F dot name on a score scope open parentheses single quotes output underscore layer colon at
the end and then select all of these lines of code and hit tab on my keyboard to invent them.
And what this does is it makes all of these lines part of the same block.
All of these lines of code now have the same context and that context has a name and that name is going
to be output layer.
So so all of these operations that I've got here that all belong to my output layer should not be grouped
in tensor bold so let's see if this works.
So let's see if I can come here to set up tensor flow graph and go to cell run all below and update
my graph looks the same as before.
Well it's because it is the same as before I found my model at 15 0 6 and at 15 25.
So that's why I have to change this to my most recent run.
And now you can see all the operations that have the same name scope grouped in my output layer.
If I wanted to peep inside and see what operations there are actually inside all I need to do is double
click and then I can see all the operations all the calculations that I've grouped under the same name
scope.
So what I'd like you to do is pause the video real quick and group all the other operations that belong
to the second hidden layer.
And the first hidden layer to clean up our graph.
I'll give you a few seconds to pause the video and give this a go.
Ready.
Here's the solution.
So we have to do a little bit of reorganization in our code to add a name scope to our second hidden
layer.
It's pretty straightforward because all we need to do is write with T F name underscore scope and we
can give us any name we want hidden the square to semicolon and use this right here and indented.
So that's our second hidden layer done for our first hidden layer.
We're gonna have to move some of our code to the same cell to make our life easier.
So I'll take this one him with him take this bit of code him move it below.
Same with this one.
Move it him and I'll take these three cells here and I'm going to delete them and now I can add the
same line with TAF dot name scope open parentheses single quotes hidden under square one and colon at
the end.
Take this indented and then what I can do is simply go to this markdown cell here and see run all below
back in tense of board refresh my page and then I'm gonna go to my most recent model this one right
here and here we can already see that our layers are grouped very very nicely now we've got our inputs
our axis going to our first hidden layer and the output of our first layer is going to our second hidden
layer and the output of the second hidden layer is going to our third hidden layer our output layer
but I tell you what looking back at our code it's starting to look a little bit repetitive right we've
got this block right here we've got this block right here and this block right here I think that we
can all see that most of this code is actually the same so why not create a single function that will
set up all our layers for us I'm going to define a function kind of call that function setup underscore
layer and it's going to take maybe three inputs that first input I'll just call input the second input
I'll call it Wait underscored dimensions the I am the third input I'll call bias underscore dimensions
and that fourth input I'll call name and what I'm going to do is I'm going to half with statement with
block here that reads TMF dot name scope open parentheses name colon and then inside that block I'm
basically going to do this I'm basically going to take everything I normally do to set up a layer and
I'll do that inside my function.
I won't have to tab over a little bit and actually fix this to having him but going back a little bit
so shift temp will go back one.
And now I can start replacing things that will vary with my parameters.
So this shall read wait I mentioned the.
I'm actually just gonna call W and the biases.
I'm just going to call a capital B the shape for my biases are gonna be my bias dimension and then I'll
make my variable names a little bit more generic.
This is gonna be initial w this is gonna be w and there's gonna be initial w this is gonna be initial
B.
This is gonna be initial B and this will just be b layer and here's where it gets interesting again.
The input to this layer is gonna be equal to this parameter here.
My input parameter and it's gonna multiply those input times the weights plus the biases and then the
output has to vary right.
The output has to be different for the output layer versus the first layer or the second layer.
So I'm gonna do this.
Well I'll tell you what if the name is equal to W equals two out for output.
In that case my output will be equal to soft Max.
So T F Dot and Dot soft Max parentheses layer underscore in.
However if the name is different from the word out then we're simply going to say the output is going
to be equal to T F dot and then die.
Really.
Which is what we have for all the hidden layers and then layer underscore in my function will then return
my output.
So that should be fairly generic.
Right.
This will allow us to insert a cell below and set up all of our layers and a couple of lines of code
layer underscore one.
It's gonna be equal to set up underscore layer open parentheses and then what's gonna be the input the
input the first hidden layer.
It's gonna be our place holder capital X. The Wait underscored dimensions are gonna be equal to square
brackets.
The total number of inputs and then and then the other dimension for this one is gonna be and one hidden
one so five hundred and twelve.
The bias on a score dimension is gonna be equal to square brackets and hidden one.
So the number of neurons again five hundred and twelve.
And then we're gonna call this name as equal to layer on the score one must split this over a couple
of lines two lines.
Here we go.
That's the setup for layer 1.
Done.
I can do the same thing for layer 2 and for our output layer.
So there are 2 and output and I'm actually going to go back to my setup layer function here.
And I'm going to call this thing layer on a score out that way we avoid confusion.
So I'm just gonna rename these variables to not clash with this one.
I want to keep the output of my model with the name output.
Now let me update the weights and the biased dimensions.
Our second hidden layer is of course going to have 512 inputs but it's only gonna have 64 neurons so
an underscore hidden to the output layer is gonna have 64 inputs and underscore hidden 2 and 10 outputs
and R underscore classes for both of these layers.
The number of neurons also has to be reflected in the biased dimension.
So for the second hidden layer it was an underscore hidden too and for the output layer it was again
number of classes the names for layer one and layer 2 where you're going to keep with layer on the square
one and layer on the square 2 and then for the output layer in order to trigger this if statement.
I'm just gonna name it lowercase out.
So this is gonna be the same as this.
So my output layer is gonna have these soft Max activation function and that leaves us with one last
thing to do.
The outputs of the first layer are gonna be the inputs for the second layer and the inputs for the third
layer are gonna be the outputs of the second layer.
So this place will look at this capital X it's only gonna be input for the first layer the input for
the second layer is actually gonna be layer number 1 and the inputs for the output layer is gonna be
layer number 2.
Now what I'm gonna do is I take these lines of code and I want to move them to the very bottom of our
notebook.
So I'm going to keep hold of them.
I'm not going to delete them but I'm going to move them all the way down and I want to set a markdown
cell here.
That's kind of read code for first part of module.
I think what I'll do is I'll come at the Mount.
So we're keeping them around as a reference but we're not using them anymore.
We're not executing them anymore.
Now that we've done all of that we can come back up here to where we've got our mark down set up tensor
flow graph and we're to run all below and what we get is the crash.
Why.
Because we've still got some old code floating around this line here.
Well now.
Error out because there is no more tensor called B so I'll move this line of code down to the very bottom
as well and I'm going to comment it out before I rerun everything I'm going to execute this cell here
which closes our session and resets our graph.
And then when I come back here and try again so I'll run all below.
Now our model starts training again.
We can come back into tensor board.
So now that we've successfully created some named scopes for our layers we can also create some name
scopes for the other calculations that we're doing.
In particular I'm thinking of our optimization because our optimizer is actually connected to all of
our layers and I'm thinking of our lost function and our accuracy calculation.
There's a lot going on here.
There's a lot of nodes and a lot of edges and we can consolidate those.
Let's tackle one at a time.
I'm going to hit up the accuracy calculation first.
So for the accuracy calculation I'm going to provide a name scope that reads with T F dot named scope
open parentheses single quotes accuracy underscore calc colon and tab this over so that it's inside
the body of this scope.
I'm also going to come down here where we've got our two summaries and I'm also going to provide a name
scope for these two with TFT.
Name scope open parentheses and I'm gonna call this performance.
This is gonna be the performance named scope.
Next I'm gonna tackle the optimizer the optimizer.
I'm just gonna give the name f named scope open parentheses optimizer.
I think that's pretty straightforward.
And also I kind of like imagination.
So there we go.
That's the name scope for optimizer done.
And while we're at it let's add a name scope for our loss function as well.
So with TAF dot name underscore scope parentheses loss on a score calc colon at the end tab everything
over and now let's go to the very top and run all the cells again.
So will it run all the cells below.
They'll go into tensor bode well refresh our graph will refresh our page and we'll take a look at our
latest graph.
Remember changes dropped on here to your most recent run after you've made all the changes and now you'd
see something like this.
You should see that we've got everything a lot more tidy we've got our inputs here x those go into a
layer 1 but they also are connected to our optimizer.
We've cut our first hidden layer feeding through to our second hidden layer which is feeds through to
our output layer and our outputs go into the accuracy calculations as well as the lost calculations
the lost calculations of course also feed through to the optimizer because the optimizer is connected
to all the weights in all the layers and of course it's also connected to the lost calculations.
It's not connected to the accuracy calculations.
It doesn't have anything to do with that.
So looking at the graph in this way when everything's grouped together it's a lot more clear.
But one thing that we can do is right click on this and we can actually remove this thing from the main
graph because it's a little bit distracting it's connected to everything right.
So if we move it to the side then we can see that's still there.
You can still see it connected to the optimizer but we see that this now represents our neural network
in the clearest way yet.
We have our inputs our features our Xs feeding through from the bottom going through all the layers
and at the very top of our graph we have the two outputs that we care about the loss and the accuracy.
Now that almost covers everything I want to talk about with the graph but before I end this lesson I
want to show you one other cool thing that tensor board can do tensor board can actually also show us
some images namely the images of the numbers we have in our dataset to show you how.
We just have to add a single operation here and we add this operation here just above the code we're
running our session.
So add a quick markdown cell here that's gonna read check input images in board and the way we're going
to do this is with a summary.
So there's a tense about summary called Image T F dot summary dot image and then open the parentheses.
And then I want to provide a name for the summary.
Going to call this image underscore input and then I'm going to provide a value.
Now this is the tricky part.
I have to get an image pack out of our data.
So I'm going to leave this for now.
And I'm going to have X underscore image as the variable.
I want to feed into my image summary here.
That's gonna be equal to some data from our placeholder tensor.
All the data and the place holder tensor is like in a single row right.
Seven hundred and eighty four values.
In order to display the image which is 28 by 28 pixels we have to reshape the thing so as CTF data reshape
and I'm going to grab my placeholder tensor and then I have to say how I'm going to reshape it.
So when I add some square brackets him and I'm going to reshape it into the following dimensions minus
one comma 28 comma 28 comma 1 oneness the color channel 28 is the width and 28 is the height at minus
1 is that because I've only got three dimensions and this other dimension would be for all the samples.
In other words I would have more than one image that's 28 by twenty eight with one color channel.
Now that I've got this I can come down here and I can supply X on a square image to my summary and I'm
going to supply one more argument here namely the number of images I actually want to show up intensive
mode.
So Max on the score outputs is the name of the parameter and I'm going to say for I don't want to see
all sixty thousand of them right now since reshaping stuff and working out the summary are an operation
and tensor flow.
We can also give them a name scope.
So I'm going to say with space T F dot name scope open parentheses and then I'll say show on a score.
Image as the name scope for this thing and alt tab this over so that it's in the body of this with block.
Now both this reshape calculation and my summary have the same context.
Let's go back up to where we set up tents aboard and run everything below.
So our models already training and we refresh tents aboard.
And what we should see is a new type of summary.
Previously we only had the scalar and the graph but now we've got another type of summary here namely
our images summary because we added an image summary here.
If I change my graph to my latest run then you will also see this additional node here.
We've got our show image name scope for two calculations namely our summary and our reshape calculation.
These in turn are connected to our features tensor are x.
Now let's check out our images tab here in my images tab.
I get an image summary for every run that I've done and what I'm looking at here are four images from
my training run and my validation run in my training run.
I've got the number six in my validation run I've got the number five showing up if I just wanted to
see one of them I could hit this radio button him so we'll see the screen one I've just got my validation
data set for which I'm looking at these four images why four.
Because that's our max output right.
So that's pretty cool right.
Not only did we learn about name scoping and grouping operations in our tensor flow graph we also got
to use a different kind of summary we've been using this scalar type summaries many many times before
and now we got to see the image summary as well in the next lesson.
We don't try out a few different models and this will allow us to look at a new kind of summary as well.
The histogram the histogram is incredibly powerful because it will allow us to see how our weights are
changing over time as we're running the model and training it over the course of several epochs.
So stay tuned for all of that and more in the next lesson.
I'll see you in a bit.