All right.
So in this lesson what we're gonna do is going to vary up the hyper parameters and an experiment with
a dropout layer.
We're also going to look at a news summary intensive bought namely the histogram summary.
All of this will give us a chance to run our model for a different number of epochs.
We can see how our accuracy changes as we continue trading our model.
We can also vary up the learning rate and we can change the architecture.
So let's get on it.
The very first thing I'm going to do is I'm going to go up to the function where I'm creating all my
layers.
And in this function I'm going to add two more summaries because I'm very much interested in what happens
to my weights and my biases as my model continues training in order to see what happens with our weights
and our biases we're going to use the histogram summary so TAF does summary don't histogram open parentheses
and then I'll have single quotes weights as the name for the summary and then the values are going to
be my lower case w want to be tracking my variables for all my layers in this histogram summary and
then also going to track my biases.
So once again this is gonna be a histogram summary.
I'm going to call this one biases and the value is going to be lowercase b.
The next thing I want to do is to scroll down a little bit.
And since this here is actually where we're setting up our model I'm going to create another variable
here called model on a score name.
And this one is gonna be an F string.
And in this model name I'm going to be very descriptive.
I'm gonna put down the architecture of this model.
So an underscore hidden one close in curly braces hyphen curly braces and underscore hidden to allow
for learning rate curly braces learning on a school rate.
That's this one right here.
And at the end of May maybe have capital E curly braces and R underscore epochs.
So that way would have a very specific model name him that includes the number of neurons the learning
rate and the number of epochs some capturing some of these hyper parameters in the model name.
What I'll do next is where I'm setting up the directories for tensor board.
I'm actually gonna change this from Model 1 to kind of braces and then model on a score name.
So now when we're setting up our Directories we get the model name in the directory.
And this means the model name is gonna show up here and it's going to show up on the runs on our graph.
Now we've done quite a few runs already.
So what I'm gonna do is I want to delete all the folders inside my amnesty digit logs directory and
that way when I read run my cells.
I have a blank slate.
The next thing I'm going to do is I'm going to change the number of epochs from five to 50.
So we're gonna do a whole lot more training and we're going to be able to see how our loss and our accuracy
evolve over time as we continue training our model.
So let's get on it and select this cell here and then I'm going to say run all below.
And now we play the waiting game as you can see training for 50 epochs.
Takes a lot more time to just training for five.
If you're running all of this in the Google collab notebook then this is gonna run a lot quicker but
you'll have the disadvantage of not being able to follow on tensor bored so easily.
All right.
So let me refresh tensor board I get a little error message here but that's not a problem.
The reason is is that we're still in the process of training.
So I think it's a bit confused since we've already talked about our graph and we've talked about the
images I'm going to look at my scales.
And here I have my two metrics my accuracy metric and my cost.
This is my loss remember grouped in the same section the section is called performance because performance
is the name scope for both of these summaries.
I'm going to enlarge my chart here in Orange.
I've got the accuracy on the training data set and in blue I've got the accuracy on the validation data
set and what I can see is that most of the learning takes place in the first ten epochs and then it
takes longer and longer and longer to get that extra little bit of accuracy.
So going from 86 to 87 percent accuracy takes a lot longer than going from say 60 to 70 percent accuracy.
In other words this is the classic shape of a chart.
When you've got diminishing returns now let's scroll down to our losses in the last module we talked
about how it was a problem when our validation loss stopped going down when the validation loss starts
to stabilize and even starts going up then we're over fitting our model.
So in this case we can see that it's still going down at epoch number 50 but not by much.
Now what about these other tabs that showed up distributions and histogram is the reason we get those
two new tabs is because we've added the histogram summary these two lines of code are now going to show
us a histogram and a distribution for our weights and our biases over time check it out.
Let's start out looking at the histogram and what I'm going to do is I'm just gonna take our training
run here and then I'm going to make these charts a little bigger and bigger and bigger and I mean expand
the output layer here.
What's hidden before.
And then make that one bigger.
Now let me scroll to the very top and let's take a look at what this is actually showing us because
it's very pretty but what does it mean.
Well what it's showing us is a distribution.
It's showing us a distribution of our biases for the first hidden layer over time.
Now if you remember all our biases started out at zero.
And by the end of the first training run our biases got a little update.
And then one hundred and eighty three of them were at about zero point 0 0 5 0 5 and about sixty three
of them were around minus zero point zero zero three nine nine.
So in the very back here we've got the oldest histogram at the end of epoch number zero.
Our biases updated and some of them about smaller and some of them got larger.
Now for hidden layer number one there were a five hundred and twelve different biases.
So what we get in this picture is an idea of how the distribution of these five hundred and twelve different
biases changed over time on the y axis.
You can see the steps.
So we've recorded a summary in every single epoch in our for loop.
So here we're moving forward in time until we get to the most recent epoch.
So this was the one right at the end of the training.
And here we see that some buy's values that are quite large and some values for the biases got quite
small.
The reason why these little steps correspond to one epoch is because this is when we decided to add
our summary and tell our file writer to write to our disk.
How do we structured our for loop a little differently than we might have written a summary every two
epochs every five epochs.
So the way to think about what you're seeing on this axis is the step size how often during the training
you're writing to your file.
So in our first layer our biases changed quite a bit during the course of the training.
What about our weights.
Now this is quite interesting right.
Our weights started out as a truncated normal distribution so they started out as kind of bell curve
without the tails on either end.
And what we see is done by the end of the training all the weights still seem to more or less fall into
this truncated distribution.
They still seem to have the same distribution as when they started.
So to me to suggest that all the learning that's been taking place in that first hidden layer actually
happened in the biases and not in the weights.
What about the second hidden layer.
So this is quite interesting.
Their biases changed dramatically.
They all started out at zero.
We had 64 of them to begin with.
And by the end they had this kind of distribution.
So there's some over here quite negative somewhere here quite positive.
So once again all 64 weights in the second hidden layer started out with a truncated normal distribution.
And by the end there hasn't been that much of a change.
What about the output layer though.
Here we only had 10 biases and these.
Yeah they moved around a little bit since there's only 10 values here.
This visualization looks very very jagged.
It tends to be a little bit more realistic when you've got a lot of values but for 10 you kind of get
this like mountain landscape.
But regardless what we do see is there was a change from the beginning of the training towards the end
of the training and the same can actually be said for the weights.
In this case we've had 10 weights and as you can see by the change in the shape here there was some
change to the weights in the output layer now tens of what actually gives us so many different ways
of looking at the same kind of data.
For example we can change the mode here.
You can change the histogram mode from offset to overlay.
And when we do that we rotate the whole thing by 45 degrees and we're looking at from the side.
So this is another way of looking at how things change over time and mousing over it.
We can see the different steps the different histogram as the different curve gets highlighted and all
the slices are no longer spread out over time but instead they're all plotted on the same y axis.
So if you prefer looking at the data in this way then overlay is your friend.
But let's take a look at this other tab here as well.
This is quite interesting because what this shows us as well shows is the same data as before.
So I'm going to look at only the training data this time and I'm going to take a little look at what
it is that we're looking at here.
So you're calling this one distributions and what we're actually looking at here are the percentile
changes in the middle of this distribution.
We've got the median value and on the top we've got the maximum value for lightly colored.
And at the bottom we've got the minimum value also very lightly colored.
And on the x axis here we have the steps.
So at Step Number Zero we were here.
And step number 50 we were over here the information on the distribution tab is essentially some high
level statistics each line in the chart represents a percentile in the distribution of the data.
So what I'm seeing on this distribution of the weights in the first hidden layer is that nothing much
happened in the middle of my distribution right.
So all the weights that were between zero and zero point one nothing really changed for them between
starting out and finishing training.
But I can see that there was some change at the extreme.
So the smallest weights that started out at minus zero point two.
They did change.
They got more negative and the largest weights at positive zero point two got larger.
This was something that was quite difficult to see on the histogram but if you look very closely you
can see that the tails off the distribution got a bit longer.
Right.
So initially the truncated distribution kind of was very steep and it ended here but then it kind of
fanned out a bit by step 49.
In other words the changes in that first hidden layer were happening on the edges.
So in summary this histogram tab and its distribution tab gives you a chance to see what's happening
with your weights what's happening with your biases over time.
And when I say time I mean over the course of the training is your network learning whereas it learning
and such tensor board really acts as a flashlight to help you better understand the goings on of your
neural network.
So what I want to do now is I want to make a change to my hyper parameters and then I want to see how
these histogram is and how does distributions and how my scales change in response the hyper parameter
that I'm going to change is going to be my learning rate I'm going to make it 10 times larger than what
it is currently.
I'm going to change it from E to the minus four which is zero point zero zero zero one to e to the minus
three which is zero point zero zero one.
So without further ado you change this parameter here.
Now I'm going to click here and say run all below.
So let's run this thing again for 50 epochs and we'll take a look back intensive board and when I refresh
the page just kick start this thing.
So we've got a higher learning rate now.
And what we can see and what we can see is that we're approaching a much higher accuracy much faster.
So this is quite interesting right.
Using a higher learning rate we've been able to train our model a lot quicker in fewer epochs than with
this lower learning rate.
So that's a massive improvement in performance just from changing this one hyper parameter check out
the performance in blue here on the validation data set with each of the minus four versus e to the
minus three.
If we look at the loss we had a massive decrease at epoch number five.
And then slowly starting to stabilize slowly starting to decrease at a much much lower rate.
So the question is really what it's start over fitting by the end maybe we have to change our model
of and add some regularization or a drop out layer to prevent it from over fitting.
Okay so I'm done training my accuracy on my training data set is very very high.
But that's not surprising.
The key is the accuracy on the validation data set.
Looking at the performance this seems very promising.
Towards the end of training we've got an incredibly high accuracy around ninety eight percent on the
validation data set and looking at our loss.
It doesn't seem to be going up.
It does seem to have stabilized.
It doesn't seem to be decreasing that much anymore.
As we're continuing to train but it doesn't seem to have started over fitting yet in particular but
I want to pull up side by side are my histogram is for that first run with that really small learning
rate and my histogram is for my second run in with my learning rate that was ten times larger.
So here are my biases.
And here are my weights and what I can see that is that the distribution changed massively between the
two runs right now if you remember on our first run by the end of the training our distribution had
a much wider tail but on the second run the change in the weights even in the very first layer was much
more dramatic.
We see the same pattern emerge where previously the weights weren't really changing all that much.
With that lower learning rate now the all the distribution is shifting from epoch to epoch.
And the same is true for the output layer.
So previously when the learning rate was low most of the adjustments seemed to have been made in the
output layer and then fewer and fewer adjustments happen in the second layer.
And then the first hidden layer this time around there were also big changes in the output layer.
So given that the distribution of these 10 weights started out like this.
And by the end we had something more like this more and more of the learning seems to have filtered
through to the second hidden layer and the very first hidden layer when the learning rate was higher.
But that's just the weight side of course.
In both cases the biases changed dramatically.
So even with the learning rate was low the biases changed quite a lot in the output layer in the first
layer.
And the second hidden layer and we see the same pattern emerge with the higher learning rate the biases
also changed quite a lot which is a good indication that our model is learning having looked at the
histogram side by side for the two runs allowed us to see the differences from the beginning and the
end for the first run in the second run.
But we can also look at the change in the percentiles for the distributions.
The smallest changes occurred in that first hidden layer there we had bigger changes in the biases and
smaller changes in the weights with the higher learning rate.
We still had quite big changes in the biases and smaller changes in the weights but comparing the distribution
in the percentiles between the second run and the first run we can see that in the second run we've
got much more of a spread.
The same is true for the second hidden lamb.
Notice for instance the scale here on the y axis for the first run we're looking at values between zero
point three and minus zero point three.
And on that second run we're looking at values spewing minus zero point six and zero point six almost
double the range in the previous module.
I showed you how to use a tripod layer in Caris.
Now I want to show you how to include a drop out layer just using tensor flow.
This is more for illustration than necessity.
I don't think this model is over fitting but I would like to cover this nonetheless because you're going
to need it on your future projects.
I'll copy paste this cell here and I'm going to comb it out.
This line of code here.
So this was our model without drop out.
And here we're going to include a drop out layer and we're going to include that droplet layer after
our first hidden layer.
So I'm going to change my model name to include this.
Oh here for dropout and then what I'm going to do is I'm going to add a dropout layer here.
Call this layer on a squat drop and the way I'm going to add this dropout layer is using T F dot and
then dot dropout.
This will create the dropout layer for me.
All it needs is an x and something called a keep probability the keep probability is the probability
that an element is kept.
So if I wanted 20 percent of the neurons in the previous layer to drop out then I would set my key probability
at 80 percent.
So let's try it out.
TMF Dot and Dot drop out parentheses and then it needs the input the input will be the previous layer.
So layer under score one.
Then we need to specify the.
Keep probability and let's call this zero point eight.
And finally let's give this whole thing a name right.
Name is equal to or drop out on a score layer but since we added this Trump on him we have to change
the input for Layer number two.
In this case the input is no longer in a row one.
But it's this tripod layer in between.
So it's not really a real layer but it is what forms the input to that second hidden layer.
All right.
So my training has finished and I've refreshed tensor flow.
So in green we have the validation run with dropout and in blue we have the validation run without dropout.
What we see is kind of as expected when you include drop out then the learning takes place a little
bit more slowly.
That's why the performance of the model with the drop out is a bit below the previous one.
But it does catch up and towards the end both are pretty much identical in terms of the loss.
We didn't really have a problem with over fitting previously and we most certainly don't have one now.
So that's concludes this lesson.
We've explored tensor board extensively.
We've looked at a lot of different summaries from scales to images to our graph to the history grams
and the distributions and we were able to see what was happening as we were training our model.
We were able to see our lost decrease and change over time.
We were able to see our accuracy increase over time.
We're able to see how much of a difference the learning rate can make and we're also able to see how
our biases and our weights changed over time as we were training our model.
The changes were most significant in the output layer which had the fewest neurons and then the changes
got smaller and smaller as they propagated down the network to that first hidden layer which had 512
neurons.
Initially it seemed that the biases were doing most of the adjustment but when we cranked up the learning
rate we could see that there were also bigger changes happening with the weights in the next lesson.
We're gonna be evaluating our model on our testing dataset and we're also going to make a prediction
on a single image for all that and more.
I'll see you in the next lesson.