In this lesson we will train our model.
We will fit the model to our data and we will start to update the weights in our neural network so let's
add a markdown cell in Jupiter notebook and commemorate this.
So this market on cell will read fit the model.
But how do we actually go about doing this.
Well the first place to check is the caris API documentation.
The relevant section is called Model parentheses functional API.
And if we scroll on here we see that compile method.
But we also see this fit method and this fit method can take a number of different arguments so let's
call this fit method and supply our training data for the X and our labels for the Y.
This is where our really tiny data sets will come in handy because we're gonna use these when we're
iterating and trying things out.
So let's write model underscore one dot fit parentheses x underscore a train underscore X S for extra
small and y on a school train on a score.
Access Now at the very top of the cell before his shift enter.
I'm going to add some micro benchmarking Code 2 percent signs and time will tell us how long.
Jupiter note book takes to execute this line of code.
Now let me hit shift enter on the cell and what I see is that it took about two seconds to execute this
cell but how did that go.
Did we do the job.
Well at this point I have no idea.
So this is where we're gonna use tensor board tensor board will help us figure out what's going on behind
the scenes.
Now the question is how do we get the information from our model into tensor board.
Well let's look back at this documentation here and here we see this parameter called callbacks which
by default is set to None.
But we've seen this would callbacks somewhere before right.
So I recall in our import statements tensor board was actually a callback.
So this means that we can actually use tensor board as a callback.
Now is the time that we get to use our get tensor board function that we created earlier.
Let's add that callback argument to our fit method now.
So I'll add a comma.
Right callbacks equal to and then square brackets get underscore tensor board open parentheses and our
supply my model name.
I'll call this one model 1.
Why do I have the square brackets here.
Well because callbacks is expected to contain a list and we can see this if we go to the documentation
and scroll down to where it says callbacks.
And here the carries documentation very clearly states that a list of callbacks is expected.
So let's head back to our Jupiter notebook and hit shift enter on the cell and our output now reads
successfully create a directory which we can see right here.
My training time this time was around one point two seconds but I don't see tensor board anywhere.
So how do we get hold of it.
Well I actually have to open a terminal or the windows command prompt again at this point.
So if I open up another window here new window and then I write tensor board and then as an argument
I supply my logging directory so minus minus log.
Dear is equal to.
Well this folder right here.
This is my logging directory that we've created since I'm on a Mac.
All I have to do to get my path to this folder is drag and drop it into my terminal and there I see
my path users and all projects tensor board sofar underscore logs let me hit enter on this right now
and I'll show you how to get this path very quickly on windows in a minute.
Once my command executes successfully what I get as a result is this here you are L and I can copy this
your l go back into my browser pasted in and this is where tensor board lives so here's how you can
get the path very quickly on Windows so go to your tensor board underscore CEF are on a underscore logs
folder open it and then up here you see this address ball.
And here you can click and then you see the path for this folder which you can copy and then when you
open your an anaconda prompt all you need to do is of course type tensor board space hyphen hyphen log
D.A. R equals and then right click to paste right click to paste is the trick.
Okay so now that we've got tense about running what are we looking at.
Well we've got two tabs up here graphs and scales in this module.
I'm going to be focusing on these scalar section.
The next thing is is that we get some data here.
The accuracy and our loss.
I want to make this a little larger and you can see here we've got a single data point here.
Our model currently has an accuracy of about eleven percent in sample.
So it has a 11 percent accuracy on the training data set.
The next thing here that we see is a loss and for the loss.
We also only have a single data point.
Now this is a little strange.
This is not exactly what we expect.
So let's dig into the documentation and see why that might be when we look at our fit method.
Then we see two additional parameters batch underscore size equals none.
And epochs equals one.
So what does that mean.
Let's tackle the epoch first.
What's an epoch.
Well an epoch is when the entire dataset has been passed through the neural network a single time considering
that the default value for epoch is 1.
It also makes sense to see only a single data point on our tensor board now passing the entire dataset
through the model a single time almost sounds like it should be enough right.
Well unfortunately the answer is no.
As we saw in the gradient descent module the optimization process really is an iterative process meaning
yes the weights were updated that one time that we ran the fit method but a single pass is actually
not enough.
We have to pass the entire dataset through the network again and again and again for the neural network
to differentiate amongst our images.
But wait why don't we have lots and lots of training data.
What if we have a huge dataset.
Well here at the practicalities of training a neural network set in.
If your computer is powerful enough to handle your entire dataset or your dataset is small enough then
yes you can probably process everything at once.
But usually what you have to do is you have to split up your data set and process one piece of it at
a time a single piece of your data set is called a batch.
And if you're splitting up your data set like this then you're training your model on one batch at a
time.
So that means that it will take multiple iterations to actually go through the entire dataset.
The formula for the number of iterations that it takes to chew through the entire training data is actually
pretty straightforward.
All you need to do is divide the number of training samples by the number of samples in the batch.
That's how you can work out how many iterations it takes to feed the entire data set through the model
a single time.
In other words if you have 100 data points in total and your batch size is equal to 50 then you would
need two iterations to go through the entire dataset.
It would take you two iterations to complete a single epoch.
So the number of iterations is just the number of batches needed to complete one epoch with this vocabulary
out of the way let's head back into Jupiter and change both the batch size and the number of epochs
that we're using to train our model.
So what I'll do is I'll add another cell here and I'll create a variable called samples on a school
per on a score batch and I'll set that equal to 1000.
Now we've got some micro benchmarking code here in the cell below where we're fitting our model and
what I encourage you to do is to try out different patch sizes and see how long it takes the cell to
execute.
Now let's set the number of epochs equal to 20.
Let's train our model for 20 epochs.
I'll create another variable in this cell that reads and hour underscore epochs and I'll set that equal
to 20 and then inside our fit method.
I'm going to add another comma here before callbacks and here I'll add batch underscore size and I'll
set that equal to samples on a scale per underscore batch.
My autocomplete isn't working because I haven't actually hit shift enter on the cell.
Now I can also add my epochs which had the parameter name while epochs and that I can set equal to an
hour on a score epochs.
Now the single line is getting very very long.
So what I'll do is I'll put my callbacks on the line below.
Now let me hit shift and her on the cell and we can see our model being trained throughout the 20 epochs
now as part of the output.
You can see the total amount of training data you can see the loss that was calculated and you can see
the accuracy in sample per batch.
And one thing that you'll notice is that in this case our model isn't learning anything it's just guessing
randomly.
It's getting about 12 percent right.
Meaning if it has 10 things to classify and it's getting about a 10 percent accuracy it's just randomly
guessing.
Now we would have hoped to see some learning happen by epoch number 20.
The fact that we armed means there's a bit of a problem now.
We could have this problem for a number of reasons but one possible reason is that maybe we have a bad
starting point.
Maybe our optimizer is stuck and it cannot minimize this loss any further.
So one thing we can try is we can go up here to where we're defining our model and recompile it.
So just hitting shift enter on the cell coming down here and rerunning our fit method.
What do we see now.
In this case we see some things are happening as we scroll down our model.
Indeed appears to be learning.
It started out with about 10 percent in epoch number two and by epoch number 20 it had improved this
to about 16 percent.
Now here's the big advantage of tensor board.
It makes us a lot more clear if we refresh our page it's going to redraw all our graphs and what we
see here now is if we enlarge this slightly we see the R three runs right.
Our very first run was just one epoch.
Our second run our model did not learn anything.
The training accuracy stayed constant in our third run.
We can see how the training accuracy evolved over time and what we see as this improvement from around
10 percent to around 16 percent.
The way that 10 support works is that it read some files from our disk and it reads these so-called
event files.
This is where it pulls the data from to throw up onto these charts.
Now I can actually see I've got four folders here but I've only got three charts.
And that's because last night I created this empty folder here which does not contain one of these event
files so I'm just going to delete this folder and of course if I delete one of these event files here
and I refresh my page then the data will also disappear from 10 support.
So let's take that blue second run here at eleven thirty nine.
I'll delete the folder and then I'll refresh my page.
And here we go.
It disappeared from tensor board.
Now back to some more urgent questions seeing our model learn over the course of 20 epochs seems promising
right.
Perhaps all we need to do is run our model for longer.
So maybe let's come up to where we're training our model and change this whole thing from twenty to
one hundred and fifty epochs.
Now if I don't want to see all this output then I can actually mutate looking at the carries documentation.
There's a parameter called verbose by default it's set to 1 but we didn't have to have it set to 1.
If we said boats equal to zero then we mute a lot of this output.
We're getting this intensive Ward anyhow so let's try this.
Let's see what happens when I run my model with one hundred and fifty epochs in this case it's going
to run a lot longer.
I've successfully created my directory my model is running and if I go into tensor board and I refresh
it then what you should see is your new graph appearing.
The red line here is clearly above the blue line.
Now one thing I don't really like is how it's going off the chart here.
But more importantly have a look at the starting point of this red line here.
My latest run actually starts at around 16 percent and this is no coincidence the previous run actually
finished around 16 percent.
So if I refresh this cell over and over again the model actually picks up training where it left off.
So if we actually want to compare the different runs starting from scratch starting from zero instead
of adding to them what we have to do is we have to come back to the cell here refreshed and then come
back down here and fit our model.
In this case when I go back to tensor board and I wait a little bit then I can actually see that it
picks up the new file that was created with the latest run right here at eleven fifty two.
And it actually starts plotting the thing while my code is running.
So every once in a while this chart will actually update with the latest model grabbing the latest file
from the disk even as the training is in progress.
So what we see now is that we're starting at a similar starting point as the previous runs where we're
starting from scratch.
Then the optimizer is working working working and improving the accuracy on the training data.
At the same time when we scroll down here the loss on the training data is decreasing and decreasing
and decreasing.
So this seems promising right.
After about 150 epochs we're on 40 percent accuracy but that's on the training data right.
What about our validation data.
What about on images that this model has not seen yet.
Looking at the caris documentation we can see that there is another parameter that we can supply called
validation underscore data the fit method expects to receive the validation data as a tuple.
So let's add that to our fit method and refresh the cell.
I'll add a comma and then I'll type validation on a score.
Data is equal to parentheses to create that tuple X underscore Val comma Y underscore Val.
Now what I'll do is I'll quickly come back up here.
Refresh this cell and rerun our fit back intense aboard.
I'm going to refer to page and now I can see my latest run in pink.
I quite like the fact that tensor board scale this graph a little better and I can see my computer doing
work and that little pink dot moving up as the model is being trained.
You see that after fifty five steps fifty five epochs we've got about 32 percent accuracy on the training
data back in my Jupiter notebook I can see that this whole run took me about one minute and 12 seconds.
Mind you my computer actually does not have a dedicated GP you.
So this is definitely taking a little bit longer on my four or five year old machine than it would on
something a little bit more powerful.
So let's see how this is going intensive board the training accuracy once again seems like it's off
the charts right.
So let me try and refresh this page again see if we actually can redraw the plot so that it shows us
where it actually ends.
If we're lucky it does just that and we are brilliant.
Now one thing that you might ask at this point is well hold on a second.
Why is it that all these lines are different.
We're using the same data and we're using the same number of epochs.
Why am I always getting a different chart.
Well the thing is there's some randomness involved in the optimizer one way the randomness for instance
comes in is the starting point where the optimizer starts to optimize and what we saw the very first
time that we fitted our model was that our model didn't learn anything at all.
And so at this point you can see how this might just be a fluke.
Right.
This might be just bad luck by fitting the model a couple of times we can actually see how each training
run is going.
This latest one ending with 55 percent accuracy after one hundred and fifty epochs seems to have gone
particularly well.
Right.
But remember this is the training accuracy.
Our model is learning to classify our training dataset Let's scroll down here and what we should see
is having added the validation data said to our fit method we have two new sections down here intensive.
The first one is the validation accuracy and because our previous runs did not include the validation
data.
They are not appearing on this chart.
So let's see how we're doing.
After about 150 epochs we're getting about 30 odd percent accuracy on the validation data set and I
can tell you that the validation accuracy is always going to be lower than your training accuracy but
let's take a look at this last section the validation loss.
Now this is very interesting.
What do we see is that our validation loss starts high and then it starts to come down as we're training
right to point to with step 18 to point one on epoch number 48 the validation loss is increasing and
decreasing but it's getting a little bit more jagged down.
And by epoch 100 but we're actually seeing is that our validation loss starts to creep up again.
Our validation loss starts to increase.
This is actually a problem.
And this is why evaluating your model on the validation data set is so important.
The problem that we're seeing here has to do with over fitting and in the next lesson we're going to
see how we can tackle this.
Now even though we haven't written a lot of code in this lesson we've actually covered quite a few different
concepts.
The first thing that we saw was how we can fit our model.
And that was using this fit method.
But one thing that we learned about the fit method was that it starts right left off.
So if we've trained the model for 20 epochs and we call the fit method again it will remember what the
values of the weights were during the previous run.
The next thing that we learned was that not every run is the same.
There is some random this baked in and the optimizer will start at a different starting point every
time you run the fit method.
The third thing that we learned was the process of passing the data set through the network.
And this process was iterative.
Not only did we have to pass the entire data set through the network more than once in order for the
weights to become meaningful.
What we also learned was this technique of splitting up our data into batches and finally we learned
a little bit more about tensor board.
We learned how 10 support creates these files on our desk and then reads from these files to create
our chance.
Now I know this was a very dense lesson with a lot of information to take in.
But don't worry we'll be reinforcing a lot of these concepts in the upcoming lessons.
I'll see you there.
Take care.