In this lesson we're getting up to the final and last part of this module the evaluation stage of our
model.
It's been quite a journey.
We've formulated our question.
We gathered our data.
We've pre processed and clean their data and we've explored and visualize that as well.
Then we spend quite a bit of time training three different versions of our model looking at dropout
regularization early stopping and examining the performance of our neural networks.
Now it's time to move on to the evaluation stage.
Let's analyze our favorite neural network in a bit more detail in the module where we covered our naive
bayes classifier.
We looked at three metrics in addition to the accuracy for evaluating our classifier.
The first was the recall school.
The second was the precision and a third was a combination of the two namely the F school.
So let's tackle each of these in turn in our Jupiter notebook.
First we'll take a look at the accuracy and then we'll take a look at our false positives and false
negatives in a confusion matrix.
And finally we'll calculate our precision recall an F school.
I'll create a subsection here with a markdown so and the first thing they'll do is select a model that
we're going to look at.
Now looking at tensor board it's a pretty close call.
Our most accurate model on our valuation data set was actually model one with about 49 percent Model
1 if you recall did not use any regularization.
It had no dropout layers.
As such it ended up with the largest difference between the valuation accuracy and the training accuracy
on the validation set.
Model 1 got about 49 percent but on the training data set it got around 60 percent.
Model number 2 with one dropout layer was much closer.
It had a bit of a rocky start and probably could have performed a little better.
On another training run but its training accuracy and its valuation accuracy are pretty close to 50
percent and looking at the stats it's not far behind model number one.
So this is the one I'm going to go with.
As you can see from tensor board they're already two metrics that were calculated as part of the training.
One was the loss and one was the accuracy.
We can reconfirm what these metrics were by putting our modelling then adult and then metrics underscore
names.
Here's the list of metrics that our model can calculate for us if we wanted to get the loss and the
accuracy on the test dataset.
We would use the Evaluate method.
So looking back at the carrier's documentation we can see the Evaluate method listed here on our model.
Functional API.
The description here reads returns the loss value and metrics values for the test model in test mode.
And again if you supply a lot of data then it will do this computation in batches automatically which
is quite nice.
If we scroll down a little bit then we can see what the return values are.
Here we see that the attribute model dot metrics underscore names will give us the display values of
the scalar outputs and we actually get more than one output right.
We get the test loss if there are no other metrics but if there are then we get a list of scales scales
is the same word that you saw here intensive board.
So things like accuracy and loss are scales since we can get to return values from our evaluate method.
Let's store these in two separate variables.
I'll call the first one test on the score loss.
Put a comma and then I'll write test on the score accuracy and I'll set that equal to model on a score
too don't evaluate and between the parentheses I'll supply our test data set X underscore test and our
test labels y on the score test next I'll print this out so I'll let a print statement that reads test
loss is curly braces test under score loss and test accuracy is curly braces test on a score accuracy.
Now let me add shift enter.
Let's see what we get.
Caris will run this evaluation on the entire test data set which if you recall was 10000 different samples.
This calculation took me about 3 seconds to run and here's our output.
We've got an accuracy of around forty nine percent on our testing dataset and this is also what we should
have expected given that we had about 49 percent on our evaluation dataset.
If you think this print statement is a little bit hard to read then of course you can format these numbers.
So with a semicolon and zero point three I can format my loss so that only shows three decimal numbers
and if I want to show my test accuracy as a percentage then I can see zero point one percent.
Then I'll get my accuracy formatted to a percentage with one decimal point.
Let me show you what I mean.
That's a lot easier on the eyes.
Right now let's take a look at our false positives and false negatives.
If you remember from our boy who cried wolf story the false positive is when the boy cried wolf.
And there was no wolf and the false negative would be if the boy had cried.
There is no Wolf.
And that was indeed a wolf that would have made for a very confusing story.
But this is where the confusion matrix comes in to make things a lot more clear already small subheading
here that reads confusion matrix and what we'll do is we'll go to the very top and we'll actually add
another import statement.
We'll import the confusion matrix from psychic learn as K learn don't.
Import confusion underscore matrix let me hit shift enter here scroll back down and now we can create
our confusion matrix so I'm going to store this under conf underscore matrix and I'll set that equal
to confusion on this go matrix and here I have to supply two things my actual labels are actual classes
so why on a score test and my predictions how do I get all of my predictions.
Well if we scroll back up then we can see that we can take our model put a dot after it and say predict
on the score classes and supply our entire testing dataset.
So this is exactly what I want to do.
So I'll see model on a score too don't predict on a score of classes parentheses x on a score test.
Now if this is proving hard to read then what I'll do instead is I'll take this out I'll create a variable
called predictions set that equal to my predicted classes I'll put predictions here and I'll also add
the argument names so y underscore true is equal to y in a skirt test and lie on a score pred is equal
to predictions.
These are the argument names for the confusion matrix let me hit shift enter on the cell and let's take
a look at what we've got.
The interesting thing about this confusion matrix is that it has a shape right.
It's a 10 by 10 matrix so the number of rows right in this matrix would be in our underscore rows.
That would be equal to the confusion matrix thought shape square brackets zero and the number of columns
would be the confusion matrix dots shape square brackets 1 The other thing that we can do is we can
look at the largest value in this matrix.
So confusion matrix dot Max will give us the largest value in this matrix and that's six hundred and
forty five.
In contrast the smallest value in the matrix is five and we can pull this out of the confusion matrix
with thought Min.
But what I'm actually interested in is creating a visualization.
I want to create a chart that way we can see this confusion matrix a lot more clearly.
So when he's mad plot lib for this with BLT dot figure parentheses fixed size set that equal to 7 by
7.
I think they'll look pretty good on screen and the way we can show the confusion matrix is with a BLT
dot I am show conf on a second matrix will plot our confusion matrix on a chart and we can show this
with Pulte Don show let's see what this looks like Tara it looks absolutely horrific completely unintelligible
so we're gonna have to do some things we're gonna have to do some work on formatting the very first
thing I'm going to do is I'm going to come in here and I'm going to fix the title and the labels so
with Peel T dot Title I got a title for this thing so that we can see what it actually is.
See it's font size is equal to 16.
There we go.
Here is a confused matrix which at the moment is a very confusing.
The next thing I'll do is I'll add a y label so I'll add a label for our y axis.
So on our y axis we're gonna have our actual labels our actual categories.
So here we go.
Here's our y label and the ex label should be added of course as well with predicted labels now the
worst thing at the moment are still these little take walks 6 4 2 0.
Now these tick marks.
Actually it meant to correspond to our classes.
Right.
These range from 0 to 9.
This is why we've got these funny numbers here.
So what we actually need to do is we need to format Arctic marks so tick on a score marks shall be the
numbers from 0 to 9.
So we can create this with NUM pi with NPD and arrange and these should start from zero and then at
nine.
So our supply are constant here.
The number of classes NPD arrange.
And then this is equal to 10.
And then what I can do is I can take the 16 BLT dot y ticks and in the parentheses I can supply my tick
marks and I'll hit shift enter and I've got a tick mark for each and every single one of my classes
but even though I really like numbers what I actually want to see are the names of our classes and I've
stored these as a list at the very top with plain com bird cat and so on.
So that's gonna make our access a lot less confusing.
So what I want to do is supply another argument to this why ticks method and that's going to be my label
on the score names constant.
If I had shift enter on this then I can see that the tick marks will now correspond to the items in
my list of labels.
So instead of zero up here I have plane instead of one here I have com and since I've done this on the
y axis I'm also gonna do this on the x axis.
So with party type x ticks I can copy paste this line it shift enter and I'll get my labels here as
well.
Now what to tackle next.
The first thing is is that I want to change these colors that are being used here.
And the easiest way I can do this is with something called a color map and matte plot lib actually gives
us some sample color maps that we can pick from the kind of color maps that will work rather well here
are these single color maps for confusion matrix so grays purples blues greens oranges you can take
your pick.
I'm going to go with you.
This is gonna be a tough choice.
Facebook blue perhaps or purple.
Maybe I'll just go for green coming back here and going to this line.
I am show we can supply a color map with this sea map argument so let's try this out see map is equal
to and then color maps I can get through BLT which is our map plot lib and then see m which stands for
a color map and then we can go for a color map of our choice.
So I would go for green.
Yeah.
The name has to correspond to what we see in the reference here.
And if I hit shift enter then the color of my confusion matrix will change.
Now why are some of these fields darker than other ones.
Well that's because the color is supposed to signify a value.
So a light color would be a low value and a dark color would be a high value.
To make this a lot more clear what we can do is add a so-called color bar on the right hand side and
with PDT dot color bar we can do just that.
Don't forget the parentheses at the end and with the shift enter we can refresh ourselves and then we
see this beautiful color bar next to our confusion matrix.
One thing that we checked earlier was the maximum value in the confusion matrix and this was around
six hundred and forty five and the minimum value was around five.
So if we look at our confusion matrix again then we can see that this is the darkest color here.
So this square here should correspond to six hundred and forty five and then any of the very white squares
should correspond to very low values like five or 10 but instead of using our imagination to interpret
what's going on here let's actually print the individual values on each of these squares.
To do that we're gonna write a for loop we're gonna have to iterate along each row and along each column
to write down and print out the value onto this matrix.
So what we would need is a nested for loop.
That's one way of doing this.
We've worked with nested for loops before.
So in this lesson I want to show you an efficient alternative to what we've done previously.
And when I say efficient I mean computationally efficient.
Python has something called it or tools iteration tools.
So these are functions for creating iterator is for efficient looping.
Now that's quite a mouthful but if we scroll down here and we look at these tables here we can see that
there are all sorts of different types of problems where the acceleration tools have come up with a
solution one of these is the so-called nested for loop and we see that this product method here is equivalent
to a nested for loop and will provide us with a way that we can loop through our confusion matrix using
these iteration tools.
So let's try it out.
No scroll to the very top to our import statements and of course as always I'm going to have to import
my module before I can use it.
It's want to import it or tools that shift into.
Now I can scroll back down to my confusion matrix and add my code.
Here's how we're going to write our nested for loop using these iteration tools.
So I'll say for I.
Come on J.
Remember I've got two dimensions.
Rows and columns.
So I and J.
In it your tools don't product open parentheses range 10 comma range parentheses 10.
Why are we using range.
Well just like with a normal for loop we're going to start at zero and we're gonna add a 10 minus one
or nine.
And because we've got I and J we front two ranges here as arguments for this product method.
Now if we didn't want to use this magic number here 10 who could actually pull the dimensions directly
out of the confusion matrix which we've actually done up here.
So confusion matrix dots shape zero and confusion matrix don't shape 1.
We stored in number of rows a number of columns.
So instead of the 10 here we could have no underscore rows and in our underscore columns.
So that's how we're gonna set up our loop.
Now what are we doing inside the body of our loop.
Well the goal of this whole thing was to print out the actual value in each of these cells and we could
do that with Pulte dot text.
So what are the values that we actually want printed in this first row.
Well let's try and take a look at the confusion matrix and pull it out.
So conf on this call matrix.
Square brackets zero.
Well pull out that first row this row should have the values five hundred and eighty one 33 71 17 and
so on.
So how do we get this printed here.
Well we have to iterate through our confusion matrix and with PDT dot text we have to supply an X and
a Y on the coordinate system of Pulte dot text and a string.
So this is going to be j for the X eye for the Y and for the string for now.
I'll just write a lower case Oh.
Now let shift enter and what we see is that lowercase 0 is printed in all of these cells.
Now this lowercase 0 is not what we want.
That first row should actually have these values.
So we need to go back into our confusion matrix and we need to pull this out the way I'm going to do
this is with conf on the score matrix.
Square brackets and I'm going to use I come a J to pull out the individual values from this two dimensional
matrix.
Let me hit shift enter and let's see what we get.
So this is good news right.
We've got the values that we expected in that first row it starts with five hundred and eighty one and
ends with fifty eight exactly what we've got here.
The only thing is this is really hard to read.
First of all it's shifted so it would be nice if we could center it.
So it actually shows up in the square.
There is an argument called horizontal alignment that we can add to the party dot text and we can set
that equal to center and heading shift into well center these numbers.
But of course it will only do that hey if you've actually spelled this correctly Hari Zon total alignment.
Let's try again.
Here we go.
Now our numbers are centered but there's still one slight problem.
The higher the number the darker the cell and the darker the cell the more difficult it is to read.
So ideally we want the color of this text to be black.
If the cell is white or very light and we want the color to be dark if the cell is very dark and we
can do that by supplying a color argument to this text method.
So but a common him hit enter and see color is equal to.
And now I can actually include a little bit of logic here which is very very cool.
I can say that the color should be white.
If the value in this cell.
So come off on the scroll matrix I comma.
J is greater than some number.
So which numbers are kind of hard to read.
Maybe all the numbers above.
I don't know 450.
So if the number in the cell is greater than 450 then the color will be white but otherwise else the
color should be black.
Let's try this well.
Perfect right.
If we wanted to make this number here a little bit less arbitrary and say oh we use the cutoff point
in the middle of this confusion matrix.
So this would depend on the maximum value then what we could do is we could replace this with conf on
the score matrix.
Dot Max divide it by two hitting shift enter on this will give us this cut off at around three hundred
twenty.
Brilliant.
So what are we looking at here.
We've spent quite a bit of time creating this visualization but we actually haven't talked about how
to interpret it yet to make it a little bit easier on the eyes.
But I might do is I might scale it up a little bit.
So what I'll say up here under plotted out figure is a lot of karma and I'll scale it up with GPI equal
to 2 2 7 which is the resolution of my screen now the whole thing has a much higher resolution and should
be much easier to read for you watching this video.
What I'd like to do at this stage is pose a challenge to you.
I'd like you to have a think about the interpretation of this confusion Matrix for example what do the
numbers on the diagonal represent and what do the numbers in a single row that are not on the diagonal
represent.
So this thirty three seventy one seventeen twenty nine and so on my challenge to you is try to identify
the false positives the false negatives and the true positives in the confusion matrix.
All right here's the solution I've scaled the matrix down a little bit so you can see both axes on the
entire screen let's tackle the true positives first.
So this is the case when our model predicted the correct outcome for example it predicted a plane when
there was in fact a picture of a plane and it predicted the car when in fact there was a picture of
a car.
So the values along the diagonal are the true positives.
Now what about the values down the column.
In this case our model said there was a plane when in fact there was a call and here.
One hundred and six times our models said there was a plane when it was in fact a bird.
The definition of a false positive is a false alarm.
Crying wolf when there is no wolf crying plane when there is no plane.
So this number 39 represents the number of times our model cried plane when in fact there was no plane.
In other words the values down this column are false positives.
And if we sum all those values excluding the value in the diagonal then we get all the false positives
for one particular category.
Now what about the false negative.
In this case our model is saying there is no plane but in fact there is a plane.
Where would we find that value in this case.
We have to look at a row in 33 cases.
There was a picture of a plane but our model predicted a car.
It said there was no plane.
There's something else.
So all these values represent the false negatives.
Summing up the rows excluding the diagonal will give us the false negatives for a particular category.
And summing up the columns apart from that angle will give us the false positives.
Now one thing that's really really interesting about the confusion matrix is looking at the categories
that were most often classified incorrectly.
For example our model confused trucks and cars with each other more than any other category.
Similarly dogs and cats were very difficult for a model to tell apart as we're ships and planes or planes
and ships and even for some reason birds and deer armed with this knowledge we should be able to calculate
both our precision and our recall for both of these.
We need the true positives in the denominator.
How do we get hold of the true positives in our confusion matrix.
That's actually fairly straightforward.
So the true positives are going to be along the diagonal and we can use num PIs MP dot dialogue and
then supply our confusion matrix to get a hold of these values.
Check it out.
5 8 1 5 6 5 3 0 9.
You can see that these are the true positive values along that agony.
Now looking at our recall score we need the true positives and the false negatives in the denominator.
So how do we get hold of this.
Well that'll be the value in the diagonal plus all the values not on the diagonal but easier yet if
we sum up all the values in a row which includes the true positive we should get the denominator for
the recall score.
So our recall is actually equal to end p dot Diack conf on the school matrix divided by end p dot sum.
And now we have to sum along the row.
Right.
So that's come on a score matrix comma axis equals one axis equals one will sum all or rows.
So let's check it out.
Our recall is equal to an array with these values.
We get one recall score for every single category now let's calculate the precision for each and every
single category.
Once again we need the diagonal values.
But now we need to get the true positive and the false positives the false positives.
We said where the values along an entire column excluding the value in the diagonal.
But since we actually need to add it back we can actually sum all the values in a column.
So let's go for it.
Precision.
Going to be equal to end p dot diagonal.
Conference call matrix divided by NPD out some conference call matrix come on axis is equal to zero
in this case.
This is how we sum along our column our precision for each and every category.
It's gonna give us an array and it's gonna be 10 values like so.
So now that we've got 10 recall values and 10 precision values.
How do we calculate the precision or the recall of the model overall Well the easiest thing to do is
to actually just average these values.
Averaging all the recall scores for every category.
Well give us the average recall score for the model as a whole.
The average recall is equal to end p dot mean recall.
And this means that if we print this out and see model to recall score is curly braces average on a
score recall and we can format this as a percentage such shift into c what we get.
It's about forty nine point one four percent.
Now as a challenge can you calculate the average precision for the model as a whole print out this value
below yourself afterwards calculate the f score from Model Number Two.
I'll give you a few seconds to pause the video and give this a go.
Ready.
Here's the solution.
The average precision is going to be equal to N p dot mean and then that array of all the precision
values.
So that's going to be precision and we can print this out.
Use the F string again formatted the same way as before looking back at the definition of how we calculate
the f score.
We see that it is equal to two times.
Precision times recall divided by precision plus recall having already calculated the average recall
score and the average precision score we can calculate the f score or F one score simply by using these
values two times and then it was average precision times average recall divided by average precision
plus average recall we can print the sound and there we go.
Our f score for this model is forty nine point zero four percent.
So now we've calculated all our metrics.
We've calculated our accuracy.
We've calculated our F school we've calculated our recall and we've calculated our precision school.
How are we stacking up.
Well considering that we're getting about 50 percent correct I'd say this is not bad for a very first
try and for such a simple model.
If we take a look at this Web site that's hosting our datasets we can see that the baseline result is
closer to an 18 percent test error.
We got close to 50 percent incorrect.
Now at this point you might ask well why is our error so high.
Well one thing that we saw was that the more data we had the more accurate our model became.
But there are other ways to improve accuracy as well.
You see a more specialized neural network for computer vision would definitely fare better with the
same amount of data than our multilayer perception.
In fact a model structure closer to the inception resonate that we've used with pre trained waits which
is a convolution or neural network or CNN would achieve a much higher accuracy.
However I think we should give the multilayer perception another chance.
It's a very very simple model after all.
And it'll be very interesting to see how it stacks up on a different dataset.
Another reason I want to do this is because these past few lessons in this module were very very theory
heavy and we've covered a lot of new concepts and in the next module what I want to do is I want to
focus more on tensor flow itself.
I want to take you into a deep dive on how to use tensor flow without Charisse.
So what we're going to do next is we're going to continue building on our understanding of the multilayer
perception for the time being but we will focus more on how tensor flow actually works on what a tensor
actually is and how to set up your layers and your weights intensive flow and how to batch your data
during your training session.
Plus we're going to be exploring tensor board on a whole new level.
So well done for persevering through this challenging module and I'm looking forward to seeing you in
the next one.
Until then take care.