Wonderful.
We've seen how to quickly get a sniff of how our machine learning models doing and evaluate it using
the score method and that will return a default evaluation metric depending on the problem we're working
on in regression it's returns the coefficient of determination and in classification it returns the
mean accuracy.
However when you get further into a problem it's likely you'll want to start using some more powerful
metrics to evaluate your model's performance.
And so naturally the next step up from using score is to use a custom scoring parameter.
So if we see here we've looked at this one estimate a score method and now we're gonna have a look at
using a scoring parameter model evaluation tools using cross validation or we might have to see what
that is.
So let's dive into it.
Hey what we're going to do is to work this out.
Now you'll have to bear with me for the next few videos because we're going to cover a fair bit here.
Reason being is because evaluating a machine learning model is such an important step.
It's one thing to call a fit function on some data but the next most important thing is going hey is
that model actually working is it learning something.
Could we use that to predict in the future.
And so that's why making sure that we're covering a lot of ground here.
So let's get started.
What we're going to do is go from S.K. learn not model selection import cross vowel score.
And as always we're gonna run the code first before we dive into what's actually going on and what we
do need is some classification code.
So what we're going to do.
We can copy this.
You're allowed to copy this by the way because we've written in a fair few times now.
So only thing different here of what we've done is we've imported cross vowel score from cyclones model
selection.
Now next thing we're going to do because we've called FET there whereas put a little semicolon so we
don't get a big output.
You know what this warning keeps coming up so I might just keep changing that and estimate as equals
100.
Wonderful.
And so cross Val score it has the word score in it but we've seen what does this cross Val doing.
Well let's have a look.
Look I see a dot score we're going to do the test data first so we can compare and then we'll do the
same but this time using cross Val scored cross Val school and cross Val score takes our classifier
it takes X data and it takes a y data on not the test and not the train we'll see what's going on in
the second huh what's happening here.
And again we're getting another warning this is just to say that the default value of CV will change
from three to five.
So if we have a look at this shift tab let's read the doctoring evaluating a score by cross validation
What even is cross validation and so CV is gonna give us a warning.
This is where that warning is coming from.
If we change this from CV Eagles the default is three if we change it to five.
That warning is going to go away and then we're going to get an array back of five different scores.
So that's really the first difference that you'll notice cross Val score returns an array whereas score
only returns a single number okay.
So how can we figure this out.
We need to figure out what cross validation is because that's after all what cross Val score is doing
come back into the doctoring value add a score by cross validation on lucky his on the not prepared
earlier to demonstrate.
And as always I understand things better visually you might do the same to demonstrate what cross validation
is doing.
So what we've done before in our normal training test splint.
So we've split our data into training so this could be a stay we started off with 100 patient records
we'd split it into a training split in our case we've used 80 percent and this would contain X train
and Y train 80 samples and I've used the number 100 here we've really got more than that but this is
just because the numbers work out visually better here and in our test data set we've got 20 percent
of the data which would contain x test and Y test.
Now the difference here with cross validation and in our case this image here is demonstrating five
fold cross validation.
What you probably see cross validation referred to is K fold where k is an arbitrary number and the
reason why we're using five is because we've come back here CV equals five there.
So that stands for cross validation and this is what I'm talking about splitting our data into training
and test using 20 per cent for the test size.
So that means naturally the other 80 per cent goes to the training data which is where this level split
here comes from.
Now what cross validation does is it does five different splits.
So it will use the first 20 percent.
So see how here this is kind of using the last tail 20 percent here.
So it'll create a test data set here and a training data set here then we'll move over here and use
this as the test data set and then again and again and again.
And so what happens is that cross validation trains five different versions of the model and then it
evaluates that those models trained on each training data on five different versions of the test data.
So what's the purpose of this.
Well as you could imagine if we're only training one model it could be a lucky split like say this 80
percent of rows say that had a whole bunch of information in the model was able to learn really well
on these 80 rows on these 80 patient records and then it got a really good score on this test set.
Is that a true reflection of how our model would understand the data or figure out the patterns in the
data.
Well not really right because it's just luck right.
If we're just splitting this randomly and somehow a bunch of easy patient records have gotten here and
the models figured out certain patterns and it's gone over to here this test data center it's got an
amazing score we could be tricking ourselves we could be fooling ourselves into thinking that our model
is far better than what it actually is.
So that's where cross validation comes in to play.
It aims to provide a solution to not training on all the data and avoiding getting those lucky scores
on just a single split of data.
So it'll create five different splits.
So no matter what our model is going to be training on all of the data and evaluated on all of the data.
And so this is why if we come back to C cross value score it's why gives us back five different scores
here.
So that's starting to make sense of we call the score parameter on only our x test data and our y test
data is gonna give back one score.
But if we call cross value score because we refer back to the graphic here it's going to make five different
splits and remember five fold is just an arbitrary number you could do 10 fold you could do three fold
you could even do 100 fold but five fold is the default of the library it's usually pretty good depending
on the size of your data.
So we'll use fivefold to demonstrate here just to prove it to you.
We can go in here cross Val score.
We could even do 10.
So this means it's just gonna make 10 different splits exactly the same as this and then return 10 different
scores so you see here this is a great example on split 1.
It's got a score of point nine.
So that could be 90 percent which is higher here but on a later split it's got something much lower
72.
And so what we do here is to figure out in a more ideal performance metric or evaluation metric for
our model is that we can take the average of these five scores let's see it happen we'll do it all in
one cell random seed and what we're going to do is get a single training and test lit score we're going
to make sure see a left single score equals CnF score x test it's gonna use the same data here why test
wonderful.
And then we're gonna go take mane of five fold cross validation school CSF cross vowel school Eagles
NDP mane then we need cross vowels score X on No we need to pass it our classifier x y and will you
see the Eagles five and then compare the two so I see a live single score and then see a left cross
Val score burn.
So what do you see here.
Well in our case our original single score which is now down here just the exact same number because
we're using a random seed using the same test data our single score is point eight five but when we
use cross validation when we use five splits because CV equals five we get a score of point eight two
so it's slightly lower but in this case if you are asked to report the accuracy of your model even though
it is lower you'd prefer the cross validation metric over the non cross validation metric.
Now wait we haven't even used the scoring parameter at all.
Well that's because by default it's set to None.
Let's have a look at the scoring parameter set to None by default.
So if we call cross Val school seal Elif x y CV we can pass scoring here and it's gonna be set to None.
So if we do shift tab on this how do I know this.
Well luckily the docs string comes in handy C by default it's set to None.
So if we keep scrolling down move our notebook.
So scoring string callable on one optional default.
None.
Okay a string.
See model evaluation documentation.
That's what we've had to look up here or a scorer callable object slash function with signature scorer
estimated x y wish a return only a single value if none the estimate is default scorer if available
is used.
Okay now this is why we know that this is accuracy because if the scoring parameter of cross vowel score
is none it uses the default scoring parameter of our estimate.
In our case what is a default
scoring parameter of classifier equals mean accuracy.
And where do we see that before we saw that in last video.
Go see a left score and we hit shift tab returns the mean accuracy on the given test data and label
so that means when we have scoring set to None.
It's gonna use the default evaluation metric for cross validation on our classifier so if we hit shift
and enter it's gonna return the same values all might be slightly different right because we haven't
set up a seed in this cell so these values are gonna be different to the cross fail score we see out
there.
If we'd run it in here we would have seen a similar values.
Who would have we covered here.
Well as you might have guessed the scoring parameter can be changed right so we can as the docs shrink
says we can input our own scoring parameter here.
We can change this to something other than none.
That is what we're going to start to cover right.
We're gonna have a look at in the next few videos.
Some other classification model evaluation metrics that we can use with cross validation.
And so while we use cross validation well as we saw in the picture cross validation aims to solve the
problem of not training on all the data.
Right.
So we're creating five models and we end up having a model trained on all of the data and avoiding getting
lucky scores.
So training on a single split.
And we saw that in action with our single split score here getting a slightly higher score than our
cross validation score.
Who.
That's a lot.
Now we've still got a bit more to go.
Let's get some depth classification metrics happening.
I'll see in the next video.