Oh we've really covered a bunch but we haven't even touched the scoring parameter.
And as a refresh the scoring parameter is one of the second ways that we can evaluate a model right.
So the scoring parameter is used with model of our nation tools that use cross validation such as model
selection across our score.
We've seen that one before and model selection dot grid search CV.
Now we haven't seen that one before but what we're going to do is check out how we can use the scoring
parameter with cross value score before we do so let's set up a little heading.
Let's remind ourselves finally using the scoring parameter and I know there's a lot to take in right
and I don't expect you to know everything right straight away.
It's gonna take a little bit of practice a little bit of time that be the effort for these things to
start to sink in right.
So here we go finally using the scoring parameter.
And speaking of practice it's one of the reasons why we keep practicing writing out building machine
learning models and splitting our data.
So from model selection we're going to import cross value score because that's where we can use a scoring
parameter.
And then we're going to import our random forest classifier import random forest classifier yep and
then we can go empty at random seed 42 our favorite number again you could use almost any number here.
I actually really like the number 72 but 42 will do X equals.
We'll set up our X data heart disease not drop target access equals one beautiful Y equals heart disease
target.
There we go.
And now we can set up a classifier and we're going to use of course random forest classifier.
Our favorite the warhorse of machine learning will set number of estimates to be in one hundred so we
don't get that pesky little warning that we've been getting.
Now we'll do that.
Beautiful.
So we've instantiated our machine learning model it's ready to be fit so we can do that here.
MP random say we'll do it again and we'll go.
CV ACH why we don't see the act.
We'll see in a second run cross.
Now school S.L. f x y.
And then we'll use fivefold cross validation save CV AK wonderful is gonna try and five models evaluate
them on different test sets and then of course we could take the main of this.
So why don't we do that we'll go cross validated accuracy so we can go print f the cross validated.
Accuracy is need some space.
Daniel come off the main and P main CV.
And then we're going to take that too.
We'll only use times one hundred and two decimal places and we'll put a little percentage sign on the
end and we need a final String delimiter.
Beautiful.
So now across validated.
Accuracy is eighty two point four percent now we've seen this before right.
We've seen this line we've seen this line and the reason why is we're using this is because by default
the model it uses accuracy cross Val score because in essence this is scoring equals num so this is
gonna produce CV ACH and then if we go here.
So now if we just copy this right actually what we might do is just test it out by only changing one
thing.
So if we take this we'll copy this line just to prove it.
This is where the scoring parameter comes in.
So we can use accuracy here.
All we're doing is changing num to accuracy because remember our random forest classifier by default
uses mean accuracy.
So remember the scoring parameter when left to none uses a classifier or estimate as default parameter
from score to evaluate using cross validation.
So we'll get rid of that now all we've done is we change scoring equals accuracy and then we're gonna
take this line here we'll just copy this and if this is true right if I'm not telling lies it should
print out the exact same thing here because CV ACH equals cos Val scores CSF x y CV equals five scoring
goals accuracy should be the same as this.
Well let's say it.
You know why it was different was because we forgot our p random seed
yes there we go bomb.
See that's another reason highlighting the use of MP random seed round is making our experiments reproducible.
All right.
Now to see another metric this is where the scoring parameter really starts to shine How about we use
precision so CV precision to evaluate our classification model across fell score because remember precision
is one of the metrics that we can use to evaluate our classification models.
Now all we're going to do is change scoring equal to precision beautiful.
And we'll see what savy precision is
wonderful.
But we probably take the empty main
beautiful so point out too on the precision front.
We want to hear precision.
Now how about recall.
So go see the recall Eagles cross vow score S.L. f x y CV equals five scoring can you guess want what
the scoring parameter might be equal to every wanted to calculate recall not to worry if you can't do
it here recall and then we want NDP main to get the cross validated recall
wonderful point eight for recall now how about for F1 how about we do that yeah why not save a f one
equals member F one is a combination of precision and recall x y CV equals five scoring equals F one
we want P main don't CV F one.
Wonderful.
There we go.
Okay now we've done some classification metrics here.
How about now regression model.
Well why don't we make a regression model and check it out.
Okay.
From a skyline that model selection import cross Val school.
So we want to cross validate our regression models predictions.
So from SCA learned ensemble to get all the benefits that cross validation offers random forest regress
a look at us go typing machine learning code like it's nothing.
Go and pay random seed 42 will create out x data.
So Boston D F Yes drop target access.
Can you guess what axis axis because 1 guess it goes Boston DNF.
So this is why actually tog I'm getting him myself my voice can't keep out with my coding model equals
which is very unusual for me actually.
Random Forest regress.
We'll set an estimate as again to being tablet a complaint to being 100.
Wonderful.
And now do you remember what the default metric is for score for a regression model.
So if we do model dot score the default evaluation metric is returns the coefficient of the termination
r squared of the prediction.
Alrighty well let's check it out.
So NDP random seed and then we're gonna go save a squared equals cross Val score model this time X Y
CV fivefold cross validation let's do scoring and equals none.
Right now we're going to see the R squared and we're gonna copy this exact code bomb.
And because remember cross vows score uses the default evaluation metric in the score function.
If set to None.
So if we just change this to be R2 what do you think's gonna happen.
Because this is shofar squared right.
So if we got here
foam we get the same values.
Look at this.
Now this is pretty pretty drastically different right.
So we've got over here.
This is point eight five.
This is what I was talking about.
Getting that lucky split.
So cross validation on this one it's done pretty well.
So point eight five.
But on this split it's only down point two five.
So if we got here if we did the empty main
point six to two.
Now that's that's quite a bit different to the first r squared score we saw just using the score function.
Well let's keep going.
Let's not stop there.
How about we do mean absolute error because that's another regression metric.
So let's try it out.
So CV maybe we'll put it in here mean absolute error.
Wonderful CV MIB equals cross value score we'll pass it our model which is a random forest regress.
X Y a CV of five scoring is going to be.
What might this be we might type in guess this string mean absolute error.
Let's just see see them all laid out because that was pretty cool to see with our r squared score right
just to see how different they can be across five different splits of data using cross validation key
error mean absolute error.
Now why is that.
And I'll tell you why it's because we need neg in the front here or and you might be asking again why
neg but I'll share that in a second.
Okay so that's interesting.
So if we come up here we can tell why it means neg is because all scorer objects follow the convention
that higher return values are better than lower return values.
Right.
So that's that's a key point to remember here.
This is the documentation all scorer objects follow the convention that higher return values are better
than lower return value.
So if we come down here this is why we have Mike zoom out because that's that's looking a bit too big.
This is why we have Meg mean absolute error.
Looking back in a notebook.
See these values are all negatives.
And the reason being is because all scorer objects follow the convention that higher return values are
better than a lower return values.
So that means because these are negative the higher these are so the closer they are to zero.
It's better than the lower they are.
So these are pretty low.
So again our model isn't perfect but you can see the variations in the mean absolute error values just
like we had in the r squared values is that some splits like in 5 splits are offering negative point
to 1 2 we saw two point one two up before but other ones are getting we're getting a much higher one
he has a negative three point eight.
So this is where cross validation really comes into play Ryan is figuring out Okay are we just getting
a lucky split on our model and training it on some easy training data to understand and then evaluating
on the test data set we might be going if it's lucky split it might be getting metrics that aren't really
congruent with how the model is actually performing so let's finish this off we're doing mean squared
error and now I bet you can guess how to do this one.
So CV MSE equals MSA for main square cross score will pass it the model do X Y and we'll do CV equals
five scoring equals.
Now if we'd had to do NEG mean absolute error what do you think it is for Main squared error and if
you guessed neg main squared error you would be 100 percent correct which is the opposite to what our
model is.
CV MSA shift and enter and again we're seeing some quite quite a wide range of values here right.
So we go from negative seven point seventy seven right up to the highest one here is negative forty
six point two.
So we could take the mean of these to see what it looks like across five different splits
so negative twenty one point five so that's a bit different again to our previous value when we weren't
using cross validation right of means squared error is nine point two for Mm hmm.
So naturally if we're looking at these evaluation metrics and we're starting to see okay we want to
get better metrics you might be starting to think is like how can we start to improve these metrics
how can we make them better.
And we're definitely going to look at that but finally to finish that evaluation metric section off
we're going to have a look at the third and final way of evaluating a model and that is using metric
functions.
Spoiler alert you've seen some of these in action before is to make sure we're evaluating our models
correctly we're gonna have a look at those so take a little break we covered a lot in this video try
adjusting the scoring parameter yourself Don't forget if you're getting different values you might be
using a different seed every time and I'll see you in the next video.