It's in the last video we left off saying that these metrics had precision recall and f1 score are only
calculated using one split train test split.
So only calculated using one but we want to prefer using cross validation where possible.
So that's what we're going to do.
Let's make a little heading we'll go calculate evaluation metrics using cross validation.
All right.
Here we're going to calculate precision recall in f1 score of our model using cross validation and to
do so we'll be using.
Let me put this one in code cross Val school.
So we've seen this one in the in the socket line section.
Cross Val school and if we look it up here as we always want to do so cross Val school what's it say
so it takes an estimate.
It takes an X it takes a y evaluate a school by cross validation the thing we want to pay attention
to here is scoring.
So if we get a scoring a string callable a string or score callable object function with signature which
should return only a single value.
OK.
So this is the parameter we're going to use to evaluate our model using cross validation with different
metrics.
So we're gonna change the scoring string so let's say that we'll set up a new logistic regression model
instance with our best type of parameters.
What were our best hybrid.
So let's go check the hybrid parameters because we want to set up one of the best models that we can
and then use cross validation to evaluate it.
G.S. log rig dot based programs.
Wonderful.
So now we're going to create a classifier go here create a new classifier with base parameters that
we've found so CnF equals logistic regression Sea Eagles we're going to just copy that here.
Wonderful.
Whoops I forgot the 0.
That's a very long number there and then solver equals.
Now again these parameters might be different if you've done your own hype of hammering a tuning and
found some better ones but that will do for us for now.
Wonderful.
So now we've instantiated a model with the best high parameters let's use Crossrail score along with
the scoring parameter to get some cross validated metrics.
So what we do I've actually forgotten here.
We're going to calculate yes there we go.
So we want accuracy here.
Cross validated accuracy.
So we might do four cells and we go here cross validated precision
cross validated recall and then cross validated
EF 1 school.
Wonderful.
So let's do that.
So we got gonna CV ACH which stands for cross validation accuracy equals cross vowel score and we're
going to pass it out classifier which we just instantiated here.
We're gonna pass it all of the X data because we can now because we're using cross vowel school and
all of the y data and we're gonna set CV to five and we're gonna set scoring to be accuracy beautiful
and then we're gonna check out what CV act looks like and we can take the main of this because remember
but what is it done it's evaluated our model over five different splits.
So if we take the mean of it all right that's gonna get the average accuracy across these five different
splints.
So let's do that.
So we go empty You don't mean save a wonderful.
So that's our accuracy there.
So we might say that actually you might just override our value and then we'll go save a beautiful.
And now let's do the sign with these elements.
We'll go here we could just copy the code here again.
You don't want to get into the habit of copying code.
We should really function like this but we're just going to keep rolling with the punches what we're
doing now.
And by functional is this I mean because we're doing relatively similar calculations the whole way through.
We could just make it function.
So that is the precision might override this to be precision equals MP main and then see it again wonderful.
And then we could do the same thing here except for recall what we might do is be a little bit tricky
put in spaces here when you hold command you can put little curses here.
So if we just change this old to recall how happy days look at that I got five cases on the go recall
one for oh that's a nice recall score cross validated as well we might do the same for F one and we
go here we're gonna change all of these bad boys to F one.
There we go.
We might put a little space.
So our code is looking polyphonic wonderful.
All right.
And so now we've got all these cross validation metrics across validated metrics that our boss is requesting.
It's not really good to just happen Pierson and Jupiter and I book like all of our other valuable staffers
in a nice neat little table is our classification matrix that looks really good in a presentation.
You know this is something that someone could look at and go yep I can see that value is pretty high.
I can see what's happening here.
Let's do the same.
Rather than just having them all spread out let's pull into a graph of sorts or a visualization.
So we go visualize our cross validated metrics.
So we're going to create save a metrics as a data frame.
I mean yeah we've probably really should have functional this to begin with but that's right.
Sometimes you have to just go through the old fashioned way and then realize why you've gone wrong and
see how you could improve and maybe that's a little extension you could try yourself and see how you
could function or some of what we're doing here.
Maybe I just cross validate some X and Y using different metrics and puts it all into a nice little
presentation all in one hit that would be a good practice save a recall and then we go if one is going
to be our CV F one wonderful we need to set an index equals zero beautiful and then we can create a
plot save a matrix.
Now I've done this before.
Spoiler alert.
So I know I need to transpose it.
Bah.
Title equals cross validated classification metrics.
And do we want a legend.
No we don't.
False.
Let's see what that looks like.
Beautiful.
We're gonna put a little semicolon here.
There we go.
So maybe we could probably add like the numbers on hand so people know exactly what they are but we
see our models doing pretty well on recall.
And again we could probably add our cross validation results from our random forest model or our Kenya's
neighbors classifier or any other classification model that we may have tried during our experimentation
phase.
Remember this experimentation phase is just it's an iterative process right.
Just going back and back and back and forth through here so we could add that to here and see which
model performs best.
But for now we're sticking with logistic regression and this looks like something that we could share.
So what do we have left now.
Well let's go back out to where we put down what we've what we've ticked off.
Have we ticked off everything the things our boss was asking.
The things we previously didn't know how to do.
But now we've seen them in action.
All right.
So we've got cross validation precision recall Yep classification report tick ROIC curve tick area and
look have fusion matrix.
Yes.
The thing we're missing out on here is feature importance.
Mm hmm.
All right well let's see how we do feature importance in the next video.