Welcome back.
The last video we saw had a tune a model's hyper parameters using randomized search CV.
And what that did was we defined a space or dictionary of different hyper parameter values we'd like
to try and then randomize socially we combine these parameters at random and built a a classifier with
different settings different of parameters and then evaluated them all using cross validation or we
might change this back to 10.
We don't want 50 models running off.
And then after trying 10 different combinations it found the best parameters to be this set of values
here.
Then after evaluating them we saw it didn't really do as well as our previous model up here where we
choose just by hand and so finally a workflow you might end up with when you're turning off parameters
is starting off by hand and then using randomized search CV across a space of high parameters.
And then once you found some pretty good hybrid parameters you'll probably finish up with grid search
CV.
Let's see that in action here.
So five point three hyper parameter tuning with grid search CV beautiful.
And let's check out our grid so our grid at the moment is you might notice the grid search CV has the
word grid in it and our grid of hybrid parameters at the moment are these values here.
And the key difference here between randomize search and grid search CV is that randomize search CV
has a parameter called n ita which is what we saw up here which we can set to limit the number of models
to try so in our case we use 10.
So if we look back up here the top of this code fitting five folds for each of 10 candidates totalling
50 fits.
The key difference between randomized socially V and grid search CV is that grid search CV is kind of
like a brute force search it means it will go through every single combination that is available here.
So if we have a look we've got six values there.
So we six times five times two times three times three.
So there's six parameters here.
There's five here two here three three in total 540 different parameters that it would try or combinations
of parameters.
That's a lot that's and then cross validated So time's up by five.
So two thousand seven hundred model.
That's a lot right.
That's going to take a lot of compute power training one model might take long enough time but training
twenty seven hundred of them especially if you're working with a large dataset probably not going to
be suitable for your laptop you may need a big computer or we can reduce the search space so let's see
that in action.
So what I mean by that is we'll create Grid 2 using a little bit more of a confined hyper parameter
space.
So in essence just reducing the number of hyper parameters grid search CV has to go through.
So let's see this in action what we might do is just copy this and now how would you create Grid 2 in
practice.
We could just copy and paste it which we've done.
I want to line these up so it all looks pi formic.
There we go.
And we could just copy and paste in and then just delete these at random delete delete we might not
do that say we've gone through randomize search CV and it's gone through let's say we up this to 50
different models like we had before but in our case we've only actually done 10 we might take these
parameters the best parameter from our randomize search CV and use them to influence where we'd like
grid search CV to run.
So let's do that our best parameters is two hundred six to square root.
So what we might do remove 10 keep 500 day and remove these two yep max depth is none.
So we might just keep that as none so it's only got one option there and then for square root we might
actually keep that is an order.
We'll square root then what else do we have means sample split is six.
We might get rid of four in two and then what we might do mean samples leaf.
So we might keep that is one and two.
So what have we done here.
Well what we've done is we've reduced our search space of hyper parameters so now you remember we had
540 different combinations there.
What do we have.
So we have three times one times two times one times two so twelve.
And then it's gonna be cross validated five times so we can adjust this using the CV parameter.
So 60 versus 2700.
So that's a lot less parameters.
So how would we do this in practice how would we use grid search CV to find the best hyper parameters
in this space.
Well as always let's see it in action from S K learn the model selection import grid search CV and we
want to import as well.
Train test split we probably already have that we can put that there the random seed wonderful split
edit what we might do actually is just bring this code up here because it's the exact same code that
we want to use before.
So we'll just take this said we're going to change one thing come down wonderful.
So we're using the same dataset as our randomized search CV making the same split we're instantiating
a random forest classifier.
But this time we're gonna set up grid search savy grid search save a wonderful grid search saving and
then we're going to fit it the grid search savy here and now there's a different parameter here.
So this one is going to be param grid I believe it's not going to have an ETA because grid search CV
is like brute force it tries every single combination and we want it to try out grid 2 not grid one
because it's got a few less options there and grid 2 as we saw before.
So we'll come down here.
Keep cross validation as fivefold verbose equals that.
Okay.
Now let's step through what's happening.
So we're importing grid search TV.
We've created Grid 2 which is a refined search base of different hyper parameters so different settings
so much like if you were working on your favorite meal adjusting your oven.
If you set the temperature to 400 degrees that's way too high.
So if it starts off 180 degrees and it does okay You might go why would I go right to 500 or 400 degrees
when I can just make a little jump to 400.
So that's why we've created Grid 2 We've based it off the best parameters or the best hyper parameters
that our randomize searched CV or randomize search CV has found so let's do it let's build a random
fast classifier using grid search CV.
So here's how it's gonna go again.
Same thing in feeding five falls for each of 12 candidates totalling sixty fits.
Now what grid search CV is going to do just like randomize search CV.
It's going to go through all of these different hybrid parameters and pass them to our random forest
classifier trying each different combination and then eventually when it's finished we'll be able to
go G.S. CnF best Grams
wonderful so it might take a little while with grid search CV.
This is something to keep in mind the more hybrid parameters you have in here the longer it will take.
Grid search CV to run because it has to try every single combination of high parameters in here.
So now as we did before we can evaluate our grid search classifier by making predictions with it and
then using our evaluation function to figure out how it did.
So let's do that.
G.S. CLSA I actually got to guess why friends equals G.S. Steel f don't predict will predict on the
test data as well.
Ex test and then we'll go evaluate the predictions.
G.S. metrics equals evaluate Fred's our function when that we defined a couple of videos ago and then
go.
G.S. y Fred's wonderful and now we again we see a slight decline in accuracy.
Mm hmm.
And that just goes to show how much of a process churning hyper parameters can be.
Right.
It's all about that trial and error.
What you could do to fix this is go back up your workflow would probably be try a few settings by hand
as we did right back up here churning hard parameters by hand.
This is the workflow you'd probably do try.
Just a few of these by hand see if you can figure something out with the validation set like we've done
and then try some high parameter tuning using randomized search CV you might in this case up this number
to 20 or 50 or so or whatever arbitrary number that you find again.
This is all experimentation all to try and figure out what type of parameters are better and you might
even change the grid that you're searching over based on the hyper parameters you find in the psychic
learn model documentation for the model that you're using and then once you've found some good type
of RAM and is using randomized search CV you might take that those hyper parameters these ones here
best parameters and then create another grid to grid search over like we've done here in our case it
hasn't improved our models metrics but this is where we'd probably start to experiment a little bit
more.
Try a few different other metrics try a few different other hybrid parameters but for completeness what
you probably do then after you've tried a fair few different parameters you've created a baseline model
with the default settings you've tried to adjust and by hand you've tried to adjust and by randomize
search TV and you've tried to adjust and by grid search you're gonna want to compare your models so
let's compare
how different models metrics
wonderful and to do so we're going to create a data frame so payday.
This is why we've been returning dictionaries from all of our all of our models and I'll evaluate Fred's
function so baseline is going to be baseline metrics and then a 2 is going to be left to metrics type
and ends here when we don't need them random search is going to be our race metrics and then grid search
is going to be G.S. metrics wonderful and then to see it in action we're gonna go compare metrics which
is just a data frame that we've instantiated here or we might need to move this up there is gonna be
an error there in there that's actually squiggly bracket that we didn't need dot plot dot the big size
equals 10 eight let's see it wonderful.
So this is going to compare all of our different classification models metrics in the one space so we
can see our baseline model is the blue so on this one it didn't get the best year it was more of a tie
between CnF 2 and random search on precision CnF 2 is out in front on recall baseline is out in front
and on EF 1 it's a tie between baseline and EF 2 and again you could adjust this graph to to make it
more visible.
This is just to demonstrate different metrics.
So this is the kind of communication you do for not only yourself but for your teammates or if you're
reporting to someone in a project is to show them the different models that you've tried and and how
they perform differently on different classification metrics.
If you are working on a classification problem of course you can do the same sort of process with a
regression problem and now depending on what you need for the project will depend on which model.
So say you wanted a model that optimized recall in this case you might use the the baseline model or
if you wanted a model that optimized precision you might choose the CSF to model.
And if none of these metrics really worked for you you might continue to trying to find some better
ones through random search or grid search.
And now we finally got to the end of how you can improve a model via hyper parameter tuning.
And again there is a lot to take in here.
The key point is remember it's an experimental process improving a model through hyper parameter tuning
is it there's no sort of written law on how you do it but the point is to just keep trying.
See if you can figure out something.
See if you can get yourself familiarized with with different hybrid parameters at different models have
such as a random forest classifier and just remind yourself that the first model that you make is more
than likely not the best model that you have.
It's a baseline model and it can be improved upon and so built a model we've tried to improve it.
Let's say you've decided you're going to focus on on recall or something like that.
You may want to export your model so you can share it or use it in some sort of deployment setup.
So in the next section we're going to look at if we look at our list here what we're covering we're
going to look at how to save and share and load a model so save and load a trained model.
The benefit of this is that you won't have to retrain a model like we've done in the past few sections
so go back through check out what we've covered with high priority training how to improve a model high
parameter with grid search CV.
Try it out with randomize search.
Try it out by hand and I'll see you in the next video.