So let's start diving into different ways we can improve our models using hyper parameter chaining.
The first one here is by hand.
So let's make a little heading 5.1 tuning hyper parameters by hand and so far we've talked about dealing
with training and test data sets a model gets trained on the training set it finds patterns and then
it gets evaluated on the test set so it uses those patterns but hyper parameter tuning introduces a
third set a validation set.
So if we have a look here is what it's gonna look like when we turn half of parameters by hand.
So if we were starting with a 100 patient records in the case of our heart disease problem.
Now again this this may change depending on what problem you're working with.
And I've just used the number of hundred because it's an easy number to picture.
So this is our starting data and what we might do is split it.
This is what I mean by three different sets.
So we split we might use 70 to 80 percent so 70 patient records as a training split we might use 10
to 15 percent as a validation split and we might use 10 to 15 percent on a test split.
Now of course you can adjust these numbers these are just some common numbers you'll see used.
And so naturally the model gets trained on these is what we've seen before and usually without the validation
set our model will get evaluated on these.
But now because we have a validation set this is where we're going to choose our model settings a.k.a.
the hyper parameters get tuned on the validation split.
And then finally as normal a model gets evaluated on the test split.
And now the analogy here is if you remember right back at the start the most important concept in machine
learning is the three sets.
So this was right back in the start.
Machine learning 1 0 1 the training set is analogous to say you're at a university course and you were
learning the course materials the validation set is where you would test your knowledge a little bit
and see what you need to adjust so you would get the practice exam from your professor and you try it
out and you go Oh wow I'm doing terribly at questions 1 2 and 4 maybe I need to adjust the way I approach
this and then you do their practice exam again.
And once you've improved your skills on the practice exam you'd feel a little bit confident and then
you'd finally really evaluate yourself on the final exam which is the test set.
So how would we do this with code.
Let's come back to our notebook churning hyper parameters with hand.
Let's make three sets.
Training validation and test.
So what we'll do is we'll just remind ourselves of our get firearms or remind ourselves of our random
forest classify as baseline parameters so these are all the baseline parameters a.k.a. the settings
on our model which we can adjust and now in fact which ones are we going to adjust.
Well after reading the random forest documentation and again you can do this for any machine learning
estimate or model with socket loan we start to get an idea of how we could adjust each setting.
And even with some of the models there'll be some notes on different of parameters socket learn suggest
to change that is kind of like the ones you want to change first to try.
And so after reading through that after going through that change.
So in our case we're going to try and adjust the following.
So we want max depth actually we'll put these in back ticks because we know that there.
So we know that they code max depth Max features mean samples leaf.
If none of these makes sense remember the definition to all of these is in the documentation for any
model.
So if we come up here means samples leave the minimum number of samples required to be at a leaf known
and you can read more in depth in what a leaf note is.
If you'd check out some resources on a random for us in depth.
But for now we're just focusing on how we can adjust hyper parameters of a machine learning model I
samples leaf and mean samples split and an estimate is wonderful.
So these are the type of parameters we're going to adjust and now we'll use our same code as before
except this time we need to create a training validation and test split which just uses these splits
here.
So the training split will create with 70 percent of the data the validation and test sets will each
contain 15.
And we'll get some baseline results.
So we've kind of already got them but we'll do it again get some baseline results and then we'll see
how we can tune the models hybrid parameters by hand.
And since we're going to be evaluating a few models I think it's important that we create an evaluation
function.
So this is what you might want to do whatever model you're you're working on is create functions.
If you know you want to do something more than once.
So it saves out writing lots of code.
But again that's probably are a bit different to what we've seen in the past right.
We know we like to write everything out by hand.
We just leave little doctoring it performs evaluation comparison on wine true labels that's why pred
labels and now as we saw in the evaluation section the main thing evaluating machine learning model
does is it compares its predictions versus the true labels.
So that's what this function is going to do so because we're working with the classifier we want accuracy
accuracy score y true why parades we also want precision and now we might actually note here on our
classification model because if we use this evaluate spreads on a regression model we're putting in
classification metrics we're going to get errors because a regression model predicts different things
to what a classification model predicts precision and then we want recall of course equals recall score.
And now again you could adjust this evaluate Fred's function for what you need but I'm just gonna include
some of the most common metrics that we've covered for the classification models some of things you
want to pay attention to the basically all of your classification models y periods and then we'll create
a dictionary so it can return the predictions so we can compare them with other predictions later round
accuracy we'll go to two decimal places.
Yeah.
Precision round precision to Now always remember this seems like a lot.
It's because first of all it kind of is a lot to take in and one hit and the second thing is that I've
had a fair bit of practice with this.
So I inherently know what to use.
Again I'm always learning but I've just had a bit of practice building these specific kinds of systems
and these patents come up time and time again.
Go Print.
We want just to give us a little bit of printout of what's going on AK accuracy times 100 and then we
only want to decimal places that will be enough.
Then we'll go the same thing.
If precision this can be precision we won't need to times out by 100.
That can just stay at two decimal places.
We need a string and a wonderful print.
Then we're going to go recall is a lot to type out to begin with but because we'll be able to reuse
it.
It's gonna save us down the track.
And then finally we want to print if one school
we're making it might finally make an excellent effort.
There we go.
Okay.
And finally we want to return our mystery date.
Now what is happening here.
Well essentially this function just takes some true labels and some prediction labels from our classification
models we know this from the string here.
It's gonna compute different valuation functions here and then it's going to return a metric dictionary
so we can save that for later and then also print out some metrics.
So we get a print out as soon as we run this function.
So just an evaluation function for our classification models.
WONDERFUL AND I'M GONNA SAVE THE NOTEBOOK here as always.
Now how exactly would we create these train validation and test splits.
Well our train test split function that we've seen before remember this one train discipline.
So this one only returns.
Train splits arrays or matrices into random train and test subsets so this one only splits into train
and test sets.
So what we need to do is we need to manually split our data into train validation and test sets.
And so to do this we can how can we do this because our data is in a data frame heart disease.
We should just be able to do it using some good old math and indexing.
Let's see how we would say from S.K. learn we'll import our model first even though we've already instantiated
a random forest classifier
let's go empty random seed and then we'll shuffle the data.
Why are we shuffling the data here.
If we're splitting into train validation and test when we want to mix it up make sure it's not just
the same.
The records are coming in order that they come in.
If we're using slicing to create our train validation and test when we want to make sure all of these
records are jumbled up.
So that's what we'll do.
We can shuffle it using pandas sample function we frac Eagles 1 4 100 percent of the data.
This is just going to take the heart disease sample it.
This is randomly and then reassign the heart disease data frame to its normal variable.
So just take this sample a bunch of rows at random and then reassign it to this variable a.k.a. shuffling
the data now.
Now the data is being shuffled will split into x and y.
Actually you might say that to heart disease shuffled as a better idea X equals.
This time we need heart disease shuffled and then we're going to drop the target column here.
This video is dragging a little bit on but this is an important concept right.
We're going to see here it'll all be worth it because we'll be able to improve our models using high
performance tuning split the data validation and test sets.
Wonderful.
So the train split.
How we gonna do this.
Well we need to create a number that we can use for slicing.
So we want 70 percent of the length of our data.
So you see what's happening here.
Train split.
We need a number.
We need 70 percent so zero point seven times the length of heart disease shuffled.
And this is going to be 70 percent of data and then we need a valid split which is going to be do you
remember.
Remember what the split is 15 percent that's what we've agreed on.
Wonderful.
So we can do that in the same way that we'll be trained split because we want to index.
We want the next 15 percent.
So try and split plus zero point five one five the length of our heart disease shuffled data frame.
This is 15 percent of data.
Wonderful.
And so x train y train is going to be the X data up to the train split and the y data up to the train
split.
Wonderful.
And then X valid y valid sorry it's going to be the X data from the train split to the valid split and
the same thing for Y is going to be the y data from the train split to the valid split
and then we're going to have x test is going to be and Y test is going to be X data from the valid split
onwards.
So the rest of the data and then Y from the valid split onwards who that was lost so far let's just
see what's going on right.
Let's.
Let's go.
Len print it out x train and then we want Len X valid and then we want Len x test
so that training center has 70 percent of the data.
Okay.
212 rows.
The validation set has 46 rows.
15 percent of the data and the test set has 46 rows.
So that's 15 percent of the data.
There's different amounts here even though both 15 percent because we've used round we have an odd number
of samples in the heart disease shuffled data set.
So there's gonna be a spillover of one somewhere that is perfectly fine.
Okay.
Now we have our splits.
We can do CSF or instantiate a random forest classifier.
And now because we've passed nothing here this is going to instantiate it with the baseline parameters.
So if we do this CSF don't get Paramus.
That's an attribute that's a function to without passing anything to hear our random forest classifier
and instantiate the random form classifier with the baseline parameters a.k.a. this.
And so that's what we want because we want to make some baseline predictions.
So we go see a left up fit we're going fit it on the training data as usual and then we're going to
make predictions.
So why spreads Eagles see a left up predict.
No we're gonna predict on the validation data because we come back here we want to tune our model on
the validation split.
So that's what we're basing our metric on is will first create a baseline metric which is by running
our evaluation function on the validation split.
Then we'll adjust the hyper parameters and try our model again on the validation split and see how they
compare so let's go here and make predictions.
We'll call this baseline predictions and then we'll go evaluate the classifier on validation set.
So we go baseline metrics and now this is where our evaluation function will come in handy metrics Eagles
evaluate grades why valid so see how we pass it the why the validation set and why grades and then we
want to go baseline metrics Well what have we got local variable accuracy reference before this is what
we've missed out on this needs to be accuracy score try this again it's done terrible now what is going
on here.
Accuracy in point to 2.
Have we done this correctly.
Oh there we go that's why we've used our original C that was like that we need to use heart disease
shuffle because otherwise our model was predicting the original labels rather than now shuffled labels
so that's that's what we missed out on the there we go.
So we're getting this warning here that the value of an estimate is will change from 10 inversion zero
point two to one hundred and point to two.
So this is our baseline metrics now what we've done is we've created a train validation and test split
we've we've instantiate and model just if we had done before we fit it on the training data here and
then we've evaluated the baseline parameters baseline hyper parameters on the validation set and got
these metrics.
So if we were to try and improve our results if we were to try and adjust our models hyper parameters
these ones here on our random forest by hand because that's what this section is churning hard parameters
by hand how would we do so.
Let's take this warning as an example of changing our hyper parameter let's change and again if you're
using psychic point to two or above you won't get this warning but since we are we're getting this warning
we're going to try and adjust our hybrid parameters will change and estimate is which is this one here
we'll change it from the baseline of ten to one hundred and see if we get a different score on the validation
set greater random seed random seed forty to what we're going to do is create a second classifier with
different type of parameters because remember instantiating our random forest classifier up here passing
it nothing instantiate the classifier with the baseline hyper parameters it comes with right out of
the box so like using your oven's predefined settings when it comes out of the box to cook your favorite
dish these scores aren't really helping us out here we want a better model we'll create CnF too and
we're going to adjust the hyper parameters like you would adjust your oven to try and improve that favorite
delicious roast chicken dish that you're making maybe you're preparing for a big function that your
or your friends are coming over and you've promised them a great dish.
So you're trying to perfected.
That's what we're doing with our machine learning model so again same thing same data fitted on the
training data but this time we've got CnF too which is using an estimate is as 100 rather than 10 and
now again we could try different settings for all of these features here the ones we're going to try
and adjust but we might start with just one see an example of it and then we'll go and make predictions
so why parades too.
Because we're using CnF to CnF to help predict X valid.
We're doing the same thing is up here.
The baseline predictions are on the validation set and now make predictions with different type of parameters
on the same data.
So different models same data.
Now we're going to evaluate the second classifier CSF two metrics equals we're using our evaluation
function again already.
The one we created before and we're saving a fair few lines of code by just calling it like that.
All right.
Now let's check it out.
OK so now if we compare using the baseline hyper parameters that our model came with.
What can we say that's different here or we've changed is an estimate as equals 100.
So we've adjusted one dial on our model.
So like adjusting one dial on your oven we can see a slight boost in accuracy on the same data.
So CnF F2 has a slightly higher accuracy it's got a higher precision but a lower recall.
And at the same EF 1 score.
Mm hmm.
Okay.
That's giving us a little inkling that maybe if we kept going with different type of parameters we kept
trying to adjust them by hand we would eventually just hopefully keep improving these metrics.
So what would you think we would do next if we got here.
Maybe we tried to change the max depth.
So what's the default max depth.
We find it in here max depth is none.
So maybe we'd look at the documentation and go to the different variables that max length can take.
So it can take integers or none.
Default is none the maximum depth of the tree and doing our research reading documentation we see some
different values for max depth.
So then we'd go back to our model here and maybe we create CSF three Eagles random forest classifier
this time we'll keep an estimate as is the same equals 100 and then we'll change max depth from none
to equal 10 now.
I haven't made this number up.
I've done some research read the documentation and figured out different ideas for max depth and then
we might do the same thing as what we've done here evaluate our third classifier as you might have guessed
here as you might have thought if we're going through and adjusting all of these by hand adjusting all
the numbers by hand.
That's going to take a fair bit of work.
Right.
So just like perfecting your dish in real life like if you're making favorites you can dish everyone's
coming over for dinner you're trying to adjust the settings on your oven.
You may have to go through a bit of trial and error right.
You might cook it once and then it's not that good and then by the tenth time you starting get really
good but that could take a lot longer.
With what we're trying to do here writing code if we have all these different settings up here trying
to change the trying to find the best settings could take far longer than what we have and the rule
is in code don't repeat yourself.
So you might might have guessed.
So I can't learn has a way a method inbuilt that can do this for us.
Try a bunch of different settings for us.
And that's randomized search CV where we're going to have a look at that in the next video.
This one's already getting far too long.
But the main takeaway here is that when we're choosing a model's hyper parameters you can think of the
training set as like the course materials where your you're learning the Foundation data.
That's the training set and then on the practice exam you're refining what you know.
So you've already learned all the baseline patterns like our model.
And then with the validation set we're adjusting the settings so we're refining what the model knows
how it learns different things before we test ourselves on the final exam.
So in a.k.a. how before we evaluate our model on the test set.
Now don't worry if you're feeling a little bit overwhelmed go back through the code that we've written
check it out.
You could try out adjusting a few different high parameters yourself but if not not to worry we're going
to see in the next video how we can use a module in socket learn to try different hyper parameters for
us.