All righty.
So we found a logistic regression model that performs pretty well but we went and showed that to our
boss and then she said we had to find a whole bunch of things like half revenue turning feature importance
confusion matrix cross validation and these things could sound made up to someone who is not a budding
data scientist or machine learning engineer like yourself but being the bonding machine learning engineer
and data scientists that you are you know that data scientists make up words all the time.
And the good thing is that these aren't made up.
These are things that we can actually find.
So now we've got a baseline model and we know a models first predictions aren't always what we should
base our next steps off.
What should we do
if we remind ourselves of the classification metrics and regression metrics.
We're not worried about regression right because we're working on a classification problem.
We've used the default accuracy.
Then there's precision there's recall there's F1 which are all part of the things that the boss asked
us and we got a confusion matrix.
Okay we should probably make one of them since we're working on a classification problem.
Then we've got a classification report which has a bunch of information here like precision recall f1
score support accuracy macro average far out.
And then we've got hyper parameter tuning we've covered before but we've got a lot of steps.
So first we should put down what we're going to tackle let's look at the following because we're a part
of this experimental phase.
That's what we're going to be working towards.
We're going to be taking our models that we've got.
These are our baseline models here and we're going to be experimenting with them further seeing if we
can improve them and seeing does logistic regression perform best on accuracy or could attuned or hyper
parameter tuned version of a random forest beat it out or maybe K and M could improve.
Who knows.
Let's write to analysts.
Let's look at the following hyper parameter tuning
feature importance confusion matrix
cross validation
precision
recall if one school classification report ROIC curve area under the curve.
So as we saw on the psychic Loan Section These are some of the things that you should pay attention
to when youre working on a classification model.
These two here are actually part of almost any machine learning model that you'll be working on.
But these are specific to classification.
We've seen the difference between regression and classification metrics but since we're focusing on
the classification problem whether or not someone has heart disease based on their health parameters
we're gonna be focused on looking at these.
So let's jump into it.
So first things first.
Why don't we try hyper parameter tuning and cross validation and remember what's hyper parameter churning.
Well to cook your favorite dish you know that to set the oven to 180 degrees and turn that grill on.
But when your roommate cooks their favorite dish they set the oven to 200 degrees and use the fan force
mode same oven different settings different outcomes.
This is the same for machine learning algorithms you can use the same algorithms but change the settings
a.k.a. hyper barometers and get different results.
But just like turning up the oven too high if you do that for machine learning algorithms you tweak
the settings too much you can change the settings and it works well or so well that it over fits.
So we're really here what we're doing to do is look for the Goldilocks model one which does well in
our dataset but also performs well on unseen examples should we turn AK neighbors classifier.
Mm hmm.
Yeah I might do that even though we've said goodbye to the cayenne and classify we might just see how
would you tune it.
And if you're wondering where I would start if I was trying to approach this this is what I'd start
with how to tune a K neighbors classifier model we go here in depth parameter tuning for cayenne in
model selection tuning and evaluation and came nearest neighbors.
So if we were to read through these we would probably find some information that we're going to cover
here so let's start it out.
Let's make a little heading here we'll go here.
Hyper parameter tuning let's change.
Kanan
let's go a list of training schools an empty list so try and scores and then a list of test scores.
Empty list because what we want to do is compare different versions of the same model.
So model with different settings and compare their scores and the two different datasets.
Looking up what we looked at before we would to read into these again you can do this with almost any
machine learning model or basically any machine learning model.
This is a part of the research.
This is a part of the experimentation step of many machine learning problems is is searching up things
like this then reading what you can find and bringing it back to a scenario like what we're doing here.
But let's just pretend we've gone through that.
So we're going to create a list of different values
for and neighbors.
Now why and neighbors.
Because as I said before when we did our reading we found that if we go S.K. loan K and in K neighbors
classifier we see that n neighbors is one of the parameters and we could see this by reading through
the documentation of psychic loan as well.
So the number of neighbors used by default for K neighbors queries in our research on how to tune a
and and model.
We found that we can adjust the number of neighbors and if you want to find out more about this you
can check out the documentation.
You can read it up here.
So that's what we're going to do.
We're going to create a list of different values.
So the default here is five.
So maybe what we might do is try different values from one to 20 or something like that.
So let's see here neighbors equals range 1 to 21.
Wonderful.
And then we're going to set up K an instance so go Mary Kay and N equals what is it.
You know this kind enables classifier.
Beautiful.
And then we might go loop through.
We need to loop through different and neighbors
which is our ace that we made here for I in neighbors thinking out loud here.
We'll go Cannon Dot.
Set parameters.
Remember you can adjust the parameters of a machine learning model and socket line using dot set parameters
equals I because I'm going to loop through this range here.
Wonderful.
And then we're going to fit the algorithm what we're trying to do here is improve on this baseline score.
So this is with the default of 5.
And so we're going to try 20 different versions 1 to 20 to see if it's any better I do anything I accidentally
hit shift into trigger happy you know my fingers are just they just default to shift in into whenever
they want.
Now we want to update training scores list so we're going to go training.
We want train scores up here train scores dot append.
Now we want our canine Ensign stopped score on X train y train.
Wonderful.
And then we want to update the test scores list and we'll do that test scores dot append K and N dot
score and you can probably guess this is going to be on the test set.
Beautiful bonus points if he did.
Now what this is going to do is just gonna loop through this range of 1 to 20 and then it's gonna create
20 different canine models and append their scores to lists.
So let's check out those lists train scores shift into wonderful and then go test scores.
Typo.
Now we've got 20 different schools or so.
But these are probably best visualized.
Let's see how we would do that.
Let's go penalty.
No lot neighbors train schools and then we'll get a label equals train score.
Of course.
Yes.
BLT will add another plot neighbors test scores label equals test score.
Wonderful.
And we want to go peyote actually.
Let's just see what this looks like ex label will go a number of neighbors because that's what's on
the X spelling neighbors.
The Australian way here with you there.
Well maybe it's not Australian.
Maybe it's just a different version but neighbours in a socket line is spelled with no u.
So this is the x and then the dot y label is going to be the scores so this is X Y so this is going
to be model score and then we want P LTE legend and we'll see what comes up.
Actually we want little tidbit here Max F so maximum Kane and score on the test data.
We're going to go Max test scores.
It's gonna be accuracy so we want times one hundred point to f close parentheses and then close string.
That should work wonderful OK.
So we can see here that we're really paying attention to this test score here.
So we get the highest value here is around about 11 for K nearest neighbors.
So if we look at the graph actually that's let's adjust these x2 so we can see where it actually is.
X takes dot NDP range 1 to twenty one space of 1.
So what this is gonna do is just bring out a list of range of 1 to twenty one of space 1 which is the
exact same as what we're using here.
So neighbors equals range 1 to twenty.
So let's do that and yes our assumption before before we adjusted the X ticks which is these just little
labels down here.
Was that okay.
And then or n neighbors value of eleven yields the best score on our test data set.
So if we come up here the default we remember is five.
So we've just done a little bit of hyper parameter tuning and we were able to improve our cayenne and
classifiers results on the test data set by changing it from the default 5 and and neighbors parameter.
So we're doing hyper parameter tuning here change that to 11 and we get seventy five point four one
versus our initial result which is back up here of sixty eight per cent Mm hmm.
Well even then even after improving it to 75 or whatever it was it's still far below Alo drastic regression
and random forest.
So I think this has put the nail in the coffin for our K N N model is that because even with the type
of parameter tuning it still hasn't reached the scores that we got with logistic regression or random
forest.
And so because of this we're going to discard K and end for now.
This is part of our experiment.
A big part of experimenting a machine learning step six is going through different machine learning
models and figuring out which work and which don't.
So what we're trying to do here is as quickly as possible work through different little experiments
like what we've just done tuning K and N and seeing which model performs best on our data.
So what we might try and do is we've churned Kanan by hand like we wrote out this little for loop here.
That was a bit tedious right.
So you had to do that for every single machine learning model you might run into some problems.
It was okay here because we had one parameter tuned but if you had more parameters writing these for
loops is going to be very inefficient.
So what we might do next see how we can tune logistic regression and random forest classifier using
randomized search CV which stands for cross validation.
So instead of us having to manually try choosing different hybrid parameters by hand randomize search
CV which we've seen in the psychic loan section it's gonna try a number of different combinations of
hyper parameters for us and evaluate which ones are the best and then save them for us.
So that's what we're going to have a look at that next video.