Alrighty let's finish up this section on evaluating machine learning models so we can do so by tackling
the third and final way to evaluate a machine learning more metric functions.
So in essence we've kind of already covered this because all the metrics we've previously seen have
their own function in psychic loan.
Let's see what I mean by this.
So using different valuation metrics as psychic loan functions four point three.
Wonderful.
So to do so for classification we had accuracy we had precision we had recall we had EF 1 and then for
regression we had r squared mean absolute era and means squared error.
All right.
So let's do what we always do and say the code first and then we'll talk.
Right.
So from SBA loan import metrics I'm gonna do a full example as we always do right.
Because that's what we like to do.
We like to be complete with what we're working on precision score recall score you might be out to figure
out what the one is for F1 score or typed in a bit too quick.
So from S.K. loan we'll also import our model because what we might do is create a section here like
classification evaluation functions.
Now again this section here is just another way to do what we've done before.
So ensemble but it's good to practice.
It's always good to practice random forest classifier and then we want S.K. learned of model selection
import train test split MP dot random seed 42 lovely X equals heart disease dot drop we can almost write
this in our sleep.
Now y equals heart disease target and if you can't that is more than okay write it then the reason I
can write this sort of out is because I've had a fair bit of practice with it and you'll be the same
too.
If you're starting out now you might be looking at all these functions or this code and going whole
league goodness there is so much to remember but the beautiful thing is is that it's here it's available
for you you can run it in Jupiter a notebook and you can practice as much as you like.
So really your only roadblock is just put in the work and practicing learning and the Don't forget learning
something new especially machine learning takes time and it's not going away.
So you've got plenty of time.
So what we're doing here we've seen this code before importing some metrics specifically accuracy score
precision score recall score f1 score and we're importing a model and we're importing trying to split
we're splitting our data into x and y we're splitting it into train and test sets where instantiating
a random forest classifier and fitting it to the training data beautiful running machine learning code
and we'll make some predictions
and then we'll go y spreads because remember what is an evaluation metric doing.
If you said comparing our model's predictions to the truth labels to the actual labels you would be
correct evaluate the classifier so now what we're going to do is going to take advantage of these inbuilt
functions here.
Right.
We could use Skoal we could use this going parameter but we've already covered those.
This is using psychic loan functions.
So evaluate the classifier what we'll do is we'll print out something nice maybe classifier metrics
on the test set wonderful and then we're going to print out we'll do a f string.
Accuracy is going to be we use the accuracy score function on y test and Y parades.
Wonderful.
Then we're gonna times out by 100.
So it comes out in a nice neat percentage because I prefer that or we'll prefer that than the decimals.
Maybe you don't know we got here.
We're getting a precision now.
Again we could function something like this and we probably will in a future video but just for examples
sake we'll type it out we'll practice typing it out back and say it like that well it needs the end
of a string.
There we go.
Now we're gonna do a recall.
How would you do this one if I start you off with the F string that's right.
We'll keep going.
So recall score why test.
Remember just comparing our predictions to the test labels to the truth labels.
And then finally we're going to go F one.
So this is something you might do like if you're reporting to your colleague or to your boss or to your
manager or something like that or to the greater public like how your model is doing you might give
them all these different evaluation metrics so they can start to understand.
Okay.
The accuracy is a certain thing but the precision is there so they have an idea of how many false positives
there are and the recall is there so they have an idea of how many false negatives there are.
And the F one is kind of a combination between the precision and recall.
So we'll hit shift and enter walla.
Now we've taken advantage of the third method of evaluating models and that's by directly using functions
such as accuracy score precision score recall score an F 1 score we're to appear in the documentation.
This is a metric function right classification metrics.
Here we go as a whole bunch more there if you want to check them out.
But these are some of the main ones that we've covered and the principle is still the exact same for
the rest of them.
And so if you come in here we're going to do regression evaluation functions turn that into markdown.
So same thing again you could almost do this yourself I reckon and if not don't why we're about to type
it out but how would you go about it.
If we look at our classification evaluation functions what you might do is from S K low end up metrics
import some regression functions then import the regression model then import this trying to split create
the data split it into training and test instantiate your regression model make some predictions and
then evaluate them but this time instead of valuing classifier you're evaluating the regression model
using regression metrics but just for completeness Let's type it out again metrics import to score wonderful
mean absolutely around and we've seen these before mean absolute error mean squared error.
BAIER You travel S.K. learn dot ensemble.
Import random forest regress I so this is a kind of workflow you might do for your own problems right.
If you're working on a regression problem you might have some sort of import statement at the top.
Your notebook like this import train test split.
In our case we've already got our data in a data frame.
You may have some more lines of code getting your data to a proper data frame.
Oh we almost forgot.
MP random seed so that out you actually don't need a random seed.
I'd just like to have one and you'll see them all over the place just so if you run the results the
same as what someone else was getting target access equals one line equals Boston DLF target.
Beautiful.
So and another reason the random seed right.
So if I ran these cells and then you took this notebook as a resource for the course and then you wanted
to compare your results to mine without the random seed they'd probably be different because all of
the randomness in this notebook such as train test split randomly splitting our data into training and
test sets would use different samples for each and so we'd get different numbers and that would cause
confusion which is not what we're about writing for all about communicating what we're finding.
So we're instantiating a model here.
So random forest Progresso.
Wonderful.
When we go model dot fit a train line train beautiful make predictions using our regression model model
dot predict x test.
Yes yes yes.
And now evaluate the regression model.
So I'm going to go here print regression model metrics on the test set again you could function eyes
this to pass your regression model as well as metrics but we're just gonna ride it out here just for
just for good practice to school.
Why test y parades.
Wonderful.
Now we just need to end the string.
We can do the same for main absolute error so m80 equals mean tab complete that one.
Of course we will.
Why parades
wonderful guy.
Print f MSE.
Main squared error comparing it predictions to the actual labels finish and off with the string and
boom.
Oh we're going there of course we did.
And this is gonna give us a warning because out an estimate is not equal to 100.
And what is our other era found input variables with inconsistent number of samples 102.
What has happened here.
One hundred and two boy test y prints vs. sixty one.
You know what it was happening.
Sixty one is the number of ah here we go.
There we go.
You know how I knew that is because if we go up here and if our classification problem if we go Len
why spreads before we instantiate our regression problem it's sixty one.
So because we didn't set y spreads here.
Right.
Previously this was just this.
It was using Y spreads from above.
That's where I got called out from using the same variable names throughout the notebook right.
Ideally we'd have different variable names for our classification and regression problems but just to
illustrate purposes this is usually called Y produce something of the like.
And there we go.
Regression Model metrics on the test set.
Now we've seen similar metrics before.
What can we do now.
Well we've covered a whole bunch right.
And the reason being is because evaluating a machine learning model is paramount.
It's one thing to train one but then again there's nothing worse than training a machine learning model
and optimizing it for the wrong evaluation metric.
So keep the metrics and evaluation methods we've gone through when training your future models.
Make sure you keep them in mind go through them and have a little read here.
This is probably the most important section that you read in the entire psychic loan documentation but
after you've done that you'll naturally probably start to ask is that how do we improve these numbers.
How do we make them better.
They're kind of stagnant.
We've been using a random seed and we've been seeing just the same numbers for accuracy precision recall
in F1 over and over and saying with r squared MBA and MSE So the next section that's what we're going
to cover.
So we look back our list what we're covering.
Number five we're up to improving a model.
All right.
So take a little break.
Have a look at the psychic loan documentation for metrics and scoring quantifying the quality of predictions.
You can find it by just going to this you are all here or searching SBA loan evaluator model but otherwise
get ready for the next section.
We're gonna see how to improve our models.