Modelling part for is the final part of modelling and its comparison wanting to know here's how will
our model form in the real world.
So after you've tuned and proved your models performance through hyper parameter tuning it's time to
see how it performs on the tests in the tests and is like the final exam for machine learning models.
If you've created your data splits correctly it should give you an indication on how your model will
perform once deployed in production means customer facing rather than just being on your local computer.
Since your model has never seen data in the test set evaluating your model on it is a good way to see
how it generalizes and remember why generalizing I mean adapts to data it hasn't seen before such as
how heart disease prediction machine learning model would perform at classifying whether a patient has
heart disease or not.
On a patient who wasn't in our original data set a good model will yield similar results on the training
validation and test sets.
And it's not uncommon to see a slight decline in performance from the model on the training and validation
set to the test center.
For example your model might achieve 98 percent accuracy on the training dataset and 96 percent accuracy
on the test set what you should be worried about is if the training set performance is dramatically
higher than the test set also known as under fitting.
And if the test set performance is higher than the training set performance also known as overfishing
overfishing and underfunding are both examples of a model not being able to generalize well which is
what we don't want the ideal model shows up in the goldilocks zone it fits just right not too well but
not too poorly.
You see here this this machine learning model if this was your data these green data points.
This line here it kind of fits the shape but it's it's.
This would be classified as as under fitting.
This is not what we want our model today.
And this one over here.
Well it's doing a good job of fitting all the data points but it's going far too close it's almost too
perfect model it's just snaking between them so this example would mean that the model has learned the
patents too well in this dataset it would be like seeing the final exam before actually taking the final
exam.
This one has the goldilocks zone right.
This is an iterative process where exactly this Goldilocks Zone is like a balanced model will really
depend on your data and the problem you're trying to solve.
That's why again it's a it's an iterative process finding this this balanced zone and now after some
experience and practice working on different machine learning problems you'll be out to start to tell
whether your model is overheating or on defeating now.
There are several reasons why on defeating an opening can happen but the main ones are data leakage
and data mismatch.
Data leakage happens when some of your test data leaks into your training data.
This often results in overfishing or a model doing better on the test set then on the training dataset
it's like.
Remember it was like if you were to have a look at the final exam or everyone had to look at the final
exam as the practice exam you're machine learning model has just learned what it's about to be test
on.
So when it comes time to modelling it just learned it way too well and it starts to fit the data like
this.
And this is why it's important to do your splits correctly and ensure that machine learning model training
happens only on the training data set.
Validation and model tuning happens only on the validation or training data set and that testing and
model comparison happens on the test data set.
And I've remember some different approaches use only a training and testing set and do model tuning
on the training set but testing always stays the same.
It's like when you when you go to university you're doing a course you want to make sure that the final
exam is kind of an indication of how well you understand things.
Same with the test data set for machine learning the test dataset is used as an indication of how well
your model will generalize in the real world.
So you want to avoid data leakage data mismatch happens when the data you're testing on is different
to the data you're training on such as having different features in the training data to the test data.
Having this kind of mismatch can lead to models performing poorly on test data compared to training
data.
This is why it's important to ensure that training is done on the same kind of data as you'll be testing
on and as close as possible to what you'll be using in your future applications other ways to combat
under fitting include using a more advanced model.
This could mean a totally different model or increasing the number of hyper parameters on your current
model.
Remember when we were cooking our chicken dish we might alter one of the hyper parameters of our oven
by turning it up.
That might be something that you might do on a machine learning model instead of only using two layers
in a known network.
You might use for.
We'll see more of this in the future project you could also reduce the number of features you're trying
to model.
Maybe your data has too many features and the model you're using is struggling to find patterns in them.
Finally you could train your model for longer.
Sometimes models take longer to train or longer to learn than you'd expect.
One of your experiments may involve a longer training phase to reduce overfishing.
Useful solutions are to collect more data.
More data will provide more potential patterns for a model to find and thus lower the potential for
it to find them all.
Or you could try use a less advanced model.
This is uncommon but it's a possibility.
The model you're using is too good at learning and it models your data too well be cautious of models
performing too well as they might lead to incorrect predictions.
Remember no model is perfect so be sure to check your good results as much as you check your poor results
finally when comparing two different models to each other.
It's important to ensure you're comparing apples with apples and oranges with oranges.
For example Model 2 trained on dataset 1 vs. Model 3 trained on data set.
1 During comparison you'll want to make sure you take into account not only the final result but what
it took to get there.
If Model 2 takes one second to make a prediction at ninety three point one percent accuracy Model 3
takes 4 seconds to make a prediction at ninety four point seven accuracy is that extra 3 percent accuracy
worth that extra 3 seconds of prediction time.
Now this will depend on what the goal is but if you are optimizing for prediction time you want to make
predictions as fast as possible.
You might choose Model 2 because it makes predictions 4 times faster than model train but at a higher
accuracy level.
Again this will be different depending on what kind of application or production you use case you want
to use but just some things to keep in mind is more than just how a model performs that goes into choosing
which one you should use.
A couple of things you want to remember from this lesson.
Avoid overfishing and under feeding you want a model that heads towards generality.
It's like when you do your practice exam.
If you saw the final exam you might just become an expert memorization machine rather than someone who
could use that knowledge in the real world.
Keep the test set separate at all costs.
When you split your data you want to have a training set and then throw away the test data set and lock
it up.
And once your model has been trained then you can open up the test data set you can unlock it take it
out of the safe and see how your model performs when comparing models compare apples to apples have
Model 1 and Model 2 on data set 1 and dataset 1.
You want to make sure the two models you're comparing have been created in the same sort of environment
so that you can ensure that what you're comparing is is legitimate.
Comparisons finally won Best Performance Metric does not equal the best model.
Remember now example you might be optimizing for prediction time.
So although a model that makes a faster prediction doesn't get as high accuracy is as another model
that takes a little bit longer.
It might not matter to you because you need something that can predict as fast as possible.
That was a lot but we'll see plenty more of this in action throughout the course.
You'll also be using it throughout your entire machine learning Korean so it's important to remember
these concepts.
Let's push on to the next step and seeing how we can put all of this previous steps together in Step
6.
Experimentation.