Now we've gone through Step 1 into problem definition and data.
It's time for step 3 evaluation every machine learning problem you come across.
We'll have the similar goal of finding insights in data to predict the future in some way an evaluation
metric is a measure of how well a machine learning algorithm predicts the future.
And in this step the question you'll want to answer is what defines success for us.
For example if your problem is to use patient medical records to classify whether someone has heart
disease or not you might start by saying for this project to be valuable we need a machine learning
model with over ninety nine percent accuracy because predicting whether or not a patient has heart disease
is an important task.
So you'll want a highly accurate model and as you could imagine there are different evaluation metrics
for different problems for classification or predicting whether something is one thing or another.
Accuracy precision and recall a common for regression or predicting a number such as how much a car
will sell for.
You'll probably want to minimize how different the number your model predicts to the actual sale price
for this mean absolute error and main square area are common options or for recommendation problems.
You may have thousands of different products to recommend to someone but in reality you only care about
the top 10 recommendations and how well they align to a customer's potential interest.
To measure this you could use precision at K where in our case k is 10 sitting down and thinking about
an evaluation metric.
At the start of a project ensures everyone on it has a similar goal to work towards but it's important
to remember these don't have to be exact either as you find out about more about the data you might
find the evaluation metric changes as the project goes on for now don't worry if you're not too sure
about each of these we'll see different examples of these.
As we build projects as an example of an evaluation metric being used in practice we had a project where
we wanted to use the text from car insurance claims to predict who caused the accident the person submitting
the claim or the other person involved the car insurance company we partnered with wanted at least a
95 percent accurate model to consider the project worth continuing.
This meant the model I was building had to be able to read a car insurance claim and predict with 95
percent accuracy who caused the accident.
This meant it was only allowed to get it wrong 1 out of 20 claims.
Again we'll learn more about evaluation metrics when we get hands on for different projects.
But the thing to remember is as you go you'll start to define a problem like in Step 1 then you'll start
to remember all this is a classification problem I should use accuracy as my evaluation metric to get
an idea of how my model is doing before we go on to Step Four have a think about different things you
measure everyday.
How do you measure them.
Are there different kinds of measurements for different tasks.