1 00:00:00,610 --> 00:00:04,860 Now we've gone through Step 1 into problem definition and data. 2 00:00:05,020 --> 00:00:11,020 It's time for step 3 evaluation every machine learning problem you come across. 3 00:00:11,020 --> 00:00:17,860 We'll have the similar goal of finding insights in data to predict the future in some way an evaluation 4 00:00:17,860 --> 00:00:23,200 metric is a measure of how well a machine learning algorithm predicts the future. 5 00:00:23,320 --> 00:00:29,260 And in this step the question you'll want to answer is what defines success for us. 6 00:00:30,740 --> 00:00:37,040 For example if your problem is to use patient medical records to classify whether someone has heart 7 00:00:37,040 --> 00:00:42,980 disease or not you might start by saying for this project to be valuable we need a machine learning 8 00:00:42,980 --> 00:00:50,930 model with over ninety nine percent accuracy because predicting whether or not a patient has heart disease 9 00:00:51,020 --> 00:00:52,500 is an important task. 10 00:00:52,640 --> 00:00:59,970 So you'll want a highly accurate model and as you could imagine there are different evaluation metrics 11 00:01:00,030 --> 00:01:05,380 for different problems for classification or predicting whether something is one thing or another. 12 00:01:05,400 --> 00:01:12,180 Accuracy precision and recall a common for regression or predicting a number such as how much a car 13 00:01:12,180 --> 00:01:13,040 will sell for. 14 00:01:13,380 --> 00:01:19,080 You'll probably want to minimize how different the number your model predicts to the actual sale price 15 00:01:19,710 --> 00:01:26,670 for this mean absolute error and main square area are common options or for recommendation problems. 16 00:01:26,670 --> 00:01:32,220 You may have thousands of different products to recommend to someone but in reality you only care about 17 00:01:32,220 --> 00:01:38,380 the top 10 recommendations and how well they align to a customer's potential interest. 18 00:01:38,520 --> 00:01:46,120 To measure this you could use precision at K where in our case k is 10 sitting down and thinking about 19 00:01:46,120 --> 00:01:47,400 an evaluation metric. 20 00:01:47,410 --> 00:01:54,620 At the start of a project ensures everyone on it has a similar goal to work towards but it's important 21 00:01:54,620 --> 00:02:01,520 to remember these don't have to be exact either as you find out about more about the data you might 22 00:02:01,520 --> 00:02:08,900 find the evaluation metric changes as the project goes on for now don't worry if you're not too sure 23 00:02:08,900 --> 00:02:11,700 about each of these we'll see different examples of these. 24 00:02:11,750 --> 00:02:19,270 As we build projects as an example of an evaluation metric being used in practice we had a project where 25 00:02:19,270 --> 00:02:26,920 we wanted to use the text from car insurance claims to predict who caused the accident the person submitting 26 00:02:26,920 --> 00:02:34,890 the claim or the other person involved the car insurance company we partnered with wanted at least a 27 00:02:34,910 --> 00:02:40,320 95 percent accurate model to consider the project worth continuing. 28 00:02:40,320 --> 00:02:48,480 This meant the model I was building had to be able to read a car insurance claim and predict with 95 29 00:02:48,480 --> 00:02:51,980 percent accuracy who caused the accident. 30 00:02:52,020 --> 00:02:57,960 This meant it was only allowed to get it wrong 1 out of 20 claims. 31 00:02:57,960 --> 00:03:03,780 Again we'll learn more about evaluation metrics when we get hands on for different projects. 32 00:03:03,780 --> 00:03:10,350 But the thing to remember is as you go you'll start to define a problem like in Step 1 then you'll start 33 00:03:10,350 --> 00:03:17,790 to remember all this is a classification problem I should use accuracy as my evaluation metric to get 34 00:03:17,790 --> 00:03:25,050 an idea of how my model is doing before we go on to Step Four have a think about different things you 35 00:03:25,050 --> 00:03:26,570 measure everyday. 36 00:03:26,730 --> 00:03:27,990 How do you measure them. 37 00:03:27,990 --> 00:03:30,390 Are there different kinds of measurements for different tasks.