In the last video we left off scoring our machine learning model that we fitted to the data.
After we'd just filled all the numerical values filled all the categorical values and turned all our
data into numbers model worked our data had no missing values and it was all numeric but we left off
with the question why is this metric reliable.
Let's put that here because that's worth highlighting because remember always remember feeding a machine
learning model or evaluating a machine learning model is just as important as fitting one.
And that's where we're up to now because we've just fit a machine learning model we have to make sure
that our results hold water right because we don't want to be promising things that our model can't
do.
So question why doesn't the above metric hold water
hold water is an expression for why isn't the metric reliable.
That's just a way of speaking.
So why doesn't it.
Oh water why isn't it reliable.
Have you had to think about it.
I'll show you a diagram that may refresh your memory if we come back to our keynote.
Remember the from right back to the beginning.
The most important concept in machine learning a.k.a. the three sets.
So what we've done is we've trained a machine learning model on one dataset.
But then what have we done.
We come back to the code.
We fit the model on the data set on some data but then we've had value why did it on the exact same
data.
Okay so if we come back here essentially what we've done is if we were thinking about this has been
a university course we've learned the course materials.
But what we've done instead of testing our model on the equivalent of like a final exam a.k.a. a test
dataset we've just evaluated our machine learning model on the exact same materials that it's learned
on you are in one class and you got given a book to read and then at the exact same time and just like
you're gonna be tested with the questions directly from that book you're reading.
Okay.
But we after our model's ability to generalize and a.k.a. the ability for a machine learning model to
perform well on data it hasn't seen before.
That's what we're after.
If we go back to Kaggle there's a training data set which contains data through the end of 2011.
There's a validation set and there's a test set.
So that's what we're after.
We need to evaluate our model not on the data we've trained it on which is this we need to evaluate
it on the test data but before we do that if you remember right back up the top let's go right back
up the top to we imported our first data frame I said we'd revisit this.
And now the time has come.
So we go here we imported train and validate CSP.
Wonder why did we do that.
While it's all gonna be come clear in this video.
So we train invalid CSB.
You are valid CSB and we've got train CSB why don't we just import them separately.
Well I wanted to demonstrate what it's like creating your own validation set rather than someone else
creating it for you with a time series data set which is what we're working on.
So this is a perfect playground for that.
So if we read here.
Train CSB is the training set.
So basically what we've been working on and valid CSP is a validation set.
The key point here is because it's a time series.
The training data set has data up to the end of 2011 whereas the validation set has data between January
1 2012 to April 30 2012.
So what we will do to better evaluate our model we need to split our data up into training and validation
set so let's do that.
Let's go here.
Splitting data into train validation sets because that's what we're after now we've got a model we can
build models now.
But we need to rather than just build better models we need to make sure that what we're evaluating
with makes sense.
So let's check our data again as we always do.
So maybe we can use the year.
What was it called sale year Media Temple sale year There we go.
Okay so maybe because our data frame is in order.
Yes this is going to help us when we import loud at a frame we ordered it in the order of the sale dates.
So now reading this to create our own validation set we want to split our data.
So all of the rows up to 2011.
So the s year 2011 that can be in the training set and then all of the rose in 2012 can be the validation
set.
We won't worry about the test set for now because that's in a separate data set.
The reason why we're only working on train and valid CSP is because those we imported the train and
valid CSC at the start of the notebook.
This file here so we have to create our own validation dataset.
So let's do that.
So we've got data templates check the sale year dot value counts.
There we go.
2012 so there's eleven thousand five hundred seventy three samples in 2012.
All right.
So to split our data into training and validation it should be as easy as going.
Okay.
Let's introspect the s year column and every column that's equal to 2012 will be a validation set for
our year and example.
And every row that's not equal of s year.
That's not equal to 2012 will be in the training set.
So let's stop talking about it Daniel.
Let's see the code to split data into training and validation.
This is something that you might have to do with your own time series data right.
Because you're not always going to be given in a casual format right.
When you're working with a client or working on a project you're not always going to automatically have
your data in train valid and test.
These are things you're going to have to create for yourself.
Looking at this sale column or looking at the time column looking at the date column and figuring out
how you can make your own training a validation set.
So that is exactly why we imported them as one set to begin with so we could practice making our own
training and validation sets.
And I feel like I'm saying the words set a lot but that's important because we go back to our keynote.
This is the most important concept in machine learning because whatever we train our model on we want
to make sure we're evaluating it on something else.
So come here to the validation set is every row in DFT temp where the s year column equals 2012.
Yes that's correct.
And the the day of training set is every row in IDF temp where the same year column is not equal to
2012
WONDERFUL.
AND THEN WE MIGHT GO THEN DLF down and land the F train.
So all this is going to tell us is the length of these two data frames that beautiful.
So now we have a validation set which contains eleven thousand five hundred seventy three rows and a
training data set which contains four hundred one thousand one hundred twenty five rows or samples and
they're split on date beautiful.
We're taking boxes here.
We are taking boxes here.
So now what we might do is split data into x and y.
We've seen this before and so that way we have an X training set and a Y training set and a X valid
set and a Y valid set which is our data and labels.
So we'll have X train y train equals D.
Now we're working with the training set here drop we want to drop the sale price column on the first
axis.
Okay I just dropped the column and we're gonna set it to D train y train is going to be equal to sale
price so see how I'm doing it with a comma here I'm being a bit tricky by just doing in one line beautiful
and gone here valid why valid not sure and dear Val looked wrong sale price just the same again but
this time with the validation set def thou Dot this is actually going to be Sal Price There we go and
now we have a little game inspect our data just to make sure we haven't made a little error somewhere
they should all be comparable shapes to each other
why valid dot shape so we're just taking these data sets that we're creating here and we're finding
out the shapes of them beautiful.
So our training set is 400 1000 rows or about that with 102 features 102 columns our y training is 400
1000 rows and then validation is about eleven and a half thousand with the same amount of columns for
X and no columns for Y because Y is just one column let's have a look.
Now look this is what we do make sure all of our data.
Okay so these are all sale price.
Beautiful.
Well okay.
So we've got our data into train and validation sets we've taken care of two of the three most important
things or most important sets we'll have a look at the test set later but this is what we've created
a training set and a validation set it's time to keep building some more models so we might end this
video here and we've got to figure out a way to evaluate how a machine learning model so we'll probably
have a look at that in the next video.