Now we've got an ideal model that is performing pretty damn well on the validation data set so valid
are am SLA is the metric we've been paying attention to because that is the evaluation metric for this
particular cargo competition.
Of course if you're working on a different problem or a different set of data you may have a different
evaluation metric that you're trying to improve.
But now we're doing pretty well.
So let's use our ideal model to make predictions on test data.
There's a little tidbit in this section that's really important to remember and we're going to come
across it in a second.
So first of all we're going to have to import the test data.
Now if we have a look in now data folder to this is just in the Jupiter home because we've created a
project folder for this particular project and we've downloaded our data put it in here.
Inside here there's a data set which is test dot CSP.
So this is what we're going to be using.
And if we come back to Kaggle it tells us tests CSP is the test set which won't be released until the
last week.
Competition Well we've got at the moment since this competition has already passed it contains data
from May 1 2012 to November 2012.
OK.
So it's about the six months or so after our validation set.
Beautiful models performing okay on the validation set.
So what you might see is see how we can make predictions on this future six months of data.
So we'll get a IDF test Eagles PD or read C as V.
Which is gonna do it the exact same we did at the beginning with our train and valid data blue ball
for bulldozers.
Wonderful.
And it's called Test dot c as we will pass in low memory equals false and we're also going to pass dates
because remember we haven't done any manipulation to this data set like we have to our other ones.
We'll see where that comes in in a moment and then we'll check DNA test.
Go ahead huh.
What's happening.
Oh typos beautiful.
So this looks familiar.
It's something that we've seen before.
Except now this time the test data set is missing the sale price column.
And the reason being is because that's the column we're trying to predict.
So why don't we see if we can do that.
So we go test pred equals ideal model we'll use our ideal model this time model not predict an ideal
model has been performing the best and we should be able to just go pass it the whole data frame.
Right.
Because this will be our x values and we don't need to drop any columns because it doesn't have a sale
price column.
So let's see what happens.
Actually put a little comment here and make predictions on the test dataset and I just sort of a little
challenge before we run this cell.
I want you to kind of think about what might happen.
So our machine learning model it's been trained on a particular dataset and now we've imported the test
data set.
What's gonna happen if we try to make predictions on this dataset based on I'll give you a hint based
on the data that our model has been trained on
and if you're not sure that's fine we're going to see it in a sec.
We should see it right now.
Oh there we go.
Value era.
Okay.
What's happening here.
Could not convert string to float.
Okay so if you remember right back up the top let's scroll back up through our beast of a notebook that
we've been working through.
What did we do before we could train a machine learning model we saw that a similar error when we first
started right up here when we started modeling would be great if I could find it.
Our model didn't work.
Here we go.
Okay.
Build a models IDF temped or drop could not convert string to flight so the same error messages what's
below.
Now the reason being this has happened is why do you think
well it's because when we imported our original data set we did some manipulations toward what did we
do we made sure our data was numeric and we made sure there were no missing values.
So what we've done now is we've imported DNA test
may have some missing values which it does and it might not be all numeric
and far out we've done a fair bit of data processing here.
And if we check the columns so these are all the features.
But if we check out x train columns that need brackets a we notice that the column lengths are different.
So there's 102 columns in now x train whereas in DFT test what do we got here.
We saw it up here before when we did.
Dot had 52 columns.
So the reason why our machine learning model can't predict on the test data set at the moment is because
it's not in the same format as the model was trained on whereas the data set the model was trained on.
So that's what we're going to have to do.
All right let's get rid of these here and so what.
How could we do this.
So we might go pre processing the data.
In other words getting the test data set in the same format as our training dataset.
Wonderful.
So how might we do this.
So what we might do I think is create a function for doing so.
So if we were to import a new dataset say a test data set we actually did some manipulations to our
training data said we could take those functions that we did on the training data set and just do the
exact same thing on the test data set.
So that way they're both the same.
That makes sense.
Let's see it first let's see if we can do this.
So pretty process data will pass at a data frame.
We'll leave a little note here so performs transformations on the F and returns transformed the F simple
right.
We want to create a function which takes a data frame of some sort.
It does a bunch of manipulations on that data frame and then eventually it's going to return that data
frame.
So now we're going to have to fill in all the transformations that we did originally to DFT temp in
here and so we can do that by going up we might have to do a few copies and paste here.
So remember right back up the top.
So this is a little rule a little tidbit.
What you do to the training data set you're going to have to do to the test data set.
That's a that's a machine learning tidbit there.
Okay.
So here's the first big set of manipulations that we did that might copy that wonderful and we can just
remember this line.
So we've we've added some information here based on the date time column sale date and then we removed
it.
So the first two steps that we have to do like will scroll right down to the end of our notebook.
It's gonna be a lot of scrolling in this video because we did three major things right.
We added some information here to the date time or we extracted.
That's probably a better idea.
We extracted information from our Date Time column and added it to the data frame.
So Mike IDF does drop and after we've done this we'll drop the sale date column and we'll go access
equals 1.
And in place it goes through.
Wonderful.
And we have to remove dot temp from all of these because we're only using DNF.
So a little trick here is by moving your cursor and holding down command.
If you're on a Mac or maybe it's control if you're on Windows and we can just go backspace here.
Wonderful.
Right.
Because we're not working with Def temp.
We're working with DEF.
So any arbitrarily named data frame will work with our function.
So the next thing we did was we need to fill the numeric rows with median and we also field categorical
missing data and turned into numbers in turn categories into numbers.
So that's what we have to scroll back up and find.
Okay let's let's do that.
First we'll get the missing missing new miracles.
Luckily we've laid our notebooks.
We've added these headings here so that we can see what we were doing.
This is all part of helping yourself out right.
So fill numeric rows at the median.
We actually need all of this code here.
We'll copy that.
Beautiful.
So this is where comments and different headings in your notebook can really help your future self out
is when you want to function ize things and clean things up.
So there we go again we have to remove DFA tent.
So a good deal of temp.
We don't want that day of temp.
We only just want def okay beautiful and we still need to fill categorical missing data and turn categories
into numbers.
Now let's just see if we can remember how to do this rather than having to scroll back up because we've
done enough scrolling to be honest so we can do this.
If not for you remember if it's not a numeric type that's how he did it.
Time is numeric PD type content because we're looping we're in the same loop here.
Content we want to go the F label plus the score is missing.
Yes equals so it's gonna check if it's now.
If it's now we want to add a label that a label is missing that particular data point is missing and
drove how we add plus 1 to the category code because pandas and codes the missing categories as negative
1.
So this is where we can go IDF label Eagles PDA categorical so turn it into a categorical so turn the
content of a particular column into a category and then access its code.
Okay turning it into a number and then plus 1.
All right.
Not a bad function.
So this what kind of happens in a notebook workflow right is that up here.
You could probably classify most this is pretty messy we've helped ourselves out with different headings
and whatnot in comments.
However as you sort of get further into the process you'll start to build these helper functions so
you're collating a whole bunch of different steps into a function so that it really syncs things in
the notebook and you can just call a function like this.
Much like our show scores function and it does a process that's going to be the same every time rather
than having to scroll up through the notebook and find something in a different cell.
So now we can hit shift enter on that or what's happened expected not missing.
BLOCK I didn't think we needed 1 0 here.
That's where we need it.
We needed indents here.
There we go.
Classic.
This is what happens when we copy and paste code right.
The formatting isn't always perfect beautiful so we have a function here that's going to pre process
some sort of data frame.
It's going to add some information based on the sale date column.
It's going to fill numeric rows with the median.
It's gonna turn the missing categorical data into a zero value and the rest of the data into then category.
Code number.
Beautiful.
So it's going through a few steps but that's the right way function eyes it's it's all gonna happen
together.
And now what might we do here.
Actually a good question is where might this function break.
I might leave you on that for this video to save it from getting too long.
I want you to have a look at this big function that we've created and think about if we pass it out
training data frame.
How might it work.
And then if we passed it our test data frame right DFT test which is up here we passed at this data
frame.
How might it break we'll answer that next video.