OK so now we've got this behemoth of a function that's going to pre process our data.
Let's try run our DNA test that we just pull up here and tried to make some predictions on.
So we import it here.
Tried to make some predictions on it but it didn't work because there's not the same format as the training
data.
Let's run it through our pre processed data and hopefully if we've done it correctly it's going to make
the same changes on DNA test as we have on the training data.
So let's go here.
DNA test equals are actually we'll put a little common here.
Process test data and we might just re instantiate to make sure that we've hit shift enter here shift
into here just to make sure that these cells have run I've been notorious in writing a function and
then not running the cell so it actually doesn't work.
Pretty process data DNA test and then we should be able to check it out like DNA test or ahead Okay
101 columns all that's looking better.
So we're up we're up a fair few columns from what we started with we go back in and we start with 52
columns.
Now if we go through here guy on the end there's a fair few missing.
All right.
OK.
So it's going to be missing the sales price column.
OK.
That makes sense but still a different shape to X train.
So we've got one hundred and one versus one hundred and two columns.
And now if you remember in the end of last video I asked to have a think about where this function might
break and if you're not sure that's fine and I've stumbled across this many of times right in the process
of trying to make our training data or our test data in the same format as our training data and what
it is is that the test data when we imported it is slightly different to the training data as in Let
me demonstrate we're doing model driven idea.
So we'll try to make a prediction again.
So make predictions on updated test data.
We'll go test spreads equal ideal model don't predict the test and give value error again.
Hold on I just pre processed it.
What's wrong.
Let's check it out number of features of the model must match the import model and features is one or
two.
Okay.
That's what our model has been trained on and input features in one to one.
Well we just saw this when we went X train go ahead.
That's one hundred and two columns so there is a difference in the number of columns that our test data
frame and our training data frame map.
Hence why our machine learning model doesn't work.
So how would we figure out where the difference is.
What we can do that using the columns attribute and turning them into sets.
Let's have a look.
So we can find how the columns differ using a python set.
If you've heard of sets there are basically a list with only unique values in them.
Go Set X train dot columns
minus set D test dot columns and now this is going to tell us which columns are incorrect.
AUCTIONEER idea is missing.
So our RDF test has no column auctioneer I.D. is missing.
So what that means is when we imported our test data frame according to our function it didn't have
any auction I.D. values that are missing.
So what we can do is manually update a column on our DNA test data frame to have all false values for
auction.
I.D. is missing because the reason being there is no column auctioneer idea then this is my having if
you can't tell I'm having trouble saying auctioneer here Heidi.
The reason why RDF test data frame has no column named this right.
This this here one I'm highlighting is because all of the auctioneer I.D. values were filled in the
test.
So what we have to do is manually adjust DNA test to have auctioneer I.D. is missing column and this
is just as simple as adding the F test go or just copy this here.
So we'll create a new column and we're going to set it to just false because it had no missing values
a day after head and now Lopes DNA test go ahead.
Wrong guy different today of tests done ahead.
One hundred two columns.
Beautiful.
So if we come right to the end it should have.
AUCTIONEER Ida is missing.
Wonderful.
So this means finally now our test data frame has the same features as our training data frame.
We can make predictions let's do it.
So make predictions on the test data.
Wonderful.
So we'll go test as equals DNA test on it.
Ideal model.
So our ideal model don't predict DNA test boom that worked.
Now let's have a look.
They should all be like prices an array of prices and it's gonna be.
How big is our test data frame.
So these are all the sale prices that we've predicted.
So twelve thousand four hundred fifty seven samples.
But it's not gonna help us in an array format because Kaggle expects it to be
in a data frame with two columns sales I.D. sales price to what we can do we can turn it into that.
So format predictions into the same format Kaggle is after
we've made some predictions but they're not yet in the same format Kaggle is asking for.
Now we'll just link that here evaluation so we know we could go there.
Okay.
So how would we do this.
Well we need the sales I'd call column and we need the sales price prediction.
So what we might do is just make a simple data frame the pred equals P a data frame so empty data frame
dear friends the sales idea column is just going to be sales I.D. from the F test so we can go equals
the F test sales I'd say a well that needs to be the same capitalization and then the sales price is
going to be test parades.
So just our array here and then let's have a look at IDF parades boom.
There we go.
Look at that.
How exciting.
We've got a submission that we could submit to this cow competition but at the moment this while this
cable competition is no longer running.
But that's an example of how you could get your data in a format that it's asking for and then to export
that how might you do that.
Well if we wanted to export this to CSB we might do something like DFA breads dot to put a little comment
here
export prediction data to see as they will put it into data and we'll go test predictions.
Something easy.
Dot CSA and index equals false wonderful I actually know this is Blue Book for bulldozers.
We'll see if that'll export.
Let's go up into our data folder.
Do we have that test predictions dot CSB.
Wonderful.
Okay.
And so now we've got some test predictions we can't really evaluate these though because we don't have
the ground truth labels for the test dataset.
That's why we were doing our evaluation on the validation dataset.
So what we might look at one final thing to wrap up this project is feature importance.
So we've just used out the patterns whereas in here where we made predictions on the test data we've
used the patterns that our model has found in the training data set to make some predictions on sale
price and these patterns are how each of these columns contribute to predicting the sale price.
So a logical thing or a good idea to figure out would be in feature importance which what columns here
or what features meant the most when the model was trying to make a prediction.
So let's do that in the next video.