Okay you've made it.
Look how far we've come.
We've covered an absolutely massive chunk of the psychic line library and we're still going to finish
it off.
We're up to the last section putting it all together.
We've been through an intense workflow.
We've seen how to get the data ready.
We've seen how to choose the right machine learning model for our problems.
We fill it to the data.
We've used it to make predictions.
We've evaluated a trained model.
We've improved a drain model.
We've saved and loaded a trained model.
Now let's combine all of this all of this could we do it could we do it now let's make a little heading
7 Buono putting it all together first thing give yourself a little pat on the back right.
Because we've covered a lot and so far it seems to be all over the place in which it is but not to worry.
This is how machine learning projects often start out as a whole bunch of different experiments and
code all over the place.
And then once you've found something which works the refinement process begins.
And now what would this refinement process look like.
Well let's use remember our good old car sales regression problem a.k.a. predicting the sale price of
cars.
We'll use that as an example and to tidy things up.
We're going to be using psychic loans.
Let's have a look.
Psychic learn pipeline to tidy things up.
We're going to be using the pipeline class.
Now you can imagine pipeline as being a way to string together a number of different socket line processes
in one hit so similar to like writing a function right.
You're getting a couple of different steps and you're putting them together.
Pipeline of Transformers with the final list minor so a transformer or Transformers a.k.a. transforming
data.
Getting data ready and then a final estimate are estimated.
Remember in psychic loan terms is a machine learning model so as a refresh this is our car sales data
is a thousand rows and there's some missing data here right.
So there's missing here.
Missing here.
And what we're trying to do is use make color odometer doors to predict the sale price of cars.
Now again this data is imperfect because could you do this.
Could you figure this out given enough time.
You'd probably realistically need a few more parameters a few more details about the cars before you
could figure out how much they'd sell for.
So let's stop talking about it let's get hands on with the code to to start off we'll report the data.
Go pay data read see as v dot dot.
We've saved it in our data voter.
Car sales extended online in a data folder is not is not up the chain.
I want car sales.
There we go.
Car sales extended data missing.
That's what we want.
And then we'll have a look.
We'll have a quick peek at it wonderful.
So this is just reporting this CSB fall that we have here comeback now.
What did we say.
We said we want to use these columns so these four columns to predict the price column we'll have a
look at our data types.
So you've got make which is an object that's a string their color which is an object that's a string
their odometer doors and price which are we've kind of talked about doors being a category as well.
So odometer is our numerical column.
And do you remember what we have to do for our data.
We have to make sure that it's all in numerical format before we can build a machine learning model
on it and there's another little roadblock that we have as well before we can build our machine learning
model and that figure is an A.
So is now not some we've got no values in some of our while all of our columns we've got 50 missing
missing data points in each of our columns.
So that's also a big no no with machine learning models.
We need to make sure that the data is numerical and that there's no missing values we refer back to
our keynote or data should be numerical.
This is things to remember right.
And there should be no missing values.
Okay let's check it out.
So what we'll have to do is if we want to put this all together we've done this before we've seen filling
data we've seen converting it but this time we want to use a pipeline to do so so the socket lines pipelines
main import is steps which is a list that contains a topple which has step name and action to take.
We're gonna see this in a second what we're just talking through it and in our case our steps are what
are our steps.
Well first we need to fill the missing data and then convert the data to numbers and then build a machine
learning model on it.
So let's do it.
We'll go here.
I want to do this one so it's all in one cell.
Let's put our steps here.
What we have to do steps.
We want to do all in one cell.
Can we do it all in one cell.
Because before to do this it took a fair bit of work but let's say this is what we're doing.
This is our refinement process using pipeline.
So step one is fill missing data and then step two is convert data to numbers.
Wonderful.
And then step 3 is build a model on the data.
All right.
Cracks knuckles gets fingers ready let's do them.
And if it seems like I'm going through this pretty fast let me tell you there's nothing here that we
haven't covered except for the pipeline but what we've learned in the previous videos you'll be able
to deduce as well as reading the documentation here.
What exactly is going on.
So import panders is payday because you want to do this all in one cell and then from SBA loan dot com
pose.
Remember the column transformer.
And then from SBA loan.
This is where we're going to import the pipeline class that we're going to take advantage of here.
Import pipeline and then from SBA loan not impute we're going to import simple computer.
This is what we're going to use to impute our data.
So fill the missing data.
And then from as K line dot pre processing import one hot encoder.
This is what we're going to do to convert our data to numbers or sorry we're going to use to convert
our data to number.
So getting data ready.
These are the inputs we're going to use for that and the next one is modeling so from Skyline dot ensemble
import our trusty Random Forests regress.
Because we wanted to predict a number and then from SBA loan model selection import over there right.
Yeah that looks right.
I've been known to be prone to typos try and test blimp and we want grid search CV because we've seen
grid search being used before which is gonna search for some optimal hover parameters for our Random
Forest aggressor.
Okay now we're going to set up random seed so we'll import num pi as NDP MP random seed 42.
Wonderful now for completeness for doing it all in one cell we're going to reinstall the dinosaur import
data and we're going to drop the rows with missing labels so data equals PD CSB.
We're going to go to our data folder.
This needs to be a string to go.
Car sales extended missing data.
That's the one we're after the thousand Rose version and then to drop the rows that have missing labels.
We'll go data dot drop in a subset we'll set that equal to the price column because that's what we want
to predict and that's our label and we're going to go in place equals true.
Okay.
So what we've done is we've just replicated this here and now we've dropped the rows which contain missing
price values so a.k.a. missing labels.
Now what we're going to do here is we're going to start to define the different features of the dataset
and the different transformer pipelines that we want to take place on that data.
So if we come back up here what transformations do we need to make we need to change our making color
into numbers.
We need to fill the missing values of all the columns in here and we need to change our doors because
it's also a category we need to change it to a category of value.
So we're going to actually know forget what I said about doors.
Let's see what we're actually doing in the code.
So define different features and transformer pipelines.
So for our categorical features doors doors is a special case because it's already numerical categorical
features equals make and color.
Now what we're going to do here is create a categorical transformer.
So this just means modifying the data and to do so we're going to use a pipeline.
This is the first time we're seeing pipeline but the steps in the pipeline are going to be first we're
going to fill with missing values within puter.
So this is where the pipeline inputs take steps to take.
Right.
One after the other as a list of topples so let's see this happening.
The name is in pewter and we're going to remember this is the action to take.
So simple computer strategy equals constant.
And then the fill value is going to be missing.
So you might be out of juice what's happening there with this particular case where imputing the categorical
features with the constant value of missing string.
Now we're going to create one hot and go one hot encoder and to handle the unknowns the columns that
hasn't seen before we're going to tell it to ignore them.
Wonderful.
And then we'll make sure that we've all lined up or lined up these things that we're gonna need to we're
missing something here.
Need to check where our brackets are at.
We're missing one here.
There we go.
See half of this is our figuring out what you're missing racquets are and then door feature.
We're going to define our door feature Eagles doors.
And so the same thing again so we're gonna create a door transformer door.
Transformer equals pipeline again we're using a pipeline because this will all make sense in a second
right.
Why we're setting out pipelines so to do this we have to press impute.
We want to do the same thing again simple computer and the strategy is going to be constant and the
full value for our doors is we're just going to fill it out with force because four is the majority
number in the doors column.
Then next we want numeric features.
So our numeric features is our domain a common Cam.
Wonderful.
And then we're going to create a numeric transformer equals first saying we're going to do this pipeline
steps equal and then again we need to impute values here.
So impute a simple computer strategy equals mean.
So this means it's going to fill the numerical columns in our case the odometer with the strategy means
is gonna take the main value of the odometer column and fill all of the missing rows in odometer with
the main of the rest of the values.
That was a lot.
Now still not done.
That's right.
We're gonna set up the pre processing steps.
So remember what we need to do we need to fill missing data.
So this is part of our pre processing these to a pre processing our data.
And this is modelling the data.
So our pre processing steps are going to be fill missing data and convert data to numbers so we can
do that fill missing values then convert to numbers.
So to create a pre processor will instantiate a pre processor using column transformer tab order complete.
And we're going to pass it column transformer takes Transformers Eagles.
And now this is going to be a list of the transformations we want to do on our data and in our case
we have one called Cat for short and categorical it's going to it the categorical transformer and we
want those transformations that we've defined up here the categorical transformer to fill a values simple
computer with missing and to one hot in code making color.
We want those to be performed on the categorical features and press tab or complete categorical features.
Wonderful.
Now we pass it another topple.
So now we want it to do same steps but this time using the door transformer which is going to fill the
door column with the constant value of four.
So go door feature wonderful.
And finally I wanted to finish up using the nom transformer which is going to take our numeric transformer
and perform the imputation on the numeric features
beautiful when there's only one feature so features is as a plural is probably pulling not correct.
And now now that we've done our pre processing and so our data is getting filled the missing values
are getting filled.
And then it's getting converted to numbers using what we've done here.
The next step is to create a pre processing and modelling pipeline.
We can do this as you maybe had a guess because I've used the word pipeline.
We need to create a pre processing and modelling pipeline so now essentially what we're doing is we're
putting our pre processing steps together with a modelling step.
And because there's multiple steps we'll use a pipeline to combine them.
So model equals pipeline
deposit steps which is a list containing tar balls.
And this is the name.
So this is a pretty process a step and we're going to pass it pre processor so the first step in our
pipeline here our model pipeline is to run through this and then the next step will put a little camera
here.
We need to create another top on which is going to be our model is our Random Forest regress up wonderful.
So now we've got our model pipeline setup which will go through pre processing once the pre processing
is done.
It'll build a random forest classifier on it.
We can get our data ready so split data X equals data don't draw.
We're going to remove the price column because that's what we want to predict.
Access is going to equal one Y is going to be data price then we're going to create X train x test as
we always do.
Why train.
Why test equals try and test split x y test size.
We'll use zero point two and then we own it fit and score the model to model don't fit X train y trying
now model dot school X test y test where we got it wrong.
Handle unknown.
We've got a typo of course we do handle unknown.
There we go.
Change that.
We did it.
We did the entire pull up line in one cell taking advantage of the pipeline class so let's step through
what we did.
We imported the classes to get our data ready Panda's colon transformer pipeline simple and pewter one
hot encoder we imported the classes to do our modeling so random forest aggressor trying to split grid
search CV we set up a random seed so our results are reproducible and then we define different features
and transform our pipelines.
So we defined the categorical features here.
The door feature the numeric feature and then for each of them we created a few steps for our pipeline
to take to not only impute missing values but to one hot encode so turn all of our data that wasn't
numbers into numbers and then we set up a pre processing step that took all of our pipelines here to
take the steps that we wanted to take on the categorical features on the door feature and on the numerical
feature.
Then we created another pipeline to put all of these steps together to not only pre process our data
and then model it so then we split our data into x and y and then into train and test.
And so when we call model dot fit it's gonna call this variable here which is going to run this pipeline
which inherently is gonna call the pre process a step which is going to run this pipeline which is inherently
going to run all of these three steps.
You know I'd be thinking Daniel while this has got a lot going on but if you go back through what you've
looked at here you'll be had to say you can step through it and go okay.
I'm starting to piece together what's happening and now if you scroll right back up to where we originally
did this filling data with pandas filling data with SDK learn and then modelling it on our car sales
problem you'll see that we did it over a number of different cells and now we're doing it.
We're getting a full blown machine learning model and 55 lines of code minus some comments and we could
probably tidy this up if we're being honest with ourselves.
But now you're starting to see is an end the beginning machine learning is very experimental throughout
the whole thing and it's very experimental.
And so what it takes at the start is just code everywhere trying different things seeing how they work
and then eventually you can bring it down to something like this.
Put it all into one big cell in a pipeline going through the steps that you've figured out along the
way.
Now of course we'd want to probably improve this use hyper parameter tuning and that's what we might
look at in the next video is how to use hyper parameter tuning with a pipeline slightly different to
how we've seen it done with just a regular model so take a little break step back through what we've
done here because it was a lot.
Don't worry if you don't take it in entirely.
Have a read through the pipeline.
Class of socket loan documentation and see if you can line up what we've done in the documentation to
what we've done here otherwise I'll see you in the next video we'll see if we can improve this score.