While we're making good progress on their swords a price regression problem in the last video we tried
to build a machine learning model we learned that we've got some data that isn't in numeric format.
So see here we got some strings and if we check all Actually we're working dear temp.
My bad.
See these are things that you're going to forget that you're going to go through that you're going to
mix up TDM temp still.
Same problem right.
Because we've got some values here some columns that are numeric and a whole bunch that aren't.
So what can we do about that.
Well not even Daniel.
What do you do when you're not even working with your temporary data frame.
Okay.
Well this is all part of the game right.
Make mistakes and then going back and fixing them that's part of being a data scientist heaps of experimentation.
So one way we can turn all of how string data or non numeric data into numbers is by converting them
into pandas categories.
That's right.
What's that mean.
So convert strings to categories.
Now we'll put it here one way we can turn all of our data into numbers is by converting them into pandas
categories.
So like the date time object that we saw when we imported s date pandas has another data type called
category.
Let's have a look at the pandas types API.
Oh there we go.
This is probably a bit better.
We can see this here.
So panders to an API that types where can we have a look at this utility functions data types related
functionality.
So here we go.
Let's copy this but let's just read it out first.
What's going on here.
Do you type introspection so d types is bull type is categorical type.
Okay.
Pandas D type convert input into a is only D type object on num pi Diana type object.
Okay so we've got a fair few options here to have a look at different data types with pandas but since
we're focusing on converting strings to categories let's see how we do that.
Let's put a little link here.
We can check the different data types compatible with pandas here.
So remember a lot of machine learning is basically getting some data.
What we got here and then preparing it to run with an existing machine learning model that we've chosen
from now machine learning math is in here.
So it's about getting some inputs a.k.a. a whole bunch of data that we've got here.
All this sort of stuff here and then massaging it or essentially here.
That's a good word for it massaging it into a way that it works with a machine learning model.
So that machine learning model can find the patent.
Someone else has built this model for us and we're going to thank them very much.
And then it's going to find the patents and give us some outputs I can say some predictions some patents
some label something like that and then we can evaluate how well that's works is what we're doing here
we're manipulating our inputs to work with our machine learning model.
So let's have a look.
Let's remind ourselves of temp dot head of what we're working with.
Okay.
Yep.
We've seen this a fair few times this is just another way to look at it.
Always a good idea to look at your data.
And so what we might do is try to use one of these API type so what can we use.
Is there a string one here we go API types is string day type.
How do we use this.
Is string daytime string OK.
Let's try this out.
So paid a dot API dot types is string daytime DL temp usage band use a warning is emphasizing her use
warning isn't an to use warning automat.
Who knows.
I'm going to keep making that mistake but let's check.
OK.
So it is a string the usage ban column is a string.
Now we could do the same thing for almost every other column.
But why would we go through and write that much code.
Let's build a little for loop that does it for us.
So find the columns which contain strings because if we know which columns contain strings then we can
convert them into some kind of number.
And our machine learning model will be happy.
So for label columns in DFT help.
Now I'm gonna go through this one first and then we'll retrospect it afterwards because you might be
looking at is going I don't know what D F temped on items is but maybe you want to try it out maybe
on a pause a video and try it out and you explain it to me at times and if not that's right we're going
to go through it daytime.
Now all we're doing is we're just copying something like this because we're trying to find all the columns
with strings and we're going to pass this or actually this is more content.
That's what the documentation says and we're going to pass content here.
So this makes sense.
You don't know what the f items attempt on items does unless you've checked it.
But we're gonna go for label content and then we're going to check if the content is a string data type
yes then we're going to print the label.
Now you might be able to guess what this does but if not that's okay.
We're gonna see what does anyway.
If in doubt run the code and we got here.
Okay.
So these are all the columns that are string data types does that make sense if you're wondering what
the f temp items does.
It treats our data frame like a dictionary which is kind of what it is.
It's a dictionary of keys and values.
Okay.
The keys are the column names and the values are what you see in the in the column.
So let's give a little example if you're wondering what day after items does.
Here's an example.
So we'll create a random dict dictionary equals let's be creative.
Key 1 he calls hello.
And k 2 equals.
Well maybe you can be a little bit more creative than me but we're going to go for K value in random
dick dot items so see how we've got just on items.
Same with up here dollar items.
We're going to go.
We want print.
If this is okay and then we'll pass it K and then we'll also go.
If this is a value
value wonderful so if you've worked with Python dictionaries before you might be out to tell what's
going to happen here.
Well what have we got here.
Oh I see.
It's printed out twice.
That's right.
We can kind of get what's going on there.
So essentially what happens here is for labeled our content is basically labels are the column names
and the content is the content of the column.
So this is what this is doing.
This is this brilliant little for loop tells us all the columns that contains the string data type.
Wonderful.
So now that we know that what can we do.
Mm hmm.
Well if we were to read through our pandas data types related functionality we're not going to do that.
But it's somewhere in here or somewhere in there.
Or you could also search up how to change the data type of a pandas column.
One of the things you might stumble across with is converting strings into categories rather than talk
about it Daniel.
Come on let's just let's just code.
Let's see it happen all right.
All right.
Let's do it.
So this will turn all of the string values into category values and you're wondering why would we do
this.
Well that's gonna make a lot of sense in a second.
So for label we're going to just bring the same for loop that we wrote up for of 10 items and we're
going to go if the PD dot API is a fairly long one types is the string the type.
That's a mouthful.
So basically is the content to string if true we want to ADF temp we want to change that column.
So see here the column name.
So we want to access that column name.
We want to set it equal to the content so right keep the column name but change the content to as type
category you spelled this right.
Cat as ordered.
Oh my goodness.
What is happening here.
Well let's have a look
what is going to happen.
Mm hmm.
This might take a little while because we've got to go through a lot of columns but we're going to preload
and execute next itself.
Right.
Because when we're pumping through this beautiful of temp dot info I have a look here.
What is going on what have we changed.
I'll give you a hint it's something over here.
Well it is this.
So now these used to be objects okay.
They used to be strings.
Now we've changed them into categories.
Mm hmm.
Well that's interesting.
What does this mean.
Well Aha.
This is where we can have a look member we're trying to make sure all of our data frame is numeric so
we can't pass strings to a machine learning models.
This is what we're working towards IDF temp.
Let's check out which one did we check before.
Was it the state beautiful.
Let's check that IDF temp dot state.
Look at categories.
Wonderful.
Well there's old faithful Wyoming.
Shout out to Wyoming and every other state wherever you're from.
And every other country to be honest because this is on the Internet.
All right.
So we have categories here.
And what this is going to do is remember when we did the effort that state value counts.
Remember when we did this so if we were to go through all of these because they remember how he did
up here and there's a fair few things we've just covered here Daniel come on explain them as ordered
means these are now ordered.
Say how the alphabetical.
Ay ay ay ay.
See.
All right.
Isn't it b these aren't ordered here a while they're ordered by their value counts.
But essentially what this is done is it turned it from strings to category.
So this is going to assign when we called this it's going to assign a numerical value to each category.
So now looking at this it doesn't really make sense right because these are still look like strings.
However under the hood panders is treating these as numbers.
So Alabama might be one Alaska might be to Arizona might be three.
And we can check that by going do you have 10 state cap the category.
So we're accessing the catch.
Remember how we access the date time using DTM.
Well now we're accessing the category dot codes.
Okay.
So now we have a way that our data frame can be accessed at least all the string values right have been
now converted to categories because see here we loop through our data frame and we figured out whether
they're a string or not.
And we've changed their data type to categories and we've given them order.
So now we have all these categories here and we have a way to access the numeric value not that hey
from rubbing my hands together you may not be out of here that because I'm really excited.
But even though we have a way to access our data as numeric we still have a whole bunch of missing values.
So let's write this down.
Let's give some thanks.
So thanks to pandas categories we now have a way to access all of our data in the form of numbers
beautiful but we still have a bunch of missing data.
Let's have a look at that checked missing data.
So we got DAF temp is now got some.
Now let's get some ratios here.
Right.
So let's divide it by length the data frame so we got no missing values in these columns here because
these are all zeros and then we have.
Okay.
So not much missing there but usage band has 80 per cent of missing data.
Eighty five cent missing data.
81 per cent missing data 52 per cent so or 93.
My goodness maybe something is going on there we'd probably have to if this was a data set in the wild
we were working on with the client maybe we'd ask them but since it's from Kaggle accessing a subject
matter expert is probably pretty hard on this dataset.
So these are some things we might explore a bit more but we're on our mission to build a machine learning
model.
So while we might have to do next is fill these missing values.
But before we do that since we've made some changes on our DFT camp we'll say that to a new CSB so we
can if we were to just start this notebook from fresh we can just import our manipulated data frame
and start exactly where we are now.
So save pre processed data and we'll go.
Export current temp data frame so we'll go DFT temp dot to see as we will put it back in the data folder.
So blue book Blue Book for bulldozers and we're going to call it just something simple like trying temp
dot CSA.
Beautiful.
We want to index equals false.
Otherwise we'll come up with some funny things when we report it and then we're going to import pre
processed data.
Now you could do this for whatever stage you're at right.
If you've made some changes to the temporary data frame I just decided to do it now because it crossed
my mind.
You can do it whatever stage you're out if you've updated some columns if you've done some feature engineering
you can export it and re import it that way if you're coming back at a later session you don't have
to make those changes again you don't have to run your notebook from the top.
You could just run it from here on out which is written it to far.
Daniel come on we want to read it now.
Peter Reid says we this should work.
This should report it.
And then if we went the F Tampa go ahead we should have the exact same data frame that we're working
with now wonderful.
And so again we'll just remind ourselves that we have a lot of missing values to deal with beautiful
just because we export and re imported our data frame doesn't mean that our missing values have been
handled.
So that's really what we're gonna do.
Well it is what we're going to do in the next video as we work towards getting our data ready for a
machine learning model.
Let's start to fill some missing values.