By now you're probably thinking my goodness there is so much to remember and I'm not going to lie.
I felt the same way when I first started.
I went through some information on partners and realized Holy goodness there is.
Well I didn't actually say Goodness I said a different word there.
You could probably imagine what would but I said there's so much here.
How am I going to remember all of these functions off my heart.
And the point here is to remember you don't necessarily need to know all of this off by heart.
The only reason I can kind of do this on the fly is because I've had a little bit of practice.
Don't forget to always try to code yourself.
You can look it up fine things like this.
This is part of the research part of being a data scientist part of being a machine learning engineer
is learning how to ask different questions think about how can I do this with the data and then trying
to find out.
With that being said let's have a look in this next little section here at how to manipulate data with
partners.
So put another little heading like we've been doing maybe manipulating data.
That sounds a bit better.
This came to press M for markdown because we're staying communicative Excellent.
And now again the beautiful thing about pandas is that if you can think of it you can probably do it.
So the first thing we'll see is manipulating data is some string methods.
So Python has a bunch of different string methods like lower what do you think this will do if we go
car sales make dot string dot lower let's say shift and enter it's lowered the string because it's data
type is an object encoding the pandas but it's actually also a python string.
So anything that you can do on strings in Python you can do on string columns in pandas.
There's a bunch of different functions Amber.
Well just remember that you can access the string value of a column by doing dot string.
We did that above in the price column we converted it from a string to an integer.
Now if we have a look at our car sales data frame again with there we didn't we just lower the string
but it didn't actually save.
Now here's an important concept to remember it's that pandas requires if you want to change a column
it requires reassigning that column.
So that means we want to set car sales make equals car sales make you want to do not Schering don't
lower.
There we go car sales make don't shrink don't lower.
Now what this will do is it will reassign the car sales make column to be lowered strength.
So let's have a look.
There we go.
The string has been lowered.
So now all the levers in here are lowercase.
You'll probably come across this a fair few times when you're working on different data frames is this
this concept of reassignment what some functions have is that to save you from having to go.
This column equals this column again some sort of function is some functions have a parameter called
in place we'll have a look at that in place parameter along with some missing data you might be saying
hey I'll cross sales data frame doesn't have any missing data but in practice in the real world and
when you get real data sets I've made this one up.
Right so I've typed in this into here real data sets often contain missing data.
So they often look like this because you might have someone tracking some records.
Ideally they'd look like this all the data there but in reality when you get a new data set sometimes
it's got a few holes in it.
Right.
Someone hasn't tracked everything correctly and that's that's just life.
You don't always get perfection.
So we're going to do you remember how to import CSB file.
I've already exported this one.
So we'll go payday.
Don't read CSB.
We're gonna do tab auto complete car sales missing data dot CSC what this is going to do is save the
car sales missing CSP and import it as a panda's data frame.
So what it's gonna do is bring this into being a data frame shift and into now this is how Panda's describes
missing data in a n for Nan.
That's basically just saying hey I've got no value here not a number or something like that actually
done.
Not entirely sure what Nan stands for but all I know is that it just means there's no value here.
All right.
So what if you wanted to fill these missing values with something.
So say for example we wanted to fill the odometer so we're accessing that.
There's a function called Fill N.A. which is fill these in and values and you can pass fill n a some
kind of value and it will fill these value.
So maybe we want to fill up the odometer column with the mean of the rest of the odometer.
Which in reality isn't a very good practice but just for demonstration sake.
Maybe let's let's do that don't mean.
So what this is saying oh no we want missing because this is our missing missing missing.
There we go what this is saying is grab the car sales missing or domino column this one and fill in
a fill the missing values with the car sales missing Domino columns mean value.
So the values that do actually exist there.
We're gonna get the main and fill it.
Let's quickly see what that is.
Car sales.
Missing odometer.
Don't mean so.
If we do this correctly should fill them up with this value.
Let's go.
They're beautiful.
Now we've got all those missing values.
The third one down is now filled with the car sales missing odometer mean value.
We've got a few recurrences there of all the missing ones.
So now if we have a look at the data frame and this is a practice you'll be going through right you'll
be making manipulations here and then you'll be consistently viewing the data frames he might do head
but because it's only small.
We'll look at the whole thing.
What's happened here.
It's still in I n this is the problem right.
This is what I was mentioning before that in-place value that we can use.
So let's see it in action.
So we could do a reassignment by going car sales missing don't wait
equals that and then reassigning the values to being something like that.
But what we'll do is we'll look at the in-place parameter so you can see what I mean.
So we'll get rid of that.
We're going to copy this line of code so we don't have to retype it out.
But remember should always practice retype in code.
So you get used to it.
Now this fill in a as a parameter.
And lots of pandas functions have this in place equals it's false by default we want true.
Actually it's a good time to look at the documentation so say we didn't know this fill in a function.
What would you do to figure that out.
So maybe we go to Google and go pandas how to fill missing data in a column
working with missing data.
Yes this is a panda's documentation get very familiar with looking at these when you first start looking
at this documentation can be a bit overwhelming because there's a lot going on.
Remember this documentation is for every single thing that pandas can do so going through it can take
a while and if we're looking here maybe want to search fulfill fill fill missing values.
There we go.
Phil and I.
Now often it comes with.
Let me zoom in for you a little hyperlink to the exact function.
Fill in a can fill in any values with non n a data in a couple of ways.
Excellent.
And then give some examples here.
We're going to click on the fill in a function and see what happens.
So we've got data framed up filename which is what we've got so far data frame or domino because we
only want the odometer column not fill in a yellow tick for that value equals none.
So by default the value because none.
But it can take a few different values.
In our case our value is the main of the odometer column.
We'll come back.
We can see this little in-place parameter.
It's got a few others but we're not going to cover them for now.
So in-place bull by default false if true fill in place.
Note this will modify any other views on this object e.g. a no copies slice for a column in a data frame.
So if we set in place to true it means fill in place.
Let's have a look in place equals true before we do we'll take one last look at what our data frame
looks like still missing values.
So let's see what happens.
Fill in a place equals true.
Now let's have another look.
Missing.
There we go so we've filled the values without reassigning so without typing.
Car sales missing odometer odometer
equals that.
So that's just the thing to remember.
You can always do it this way.
Remember how I said panders has multiple different ways of doing the same thing.
One way to reassign values is to reassign them by putting the exact value you want to replace at the
start or by setting that in place parameter of a function to true.
So that'll really depend on how your workflow goes.
Personally I'm not a fan of the in-place I like just leaving it by default and then reassigning the
value at the start.
Let's have a look at one more thing before we will take a little break.
As this lecture is getting a bit long it's if we wanted to remove these missing value.
So say for example we were happy with filling the odometer column but every other row that has missing
values.
We just want to ignore them because missing values aren't gonna help us with our data analysis.
In truth they might but we want to just see how we would get rid of them.
Car sales missing now there's a function here called drop and a now we might just see what happens if
we just run that drop in a beautiful.
Now if we type in here.
Car sales missing what does that data frame look like.
Oh it hasn't happened again.
Well we've just covered this.
What do you think we can do to fix this some so I might do.
Car sales drop in a in place equals true
shift into and we'll have a look at our data frame car sales missing beautiful Now what do you think
the value is in maybe not putting in place as true now.
Our car sales missing data frame is always going to return those rows with missing value is going to
be dropped.
What if we wanted to re access them.
We'd have to re import the C as V so maybe we do that it goes PD dot read C.S..
Car sales missing so we'll reset our data frame.
We'll have a look at it.
This one's going to have missing values so maybe we create a new data frame.
Car sales missing dropped equals car sales missing dot drop in a in place equals true.
So what this will do is it returns a copy of our car sales missing data frame but with dropped values.
Well I sit in place equals true.
After a brief rerun.
So that's okay.
So you sometimes you make mistakes here.
We're getting a drop in I know a guy car sales missing dropped beautiful.
So it's going to be the same data frame but dropped the missing values then what we might do is car
sales missing dropped save this to to see SUV as
car sales missing dropped dot CSC that way we can if we did want to access the rows with missing values
we can from our car sales missing data frame.
This one here or if we wanted to manipulate our data frame with missing values dropped we can from this
data frame.
So that's the important distinction there between using in place or not using in place.
All right.
We've covered enough on manipulating data in this lesson for the time being so take a little break revise
over the few functions that we've gone through.
Have a trial out with in place versus not in place and I'll see you in the next video.