We have been through a lot but we're going to persevere.
I'm going to get through this together.
We've got a few more functions to go through.
So the next one is sample.
Now this is a great way to to shuffle your data frame.
What I mean by that is mix up the indexes.
So in future videos when we start applying machine learning algorithms to to our data one of their most
important steps is creating a training validation and test set.
And one of the steps to doing that is random izing the order your data comes in.
Because what you want to avoid is the order that these samples come in.
May have some kind of influence on patterns in the data when now machine learning algorithm is learning
things.
We want it to be as general as possible so we don't want it to care about the order.
Now there are some exceptions to the rule with time serious data and whatnot but I'm just going to show
you this example if you wanted to shuffle rows in a data frame so let's do car sales.
Don't sample and now sample is basically saying take a sample from this data frame with this parameter
frac is the fraction of that sample.
So it's gonna be in an integer form.
So for example zero point five is 50 percent of the data.
So it's gonna get five rows out of 10 but we in this case want to shuffle everything so we will use
one for 100 percent of the data there we go.
So now you can see that the index has been shuffled.
So this is essentially the same data frame just with the order mixed around.
Now the important thing to remember with PANDAS is that the information in a row will stay with its
row so row nine NISSAN NISSAN white thirty one thousand six hundred kilometers four doors.
So the rows stay the same.
All that's changing with this function sample is the order that things appear in and now let's have
a look at the data frame again silly May we keep forgetting to reassign.
So we want to reassign here equals.
There we go.
Now it's going to definitely shuffle the car sales data frame or maybe we create a new one shuffled
because we want to keep the original one in order.
So we've got car sales shuffled this return a copy but car sales shuffled will be in a different order
to the original car sales data frame.
Beautiful.
Now if we wanted to manipulate this shuffled data frame we could call the same functions we've called
before on the shuffled version and another handy thing about the sample function is that say for example
you had like a data frame with two million rows.
Ours is really small it's only 10.
But in practice you're going to be working on data sets with a lot more rows sometimes running functions
in panders takes a long time on millions of different rows what you might want to do is practice on
only 20 percent of the data.
Only select 20 percent of data.
And now this number could be arbitrary right.
In our case we've chosen 20 percent.
So that's giving us two different rows.
If you had two million rows maybe you want to practice on 1 percent of the data.
So that's still 20000 rows.
And so that will allow you to do lots of different experiments a lot quicker than doing it all on two
million rows at one time.
So that's something you'll have to consider in the future in your projects when you're working on larger
amounts of data is your computer powerful enough to continually run functions on millions of rows.
Or do you want to flesh out some little experiments on smaller amounts of data first and then upgrade
those experiments as you figure out what works and what doesn't.
Now we look at our car sales shuffled data again
we've got our indexes but they're all out of order.
How would we get those back into order.
We go car sales shuffled don't reset index this function as you could imagine is going to reset the
index here.
So we want warm and now there's a way here that you can remove this column so maybe we go in place.
Let's have a look.
Did that do that in place shuffled.
Because by default the reset index column adds the old index as a new column on the very left.
And so we've got all the numbers here but this is the shuffled index that it's made a column maybe we
don't want that.
I think we can get rid of that by in place equals true.
Remember a lot of Panda's functions have the in-place parameter shuffle in place it goes through.
It still has that.
Well the way we figure this out is if we look at the documentation for reset index let's do that pandas
reset index I don't want that extra index column there.
So let's see drop do not try to insert index in the data frame columns.
I think that's what we need.
Is there an example.
Drop equals true.
Yeah.
I think that's what we need.
So we want in place egos.
True and we want drop equals true drop is false by default yet default false.
So this is another example of how you can look out what a function does in panders.
I just googled the function name reset index again you won't know these things from the start but it's
not about knowing things off by heart.
I don't know everything off by heart.
It's about researching different ways that functions work and trying things out if in doubt run the
code.
Let's do that.
We still have that there.
Maybe we come back up here.
We're going to reset this reset.
This reset this there we go.
We had to reset our car sales shuffled data frame.
Now we've set drop to equal true.
We don't have that index column on the left.
Here is one more thing we're going to look at and that's how to apply a function to a column.
So let's say we'll have one we'll look at our car sales data frame.
Remember this is the process you're going to be taking a lot is viewing your data frame and then manipulating
it viewing it manipulating it viewing it manipulating it.
So it's looking pretty private the moment but we want to convert our odometer to Miles.
So how would we do that car sales first we have to select it.
You do that by typing in the column name and then we're going to reassign it remembered to reassign
at this time odometer kilometres.
Yep.
Now this is the apply function apply let you apply some kind of function whether it be num Pi or lambda
to a certain column.
So let's just type this in and lambda functions can be confusing to begin with.
Took me a while to learn them but we're going to step through this one and figure out what it's actually
doing so if in doubt run the code we want car sales to view the whole data frame run the code.
There we go.
Now what do you think this is just done.
These numbers have changed.
One hundred and fifty thousand.
It is now ninety three thousand seven hundred seventy six.
Well let's step through this odometer column tick.
I don't wanna column.
Tick.
Dot apply.
Yeah lambda now lambda is a keyword in python which is short for an anonymous function.
So basically this is saying apply this function to X divided by one point six.
So that's what this is saying.
Apply recent X to be equal to x divided by one point six y one point six because conversion from kilometers
to miles equals about one point six 1 0 0 9.
But I've just rounded at one point six.
That's going to divide our number of kilometers by one point six.
Because in this case x is the values in our odometer column.
So this little line here is going to look at this column and go every time it's going to pick up this
hundred fifty thousand which is x and then divide it by one point six and then reassign it to that value.
So the first column 94000 thereabouts first row sorry second row eighty seven thousand eight hundred
ninety nine is actually about 55000 miles.
So that's what this apply function is doing.
And now this is just a simple example of a little lambda function but you can get very elaborate with
the things you put in apply when I'm gonna dive too deep in that.
Now I want you to know that apply is a way that you can assign a function to a column.
She might have written another function somewhere else and then use apply to apply it to a column who
we have been through an absolute mountain of information here.
And don't fret you don't have to know all of this off my heart to begin with.
Remember this stamps.
Try it.
Run your code if in doubt search for it.
Try again.
And then if you're still stuck.
Ask in the meantime before we get into the next section this is going to wrap up the pandas section.
Check out the resources section for a little bit more documentation about pandas.
Some exercises you can try out for yourself but otherwise if you're with me let's jump in and learn
a bit more about some other tools that we can use for data science and machine learning.