All right.
So now we've got our notebook a little bit laid out and we know what steps we're taking we know what
problem we're trying to solve.
We know where our data is coming from.
We know our valuation metric and the features let's import the data and get started.
So in my case we've imported it to a folder called data.
You might have created it followed the same so when I write the file path your file path might be different
to mine.
The mine's gonna be called data.
So let's do that and we're going to import the training and validation sets to begin with.
If you're wondering why we're employing them that same time this all seem clear in an upcoming video.
So you'll just have to trust me for the time being that we're importing both in one set and we go blue
book.
We can probably use tab order complete.
Surely.
There we go.
So we want train and validate CSB wonderful hip shift and enter.
Now this might take a while because as we'll see in a second this is a fairly large data set.
We're getting a warning.
What do we have here d type warning columns 13 39 40 41 have mixed types specified D type option on
import or set low memory Eagles falls.
So if we go here we can get rid of this warning by setting low memory equals false.
So this is essentially saying is when panders Read says when this function is instantiated it depending
on how large your data set is because data frames live in RAM pandas tries to minimize the amount of
space they take up.
So what we're doing is we're just setting low memory to equal false to go Hey we know you're trying
to minimize space but don't worry too much.
We've got plenty of RAM space so this should work.
We should be out to get rid of that warning.
Wonderful.
So this is just saying store it however you want to try import it.
Don't try to minimize the space that you're using.
Wonderful.
And the first thing we'll probably do is go DFA info to get a little bit of information of what's happening
so there we go.
We're going to have a look at that.
India's biggest dataset we've worked with yet by a fairly long shot we have four hundred and twelve
thousand six hundred ninety eight rows.
That is a lot of rows and 53 columns.
Remember we have a look at our columns These are our variables and descriptions.
So we should have sales I.D..
Yes that's a non now integer machine I.D. which is the identifier for a particular machine.
Machines may have multiple sales are OK.
So that means a machine could be sold multiple times over the crossover timeframe than the one we want
to probably pay most attention to because it's a time series.
Aside from the sale price that's how much our machines have been sold for is the sale date.
So this is when the machine has been sold so we have a look at this sale date time of sale.
Beautiful.
OK.
So what can we do now.
Since the sale date is a date what we might want to do is pass it as a date but we won't do that yet
because we want to demonstrate something.
So we're going to set up the plot because this is what we usually do when we first begin any kind of
project is try to make some visualizations.
Looking at this there's a lot going on here right.
Like you could just look at this this is hard to infer if you're like me and you're just looking at
list of words and numbers and you just say ah I need to visualize things I need to look at things but
we can see actually what can we see from this.
So we've got four hundred twelve thousand six hundred ninety eight entries.
These are known now we say we got some missing values especially for usage band whatever that is.
So if we come up to our data dictionary usage band so value low medium high calculated comparing this
particular machine sale hours to average usage for the F base model so therefore based on what's really
another feature there we go this aggregation of F on model description far out we've got a fair few
pieces of information here about our bulldozers writes of this actually a pretty good dataset but we
can see that for usage band there's a fair few missing values because there's only seventy three thousand
six hundred seventy non null objects so that means there's four hundred twelve thousand six hundred
ninety eight minus that missing values and we can figure that out actually before we plot let's figure
that out so we've got Def is an eight this is how we're going to check if there's no values there we
go.
So usage band has three hundred thirty nine thousand and twenty eight missing values knowing what we
know about machine learning models that's something we're gonna have to take care of before we can build
a model but let's push forward let's plot because we're trying to predict sale price and we know that
our data is a time series what we might want to look at is something like comparing sale dates with
sale price and because as so many rows here if we were to do that if we were to try plot all of these
data points in one plot because you imagine trying to draw four hundred and twelve thousand dot points
on a plot and compute is gonna have a hard time so let's do about a thousand or so why not let's go
say all day we go here we'll do the first thousand How about that.
That's a fairly large number we want to do sale price and if you want to find out the names of different
columns remember you can always go DFT columns.
So what we're doing here we're taking sale date wherever that is sale date there which is we come to
our data dictionary this is where we want to be referencing so the time of sale so the date that it's
been sold as well a sale price.
Now is that in our data dictionary.
I think that's pretty straightforward actually.
What sales prices are there we go sale price cost of sale in US dollars so let's check it out let's
check when one of these being sold we're going to put the out on the x axis because member scatter is
X Y and sale price on the y axis.
Let's check it out.
What if we go to oh we've got our equal sign
so you see here we try and down the bottom to put Saudi at the first thousand examples so this is why
our plot is looking a bit funky and it's just trying to pack in if we have a look at our data frame
DFW sale date that's what it's trying to plot on the x axis if we have a look at the first thousand
they're trying to plot all of these eyes along here so you can see how it just looks like one big chunk
of a barcode and then just a big chunk of black text there.
So what we might do is let's just plot a histogram because we always like checking the distribution
which is the spread of data of as many variables as possible.
But in this case why not do Sal Price because that's our target variable so sale price.
And now again if it seems kind of unstructured what I'm doing here it's because it really is like this
so many different ways you can explore a data set.
One of the best I like is is DNA Info check to see if there's any null value so we know now as in an
A as in missing values.
So we know how many missing values are checked the columns plot one of the most important columns from
now I've just chosen Saudi because it's a time serious problem.
Plot one of the most important columns vs. the sale price of versus the target column and then do a
distribution of a few columns of your choice.
These are kind of the stereotypical things that I like to do when first exploring a dataset.
But what we can see here we see that most of our sale prices are below 20000 U.S. dollars.
So the biggest column here or not most but a large majority a large portion.
And then we got some below 30000.
Let's just assume that 30000 40000 and then it just keeps going off and there's less and less and less
and there's not very many that are costing upwards of 100000 U.S. dollars.
So that's what we're looking at there.
Okay.
Now what we might do is because these dates aren't really in a format that we want them in.
So see how they're in D type object when you're dealing with time series.
We want as much information to be encoded in the data as possible.
So that's what we'll do a little way that we can do this is called passing dates.
So let's have a look at what this looks like passing dates.
So when we work with time series data they want to enrich the time and date component as much as possible.
You might be thinking well how do we do that.
Well let's put it here.
We can do that by telling pandas
which of our columns has dates in it
using
the past dates parameter
wonderful.
And what this is going to do is turn whichever column we pass to pass dates into a date time object.
So let's first see what is a date time object in pandas go in here timestamp to date time so you could
find out a whole bunch of different things what date time here is daytime object timestamp is a pandas
equivalent of Python's date time and is India exchangeable with it.
In most cases OK.
Beautiful.
So basically when we run there.
So let's do it again.
Import Data again at this time pass dates remember.
If in doubt run the code speed a raid C as V and we're going to go data.
Book for bulldozers tick and we're gonna go train train invalid CSB.
Wonderful.
But this time we're gonna pass it low memory was false so we don't have that warning and then we're
going to pass it pass dates and we'll pass it the column name sale date.
Why.
Because that's where our dates are.
And so take note of what this is.
So if we go sale date it's a date type object.
We might just put a little here that day type deed type O.
So when we do this let's see what the differences.
Now this is gonna override our original data frame so we can go DFT sale date D daytime huh.
That's a bit funky.
What if we did DFW Saudi.
What does it look like.
You have got Saudi first thousand okay.
So if we come up here we got sale date.
See how a look like this.
What panders is done is because we've passed it to past dates.
It's gone over all of the values in the Saudis column and converted them into this style.
So we've got year month day and we can see that the data type is daytime 64 and if you were to Google
and compare these two because you might be wondering why does it have a different value here compared
to here.
You'll find that in num pi these two are equivalent.
So you'll just have to take my word for it you could Google and we might actually do that just so it
can prove to you.
So we go here.
Control C so what data type is
there we go.
They time 64 is a general D type while M 8 N S is a specific time.
There we go.
This is stack overflow so this is just a workflow right when you don't get something.
There we go.
MP dog day type daytime sixty four equals NPD up day type n nights.
That's what we're after but now that our dates have been passed let's do the same plot that we did before.
So fig X equals penalty not subplots.
Wonderful.
And we're gonna do the exact same thing.
AX don't scatter.
We'll finish up this video in a second actually because we are actually on a rush right.
We learn and we're exploring sale price and then we'll go here.
We do a thousand and see what looks different wonderful.
So see how down the bottom now instead of when we come back up here we've just got a whole bunch of
black because we're trying to put all these down there because our sale day column is now in the form
of the date time object map plot leaves able to intelligently look at the data type and then plotted
accordingly.
So we've got higher prices so you can see there's a lot of sales maybe 2005 something happened so there
wasn't any sales in 2005 2008.
I believe that might have been the financial crisis or something like that so there wasn't too many
sales here.
And then again that's between 2008 2009 we can see the highest sale was done just in 2007 and then there's
a lot of sales around the middle of between 2009 2010 but again this doesn't really tell us too much
right.
We're just getting an idea of what's going on.
So let's have a look one more time check it out.
Do you have to have this data and that we're working with we got sales I.D. sales price a whole bunch
of different columns here and we can see that there's a whole bunch of missing values.
And if we wanted to find out more about what undercarriage pad width means remember we've got our data
dictionary and we could have a look here.
There we go.
Machine configuration width of crawler trains wonderful.
So now we've done a little bit of exploration what we might do is sort our data frame by sale date and
then keep going from there.
Make some more changes so we'll do that in the next video but otherwise have a go at exploring the data
frame importing it and trying to pass the dates with sale date and don't pass the dates and check the
difference in data type when you do and don't pass dates.