In this lesson we're gonna do some data pre processing.
We can do three things primarily.
We're going to reskill our feature data.
We're going to convert our target values into one hot encoding and we're also going to create our validation
data set from our training data.
So in your Jupiter notebook at a markdown cell here that reads data pre processing.
And that way we have a little subsection to give us a bit more screen real estate with the toggle header
here under review and we can work a little bit higher up.
So one thing that we've already seen in the last lesson was how our values for our features are between
0 and 255.
Now considering that the learning rates for our optimizer are going to be very very small values it
helps when the inputs to our neural network are going to be between 0 and 1.
That's why we're going to reach scale our features.
And the easiest way to do that is that we divide our training data and our testing data by two hundred
and fifty five.
That's the largest value right.
So if there's a completely black pixel it'll have the value one after We've re scaled it.
We can do all of this work in a single line of code X unequal score train on the score all come up X
on a school test is gonna be equal to X and a score train on the score all divided by two hundred and
fifty five point zero come on X on a score test divided by two hundred and fifty five point zero.
This calculation will do two divisions one on all the values in our training data set and one and all
the values in our testing data set and what it will also do.
Because of this decimal point and this division is we're going to convert all of the integers that are
currently in him to floating point numbers or our numbers with a decimal point so limit shift enter
on this.
And now we can tackle our target values these currently looks something like this.
They're all numbers between 0 and 9.
What we're gonna do is going to transform all of these into values between 0 and 1 actually.
So what are effectively going to do is to take this sparse matrix right here which is what this is and
turn it into a full matrix.
Let me show you this through an example.
So if I take these first five values here and I'll just store them in an array called values what I
can do then is use a function from num pi called end p dot I 10 closing parentheses square brackets
and then values me hit shift enter on this and show you what this does.
All right.
So what are we looking at here.
Well we can see that now we find a one in the position of the numbers inside the values array.
For example there's a one in the fifth position 0 1 2 3 4 5.
There is a 1 in the first position for the second row.
There's a 1 in the fourth position on the third row 0 1 2 3 4 and so on.
What's going on here.
Well let's take this step by step.
This is actually something we've seen before.
But in a slightly different form.
So if I pull up the documentation for end p dot I then I can see here that this function returns a 2D
array with ones the diagonal and zero elsewhere and N.
So this first parameter is the number of rows in the output.
So if I come down here and I just write and put out I on its own the number 10 then I get a 10 by 10
matrix with ones down the diagonal.
Why did I use the number 10.
Well because we've got 10 different types of labels in our dataset.
We've got the numbers between 0 and 9.
So that's why I've created a 10 by 10 matrix.
So what's happening next.
Well this is not matrix multiplication.
Instead what we're doing is actually array element indexing the second bit here.
This values in the square brackets acts as the index array each number in the index array indicates
which value in the preceding array to use in the place of the index.
So check this out if I've caught n Pete and I and I use that same ten by ten matrix and then I have
some square brackets after it and I put the number two there Then I get the third row extracted from
my identity matrix my one here is in the third position I get this entire row coming on.
So I hope you can see what's going on here now if I've got my values array like so and I pull out a
particular value here with the square bracket notation like so then this is the form that's very familiar
to us here I'm pulling out the fifth value are the number nine in this case here I'm pulling out the
third row at index number two.
So all we're doing him is we're using this entire array as an index and we're pulling out several of
the rows from the identity matrix and this is how we can convert the entire training data set for the
labels into a 1 Hot encoding.
So let's do that now I'll add a little subheading here that reads convert target values to 1 Hot encoding
at the moment our target values are sparse because they just have an integer for the class and what
we're going to do is when I essentially reshape this entire thing so that it is in this format instead
the way we can do this is simply by overwriting y on a squat train underscore all with and p dot I 10
square brackets.
Why underscore a train underscore all and if we don't want this number 10 floating around in here because
we might not know what it stands for.
When we come back to it in the future let's add a constant at the top that reads an R underscore classes
and that's going to be equal to the number 10.
If I refresh the cell I can now use my constant down here where I had this number 10 earlier and run
this entire cell.
If I check out why I was quatrain on a scroll that shape I can see that it is now at a rate of 60 thousand
labels but for each label I've got this one hot encoding so I've got 10 columns and one of these columns
will have a one at the position that corresponds to the label and this is in contrast to our flattened
array that we had earlier.
That was much more sparse.
Of course we have to do the very same thing to our test labels as we've done to our training labels
to be consistent.
So we'll write y on the score test is equal to Pete dot identity.
E y e parentheses in our classes y underscore test and now r y underscore test dot shape should also
be a 1 Hot encoded array.
In this case LP 10000 thousand by 10 the last thing that we'll do is we'll create our validation data
set.
So once again I'll add a subheading here that reads create validation data set from training data.
What we want to do in this case is we want to split up our training data into our validation data set
and the actual training data set that we're gonna use.
The first thing that we'll do is we'll decide on the size of the validation data set I'm going to come
back up to my constants and I'm going to add it up here.
Validation on the score size and we'll set that equal to 10000 same size in this case as our training
data set.
I'll let shift enter on the cell and then I want to throw this over to you as a challenge.
Once again this will be some good review for race slicing techniques.
What I'd like you to do is to split the training data set into four smaller datasets namely the X underscore
Val the Y underscore val x on escort train and y on escort train.
Make use of the constant that we've created above so that you end up with the validation data set of
10000 and a training data set of 50000.
I'll give you a few seconds to pause the video before I show you the solution.
Here's how you do it.
X and a score Val is equal to x on the squat train on a squirrel square brackets.
Colon validation size the same for r y underscore Val.
So the first 10000 values in the large sixty thousand sample training dataset.
Now let's do the other bit.
So X on the squat train shall be equal to X on a squat train and a square all square brackets and then
I'll be the last 50000 so it'll be everything from the 10000 sample onwards.
So validation on this exercise.
Colon closing square bracket and the same goes for the Y levels in the training dataset.
Let me hit shift enter on this.
And now when I pull up X on a squat train shape I shall see that it's fifty thousand by seven hundred
and eighty four.
An X on a score vowel don't shape it's gonna be ten thousand by seven hundred and eighty four and that's
it.
In the next lessons we're going to be busy setting up tensor flow and setting up our neural network
architecture for all that and more.
How see in the next lessons take out.