Welcome back.
Since the last few videos we've been spending a fair bit of effort getting our our data into an accessible
format.
Now we've got that.
So we've got let's just quickly have a look.
We've got boolean labels which is our labels in numerical format.
We'll have a look at the first two so it doesn't take up too much room Navy game and true and false.
We can access those as zeros and ones really easily and the same goes for our file pods.
So these are our training images here and we left off discussing how Kaggle doesn't provide us a validation
set
so creating our own validation set.
So since the data set from Kaggle doesn't come
with a validation set we're going to create our own.
And remember from way back in machine learning when I won the most important concept in machine learning
is a three sets okay.
The training set the validation set and the test set.
So right now we've got the training set and a test set but we'd like this validation set which is okay
the practice exam.
So to reiterate we're trained in machine learning model on the training set.
We'll evaluate it on the validation set to see if our experiments that we've been doing on the training
set are going all right.
And then finally the final exam is our test set.
So we'd like this little intermediate set to validate our initial experiment.
So that's why we're gonna make a validation set and now to do so what could we use.
What have we used in the past.
I want you to think about this.
I was in the socket line section if you said try and test split you'd be correct.
Now can be a bit confusing because it says try and test split but really the functionality of the function
is just splitting data into two different sets one set of certain size and another set of another size.
Let's set that up.
But first what we might do is set up x and y variables.
Now the reason we do this is because having to remember bullying labels and file names is a bit tedious.
So what we might just do is create X which is our standard variable for your data and Y which is usually
the standard variable for your label.
So we're just gonna make copies of those beautiful and now since we're working with 10000 plus images
ten thousand two hundred and twenty two to be exact.
It's a good idea to start working with a portion of them first to make sure things are working before
we go and try and a machine learning model and all them.
This is a thing to note for your own projects is that when you're doing experiments remember what's
our goal.
It's to minimize the time between experiments so we can figure out what works and what doesn't.
And if we're using ten thousand examples every single time that's going to take a while.
And if you've never trained a machine learning model or a deep learning model on ten thousand plus images
so you don't have a perspective of how long that takes you can take my word for it that it would take
a long time so it will probably reduce it down and start with about a thousand images to begin with
and then increase it as we need.
So let's write a little note for ourselves we're going to start off experimenting a thousand images
and increase as needed.
Now this goes for not just images but for all kind of machine learning projects.
And we've seen this in the past using some samples of our data maybe working on a text classification
and you had a hundred thousand emails and you wanted to classify whether they were spam or not spam
you might only start with five thousand and then build up from there.
So let's set a little parameter so set number of images to use for experimenting.
Now I want to show you something really cool within collab.
So we'll create a little parameter here and now it's in capitals because it convention you might see
here and there with with deep learning projects is that type of parameters.
So things that you can set as a user are often set in capitals.
That's where you might see these variables that are in full full capitals.
So let's go a thousand and then in collab lab it has this little little cool tool that you can it's
like a magic function at parameter.
Now I'm still working these out.
So maybe you can leave a little suggestion on where I can find out how to make more of these.
If you figure it out because I only know a couple.
Now watch this.
So what we've done is we've just gone.
Hey this is a parameter that we can set create a type slider the minimum is a thousand Max is ten.
And this step is a thousand num images has now just appeared.
Look if we go like this.
Oh yeah 5000 4000 eight thousand eight thousand well I could play around with that all day.
That's something that collab has over Jupiter aside from being backed by a GP offering all right.
So now we've got a little parameter.
We've got X and Y.
Let's use psychic learns try trying to split so let's split our data into train and validation from
model selection import train test split.
Now again this is called train test split.
But the basic functionality of this function is to split.
You could do this manually by using indexing but we'll just use a function to help us out.
Is just a split data into two different sets into training and validation of total size num images which
is this variable we've set up here.
Run that sell X train X Val this time not X test y drain y Val equals train test split X up to num images.
So it's only going to take the first thousand file names because remember X is just our file names we
have an import about images just yet because it's a lot faster to work with just file names and Y is
our labels in boolean form
and test size.
Again this is just validation size will be 20 percent and then the random state is kind of like setting
random seed.
While it is like setting random say to 42 so random state equals forty two is the same as going any
random seed forty two and if you're not sure what any random seed does.
There's a video on that.
Otherwise you might want to just quickly Google what does any random say do stuff go land X train just
to check all the the lengths of our data are correct because you wouldn't believe the amount of times
I've been burned because my data was the wrong shape you will spend the hours battling to get your data
into the right shape.
Trust me I've been there all right.
Wonderful.
So we've got a thousand samples and we've got a 20 percent test size so that means there's gonna be
800 samples in the training split and 200 samples in the validation split.
So let's just have a quick look.
Let's have our gaze at the training data so x trying just to verify that we're still on this earth we're
still working with the right stuff.
Actually that's a bit too many.
I've got to remember that these are full blown boolean labels.
So there's a lot there.
So train we've got file pods correct.
That is what we want.
And training for labels we have bullying labels.
That is what we are after.
All right.
So now that we have our data into a training and a validation set a small subset.
Mind you.
Because remember our goal to begin with is to minimize time between experiments what we can probably
finally do is turn out images or at least create some functionality to turn out images and labels into
tenses.
Let's do that in the next video.