Okay great.
Now we have a little help of function that not only returns us our images in the form of tenses market
processes the images for us but it also returns the labels so we can get it back in a topple form.
So let's utilize that to write another function that's going to turn our data all of our data because
this only does one sample.
We need a way to turn all of our data so our x and y our data and labels into batches of size 32.
Right now we've got let's write that down.
So we're trying to communicate to ourselves here tower notebooks a way to turn our data into topples
of tenses and we go here in the form
image label.
Let's make a function to turn all of our data.
X and Y into batches All right.
Wonderful.
Now again if you're wondering where all of this came from a great place for any kind of data loading
is to read through the documentation.
Now it could be a bit boring sometimes.
But trust me reading through even if you don't understand the first time I didn't understand the first
time read through it.
Try it out try it out again and see how you go this is gonna be a big dog of a function.
Oh that's fitting because we're working on dog vision but it's gonna save us a hell of a lot of time
later on.
So let's do it.
So we need to define the batch size 32 is a good start.
Remember tensor flow.
Defaults to 32 a lot.
So if you don't set it somewhere it'll probably default to 32 but we're going to define it anyway.
So let's create a function to turn data into batches.
And now we're going to write through this function together and it's gonna do a few little things differently
depending on if we want a training data batch of validation data batch or a test data batch because
remember these are our three sets there.
So these are our three sets.
So we've got a training set.
We're gonna turn into a training batch of validation batch and a test batch.
All right.
So we want def create data batches we want it to take in an X and a Y and actually Y can be none.
Now I want you to think about Y Y might be none because Y is our labels.
This will probably make sense in a little bit but just have a think about why might we want to create
a data batch with No Labels valid data equals false and test data equals false and we'll leave ourselves
a little doctoring here.
I just to have a colleague called Ron and he told me a technique called Rubber ducking he was a lot
better coder than I was and he said if you're having trouble figuring out how to do something in code
write it out in language first or say it to yourself in language as if you're talking to a rubber duck
and rubber ducks don't understand English very well so you have to make it very clear.
That's a great tip from Ron.
Thank you Ron.
So this function creates batches of data out of image X and label Y pairs
it shuffles the data if it's training data so mixes the data around because if there were some inherent
order into the training file that we downloaded probably not because these look pretty jumbled up we
don't want our model to remember that order we want it to be as adaptable as possible no matter what
order an image comes in but so if a training data shuffle but doesn't shuffle if it's validation data.
Now I want you to have a think about why it might not shuffle if it's validation data.
Don't worry we'll cover that but just have a think about it for the meantime also accepts test data
as input.
Now here is why why might be none.
Because test data doesn't have labels of course
what we want to do is build a function that kind of does the same thing but has some if clauses or e
left causes if it's valid data or test data so we know what we kind of want to do and we've got some
conditions here that if it's valid or test state.
So let's start off with the test data because I think that's the least amount of things.
So if the data is a test dataset we probably don't.
I was about to draw I definitely had labels where we know we don't have labels if it's a test data set.
So we're gonna if test data print let's give ourselves a little little head's up I'll give myself a
little head's up what's going on.
Creating test data batches just to make sure that later on hey maybe we're making some data batches
and we've accidentally passed a false parameter to test data and it ends up doing it differently for
the test data.
We don't want that.
We want to know that it's making test batches so data equals T F data a dataset so this is the module
in tensor flow data.
We're making essentially a batch data set and we're gonna.
This is a little interesting one here from tensor slices.
So create a data set a.k.a. just a tensor flow data set whose elements are slices of the given tenses.
So basically what this says is pass me some tenses and I'll create a data set out of that little confusing
I know but this is just how it's done intensive TAF constant x.
So basically what this says is I'm going to create a tensor flow data set from the tensor slices X and
because we're passing X to T F constant it turns X into a tensor so this is only file parts.
No Labels okay.
And now data batch.
Here's where we turn our tensor of low dataset because right now this would turn all of X into a data
set.
We need to turn it into a batch size of 32 so equals data dot map.
We need to process the image and we need to turn it into a batch batch size.
Well a lot of stuff going on here.
Daniel what's happening.
Well we've talked through the line above.
But this one all it's saying is basically take this data which is intensified dataset made of x which
is just file names in the form of tenses and then map our process image function See this is why we
made it into a function so we can call it process image.
It's gonna go through all of these steps here on our file names.
So it's going to import them turn them into normalized tenses and then turn it into a batch of batch
size.
Let's go here to intensive low data batch.
This is the workflow I use whenever I'm looking at a function.
So if we go batch There we go batch combines consecutive elements of this data set into batches There
we go so if we pass it a range of eight and we want it to be a batch of three it's going to split eight
into three batches.
So that's just exactly why is this in a new tab.
That's what it's gonna do here except as is of size 32 wonderful.
So that's the test data.
Now we want to do.
If the data is a valid dataset we don't need to shuffle it.
Now I want you to think about again why would we not need to shuffle it.
It's okay if you're not entirely sure.
I got caught out on this.
A lot of times too.
So we'll print a little progress statement here and then we'll do much the same as this we could just
copy that but we're going to practice writing it out TAF data dataset from tensor slices.
Now this is where because our validation set has labels we're going to posit a couple of X and TAF constant
y.
So just turning out file names and labels into tenses and we got here.
File paths labels so now we have a tensor flow dataset
of image file parts and labels because it's a validation dataset.
The validation dataset has labels and we're going to turn it into a data batch equals data dot map just
the exact same as above process the image and turn it into a batch of batch size help getting trigger
happy wonderful and then return data batch.
And finally we want if none of these work well a valid data is false and test data is also false it's
obviously going to be a training batch so we're going to go create training data batches on the lump
and let's go.
So if it is a training dataset we want to shuffle it.
That's the only difference here.
So we want to go turn file paths and labels into tenses.
So just as we done with the validation we could just copy this.
Now there's probably a more efficient way to write this function that does the same functionality but
I've tried to write it in a way that is fairly understandable as to what's going on.
Constant X and as I said that I might have said that it's fairly understandable but if you're coming
across something like this for the first time trust me when I did I was confused.
But once I went through it step by step.
Remember our technique for breaking down a function.
Write it out line by line and then check out what it's doing in each line.
So we want to shuffle the path names and labels before mapping will go into this in a second before
mapping.
Image Processor function is faster than shuffling images I'll just write the code so we can talk about
it but data shuffle emphasize so buffer size just stencil.
How many variables do we want to shuffle then we want to shuffle the whole lot.
Well a lot going on here.
While all this is saying is hey take this data which is a tensor flow dataset of the data so the file
names and the labels and then shuffle it up and we want to shuffle the number of examples.
If we only shuffled 100 it would take the first 100 examples of x and y and shuffle them but note we
want to shuffle them all.
And the reason being you might see other tutorials shuffling later after we run the map function it
goes data map process.
Image so if you did shuffle after you've already processed the image it takes a lot longer to shuffle
a full image than it does just a file name so as I said remember we're trying to minimize the time between
experiments.
So we want to try and get any speed up that we can.
I've just realized we've made a little mistake up here in our validation dataset because we have images
and labels we need to map the function get image label which also processes our image whereas for our
test data set because we have no labels we can run directly the processed image function so let's let's
replace this get image label and it's going to be the same
for our training dataset.
So get image label wonderful and it might just write out what this is doing with a little little comment.
So create image label troubles.
This also turns the image path into a pre processed image so if you can shuffle your data when it's
in its smallest format a.k.a. in our case file names rather than full images.
And now finally we want to turn the training data into batches so data batch equals data top batch batch
size and then we'll go out here.
Return data batch hoof.
That's a bit of a behemoth of a function but as I said this is gonna save us a lot of time later on
and you'll see that remember this is the kind of development work style I want you to start thinking
about.
Hey am I going to reuse some functionality later on.
Well if I am I'm going to ride it into a function and you probably already know that because you're
a lot smarter developer than what I am.
I used to just fluster around and just write things line by line.
Now let's test our function by creating training and validation databases so we'll go train data.
We're gonna use our big dog function just above we're gonna pass it X train for the training data and
why train for the validation data and that's all we need.
And then for the vow data we could go create data batches X Val y Val is is it valid data equals true
and now fingers crossed we should see our print progress messages.
Yes that is amazing.
Okay now let's check out the different attributes of our data matches so if we go train data it's now
in the format of a data batch which is tensor flows preferred way of processing things.
There we go.
This is our training data.
The first one here what's it saying.
So you've got shape none 2 2 4 2 2 4 3.
This is our image.
None is for batch size.
I know we've set it to 32 but it's going to stay as none because batch size remember is flexible.
So this is just our images are now in batches of 32 and they've got a shape of 2 2 4 2 2 4 3 height
width color channels of type T float 32 and we've got a Tupperware here of our label.
So these are image label pairs in the form of tenses and our labels also have a batch shape of none
because batch size is flexible and they have a dimension of 120 which is because if we go y 0
there's 120 different dog breeds and then it's the same again for the validation data beautiful.
So that was a pretty in-depth one once this video is over which will be in a few seconds.
I'd go back through and just read through what's going on in this function here.
If you're not entirely sure I'd look up loading and pre processing data tutorial of images.
Have a read through that and then the best way to learn is to really just write this out break down
the function and try to run a line by line.
That's how I break down things for myself but the way our data is at the moment it's still kind of hard
to understand.
If you're new to the concept of batches it's really a very difficult concept to grasp.
It's perfectly okay not to know what's going on right now.
So to help ourselves understand rather than just having this this gobbledygook being printed out when
we're trying to check out what's in our data batches let's write a function which is going to help us
visualize what's going on.
So that's what we'll do in the next video.