Welcome back.
Hopefully, you've taken a moment
and you have gone to the bathroom
and gotten yourself a snack, and you are all settled in
and ready to start Part 2.
So in this part,
we're going to dive in and I'm actually going to show you
an example in the Azure portal.
One thing to think about as we go through,
don't get too bogged down in the details.
Really focus on that process of what we're doing
with incremental data loads,
which is, looking at our watermark,
deciding from there
how we're going to figure out what's new,
copying the data, and then finally updating our watermark.
And in this example,
all of that's going to be done in the portal.
So you're not actually going to see the watermark selection
other than when we dynamically choose
what our file structure is going to be.
That's actually where
all that watermark discussion would take place.
So, hopefully that helps.
And with that, let's go ahead and dive in to Part 2
and see this example in the portal.
So let's go ahead and jump over into the Azure portal.
So, here we find ourself in the Azure portal
and I am going to show you
how to do this incremental data load.
So, we are in Data Factory and in the Data Factory studio.
So, let's start off by clicking on Ingest
to copy data at scale.
Now, since we want to do an incremental data load,
we're going to go ahead and just select
a tumbling window to get us started.
Click on Next.
And I'm going to show you how to create this.
So, let's start off by picking our source data type.
And I'm just going to pick a Blob storage account
that I had set up.
And then, you can see here under Options,
we have File Loading Behavior.
So, I can load all files, which is that full restoration.
Or I can come in here and I can choose an incremental load,
which is what we're going to do.
We're going to pick this last modified date
incremental data load
and go ahead and click on,
oh, I got to pick an output folder here.
So, let me go ahead and just browse
and pick a folder for us.
Again, it doesn't matter too much right now
because I'm just more explaining
how to do incremental loads.
And then, we also want to choose Binary Copy.
And again, this isn't as important basically.
We're going to skip schemas at this stage because again,
I just want to kind of walk through the examples
of how to do incremental data load.
So, go ahead and click on Next.
And now, we are ready for our destination datastore.
So, we've defined the source,
and now we're going to define where it's going to go.
So, we're going to go ahead
and just pick a blob storage account again.
And now we can choose a folder path.
So, this is how we can actually go through
and create an incremental data load based upon date.
So, I could create in my blob storage a new folder
for year and month.
And every month, a new file, a new database, gets loaded in
that's going to be under that year/month format.
And so, I would just simply choose that
by again going to browse,
picking the folder where it's going to go to,
and then I would add in
year
and then month.
Let's just do that.
If I can type, there we go.
So, what it's going to do,
you can see it gives us this year and month format.
So, what this is asking me is in my blob storage,
it's going to go out and dynamically look for a year
in the format of 4 numbers.
So, 2021, and then a 2-digit month.
So, 2021/09 as an example.
So, it's going to pick the folder that I chose,
and it's going to look to put the new file into that.
So in this way, we can do an incremental data load.
It's going to look for the last folder that was updated,
compare that to the latest folder
that's in the blob storage,
and it's going to incrementally load
all the way up to that point.
If I click on Next,
I can then go through and define my settings,
and then I'm ready to review and submit.
So, finish, and you can see here,
it's going to copy from one Azure blob storage
to another Azure blob storage
using that incremental data load.
Alright, so that is another quick example
of how we could use incremental data loads.
So in review, we need to understand first,
what is an incremental data load?
It is a way of uploading the new data
that exists from our folders or files so that we can
update our database without restoring the entire thing.
Service that is going to be used
for incremental data loads in Azure.
Most commonly, that's probably going to be Data Factory.
There are some other ways you can do that,
but again, I would look at Data Factory for that.
And then finally, what is the process?
So, you need to understand the importance of watermarks,
be that a date, or be that a customer ID,
and how we can use watermarks
to do an incremental data load.
Essentially the process,
whether I'm going to the Azure portal
or whether I'm doing it manually,
is going to be looking at our new watermark,
looking at our last updated watermark,
comparing the difference, and then incrementally loading
all of the data between those 2 watermarks.
And then, we finish out by updating our watermark
so that we have a new point of reference.
That is the process for doing incremental data loads.
Alright, so that is it for this lesson.
We will talk more about watermarks
in the next section as we talk about streaming.
But for now,
let's go ahead and leave the concept of watermarks
and move a little further down our Azure Data Factory
and batch processing journey.