-: In this lesson,
we are going to talk about how to measure the performance
of data movement.
In order to do that,
we're going to start at the beginning
by talking about what that actually is.
Then we're going to jump in
and we're going to talk about estimating throughput.
Follow that up by taking a look in Azure Data Factory
with the Monitor Copy Activity
so you can see how all of this comes together.
But let's start at the beginning.
Measuring throughput is most likely going to be encountered
in Data Factory or Synapse Analytics.
This involves large-scale data transfers
by using the copy activity in pipelines.
So what we're trying to do is we're trying to optimize
how long it takes data to move
from one place to another using pipelines
in Data Factory or Synapse Analytics.
Now doing this, we're looking at serverless architecture,
whether it's Data Factory or Synapse,
both use serverless architecture.
They also use parallelism,
so this means we can have multiple streams
working at the same time.
And then finally, full utilization.
So both of these pipelines fully utilize network bandwidth
and they fully utilize your input/output operations.
So we don't need to consider that
when we're looking at the copy activity.
So to sum all of that up,
basically what that means
is we can use this bandwidth chart
to figure out roughly how long it's going to take us
to move data.
So for instance,
if I had a bandwidth
of 50 megabits per second
and I had a data size of 1 gigabit,
I would expect that it's going to take me about 3 minutes
in order to move that data
if everything is optimal.
We jump all the way over,
and let's say I have a 1 gigabit per second bandwidth
and let's say that I'm trying to move a terabyte,
well, that's going to take me 2.3 hours,
and so on and so forth.
So we can use this chart,
because we have the full utilization
and we have the parallelism
of Data Factory and Azure Synapse pipelines.
So once we have that down,
now we need to go in and see what's actually happening.
And so to do that, let's jump into the portal
and let's take a look at the copy data activity
in Data Factory to see some actual performance.
So here we are in Data Factory,
and I built just a single, one-step pipeline
that I just triggered
so we can take a look and start to see
how long it takes to move that data.
So in order to do that,
let's go ahead and jump over to Monitor
and let's take a look at our pipeline run.
So you can see that our pipeline run is successful,
and it took a whopping 12 secs.
But let's dive in a little further
to see if we can figure out what actually happened.
So you can see here we have our copy data activity,
and it copied data with a duration of 9 seconds.
If I click on my handy-dandy little glasses there,
it's actually going to pull this up.
So you can see here
that it moved a whopping 4 kilobytes of data.
It gives me my copy duration,
and then it gives me actual details
on the parallel copy.
So we have one parallel copy that was used.
It tells me also, in the duration, this is the queue
and this is the transfer.
So I can see how long it took at both steps.
So from that, I can go back here
and I can compare my throughput chart with this chart here,
and I can see exactly how long the copy activity took.
And then I can see how long it should have taken
if everything was optimized.
And then from there, I can do optimization techniques
in order to hopefully make up some of that slack
if I'm seeing a difference in what's expected
versus what's observed.
So the key points to remember from this lesson,
one, we're going to estimate
how long the data should take to move.
We're going to compare that in Data Factory or Synapse
to how long it actually took,
and then we're going to optimize as needed.
Keep in mind that full utilization
allows for that throughput estimation,
so I can use that chart
to figure out what's different
and I don't need to worry
about things like bandwidth or IOPS.
Finally the Monitor Copy Data activity
is what we're using to measure that performance activity.
Pretty simple,
if you understand those 3 concepts,
you should be good to go on the next lesson
and you should have a pretty good handle on
if you're moving data,
how long you should actually expect it to take.
With that, I'll see you in the next lesson.