So let's take a few minutes to talk about
designing and configuring exception handling
in Data Factory.
In this lesson, we're going to be talking about
how we design and think through
our exception handling strategy.
We're going to take a look at what is actually possible.
And then we're going to talk about
how we implement that strategy.
And then of course, as with most of these,
we're also going to see it in action in the Azure portal.
So with that, hey, let's get started.
First up, a few key considerations to think about
when we design an exception handling strategy.
We need to think about the activity section,
we need to think about fault tolerance,
and we need to think about retry attempts.
So when we build a pipeline in Data Factory,
we can choose what happens upon the completion
or failure of one of those activities.
So we need to think about what we want to have happen
as we build our pipelines.
Fault tolerance, another thing.
So as we look at our fault tolerance,
as we're copying data across a pipeline,
what happens when part of it fails?
There are ways to continue on
and we'll look and see what those are.
And then finally, retry attempts.
We always need to think about retry attempts
and the pluses and negatives
that go with setting a retry strategy.
With activity, we've got 4 different types.
We have success, failure, completion, and skipped.
So success, when you see a success dependency,
what we're doing is, the second activity only kicks off
when the first activity succeeds.
So we don't start the second activity
until the first one succeeds.
Failure, if we add a failure dependency,
the second activity only executes
if the first activity fails.
And we're going to go through and I'll show you all of this
in the portal in just a second,
but these are the definitions that you need to know.
Completion, so if you do a completion dependency,
the second activity only executes
when the first activity completes regardless of its status.
So fail or success, it doesn't matter.
As soon as it completes,
it's going to kick off a completion activity.
And then finally, the skipped.
So if you have a skipped dependency,
the second activity only executes
if the first one isn't executed.
So next up, let's talk about fault tolerance.
So as you look at fault tolerance,
the question you need to ask yourself
is what errors can we successfully ignore?
And we'll take a look at that as well here in just a second.
And then finally, retry.
We need to define the retry attempts and time limits.
And one of the things you need to think about
as you set up the retry, if you have a pipeline running
and you set a retry of 10, as it retries,
it's going to be holding up your entire pipeline,
and so you need to think about those time limits
and what that actually means for your pipeline
because you don't want to cause failure
or further breakdown of the pipeline later on downstream
because you've set retry attempts and time limits too high.
So that's the one side,
and the other side of that is
if you don't set it high enough,
you don't want to stop or break an entire pipeline
because of a 5-second retry attempt.
So just some things to think about
as you build those pipelines.
But with that, let's go ahead and hop into the portal
and take a look at
what some of these things actually look like.
So I've opened up a pipeline in Data Factory
and I want to walk through
some of the activity dependencies first.
So I'm going to grab a Copy Data activity,
just drag that onto the canvas.
And let's talk through
some of the things you need to be setting.
First, down here in the configuration panel,
we want to, of course, set our source and sink
and all of those things,
but from an exception handling policy,
what we really want to look at is here in Retry.
So you can see our retry here
and we can set our maximum number of attempts.
So I could say 10 attempts,
and I can set a retry interval of 30 seconds,
which means between each retry,
it's going to give us 30 seconds,
which means that we would be spending 300 seconds
or 5 minutes on this retry of this one copy activity.
Moving a little further on, down here under Settings,
this is where we can set our fault tolerance,
and so when we select our fault tolerance,
we're going to click here
and we can choose to skip incompatible rows
or skip missing files or forbidden files
or files with invalid names.
And it depends on how we set this up, of course,
so some of these are going to be grayed out
depending upon what sources we've selected to move,
but this is how we set our fault tolerance.
I would also recommend coming down and enabling logging.
If you look at the logging,
this allows us to log copied files, skipped files,
skipped rows, and so we can specify a storage connection
and then we can choose the settings there,
but I think that's something that's also helpful,
especially as we're starting to run pipelines
because it'll give you a better feel
of what's actually happening as the pipeline runs.
Alright, so the last thing we want to talk about here
in the actual portal is the activities.
So here on this little plus button, if I choose that,
I can add activities on success, failure, completion,
and skipped.
So let's say this is the first step in my pipeline
and I'm going to copy data over.
After that data copies over,
let's say that I want to kick off a Databricks notebook.
So I can drag this activity up here.
I can choose an activity on success.
Drag that little green box and pull it up here,
and now, this is my success activity.
So if the data successfully copies,
it's going to then kick off this notebook
and it's going to run the notebook.
I can also add a failure activity.
So let's say that if I have a failure,
I want to have an Azure function kicked off
that does something else.
I can drag this over here, and again,
drag the square over to the new activity.
And so now, I've got 2 different things going on.
I'll have a copy activity that runs.
If it's successful, it kicks off a notebook.
If it's not successful,
it's going to kick off an Azure function
that could alert me,
or could do, I don't know, we could build it
to do a lot of different things with a function,
but it's going to do something else
to help us out in our pipeline.
I can also come in here, and let's say
that I want to have a webhook kicked off as well,
and I want this webhook to go,
and I want it to go whether this activity succeeds or fails.
So I can come down here to Completion, grab the blue box,
and bring that over here.
And so now, you can see that I have a pipeline
that's going to do something on success, failure,
and regardless of the status, succeed or fail,
it's going to kick off this webhook.
And then finally, our last one is our skipped.
So let's say that off of this pipeline,
I don't know, let's just say we had a lookup here.
And from this notebook, I could say Skipped.
Drag this here.
So if this notebook doesn't get kicked off
because this doesn't succeed--
so let's just say that we start this pipeline,
the pipeline fails,
it comes down here to this Azure function,
if that's the case, this notebook never runs,
and if this notebook never runs,
it's going to kick off this skipped activity
because it never ran as the pipeline went through,
and so we would run this lookup activity.
So those are the 4 activities that you have.
I would suggest that you actually take a whiteboard
and map out all of the different steps in your pipelines
and think about what do you want to happen,
whether that's email notifications or webhooks
or Azure functions or notebooks or further copy activities.
Whatever it is, I would suggest that you map all of that out
because it can get quite complicated
if you have a fairly large pipeline.
So with that, let's jump back in
and talk about implementing our strategy.
So first stop,
think about what are the steps in your pipeline.
Map out those main steps in your pipeline.
Then second, I would recommend thinking about
where notifications are important along those steps.
Where do you need to receive emails on success
or failure or completion?
Where do you need to know what's going on?
What happens at an outright failure?
So if a step completely fails, what do you want to do?
Do we stop the pipeline entirely?
Do we do notifications?
Do we retry?
Do we kick off a webhook to do something else?
What are we going to do
if we have an outright failure of an activity step?
Has general troubleshooting been implemented?
Now, we'll talk about troubleshooting in a later section,
but you want to think about troubleshooting as well,
and if you go through the activity steps
and you start to have issues,
you want to be thinking about, why am I having these issues?
Is there maybe a problem with the source file
that I'm trying to implement?
Am I having a problem with an event hub
or the data that's getting ingested
into my blob storage, for instance?
Those are the kinds of things that might be causing trouble
in your Data Factory pipeline
that really doesn't have a whole lot to do
with the Data Factory activity.
So be thinking about that.
Next, think about optimization.
We'll talk about that in another section as well,
but thinking about all of those activity steps,
have you gone through and optimized each of those steps
to make sure that your pipeline is robust
and it's running as efficiently as it possibly can?
If you've done all that,
we are set up and ready for a review.
So first, look, have a plan.
It's really important that you think about
your exception handling strategy and your pipeline strategy,
and that you have an actual plan mapped out
rather than just going into Data Factory
and grabbing a few activities
and dragging them onto the canvas.
Once you have that plan, implement your strategy.
Make sure that you've implemented
and that you've tested the strategy,
and you have a really good feel for what's going to happen.
For the DP-203, you need to understand what's possible.
So if you've gone through
and you're thinking through those steps of optimizing
and troubleshooting and building in alerts
and what happens on failure, if you're doing that,
you should have a really good feel, not just for the DP-203,
but honestly, for your career as a data engineer.
So make sure that you're thinking about what's possible
as you go through
and design your exception handling strategy.
That's it for this lesson, I'll see ya in the next.