Hello, and welcome to Managing Data Pipelines.
This lesson is going to be a bit of a kitchen sink lesson,
and in that, I mean, we are going to take
some of the remaining concepts
that we haven't talked about with Azure Data Factory
and we're going to be going through those.
Specifically, we're going to be focusing on
monitoring pipeline and trigger runs,
and then talking about integration runtime,
including an understanding of the basics
of what integration runtime actually is.
And then finally, we'll wrap up
by talking about how to set notifications
and understand the basics of what that is as well,
and we'll just touch on that lightly
because we've talked about it a little bit
in some other lessons as well.
Now, one thing to keep in mind
for the Azure Data Factory lessons in this section,
almost all of those apply to Data Factory,
as well as to Synapse,
and I'll show you the Synapse version as well
and we'll kind of talk about the differences there.
So keep in mind, as we talk about pipelines
in Azure Data Factory,
most of those concepts are going to port over
to Azure Synapse as well.
All right. So with that introduction, let's get started.
All right.
So we're going to be looking at the monitor section
in Azure Data Factory,
and there's 4 main components to this.
First is dashboards.
And so we'll talk about a quick overview of Data Factory,
and basically that gives you,
your 50,000-foot view of what's going on
so that you can see if there's any alerts,
and we'll see what that looks like.
Next up, you have runs, and runs is going to be
a breakdown by detailed activity,
as well as detailing failure points within your pipelines.
And we can also break that down and look at trigger runs,
as well as just general pipeline runs as well.
So runtimes and sessions.
So this is going to include integration runtimes
and data flow debug, and this is a good time to talk about
the basics of what integration runtime is.
Integration runtime is just the compute infrastructure
that's used by Azure Data Factory
and Azure Synapse pipelines.
So whenever you use data flow
or you're using your copy data for data movement,
or you are running activities, activity dispatch,
so for using things like Azure Databricks
or HDInsight activities,
that would be something that would be used
by integration runtime.
So when you think about integration runtime,
just think about that being a compute
that is used for Data Factory and Synapse.
So then finally, debug mode.
This is just a data preview mode that we can use
in data flow for Azure Data Factory,
and allows us to do some end-to-end testing
on our data flows.
And then, last but not least, notifications.
So we can set up alerts and metrics,
and we've talked about that in some other lessons
so we're not going to spend a ton of time there,
but that is also a component as well,
and I'll show you that.
And let's go ahead and actually do that now.
I'll jump over into the portal
and let's talk through some of these concepts.
So here we find ourselves in the Monitor section
of Data Factory, and you can see here at the top,
we have our dashboards, which is the default view,
and I have intentionally
broken a pipeline for you
so you can see what happens here.
It's going to give you a list
of all the pipelines that have run.
You can see a pie chart breakdown, or a line chart,
and you can also see activities as well.
So if we have activities,
it's going to show you the failure for those,
as well as for the pipeline.
And so you can see here that we have a pipeline
that's got some issues, and if I go ahead and click on that,
it's going to take me down one section to our runs
and it's going to show me my pipeline runs.
So I can actually see that my pipeline failed,
and if I click on Error, it's actually going to give me
exactly what happened.
So you can see here that I have a missing data table,
and so it's not copying the data like it's supposed to.
And so, on our manual trigger, we had a failure.
So that is how we can see our pipeline runs,
and that's also our dashboards.
And then, while we're in runs,
we can go ahead and click on Trigger here.
And if I had scheduled triggers, I can also see that.
So I can see my clock, my wall clock scheduled triggers.
We can see my tumbling window.
We can see my storage events,
and then if I had any custom events,
we could see those as well.
So you can break down all of your trigger runs,
and it's going to look just like the pipeline.
It's going to show you what succeeded and what failed.
And of course, you could always click on it.
You can try and rerun it.
You can see more information if you need to.
Next up, we have our integration runtimes.
And so you can see here that it has spun up and is running
an autoresolve integration runtime.
Now, when we do integration runtime,
there's 3 different types that we can use.
There's Azure, self-hosted, and something called Azure SSIS.
So, most of it is going to actually be just standard Azure,
and this is going to be something that is managed by Azure.
In most cases, it's going to spin up or spin down
so we don't have to mess with that.
And you can see by clicking on Azure here,
that's where that lives.
Obviously for this course,
we're going to be using just Azure.
So we aren't going to have a need for SSIS or self-hosted.
That would most likely be covered in another exam.
So for the DP-203, I'd really just focus on the Azure
and understanding what integration runtimes actually is.
I talked about Data Flow Debug as well.
Finally, we have Alerts and Metrics.
So you can see here, we don't have any alerts.
If I wanted to create one, I could just click on that
and go through alert rules,
and we actually cover that in another lesson.
So I'm not going to focus on alert rules in this lesson,
but that's where you would find your alert metrics.
And then, last but not least,
before we jump back in and wrap everything up,
I have also opened up an Azure Synapse Analytics instance,
and I pulled up the exact same Monitor section.
And you can see here, it is very, very similar.
So we have our integration with pipeline and trigger runs.
We have Activities,
and we have SQL Pools or Analytics Pools,
which is going to be something specific to Synapse.
So it's actually very, very similar,
and you can see even the layout here is very similar
between these 2 instances.
So keep in mind, that is, the concepts
we talk about for Data Factory could also apply
to Synapse on the exam.
All right. With that, let's jump in and do a review.
First off, I want to point out the facts on the right here,
you are at the very end of a very long section.
Congratulations, you have been doing awesome,
and not only that,
this is a good milestone that marks near the halfway point
for this course.
So, congratulations there as well.
For the review, you just need to know what's possible.
So we talked about the portal,
so you were able to see the layout in the portal.
Keep that in mind. That's going to be important for you.
Make sure that you have the basics of integration runtime,
again, just compute infrastructure
for Data Factory and Synapse.
It's probably going to be enough,
but you can also go in and just go a little bit further
by talking about self-hosted, Azure, and SSIS.
If you've got those 2 things down,
the last bit is just notifications.
If you want an alert email on a failed run,
can you do that?
You should be saying yes, you can do that
by going into that monitor tab and choosing a new alert.
If you've got that down, you're good to go
and I'll see you in the next lesson.