Hey, what's up, Gurus?
In this lesson, we are going to be talking
about an introduction to Azure Data Factory.
Specifically, I'm going to teach you what Data Factory is.
We're going to talk about core concepts that you need to
know, and we're going to talk a little bit
about Data Factory architecture, so you can start to see
how all of these concepts play together.
So let's start with the Data Factory introduction.
So when we talk about Azure Data Factory,
Microsoft says it's a cloud-based data integration service
that allows you to create data-driven workflows
in the cloud that orchestrates and automates
data movement and transformation. To summarize,
we're going to create activities that moves data,
and helps to complete your cloud projects.
So when you think Data Factory, think pipelines
that help connect all of the different components
of your cloud service together.
The other piece to think
about is data pipeline orchestration.
So not only are we connecting those pieces,
but we're also helping to orchestrate the movement.
We're controlling the flow of the data, not just creating
the infrastructure that allows that to happen.
And we'll talk more about this in a few minutes,
but those are the 2 big concepts for Azure Data Factory.
So when we talk about core concepts for Data Factory,
the first is a pipeline.
So the pipeline here is a logical grouping of activities,
and these activities perform a task.
For instance, an activity could be data movement.
So we could have activity that takes an ingestion source,
or takes a source from your computer, for instance, pulls it
into the cloud, and puts it into a data lake.
That would be an activity,
and the task it's performing is data movement.
So that is what an activity is, and then a pipeline is
a grouping of all of those activities together.
So with the activity, we have 3 different types.
First is data movement, which I talked about.
Second is data transformation, so that would be connecting
to Databricks, or doing something
that's going to transform data, and then the third is
just general control activities that help us to control data
in one way or another.
Datasets is the next type.
So, datasets is a data structure within your data store.
So this is where the data that you need for inputs
or your outputs for the projects and the pipelines,
this is where that data lives.
Finally, we have linked services.
So linked services is actually the connection string
that's going to tell Data Factory how to find
or connect to your data.
So with those concepts in mind, let's see if we can put this
into some architecture.
And I'm actually going to start with the far right-hand side
here with the pipeline, and we're going to work our way
back. So the pipeline, like I said, was that large grouping
of activities, and within that pipeline, we can do things
like schedule a pipeline, or monitor,
or manage the pipeline that we have created.
Moving back a step then, activities that are grouped
together to create the pipeline, activities are going to do
things that either control, move, or transform data.
Could be something like hive, or connecting through
to Databricks, or running a stored procedure,
or copying or moving data.
Those are all activities.
Now, our activity is going to run
on our linked service down here at the bottom.
So it's going to use that connection string,
which is the linked service, to pull the data, or connect
to the service so that it can do what it needs to do.
When the activity is done, typically, it's going to move
data, and create a new updated dataset.
So we're going to transform a dataset,
and then put the new dataset somewhere else,
or we're going to move data from one place to another,
or combine 3 datasets into one,
and that activity is going to produce your dataset.
It's also going to be consuming data,
because you have your start and your end, right?
So if we're going to move data, we have to consume
the old data, transform it, and then produce the new data.
And that ties back in, because you have to be able
to connect to that dataset,
and you do that through that link service.
So really, we're kind of creating a chain here.
So all of this ties up into our pipelines,
and the last piece,
besides just building these connections, is thinking
about the orchestration.
And this is where Data Factory shines,
because you can take the steps to your pipeline,
and you can automate them.
You can create schedules around them.
You can monitor each step in the pipeline to check
for errors, and set alerts if that happens,
or set other things like a webhook, for instance.
So we can actually manage how the pipeline performs
and functions, and what happens if something fails.
So to tie all of this together, let's talk about our review.
So let's start off our review
with that Data Factory introduction.
Data Factory is really designed to create the connections
to build pipelines to move and transform data
within your cloud projects, and then it's also designed
to orchestrate the flow of that data, or those steps,
or activities within your cloud project.
Core concepts, we talked about pipelines, activities,
Datasets, and link services. And
then, finally, we talked about Data Factory architecture,
and how those core concepts fit into that data architecture.
Now, I realize that we have reached the end,
and that can be a very sad thing.
However, the good news is, coming up in section 5
we are actually going to hands-on practice for Data Factory,
as well as I'm going to be showing you some demos
as we dive deeper into those lessons.
So we'll be able to see how all of these concepts play out.
So don't worry about diving deeper.
We will definitely do that, but I wanted to make sure
that we had the groundwork, so that you understood
what Data Factory was, and why it was important.
All right, I'll see you in the next lesson.