Welcome back.
In this lesson,
we are going to be putting the puzzle pieces together
and identifying Azure services for batch processing.
So in this lesson, specifically,
we are going to take a look at what batch processing is
and how we typically use batch processing in Azure.
Finally, we're going to talk
about some common batch processing solutions
that you need to know for the DP-203.
So to start off, let's talk about the tale of 2 processes
just a little bit more deeply.
When we talk about batch services,
really think, more pizza.
So we are going to have a bunch of pizza orders
that come into the restaurant.
We take those pizza orders,
and we process all of them in this oven, right?
We bake the pizza and process it in the oven.
We are not going to take every individual slice
and run them out to the table one slice at a time.
No, we're going to take the entire table,
do all of the orders
and then take all of those orders out to the table.
Streaming is different.
When we talk about streaming, we have flowing water.
And it's just going to continually flow.
We don't stop and then start the water and then,
it's just going to keep going.
Same thing's true for the data.
So batch processing, think about bounded and unbounded data.
So bounded just means we have a start and a stop.
We're going to take that bounded data source,
we know how big it is,
and we are going to do a batch process transformation
on that bounded data source.
Streaming then is unbounded data.
We don't know how big it is,
so we're just going to start processing with more unknowns.
So batch data is not required immediately.
This is going to be something,
typically you see that like in a bank for instance,
they'll do a nightly run.
They'll take all of the orders from the day,
they'll transform, put them all into databases,
and that is going to be a batch process.
Typically, you can handle much larger transformations
with batch, and it's going to be lower cost.
So larger data, lower cost,
that's typically what we're going to see with batch.
With streaming, we need the data right now.
So things to think about for streaming
would be like a recommendation engine.
So I'm on a website, I'm looking at something,
and you want to tell me,
while I'm still on the website,
some other services that might be good for me to purchase.
That would be something
that you would need streaming data for.
Typically you're going to have
less complex transformations with streaming,
because we can't stop and run complex transformations.
Instead, we have to keep moving that data.
And so we can't do as much transformation
as we can do with batch.
And because of that, we're going to have higher costs,
because we're doing things on the fly,
and so typically that's going to be higher cost
than what you would see with batch.
So let's take a look then at where batch is used.
So I already talked about the banking example
for batch processing.
Another common use of batch would be at retail outlets.
So you take all of those orders
and you process all of the orders overnight.
And that way, in the morning or on a weekly basis,
managers of those different stores
can take a look at all the sales
and where those sales were coming in.
It's not something that needs to be done on the fly.
Hospitals, another good example for batch processing.
So taking a look at high volumes of patient information,
transaction information,
those are other things that you would see for batch.
And then finally, for marketing.
So with batch processing in marketing,
we could be looking at a marketing campaign
and taking a look at, for instance,
hits on a website,
processing all of that data together
and seeing if we can come up with some insights
to determine how we need to change our campaigns.
It wouldn't be something
I'd need to see every single minute,
but it would be something
that I would want to see every week
or every 2 weeks.
Some challenges that we want to think about
when we look at data,
first would be data format.
So is all of the data in the correct format?
And when I talk about format,
we're kind of thinking about things like country code,
for example, or name.
Is it first name, last name as 2 different columns?
Is it first name, last name as 1 column?
What is the data in?
What's the schema look like?
Is everything match up there for batch processing?
Second, would be encoding.
So in addition to the data
maybe being in a different format,
it's also possible that the data
has been encoded different ways,
and so we have to convert the data into a form
that could be used for data processing.
For instance, we want everything in JSON.
Third, dealing with Windows and missed runs.
So we've talked a little bit about Windows,
so what is the frame of the data that we're looking at?
What happens if you have data
that sits outside of those Windows?
What are you going to do with that?
And then also, missed runs.
So what happens if I try and run a nightly process
and it doesn't happen?
Are we going to run it again?
Two times? Five times?
Just challenges that you want to consider
for batch processing.
Now we'll also tell you that dealing with Windows
and missed runs and batch processing
is infinitely easier than with streaming,
but still something you want to think about.
So let's talk a little bit
about where batch processing services live.
So we start off with an input source,
a blob or a data lake,
some sort of place that's holding our data,
and then we start to look at doing batch processing.
So you could do batch processing through Databricks
or through Synapse Analytics
or through HDInsight or Azure Data Lake Analytics.
Those are all the common services
that you would see for batch processing.
And then you take that data
and you move it through one of those services.
And then it's going to come out something like Power BI,
where we are going to do
some sort of reporting off of it, typically.
And then finally down here at the bottom,
you see we have Data Factory orchestration.
Now in this section,
we are actually going to talk a lot about Data Factory,
because the critical element to batch processing,
or one of the critical components to batch processing,
is moving that data,
building a pipeline that can perform
the batch processing things that we need.
So in this section, we'll talk a lot
about how to take data from that data lake,
move it into something like Databricks
to do our transformations, and then pull it out of that
and move it into something else like Power BI.
Of these services,
I will tell you that Databricks
and Azure Synapse Analytics at the top
are by far the most common for batch processing.
I will also tell you that as we look at batch processing
in the DP-203,
you're much more likely to see questions
on how to move data, Data Factory orchestration,
those kinds of things,
than you will specific types of batch processing.
For instance, you're much less likely to see a question
on how you perform a specific transformation in HDInsight
than you would to see questions on Data Factory pipelines.
So just keep that in mind.
And that's based upon the exam requirements in the DP-203.
All right, so to review this lesson,
we talked about batch processing
and the characteristics of batch processing versus stream.
We talked about examples and challenges.
So be ready to defend batch versus stream,
because a lot of things that you see in the corporate world,
everybody wants information now,
and you need to be aware
of the differences between batch and stream
to be able to explain
why you would prefer batch over stream,
be that cost, be that the ability to do transformations,
simplicity, whatever it is,
if there's really not a need for stream,
or the other way around if there is.
So just be aware of that.
And then finally, the Azure technology choices.
So if we're going to do batch processing,
what are the services that we are most likely going to use
in Azure.
All right, that's it for this lesson.
I will see you in the next.