Hey, what's up, Gurus?
Congratulations.
You have made it to the end of a pretty long section.
In this last video,
we are going to recap some of the highlights of this section
and things that you need to keep in mind.
So in this video, it's all a review.
We're not going to be covering anything new,
so if you see anything here that you're not quite sure
what it was or what it means,
then you should go back and re-watch those videos.
Second, focus on the DP-203.
As you noticed from all of the lessons in this section,
we are focusing on the 203,
which sometimes means glossing over some details
that might be helpful for a data engineer,
such as Cosmos DB,
but aren't likely to be found on the DP-203,
so just keep that in mind as you move through this review.
And finally, if you don't know something, again, review.
If something seems confusing to you,
or you're not quite sure,
go back through and re-watch the video
or redo that hands-on lab.
It's important.
All right, here we go.
So what are the services of stream processing?
I'm going to give you a second to see if you can name them.
Hint, hint, there's 3.
If you said Azure Stream Analytics, Databricks,
and HDInsight, you are correct.
And if you'll note,
in this section we focused almost exclusively
on Azure Stream Analytics,
other than a few highlights of the other services,
because if you look at the exam skills,
they're really focusing on Azure Stream Analytics.
Doesn't mean you won't see a Databricks
or HDInsight question,
but Stream Analytics seems to be the most likely choice.
When would I choose streaming over batch?
Well, streaming is a better fit
if you need the information right now.
Remember, it's good for recommendation engines,
fraud detection, marketing applications, machine learning.
However, streaming is more costly
and a bit more complicated to do transformations
and pull data insight out.
This lesson is important.
Make sure that you understand
that there are 5 types of windows
and that you know what they are.
Remember, we talked about Stream Analytics
and setting up those windows
to look at data as it moves across?
There's tumbling windows, hopping windows,
sliding windows, session windows, and snapshot windows.
In that lesson, I recommended
that you understand, in detail, what all of those are.
I still think that's incredibly important.
So if you don't know what all 5 of those are,
go back and re-watch that lesson.
It's going to really help you.
It's an important concept, not just for the DP-203,
but also for a data engineer.
Do you remember watermarks?
Remember that there's 2 formulas for watermarks,
one that involves incoming events and one that is not,
that does not involve incoming events.
Make sure that you remember
that that top formula: watermark = largest event -
out-of-order tolerance.
That's for incoming.
And the bottom one,
the current estimated arrival time
-: late arrival tolerance
is for estimated events,
which means there's no incoming event.
Remember that there's a couple of different ways
to keep time.
Make sure that you remember that.
And then, very importantly,
remember the concept of tolerance.
We looked how to set tolerance in the Azure portal.
But remember when you set tolerance,
if you set tolerance that's too short,
you're likely to see data loss,
and if you set it to be too long, you can break process.
So make sure that you keep those concepts in mind.
And if you don't remember how to do all that,
go back and re-watch.
Question.
If I told you I wanted to upsert data via streaming,
what would you tell me?
You should tell me that you have to upsert via Cosmos DB,
that you have to set your compatibility level to 1.2,
and that it's going to require configuration
on the Cosmos DB side.
Keep in mind, Cosmos DB? Not on the DP-203.
But upserting via Stream Analytics to Cosmos DB might
be on the 203.
It's at least a requirement that they've listed.
So make sure that you remember how to do that.
Don't forget to set alerts and monitor jobs.
Hey, this is a really good common tip that you want to do,
no matter what you do in the cloud.
Whether it's data engineering, data science, DevOps.
Set alerts. Monitor your jobs.
You will thank yourself for it later.
And remember, as we look at this, you do that
by understanding what your key metrics are,
looking at events, utilization, watermarking and errors,
and then by creating those alerts.
So make sure that you're doing both of those things.
Remember also we talked
about partitioning and repartitioning.
Remember we said that partitioning was taking our data,
dividing it up into buckets, and then using a partition key.
We talked a lot about why we partition as well.
Remember the story about LEGOs,
that we use it to make our searching faster.
And we talked about how to select a partition key
by making sure that you're choosing a partition key
that is both static and has high cardinality,
which just means it has a big range.
Then we talked about repartitioning, or shuffling.
And that was for those scenarios
that weren't fully parallelized,
which basically means they're not embarrassingly parallel,
which that was a-one-to-one relationship
of input and output both being a partition.
Remember, we talked about if there wasn't,
that you can process those independently.
Make sure that you keep all of that information
in mind as well.
Oh no!
Your Stream Analytics job has crashed.
Should you panic?
The answer?
No.
But why?
I'll give you just a second.
You should have said,
"Well, the job gets started and the work is broken up
"into worker nodes.
"Something bad happens.
"The node fails.
"But remember, Azure Stream Analytics has
"an automatic recovery system
"that's going to generate a new healthy node
"and pick up based upon its last available checkpoint."
We also talked about Stream Analytics output error policy.
And remember, we're going to drop events that result
in data conversion error,
and then we're going to retry until the event succeeds.
However, that could literally take forever.
So in summary, we have talked about a ton of stuff.
I hope that that very quick walkthrough sparked
a lot of memories and that you have a clear understanding
of all of those concepts.
Some things to keep in mind as you go.
First, think scenario.
Remember, as you go through all of these lessons,
be thinking about how you would fit these into scenarios
that you might be working in,
or think about scenarios that might happen
as you go through and look at this.
Things like, "Well, I'm buying a movie ticket,
"and how would that data be processed?"
That's going to make you a better engineer.
Second, focus on Azure Stream Analytics first.
Most of the stream requirements
are about Azure Stream Analytics.
Doesn't mean you shouldn't look at HDInsight or Databricks.
It just means that 90% of the requirements
are about Azure Stream Analytics.
Finally, don't forget the labs. You're
going to need those for the exam, and for your career.
If you're enjoying the course,
don't forget to smash that thumbs up button on the lessons.
Congratulations on finishing this section,
and I'll see you in the next.