Welcome to our introduction
to Azure Stream Analytics.
Continuing on,
we're going to take a look at Stream Analytics.
And in this video, I'm going to be talking
to you at a high level about what Azure Stream Analytics is.
We're going to talk about when, where, and why it's used.
And then, of course,
we're going to talk a little bit a about how it works.
So to start off our discussion,
we are going to be talking about Azure Stream Analytics.
In order to talk about Azure Stream Analytics,
we have to talk about the tail of 2 processes.
So over here on the right,
you will see a giant circle of a pizza,
and you'll also see a picture of a stream.
So we need to talk about batch versus stream.
So when we look at batch,
batch processing is going to be much more like a pizza oven.
Someone places an order, we queue up 2
or 3 or 4 of those orders together,
we throw those pizzas in the oven.
We then pull, those is out, put it on the rack
and then we serve the pizza;
that's batch process.
Stream process doesn't build things up.
Stream process is just like this stream over here.
As the data comes through,
we're going to simply be processing that data
in near or real time
and we're going to be doing some transformations
and work on that data as it moves through,
but we're not stopping and running everything all at once.
So Azure Stream Analytics then is going
to be this stream process.
And when we talk about batch,
you're going to see that more in things like trading.
So if we're going to place a trading order,
we stop and place this trading order
and then we place the next trading order.
Or traffic patterns,
we have a light that is red turns to green,
then it turns to yellow and red and it stops,
and then the next side goes.
So it's a batch of traffic going through.
The first batch goes, then we stop,
then we send the next batch through.
Stream processing on the other hand,
you're going to see that for things like fraud detection
because we don't have time to stop
and analyze all the data.
We want to see in real time what's happening.
So we can detect where fraud might be used
and we can turn it off immediately.
So we have to be processing data
from millions of different sources
and seeing if we can detect patterns from that data.
Also, you'll see that in purchasing.
So, think Amazon here;
if I go out and I buy something on Amazon,
it's going to recommend other products for me
and that's done in real time.
So we need to have that sort of information.
You'll also see that like at AMC theatres.
So if you go to the movie theater
and you buy a ticket,
while that data is being processed in real-time
because it has to know how many seats are left
in the theatre,
which seat you purchased.
It has to be able to process all
of that information through
and spit out your tickets to you.
So those are examples of stream processes.
Be careful when moving immediately to stream.
A lot of businesses, that's what they want
because they look at this and they say,
"Well, why wouldn't I want real time?
"We want everything now.
"I don't want to have to wait and only get data once a day.
"I want everything immediately."
Be careful when doing that
because when we talk about stream,
stream can be very, very expensive
because we have to have a lot of processing
to do any transformation
on a live flowing stream of data,
especially if it's a large stream of data.
So you always want to go back
and say what is the business case?
What is it exactly that we are trying to solve?
And then let's come up
with a recommendation for that solution
that's going to save our company the most time and money
and/or at least give those arguments
so that we know where the value lies.
Is it in money?
Is it in time?
What are we actually trying to do here?
So just make sure that you have those conversations
and you kind of think through what you're doing,
before immediately recommending a stream solution.
All right, core concepts of Stream Analytics.
So Stream Analytics has basically 3 things happening.
The first is our input.
So you see our input.
We have Event and IoT Hubs
and then also you'll see Blob storage there.
Those are the 3 sources for Azure Stream Analytics
and that's important to understand.
So that's our first phase.
It's fairly simple.
We just connect our input streams into Stream Analytics.
The next is where the magic happens.
So this is our query section.
So part 2, query,
and in this stage we're going to do transformation
Transformation on a streaming source,
by definition, is going to be less
than what you can do with a batch source;
simply because with a batch source,
we can get all the data together.
We can stop.
We can pause while we run all of our transformations,
and then we can spit out our end result.
In stream, the more complicated the transformation is,
the more processing power you need.
Because again, there's more data coming in right behind it.
So we don't have time to stop and run everything overnight.
So with transformations,
it's a little bit more difficult and challenging.
And with transformations here,
you're going to use your query step
in order to do those transformations.
And just like the other lessons,
we're going to dive more into this later,
but again I want you to get the overarching picture
of what's happening here.
Lastly, output. So send things into the system,
do something to the data that's being streamed through
and then we're going to send it somewhere to be stored
and to save those results.
So that could be to Power BI
to get real-time dashboards
that could be dumping it into a Data Lake
so that something else can be done to it
or it can be pulled out and sent to a customer's account.
Lots of things can happen there,
but we have our output phase which is last.
That's really it for Stream Analytics.
It's not wildly complicated, at least in its architecture.
And we'll dive further into those transformations
which is kind of the main piece to Stream Analytics.
You can kind of see what's happening later.
The last concept we're going to talk about
is the concept of windowing.
So when we think about windowing,
we are looking at, you can see here,
outdoors, we're looking at a window.
When we think about that in terms of data,
when do you look at your data
and what is the output of the view that you want to see?
So what I mean by that is
am I interested in taking a look
at every single piece of data?
I can't get very much transformation.
I can't get averages off of every single piece of data.
Do I want to look at a window of only 10 events?
Do I want to look at a window of 10 minutes?
How do I want to handle data that arrives late?
These are concepts of windowing.
So it's looking at the data as it comes into your system
and figuring out the best way to kind of arrange
or clump that data so that we can get transformations
in the views that we want to see.
The averages, the whatever we want to look at,
so that we can get the data that we need
to make our business decision.
So we'll talk more about windowing,
but it's a very important concept to think about
when looking at anything that involves streaming.
All right, so that is it,
and I actually want to pause here.
So you'll probably notice the same as I do,
there's this very odd looking picture of a woman
in a green face mask and a Macintosh computer.
I'm not really sure what's happening here,
but I was searching for "think,"
and this came up, and
yeah, it really made me stop and think for a few minutes.
So there you go, random picture for the day.
But looking back at our review here,
let's talk about Stream Analytics.
We talked about the introduction
for every single one of these services.
It's extremely important
that you think about what the service is used for.
You can't figure out how to put it
into a data engineering architecture
without understanding what it is
and what it does at its core.
Next, how do all the pieces
of Stream Analytics architecture fit together?
This one was pretty easy.
We looked at inputs and queries and outputs.
Those are the 3 main components of streaming.
So make sure you keep those 3 concepts in mind.
Again, we'll dive further into it.
And then you also want to think
about windowing and what is a window.
And then we'll use that later
to talk about the various types of windows
and how that can be used down the road.
All right, that's actually it for this lesson.
In the next lesson,
we will talk about Databricks.
So we'll be diving in and talk about that
and I'll see you there.