Hey, Cloud Gurus.
Welcome to our lesson
on normalizing and denormalizing values.
In this lesson,
we're going to do a quick overview of what normalization is,
jump over and show a demo in Azure Data Factory,
and then, wrap up with a review.
Normalization is a common practice when analyzing data,
and the goal of normalization is to change the values
of numeric columns in the dataset to use a common scale,
without distorting differences in the ranges of values
or losing information.
Or, put another way, we're scaling the values,
so that the mean of all values is 0
and the standard deviation is 1.
And this helps greatly,
if you have, say, 2 columns with vastly different scales.
Maybe one in the 10s or 100s
and another in the 1000s or 1,000,000s,
because that can cause problems with your data work.
Particularly if you're combining a couple of columns
as features for data modeling.
You'll really see this used a lot in machine learning.
Some algorithms require data to be normalized ahead of time.
Others don't, but this helps place our data in a better way
for machine learning and other applications to use it.
And of course, denormalizing is just the reverse.
So with that quick definition in mind,
let's jump over to the Azure portal
and take a look at this in Azure Data Factory.
Here we are in Azure Data Factory Studio.
And since this is a little bit more of a complex data flow,
I've gone ahead and set it up for the sake of time.
Basically, what we have here, is 3 different sources,
but all pulling from a product's table
in order to normalize the value.
We're first taking the products and using a SQL query,
under Source Options, to find the standard deviation
of the standard cost column.
We're then caching that in a cache sink.
Similarly, for the average,
we're using a source options query to find the average,
and then, storing that in a sink as well.
Finally, we're taking these same product's information
and using a derived column.
Within this derived column,
we're utilizing all that we've put together so far
in this expression.
Now I'll open up expression builder,
so you can see it a little bit better.
But we're using this formula
to find those normalized values.
We're taking our standard cost column
and subtracting the stored standard cost average.
And you can see,
this is how we reference the information in that cache sink.
We're then taking the result of that
and dividing by the standard cost deviation,
again, referencing one of those cache sinks.
And altogether, this formula produces our normalized values.
To take a look at this,
we can come over to the Data Preview tab.
And you can see
that our standard cost doesn't look like a cost at all.
Instead, we have values in the ranges of 1s and 0s,
allowing us to easily see what our standard deviations are
and making things much easier to work with
in a standardized way.
By way of review, normalizing value scales them,
so that the mean is 0 and the standard deviation is 1.
So if the data point is greater than 0,
it means that it's higher than the mean.
Also, if the normalized value is less than 0,
then it's lower than the mean.
And so, it allows you to quickly see
how many standard deviations the original data point
is from the mean.
You can use mapping data flows in Azure Data Factory,
or modules in Azure Machine Learning,
to accomplish this normalization.
And this is particularly helpful in machine learning,
ensuring that features have similar ranges of values
to work with.
That's it, Gurus.
I hope this helps give you an idea
of how to work with normalization.
When you are ready, I'll see you in the next video.