Hello everyone and welcome to the introduction to logistic regression lecture in this lecture we'll
discuss classification problems how to use logistic regression to solve them and how to interpret the
results of the logistic regression to wreak confusion matrix.
If you want to understand the full mathematics behind logistic regression.
Go ahead and read sections 4 through 4.3 of introduction to mystical learning by Gary James.
Moving along we want to learn about logistic regression as a method for classification and machine learning
and statistics.
Classification is the problem of identifying to which of a set of categories a new observation belongs
to based off of training data.
Some examples of classification problems include trying to detect spam versus ham emails trying to detect
whether someone's going to default on their loan or not or even trying to diagnose a disease for example
trying to tell if someone has cancer or not.
These above examples are all examples a binary classification meaning we have two classes so far in
our studies we've only seen regression problems where we try to predict a continuous value such as the
price of a house although the name may be confusing at first.
Logistic Regression allows us to solve classification problems where we're trying to predict discrete
categories.
The convention for binary classification is to have two classes zero and one.
However we can't use a normal linear regression model on a binary group.
It won't lead to a good fit.
For example if we take a look at this plot below we have a Y axis which represents the probability of
belonging to a particular group.
Let's go ahead and imagine that this example plot is trying to predict likelihood of paying back a loan
.
We'll go ahead and label 0 percent probability as defaulting on their loan meaning they have a zero
percent probability of being able to pay back their loan.
And at the top we have one or a 100 percent probability as fully paying back their loan will go ahead
and mark the X axis as some sort of paycheck value.
That means if we go ahead and look at this data as your paycheck goes lower you have a closer to zero
percent probability that you're going to be able to pay back your loan as your paycheck value gets higher
.
You didn't have closer to 100 percent probability of paying back your loan.
The reason these yellow dashes are all either on 0 percent or 100 percent is because this is training
data.
Now if this was trading data and we try to use a linear regression model on it we would get a very bad
fit.
We would actually end up predicting probabilities below zero percent which doesn't really make any sense
.
Instead we can transform our linear regression to a logistic regression curve and you'll notice our
logistic regression curve can only go between 0 and 1 and that's going to be the key to understanding
classification.
Using a logistic regression curve the sigmoid.
Also known as logistic function is going to be the key to understanding using logistic regression to
perform a classification.
The key secret to this function is that it can take in any value and its output is going to be between
0 and 1.
We take a look at the equation here on this plot.
We have the sigmoid function plotted out on the z axis is going to be the bottom line.
Usually the x x there here without noting it as theta of Z and the formula is theta of Z.
So the function of z is equal to 1 over 1 plus E to the power of negative Z.
The key thing to notice here is that it doesn't matter what value of Z you put into the logistic function
or the sigmoid function.
You'll always get a value between 0 and 1.
So again if you take a look at this plot it doesn't matter that whatever value you put in for Z the
output along the vertical axis is always going to be between 0 and 1.
And that's the key the sigmoid function.
This means that we can take her linear regression solution and place it into this sigmoid function and
that's going to look like this.
Remember our linear model followed a basic y equals x plus B principle.
Here we have a linear model as y equals beta plus Beta 1 times X..
If we take that linear model and put it into the sigmoid function we finally are able to transform this
linear regression to a logistic model meaning it doesn't matter whatever the value of the linear model
output actually is.
It's always going to be between 0 and 1 when we place it into the logistic model or the sigmoid function
.
Again if you want more of you on this mathematics make sure to read sections 4 through 4.3 of introduction
to Stichel learning.
But the basic premise of all of this is that this results in a probability from zero to one of belonging
in the one class again doesn't matter what we put in on this horizontal access on the vertical axis
will always get some sort of probability between 0 and 1.
That means we can set a cutoff point usually at 0.5 and we'll say if anything below results in 0.5 or
below 0.5 that will go to class 0.
Anything above belongs to class 1.
So we're going to transform that 0.5 probability as a cutoff point.
Let's go ahead and do a quick recap overview of what we just discussed.
We can use the logistic function to output of values ranging from 0 to 1.
Again it doesn't matter what we put along the horizontal axis we get a value from 0 to 1 based off of
this probability we will assign a class by putting a cutoff point 0.5 which makes sense because we want
to say if the probability is 50 percent or less of belonging to class 1 then we should classify this
as Class 0 in our binary classification.
If we have a probability of 0.5 or above of belonging to a class 1 we'll go ahead and assign this new
point to class 1.
All right.
So let's go ahead and talk about model evaluation and using a confusion matrix after you train a logistic
regression model to classify some training data.
You're going to want to evaluate your model's performance on some test data.
You can use a confusion matrix to evaluate classification models a confusion matrix is a table that
is often used to describe the performance of a classification model on a set of test data for which
the true values are already known.
The confusion matrix itself is actually relatively simple to understand but sometimes the related terminology
can be confusing.
Let's go ahead and walk through this example.
In this case we have a binary classification problem.
So in this example we're testing for the presence of a disease where no is a negative test which is
false equals zero.
Yes is a positive test where true is equal to one.
What can we learn from this matrix.
Well there are two possible predicted classes.
Yes and no.
If we were predicting the presence of a disease for example yes it would mean that they have disease
.
No it would mean that they don't have disease.
The classifier made a total of a hundred and sixty five predictions meaning 165 patients were tested
for the presence of the disease.
Out of those 165 cases the classifier predicted a yes 110 times and no.
Fifty five times.
In reality meaning we already have label test data 105 patients in the sample have the disease and 60
patients do not.
Now let's go ahead and define the most basic terms the basic terms are the whole number it terms so
not rates just hold numbers and those terms are true positives true negatives false positives and false
negatives.
You may already be familiar of this terminology.
If you've ever had a deal of studies related to vaccine checks or disease checks a true positive.
Are the cases in which we predicted.
Yes that they have disease.
And in reality they do have the disease.
That's T.P. true positive.
In this case you can find it here at the bottom quadrant where T-P is equal to 100 true negatives are
where we predicted.
Know that they don't have disease.
And in reality again they don't actually have.
That's on the upper left hand corner T.N. and that's equal to 50.
So those are true positives and true negatives that don't have false positives and false negatives false
positives are where we predicted.
Yes that they have to Zeese that in reality they don't actually have the.
This is also known as a type 1 error that only have false negatives where we predicted.
No they do not have the disease but in reality they actually did have a disease that's not as a type
2 error.
Then we can talk about rates.
The first rate we can discuss is accuracy.
What this is actually getting at is overall how often is it correct.
A lot of times when you hear reports on studies they'll just tell you the accuracy and the accuracy
is calculated by the number of true positives plus the number of true negatives over the total.
In this case our model is 91 percent accurate.
Then we have the misclassification rate which is answering the question.
Overall how often is the model wrong.
This is going to be calculated by the number of false positives plus a number of false negatives divided
by the total.
So that's 15 divided by a five.
Overall this is 9 percent error rate or misclassification rate.
Now let me discuss a nice quick way to remember false positives vs. false negatives.
Or your type 1 error versus your type 2 error remember are false positives that I'm pointing out here
of the laser P is equal to 10 that's where we predicted Yes but they didn't actually have the disease
.
That's Type 1 error.
False negatives is what we predicted.
No but they actually do has a disease that's known as type 2 error.
Go ahead and move along.
We can think of type 1 error and type 2 errors as this funny little diagram as a nice way to remember
it.
So a type 1 error is where you're telling a man they're pregnant again.
You're predicting.
Yes they're pregnant but they actually don't have the pregnancy type.
To err is a false negative.
Here you're saying someone that's obviously pregnant that they're not pregnant.
So you're predicting.
No but they actually are pregnant.
And that's the difference in a type 1 ere type 2 error false positives vs. false negatives and statistics
are commonly referred to as type 1 or Type 2 instead of their actual names false positive or false negative
.
Hopefully this is a nice helpful and funny reminder on how to actually memorize these terms.
Let's go ahead and begin to explore an example of logistic regression using the famous Titanic data
set will use Python to attempt to predict whether or not a passenger survived based off of their features
.
Then you'll have a portfolio project with some advertising data trying to predict whether or not a customer
clicked on an ad.
OK thanks everyone and I'll see if the next lecture