Linear regression to Logistic regression a conversion | Maths Intuition | Data Science | Machine Learning | Mathematics

Photo by Chris Lawton on Unsplash

Logistic regression is a machine learning algorithm, which is used for :

1. Binary Classification

2. Multiclass Classification

Yes, it is a classification algorithm, not a regression algorithm as the name is suggesting. Basically, it is used to classify between the Yes/No, 0/1 or True/False. It uses the sigmoid function to predict the result.

Problem with the linear regression:

In linear regression, we can find the value of a dependent variable with the help of other independent variables. It works with continuous values, basically, it draws a regression line over the graph so that the distance between the regression line and the points should be minimum.

But when it comes to the binary classification then it is impossible to make a regression line through the data points, so that it can give you the best results as a 0/1 or Yes/No.

So, here the logistic regression comes into play, that it forms a sigmoid function(logistic regression curve), to classify the binary points as shown in the figure.

Source: researchgate.net

1. Linear regression can't make a regression line, which can predict the best output as 0 or 1.

2. If you got your probability below 0 or more than 1 then the linear regression will not work. If it is 0<=P<=0.5 or 0.5<P<=1 then only we can predict.

Terminology for Logistic Regression:

Let's understand some basics before learning:

Sigmoid Function:

The sigmoid function is a function has characteristic to form a sigmoid curve(S-shaped curve).

The Formula

Blues S-shaped Sigmoid curve

Basically, the sigmoid function is a special case of a logistic function, it gives us the ability to choose between the 0 class or 1 class. We can take a threshold as a 0.5 and then we can classify the values above the 0.5 as belongs to class 1 or vice versa. 

Log of Odds:

Logistic regression uses Sigmoid function to transfer linear regression into the log of odds which is also called the Logit function.

Odds are obtained by the ratio of the probability of occurring and the probability of not occurring. It is denoted by 's'.

The formula for Odds(s)

Natural log of Odds ratio is called the logit function. A logistic function or sigmoid function can be converted into the odds ratio. And by the log of odds(logit function), we can calculate the required probability to divide the area for class 0 and class 1.

Image Source: Google Images

Conversion of Linear regression to Logistics regression:

Linear regression equation: 

z=β_o + β₁x₁+ β₂x₂ + β₃x₃+ .... + β_kx_k                ....(i)

Our sigmoid function:

p = 1∕1+ e^-z ⇒ p = e^z/e^z + 1               ....(ii)

Putting the value of 'z' (i) in the sigmoid function (ii):

p = (e^{βo + β1x1+ β2x2 + β3x3+ .... + βkxk} ) / (e^{βo + β1x1+ β2x2 + β3x3+ .... + βkxk}) +1

Odds ratio:

s = p/p-1                ....(iii)

Replace 'p' in (iii) and solve further:

s = e^βo + β₁^x₁^{+ β}₂^x₂ ^{+ β}₃^x₃⁺ .... + ^β_k^x_k

Take log each side and solve:

Ln(s) = β_o + β₁x₁+ β₂x₂ + β₃x₃+ .... + β_kx_k

Solve the equation for 's', then s will be the required probability to predict whether the input data belong to class 1 or class 0. If the s is more than 50% then it belongs to class 1 and if the s is less than 50% than it belongs to class 0.

So, this is all about the conversion of Linear regression to Logistic regression, we will talk about it the actual mathematical implementation further in this post.

If you love my work then can connect me on LinkedIn and GitHub.