logistic regression gradient and hessian

Logistic Regression Gradient and Hessian: Complete Derivation & Python Guide

By Michael King
AD

The logistic regression gradient and Hessian are two of the most important mathematical objects behind one of the most widely used classifiers in machine learning. While many practitioners call logistic regression with a library function and never inspect the optimization details, understanding the gradient and Hessian reveals why training works, how second-order methods converge, and where numerical problems can appear. For binary classification, the model estimates a probability with the sigmoid function, and the loss is derived from maximum likelihood. The gradient tells us the steepest direction to improve the likelihood, while the Hessian describes the curvature of that likelihood surface.

This article walks through the derivation, matrix forms, convexity properties, and practical optimization algorithms that use the logistic regression gradient and Hessian. You will see how the gradient reduces to a simple residual-weighted sum, how the Hessian becomes a weighted Gram matrix, and why Newton-Raphson and iteratively reweighted least squares are natural training methods. The goal is to connect clean theory with real implementation choices such as regularization, stable computation, and large-scale approximations.

Understanding the Logistic Regression Gradient

In logistic regression, the probability that an example belongs to the positive class is modeled as sigmoid(theta transpose x), where theta is the coefficient vector and x is the feature vector. Training means finding the theta that maximizes the likelihood of the observed labels or, equivalently, minimizes the negative log-likelihood. The gradient is the vector of first partial derivatives of that objective. It points in the direction of the greatest increase for the likelihood, or the greatest decrease for the negative log-likelihood, depending on the sign convention.

AD
Logistic Regression Explained 📈 | Sigmoid Function, Decision Boundary & Cost Function
Logistic Regression Explained 📈 | Sigmoid Function, Decision Boundary & Cost Function
Logistic Regression Explained for Python Beginners
Logistic Regression Explained for Python Beginners
Demystification of Logistic Regression
Demystification of Logistic Regression

The gradient is not just an abstract derivative. It has a clean statistical interpretation: each training example contributes an error signal equal to the difference between its true label and its predicted probability. That residual is multiplied by the feature vector and summed across all examples. This structure makes the logistic regression gradient efficient to compute, easy to regularize, and compatible with stochastic and mini-batch optimization.

Deriving the Gradient from Maximum Likelihood

Start with the likelihood for independent examples. For label y(i) in zero or one, the probability of the observed label given x(i) is p(i) raised to y(i) times one minus p(i) raised to one minus y(i), where p(i) is the sigmoid of theta transpose x(i). Multiplying these probabilities over all examples gives the likelihood. Taking the natural logarithm turns the product into a sum, producing the log-likelihood. The derivative of the sigmoid is sigmoid times one minus sigmoid, which simplifies the chain rule dramatically.

Linear vs Logistic Regression — One Image Explanation 📈
Linear vs Logistic Regression — One Image Explanation 📈
Logistic Regression
Logistic Regression
Bayesian Analysis for a Logistic Regression Model - MATLAB & Simulink Example
Bayesian Analysis for a Logistic Regression Model - MATLAB & Simulink Example

After applying the chain rule, the partial derivative of the log-likelihood with respect to a single coefficient is the sum over examples of y(i) minus p(i), multiplied by that feature value. Stacking these partial derivatives gives the gradient of the log-likelihood as X transpose times the vector y minus p. For the negative log-likelihood cost, the gradient is X transpose times p minus y. This residual form is the foundation for gradient descent and many quasi-Newton methods.

Gradient in Matrix Form and Batch Computation

Let X be the n by d design matrix, where n is the number of examples and d is the number of features including the intercept. Let y be the n-dimensional vector of true labels, and let p be the n-dimensional vector of predicted probabilities. The gradient of the negative log-likelihood is X transpose times p minus y. Computing it requires one pass through the data to form p and one matrix-vector product with X transpose. The cost is linear in the number of non-zero feature entries, which is especially important for sparse text or categorical data.

AD

Regularization modifies the gradient in a simple way. Adding an L2 penalty of lambda over two times the squared norm of theta adds lambda times theta to the gradient. This shrinks coefficients and helps prevent overfitting. A mini-batch gradient uses the same formula on a subset of rows, while stochastic gradient descent uses one row at a time. These variants trade exactness for speed and are central to large-scale logistic regression training.

Bayesian logistic regression with PyMC3 | Towards Data Science
Bayesian logistic regression with PyMC3 | Towards Data Science
Logistic Regression Explained | Towards Data Science
Logistic Regression Explained | Towards Data Science
What is Logistic Regression?
What is Logistic Regression?
Binary classification and logistic regression for beginners | Towards Data Science
Binary classification and logistic regression for beginners | Towards Data Science
Logistic Regression (Python)
Logistic Regression (Python)
Simplifying Logistic Regression
Simplifying Logistic Regression

The Logistic Regression Hessian and Second-Order Optimization

The Hessian is the matrix of second partial derivatives of the objective. For logistic regression, it captures how the gradient changes as the coefficients change, which is exactly the curvature information that first-order methods ignore. This curvature determines the shape of the likelihood surface, the stability of Newton-style updates, and the asymptotic covariance of the estimated coefficients. Because the negative log-likelihood is convex, the Hessian has a special structure that makes logistic regression a well-behaved optimization problem under standard conditions.

Second-order methods use the Hessian to take more informed steps than plain gradient descent. Instead of moving only in the steepest direction, they rescale that direction by the inverse curvature. The result is often much faster convergence near the optimum. However, forming and inverting a full Hessian can be expensive when the number of features is large, so practical implementations often use Hessian-vector products, quasi-Newton approximations, or regularized variants.

AD

Matrix Form and Convexity of the Hessian

Define W as a diagonal matrix whose i-th entry is p(i) times one minus p(i). The Hessian of the negative log-likelihood is X transpose times W times X. The Hessian of the log-likelihood itself is the negative of that matrix. Because each probability lies between zero and one, every diagonal entry of W is non-negative. Therefore X transpose W X is positive semidefinite, which proves that the negative log-likelihood is convex. This convexity guarantees that any local minimum is a global minimum.

The Machine Learning “Advent Calendar” Day 12: Logistic Regression in Excel | Towards Data Science
The Machine Learning “Advent Calendar” Day 12: Logistic Regression in Excel | Towards Data Science
Breaking it Down: Logistic Regression | Towards Data Science
Breaking it Down: Logistic Regression | Towards Data Science
a poster with information about the functions of graphs
a poster with information about the functions of graphs
Linear vs Logistic vs Poisson Regression
Linear vs Logistic vs Poisson Regression

If X has full column rank and the weights are strictly positive, the Hessian is positive definite and the optimum is unique. In separable data, some predicted probabilities can approach zero or one, making the weights very small and the Hessian nearly singular. In that situation the maximum likelihood estimate may not exist, and coefficients can grow without bound. Adding L2 regularization changes the Hessian to X transpose W X plus lambda times the identity matrix, which restores positive definiteness and stabilizes training.

Newton-Raphson, IRLS, and Practical Stability

Newton-Raphson for the negative log-likelihood updates theta by subtracting the inverse Hessian times the gradient. Substituting the logistic expressions gives theta new equals theta minus the inverse of X transpose W X times X transpose p minus y. This simplifies to theta plus the inverse of X transpose W X times X transpose y minus p. The same update appears in iteratively reweighted least squares, where each iteration solves a weighted least squares problem with weights p(i) times one minus p(i) and a working response equal to the linear predictor plus the residual divided by the weight.

AD

Practical stability requires care. When p(i) is extremely close to zero or one, the weights become tiny, and the weighted least squares problem can become ill-conditioned. Stable sigmoid implementations, log-sum-exp tricks, line searches, trust regions, and regularization all help. For very large models, avoid forming the Hessian explicitly. Instead, compute Hessian-vector products as X transpose times W times X times v. This operation costs roughly the same as two gradient evaluations and enables conjugate gradient or Hessian-free Newton methods. Quasi-Newton methods such as L-BFGS approximate curvature from gradient differences and remain popular when the exact Hessian is too expensive.

A practical way to internalize these ideas is to compute the logistic regression gradient and Hessian on a tiny dataset by hand or with a few lines of numerical code. Compare the analytic gradient with a finite-difference approximation, then use the Hessian to take a single Newton step. Once the shapes and signs make intuitive sense, move to a larger problem and observe how regularization changes the curvature and the coefficient path. That hands-on loop turns the derivations from symbols on a page into reliable tools for debugging, optimizing, and trusting your logistic regression models.

AD