Logistic Regression: Model, Training, and Worked Examples

Verified Sources

Nov 4, 2026

Logistic regression is a widely used method for predicting a binary outcome from features by modeling the log-odds as a linear function of inputs. It outputs a probability in $(0,1)$ using the sigmoid function so that the probability increases/decreases smoothly with the predictors.

A standard form for a single observation is:

$y \in \{0,1\}$
features $x \in \mathbb{R}^d$
model score (logit) $z = w^\top x + b$
predicted probability $p = P(y=1\mid x) = \sigma(z)$ with $\sigma(z)=\frac{1}{1+e^{-z}}$

The key modeling assumption is that the log-odds are linear in $x$ :

\log\left(\frac{p}{1-p}\right)=w^\top x+b.

Training is typically done via maximum likelihood estimation by minimizing the log loss (equivalently, maximizing likelihood).

When logistic regression works well: when the relationship between features and the log-odds is approximately linear, and when classification performance is improved by regularization rather than by complex nonlinearities.

Important: Logistic regression is not restricted to “straight lines” in feature space; if you add polynomial or transformed features, the decision boundary can become nonlinear in the original input space.

Logistic regression - Wikipedia - Defines logit (log-odds), sigmoid mapping to probability, and maximum likelihood/log loss training. ↩
Logistic regression - Wikipedia: model and decision boundary - Discusses linear decision boundary/hyperplane and interpretation via log-odds. ↩ ↩²

Logistic Regression (Intuition + Math) — Sigmoid, Odds, and Decision Boundary

1) Model description with a concrete probability example

Consider a customer churn model:

$y=1$ means “churn”
$x_1$ = number of support tickets (scaled), $x_2$ = tenure in months (scaled)

Let weights be $w=[0.8,-0.3]$ and $b=-1.1$ . For a customer:

$x_1=1.0$ , $x_2=0.5$
score: $z=w^\top x+b = 0.8(1.0) + (-0.3)(0.5) - 1.1 = -0.35$
probability:
$p=\sigma(-0.35)=\frac{1}{1+e^{0.35}}\approx 0.413$

So logistic regression predicts a 41.3% churn probability. If you classify with threshold $p\ge 0.5$ , this customer would be predicted as “not churn.”

A useful interpretation comes from the log-odds form:

\log\left(\frac{p}{1-p}\right)=w^\top x+b.

Each coefficient $w_j$ represents how the log-odds change when feature $x_j$ increases by 1 unit (holding others fixed).

Odds ratio viewpoint (interpretation): if $x_j$ increases by 1, log-odds increase by $w_j$ , so odds multiply by $e^{w_j}$ .

For example, if $w_1=0.8$ , then increasing support tickets by 1 increases odds of churn by a factor of $e^{0.8}\approx 2.23$ (about a 123% increase in odds).

Logistic regression - Wikipedia - Defines logit (log-odds), sigmoid mapping to probability, and maximum likelihood/log loss training. ↩ ↩²

2) Decision boundary and classification threshold

The predicted class often uses a threshold (commonly $0.5$ ):

predict $y=1$ if $p \ge t$
otherwise predict $y=0$

Because $p=\sigma(w^\top x+b)$ , the set of points where $p=0.5$ corresponds to:

w^\top x+b = 0

since $\sigma(0)=0.5$ . That means the default decision boundary is a hyperplane in feature space.

If you use a different threshold $t$ , then solve:

t=\frac{1}{1+e^{-(w^\top x+b)}} \Rightarrow w^\top x+b=\log\left(\frac{t}{1-t}\right).

So the boundary shifts accordingly.

Example (thresholding): In medical screening, you may prefer high recall (fewer false negatives). You might set a lower threshold $t=0.3$ so more patients are flagged positive.

Training logistic regression with log loss (high level)

1
Step 1
Use $p_i = \sigma(w^T x_i + b)$ for each training example $i$ .
2
Step 2
For each $(x_i,y_i)$ with $y_i\in\{0,1\}$ , use $\ell_i = -\big[y_i\log(p_i)+(1-y_i)\log(1-p_i)\big]$ .
3
Step 3
Compute total loss $L(w,b)=\sum_i \ell_i$ (or average loss).
4
Step 4
Minimize $L(w,b)$ using an optimizer (e.g., gradient-based methods).
5
Step 5
Add a penalty such as $\lambda\lVert w\rVert_2^2$ (ridge-style) to improve generalization.

Footnotes

Logistic regression - Wikipedia: regularization - Describes regularized logistic regression objectives (e.g., L2/L1 as common choices). ↩
6
Step 6
Convert probabilities to labels using $p\ge t$ .

3) Worked examples (from intuition to computation)

Example A: Spam vs. not spam (text features)

Let:

$x$ be a vector of engineered features like word frequencies or TF-IDF weights.
$y=1$ means “spam.”

For a given email, the model computes:

$z=w^\top x+b$
$p=\\sigma(z)$

If $p=0.93$ , the model assigns high probability of spam.

Practical note: logistic regression is commonly used with high-dimensional sparse text features because it remains linear in parameters and can be efficiently regularized.

Example B: Credit default risk (calibration + threshold tradeoff)

Let:

$y=1$ means “default”
$x$ includes income, debt ratio, credit history summary variables.

If an account has $p=0.12$ , the model believes there is a 12% chance of default given those features.
A risk team might:

deny credit if $p$ exceeds a cutoff driven by business costs
or set different cutoffs to balance false positives and false negatives.

This illustrates why logistic regression outputs probabilities rather than only hard classes—useful for decision-making.

Logistic regression - Wikipedia: model and decision boundary - Discusses linear decision boundary/hyperplane and interpretation via log-odds. ↩

How coefficient signs affect probability (toy illustration)

Suppose $p=\sigma(w_1 x_1 + b)$ with $b=0$ . Then $x_1$ increasing changes $p$ according to the sign of $w_1$ .

4) Regularization and variants (why $L_2$ / $L_1$ matter)

In real datasets, features may be correlated and models may overfit. Logistic regression is often trained with regularization:

L2 regularization shrinks weights smoothly
L1 regularization encourages sparsity (feature selection)

A typical regularized objective is:

\min_{w,b} \\; \\sum_i \\ell_i(w,b) + \\lambda\\Omega(w)

where $\Omega(w)$ is an L2 or L1 penalty.

Libraries (e.g., scikit-learn) implement logistic regression with different solvers and penalties; in practice, practitioners tune $C$ (inverse regularization strength) and choose $penalty$ .

Logistic regression - Wikipedia: regularization - Describes regularized logistic regression objectives (e.g., L2/L1 as common choices). ↩
scikit-learn LogisticRegression documentation - Documents solver choices, penalty options, and $C$ as inverse regularization strength. ↩

Use “log-odds linearity” as your diagnostic

If plots of predictors vs. log-odds show strong curvature, consider feature transforms, interactions, or a more flexible model.

Probability calibration vs. classification accuracy

High accuracy doesn’t guarantee well-calibrated probabilities. If you rely on $p$ values (not just ranking), consider calibration checks (e.g., reliability plots).

5) Assumptions, strengths, and limitations

Assumptions (modeling level)

Independence of observations (common ML assumption).
Linear relationship between predictors and log-odds (after feature engineering).
Sufficient representational power via feature transformations.

Strengths

Interpretable coefficients (odds ratios).
Works well for sparse/high-dimensional inputs with regularization.
Produces probabilities directly.

Limitations

If the true relationship between $x$ and log-odds is highly nonlinear and not captured by feature engineering, performance can suffer.
Sensitivity to feature scaling when using gradient-based solvers (standard practice: standardize features).

Logistic regression - Wikipedia: model and decision boundary - Discusses linear decision boundary/hyperplane and interpretation via log-odds. ↩ ↩² ↩³
scikit-learn LogisticRegression documentation - Documents solver choices, penalty options, and $C$ as inverse regularization strength. ↩

Common exam-style questions (with examples)

  from sklearn.linear_model import LogisticRegression
  from sklearn.preprocessing import StandardScaler
  from sklearn.pipeline import make_pipeline

  model = make_pipeline(
      StandardScaler(),
      LogisticRegression(
          penalty="l2",  # or "l1"
          C=1.0,         # inverse regularization strength
          solver="liblinear"
      )
  )

  model.fit(X_train, y_train)
  p = model.predict_proba(X_test)[:, 1]  # P(y=1|x)
  y_hat = (p >= 0.5).astype(int)

tab: "Example: computing probability manually (single sample)"
lang: "Python"
code: |
  import math
  def sigmoid(z):
      return 1/(1+math.exp(-z))

  w = [0.8, -0.3]
  b = -1.1
  x = [1.0, 0.5]

  z = sum(wj*xj for wj, xj in zip(w,x)) + b
  p = sigmoid(z)
  print(p)  # ~0.413

How logistic regression is typically applied (end-to-end)

Feature engineering

1) Prepare features

Scale/transform inputs; consider interactions/polynomials if needed."

Train with regularized log loss

2) Fit parameters

Optimize $w,b$ by minimizing negative log-likelihood (with penalty)."

Select threshold & tune

3) Validate

Tune hyperparameters; evaluate with relevant metrics."

Predict probabilities

4) Deploy

Use $p=P(y=1|x)$ for ranking and threshold-based decisions."

Logistic Regression Quick Deck

1 / 5

Question · Term

What is the logit?

Click to reveal

Answer · Definition

The log-odds: $\log\left(\frac{p}{1-p}\right)=w^T x+b$ .

Knowledge Check

Question 1 of 4

Q1Single choice

In logistic regression, the model uses $p=P(y=1|x)$ with $p=\sigma(w^T x+b)$ . What quantity is linear in $x$ ?

The log-odds $\log\left(\frac{p}{1-p}\right)$

The probability $p$ itself

The odds $\frac{p}{1-p}$

The class label $y$

Explore Related Topics

Bayesian Classifiers: Definition, Naive Bayes, and Why the Correct Choice Is (i)

Machine Learning Foundations and Lifecycle

Machine learning is an AI subfield that builds models to learn patterns from data, covering its paradigms, lifecycle, mathematics, and common algorithms.

Supervised, unsupervised, and reinforcement learning describe the three main paradigms.
Standard dataset partitioning allocates 70 % for training, 15 % for validation, and 15 % for testing.
The ML lifecycle progresses through problem definition, data collection/preprocessing, feature engineering, model training, evaluation/tuning, and deployment/monitoring, with data quality and overfitting as key concerns.
Understanding linear algebra, calculus (gradient descent), and probability/statistics is essential for model development.
Typical algorithms include linear regression, decision trees, k‑means clustering, and neural networks.

Complexity Analysis: Best Case, Worst Case, and Average Case

The material introduces best‑case, worst‑case, and average‑case complexity as three distinct functions describing an algorithm’s running time on inputs of size  $n$ , explains how they are formally defined, and shows why worst‑case analysis is usually preferred.

Best case: $T_{\text{best}}(n)=\min_{I\in\mathcal I_n} T(I)$ , the minimum time over all inputs of size  $n$ .
Worst case: $T_{\text{worst}}(n)=\max_{I\in\mathcal I_n} T(I)$ , giving a guaranteed upper bound.
Average case: $T_{\text{avg}}(n)=\sum_{I\in\mathcal I_n}P(I)\,T(I)$ , requiring an explicit input probability model.
Linear search illustrates the three cases: $\Theta(1)$ best, $\Theta(n)$ worst, and $\Theta(n)$ average (expected $\frac{n+1}{2}$ comparisons).
Worst‑case analysis is favored because it needs no probabilistic assumptions and ensures reliability for all inputs, especially in real‑time or safety‑critical systems.

Browse all research articles

Logistic Regression: Model, Training, and Worked Examples

Footnotes

Logistic Regression (Intuition + Math) — Sigmoid, Odds, and Decision Boundary

1) Model description with a concrete probability example

Footnotes

2) Decision boundary and classification threshold

Training logistic regression with log loss (high level)

Footnotes

3) Worked examples (from intuition to computation)

Example A: Spam vs. not spam (text features)

Example B: Credit default risk (calibration + threshold tradeoff)

Footnotes

How coefficient signs affect probability (toy illustration)

4) Regularization and variants (why L2L_2L2​ / L1L_1L1​ matter)

Footnotes

Use “log-odds linearity” as your diagnostic

Probability calibration vs. classification accuracy

5) Assumptions, strengths, and limitations

Assumptions (modeling level)

Strengths

Limitations

Footnotes

Common exam-style questions (with examples)

How logistic regression is typically applied (end-to-end)

Feature engineering

Train with regularized log loss

Select threshold & tune

Predict probabilities

Logistic Regression Quick Deck

What is the logit?

Knowledge Check

Explore Related Topics

4) Regularization and variants (why $L_2$ / $L_1$ matter)