Designing a K-Nearest Neighbour (KNN) Exam-Failure Classifier

To predict whether a student will pass or fail the exam using K-Nearest Neighbour (K-Nearest Neighbour), we treat each student as a feature vector and the exam outcome as a class label. KNN is non-parametric: for a new student, it finds the $k$ closest training students using a distance metric and assigns the class most common among those neighbours (optionally using distance-weighting). This overall decision rule is consistent with KNN classifier descriptions in common ML references and KNN implementations. 2

From the table, we build inputs from:

• Language (Java/C++/Python)
• Passed all Assignments (Yes/No)
• GPA

The target is:

• Passed Exam (Yes/No)

KNN design depends heavily on (i) how we encode categorical variables, (ii) feature scaling, and (iii) selecting $k$ and the distance metric. Since KNN uses distances in feature space, features on different scales can dominate the distance calculations; thus scaling is a key preprocessing step.

Data preparation and feature engineering

1) Define the supervised learning problem

Let each student $i$ be represented by a feature vector $x_i \in \mathbb{R}^d$ and label $y_i \in \{0,1\}$:

• $y_i = 1$ if Passed Exam = Yes, otherwise $0$.

KNN stores the training pairs $(x_i, y_i)$; during prediction it uses the labels of the nearest stored vectors.

2) Encode categorical features (Language)

KNN requires numeric vectors. Encode Language using one-hot encoding into three binary features:

• $I_{\text{Java}}$, $I_{\text{C++}}$, $I_{\text{Python}}$ (only one of them is 1 for each student).

Keep Passed all Assignments as a binary indicator:

• $I_{\text{AssignmentsYes}} \in \{0,1\}$

Now each student becomes: $x_i = [I_{\text{Java}}, I_{\text{C++}}, I_{\text{Python}}, I_{\text{AssignmentsYes}}, \text{GPA}]$ so here $d=5$.

Pro Tip: With KNN, categorical encoding choices can change the geometry of the feature space. One-hot encoding is standard for Euclidean-style distances, because categories become orthogonal indicator directions.

3) Scale numeric features (especially GPA)

Because KNN relies on distances, features with larger numeric ranges can disproportionately influence the result; scaling reduces this effect and improves distance fairness. A typical approach is standardization or normalization applied to numeric features like GPA before training and prediction.

In this dataset, GPA is numeric while the other features are 0/1. Even so, scaling GPA is still recommended to ensure stable behaviour and prevent it from overwhelming or being overwhelmed by binary dimensions.

Distance metric and voting design

Distance metric

A common default in KNN implementations is Minkowski distance; scikit-style KNN uses Minkowski and typically supports Euclidean by setting $p=2$. 2

For two students $x$ and $z$:

• Choose $p=2$ for Euclidean distance: $dist(x,z)=\sqrt{\sum_{j=1}^{d}(x_j-z_j)^2}$ Other norms (e.g., Manhattan, $p=1$) can be explored, but the Euclidean choice is a strong baseline.

Voting rule and the role of the weights parameter

KNN classification is often implemented as a “vote” over the neighbour labels. The weights parameter typically supports:

• uniform: all neighbours vote equally
• distance: closer neighbours vote more (often using inverse distance)

Given this dataset is small, distance weighting can reduce sensitivity to noisy neighbours (e.g., a single close but contradictory example).

Warning: If you set $k$ too small, KNN becomes sensitive to noise; if $k$ is too large, it may smooth over local patterns. This bias–variance trade-off is why $k$ must be tuned using validation.

Selecting hyperparameter $k$ (and evaluating the design)

Cross-validation for choosing $k$

Since the dataset in the prompt is small, performance estimates from a single train/test split may vary a lot. Use k-fold cross-validation to select $k$ (and optionally the distance metric or weights). Cross-validation is widely used to choose hyperparameters by averaging scores over multiple splits. A typical setup is:

• Candidate values like $k \in \{1,3,5,7\}$
• Use stratification so each fold keeps similar proportions of Pass/Fail labels (important for classification).

Tie-handling

For classification, if votes tie, some implementations may depend on neighbour ordering; it’s another reason to prefer odd $k$ for binary classification.