K-Nearest Neighbors (KNN)
✕KNN Concept Mathematically
- Tell me your friends, I'll tell your nature.
- Calculate distance from test point to all training points.
- Select the k nearest neighbors.
- Predict the class based on the majority class of the neighbors.
KNN Algorithm Steps:
Calculation Example for
(2, 3, 1) with k=3:| Feature A | Feature B | Feature C | Class | Distance to Test Point |
|---|---|---|---|---|
| 1 | 2 | 1 | Red | √((2-1)² + (3-2)² + (1-1)²) = 1.41 |
| 2 | 1 | 1 | Red | √((2-2)² + (3-1)² + (1-1)²) = 2 |
| 3 | 4 | 1 | Blue | √((2-3)² + (3-4)² + (1-1)²) = 1.41 |
| 5 | 6 | 1 | Blue | √((2-5)² + (3-6)² + (1-1)²) = 4.24 |
3 Nearest Neighbors: Red, Red, Blue => Prediction: Red |
KNN Concept Visually: Step - 1
KNN Concept Visually: Step - 2
KNN Concept Visually: Step - 3
- Prediction: Red (2 neighbors are Red, 1 is Blue)
Advantages and Disadvantages of KNN
- Simple and easy to understand.
- No training phase, making it fast for small datasets.
- Can capture complex relationships if enough data is available.
- Computationally expensive for large datasets.
- Sensitive to the choice of k and the scale of features.
- Does not perform well with high-dimensional data.
Advantages
Disadvantages
KNN Hyperparameters
KNN Hyperparameters and their Effects:
| Hyperparameter | Description | Effect on Model |
|---|---|---|
| n_neighbors: Number of neighbors | Number of nearest neighbors to consider for prediction. | Small k => overfitting, larger k => underfitting. |
| metric: Distance Metric | Method used to calculate distance (e.g. euclidean, manhattan, cosine) | Different metrics can capture different relationships in the data. |
| weights | Method to weight the neighbors (e.g. uniform, distance) | Distance weighting can improve performance by giving more influence to closer neighbors. |
Key hyperparameters for KNN and their impact on model performance.