[github-actions]

✨ FAQ NEW

Course: machine-learning-zoomcamp
Section: module-4 (KFold and cross-validation appear under Module 4’s evaluation-metrics scope, so this hyperparameter-comparison best practice fits best in module-4.)
Related Issue: #480

Question

Why should we use the same KFold splits when comparing different hyperparameter values (e.g., different Logistic Regression C values)?

Decision Rationale

This specific guidance—why to reuse the same KFold splits when comparing hyperparameter values (e.g., different C in Logistic Regression)—is not fully answered by the existing KFold FAQ, which focuses more generally on what KFold does and where to instantiate it.

Placement Details

• Section ID: module-4
• Sort Order: End of section
• Filename Slug: kfold-same-splits-hyperparameter-comparison

---

🤖 Generated by FAQ Bot

Closes #480

[saeidsoheily]

We use the same KFold splits to ensure a fair, consistent, and reproducible comparison between different hyperparameter values, such as Logistic Regression's C.

1. Fair comparison
Suppose we want to compare three values of C using 5-fold cross-validation. If each value uses different random splits, the resulting AUC scores may differ because of both the hyperparameter and the particular training/validation data.
Using identical folds isolates the effect of changing C more effectively.

2. Reproducibility
Using the same folds ensures that you can repeat the cross-validation experiment with the same data partitions.
Note that reproducibility of the entire experiment also depends on controlling any randomness in the model training and preprocessing.

In simple terms, when tuning hyperparameters, you should change one factor at a time, and keep the training/validation splits identical so that differences in evaluation scores are more likely to come from the hyperparameter rather than random variations in the data splits.

[alexeygrigorev]

Thanks! I folded this into the existing KFold entry (009_24d2042733) in e89ff7e instead of adding a separate entry.