[FAQ]Why should we use the same KFold splits when comparing hyperparameters? #480

Course

machine-learning-zoomcamp

Question

Why should we use the same KFold splits when comparing different hyperparameter values?

Answer

When comparing different hyperparameter values, such as different values of C in Logistic Regression, we want the comparison to reflect the effect of the hyperparameter rather than differences in the data splits.

Using the same KFold configuration:

KFold(n_splits=5, shuffle=True, random_state=1)

makes the comparison reproducible and allows each value of C to be evaluated using the same fold assignments.

If different random splits are used for every value of C, differences in the mean AUC may partly come from the different train/validation splits rather than from C itself.

Using the same folds therefore makes the comparison between hyperparameter values fairer and easier to reproduce.

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links