[FAQ] : Homework 4 Q5-Q6 - do I cross-validate on df_train or df_full_train? #466

Course

machine-learning-zoomcamp

Question

Homework 4 Q5: for the 5-fold cross-validation, should KFold split df_train or df_full_train?

Answer

Use df_full_train, not df_train.

kfold = KFold(n_splits=5, shuffle=True, random_state=1)

for train_idx, val_idx in kfold.split(df_full_train):   # <- df_full_train
    df_tr = df_full_train.iloc[train_idx]
    df_va = df_full_train.iloc[val_idx]
    ...

Why: cross-validation is a stand-in for the validation split. df_full_train (80% of the data)
is what you normally split into train/val, so CV re-splits that same 80% five different ways.
That makes the CV scores directly comparable to the single df_val score from Q2 - and indeed
Q5's mean (0.732) matches Q2's AUC (0.732).

If you use df_train (60%) you are cross-validating a smaller dataset, each fold trains on 48%
of the data, and the numbers shift - your std will not match any of the answer options.

Two related traps:

1. Reuse df_full_train with the target still attached. Q5/Q6 iterate over df_full_train,
   which still has converted in it - unlike df_train/df_val, where you deleted it. So index
   the target from each fold (df_tr.converted.values) and drop it before building the feature
   dicts. Leaving it in gives you an AUC of ~1.0.
2. Reusing a single DictVectorizer across folds. Fit a fresh DictVectorizer inside each
   fold. Fitting once outside leaks information about the other folds into your validation set.

Also note Q6 wants you to re-initialize KFold for each C so every value of C sees the
identical five splits. Reusing a spent iterator, or creating it without shuffle=True and
random_state=1, will change your answer.

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links