Fix glmnet lambda handling: predict-time fallback and fit-time cold-start convergence failure

[ielbadisy]

Summary

Two related bugs in the glmnet learner, both surfaced by the same downstream failure: an Elastic Net learner in a real analysis pipeline (CLAVUS, VUS-reclassification prediction) produced a single constant predicted probability for every test row (AUROC exactly 0.5, sensitivity 0).

1. predict_xy lambda fallback. If spec$lambda wasn't threaded through to predict_xy, it fell back to state$state$lambda[1] -- the largest (most-regularized) value in glmnet's default path, not the tuned/intended one.

2. fit_xy cold-lambda convergence failure (the actual root cause on real data). fit_xy requested a single lambda directly via glmnet::glmnet(..., lambda = spec$lambda), cold-starting the coordinate-descent solver with no warm-started path. On a wide/high-cardinality design matrix (~35 raw predictors, ~150+ columns after one-hot encoding several categorical variables with many levels), this failed to converge for the CV-selected alpha=0.5, lambda=0.01 and glmnet silently returned an empty, all-zero-coefficient model (fit$lambda == Inf), which then predicted the same constant probability for every observation -- independent of the predict_xy fix in (1).

Fix (1) alone was not sufficient to resolve the real pipeline's degenerate result; fix (2) was required and is the one that actually restores correct behavior on that data.

Fix

• fit_xy now always fits the full regularization path (never passes a bare lambda= to glmnet::glmnet()).
• The lambda intended for prediction (spec$lambda, or the smallest/least-regularized path value if unset) is recorded at fit time and extracted via predict(..., s=) at predict time -- the supported, warm-started way to get predictions at an arbitrary path point.

Verification

• Verified directly against the real dataset that triggered this: funcml::tune() + predict() now reaches AUROC 0.781 / AUPRC 0.681 through the standard funcml path, exactly matching an independent glmnet::cv.glmnet() reference fit (s = 0.01) on the identical training/test data.
• tests/testthat/test-glmnet-predict-lambda.R: 3 tests -- non-degenerate predictions when spec$lambda is unset at fit time, correct lambda reuse when spec$lambda is set, and a direct structural check that fit_xy always produces a multi-point path (length(fit$state$state$lambda) > 1) rather than a single cold-fit lambda.
• Confirmed the degenerate-prediction test fails against the pre-fix code (AUROC = 0.50 exactly) and passes after the fix.
• tests/testthat/test-learner-audit-contract.R (glmnet fit/predict contract) and test-estimate*.R (which also exercise glmnet as a meta-learner) still pass.
• Note: I did not attempt to reproduce the convergence failure in a small synthetic example -- several attempts with wide one-hot designs, extreme-scale numeric predictors, and rare categorical levels did not trigger it, so it appears specific to structure in the real data. The structural test above asserts the invariant that prevents the failure by construction instead.

Bumped to 0.7.3, NEWS.md updated with both fixes.