Handle missing forecasts before summarisation

[seabbs-bot]

Summary

Provide utilities for handling missing forecasts before summarising scores, so that downstream analyses (pairwise comparisons, model confidence sets, ensemble importance) operate on a consistent and well-defined set of scores.

Motivation

When models submit forecasts for different subsets of targets or dates, summarisation and comparison functions can produce misleading results.
For example, a model that only forecasts easy targets will appear to perform better than one that forecasts everything.

Several downstream features depend on principled handling of missingness:

• get_pairwise_comparisons() already restricts to overlapping forecasts per pair, but this is pairwise rather than global
• Model confidence sets (Model confidence sets #1055) need a consistent comparison set across all models
• Ensemble importance (Model importance metrics: LOMO, Shapley values, and PAR for ensemble contribution #1121) needs complete forecast sets for ensemble construction

Kim et al.'s modelimportance package handles this in model_importance_summary() with three strategies: drop, worst-case imputation, and average imputation.
These are general-purpose strategies that belong in scoringutils rather than being reimplemented in each downstream feature in my opinion.

Possible approaches

• Filter to common targets: restrict to the intersection of targets covered by all (or a minimum number of) models. Simple and transparent.
• Impute missing scores: fill in missing scores with worst-case, average, or custom values. More flexible but requires assumptions.
• Flag and report: identify and report missingness without modifying data, leaving the choice to the user.

These are not mutually exclusive. A utility that detects and reports missingness, plus helpers for filtering or imputing, would cover most use cases.

Connection to existing functionality

• get_pairwise_comparisons() handles missingness per pair via overlapping forecast sets
• summarise_scores() silently summarises whatever is present
• validate_forecast() checks structure but not completeness

See also #1055 (model confidence sets) and #1121 (ensemble importance).

This was opened by a bot. Please ping @seabbs for any questions.

[seabbs]

I think this is quite worth having. Happy to take on a feature PR for it. One question is if we want this to be a function i.e. impute, documentation, a series of functions, a function that takes a function (with a few options) etc. I am thinking we want it to be a function with a series of options and then specify how to write your own (very standard data munging) as that should really be quite easy for people once they are aware they might want to do it.

[nikosbosse]

I like it. Johannes once made the point that pairwise comparisons are essentially a form of imputation. I'm not sure this is the best form of imputation, but it would be cool to add that bit of information to docs somewhere (and maybe even implement the same imputation as a standalone imputation).

[seabbs]

One thing to consider is do we want this imputation layer on the score level or on the forecast level. I assume we would want a variable to track imputed vs non-imputed forecasts if we add this?

I assumed that we would want this on the score level but perhaps we want it on the forecast level? I can't imagine we want both but I suppose that is also an option?