Find ways to help users find the most useful models from a large collection of scores

[jcken95]

One problem with {scoringutils} as a user (which is more inherent to scores, rather than the software that is {scoringutils}), is that outputting a large number of scores, across different forecasting units (time horizons, locations, location levels, ...) is there there is an incredibly large amount of data for a human to process in order to make a choice about model selection. Of course, the "best" model (or best ensemble when combining different collections of models) depends on which scoring metric is used, and perhaps other external factors. These external factors could be the quality of the prediction, but also the maintainability of a model or its runtime. These may influence if a model is worth pursuing. An easy to maintain, but non-optimal scoring model may be preferred in a fast-paced operational environment over a theoretically optimal model which is hard to implement or has lots of difficult to validate assumptions baked into it.

I am in essence asking for {scoringutils} to consider including methods that help users make pragmatic choices about models and ensembles.

{scoringutils} could be expanded to help users cut out less-than-ideal model choices. Here are some ideas off the top of my head, I don't necessarily agree with all of them, but in my opinion they are potentially useful avenues, or at least fun to think about. Perhaps they have been explored elsewhere, these are just ideas I have used myself or off the top of my head.

1. A function, or suite of functions for Pareto analysis of selected scores. See for example a {scoringutils}-{rPref} approach here. A front and plot of scores could be output. For >2 scores a graphic may be difficult. This has been done in practice by myself, and for the example I demonstrate on, is fast and simple to implement (given a set of scores).
2. Some kind of statistical model - could a Plackett-Luce model help us infer a ranking? No idea if this makes sense but it's cool?
3. An idea based on electoral systems: consider each score a preference vote for a model. Preferences can be combined to "elect" a best model (or set of good models). Again, no idea if this makes sense, but interesting intersection of scoring with other fields.
4. Guiding a user towards a bespoke utility function (score) based on a weighted sum of other scores. For example, combining WIS or RPS for different location levels/spatial granularities as in Kennedy et al (2026)

Note a challenge with 4. is combing scores with different scales, for example the units of WIS + RPS are completely meaningless. Multliattribute value elicitiation gives a method for doing this, it is a simple algorithm but imposes a large cognitive burden on the user

[seabbs]

Thanks for this @jcken95 - nice thoughts. I like the idea of all of them.

2. is mostly the avenue we have been thinking about recently I think in our own work but nowhere near at a stage where we can provide a "standard" model for people to use that would simplify things for them vs make them more complicated.

3. I like the idea of this it is very close to how some ensemble methods are thought about so from that angle makes a lot of sense. I think @kathsherratt might be more across that literature than me though.

4. Is what I have been thinking about recently but as we discussed on the call no real progress on what this looks like beyond it is kind of like prior elicitation. This one feels the most like something to put in scoringutils vs making some kind of tool that interfaces with it (which a lot of the others might be?).

5. Seems like it could be a great target for this small package extension idea to me. I still need to engage with this a bit more though to get a sense of general utility.

On that on the call @jcken95 sent me https://link.springer.com/chapter/10.1007/978-3-319-65052-4_1 which seems worth checking out.

[jcken95]

If not in {scoringutils} I think exploring these approaches could be a fun article. Let's chat sometime :)

[seabbs]

Yes totally agree. Lets do it!

[nikosbosse]

I think definitely for 4. (i.e. weighted sums of scores) it would be really cool to have some kind of vignette or use case. I remember us talking about this when working on the log transformation. I don't have a good idea, however, how to put that into scoringutils in a way that's both useful and non-trivial

[seabbs]

Yes agree and agree. This has been on my mind a lot recently. I feel like a trivial version might be okay in the first instance and then build in complexity ?