Skip to content

Added beyond-accuracy evaluation metrics - #65

Open
giuspillo wants to merge 4 commits into
enoche:masterfrom
giuspillo:master
Open

giuspillo wants to merge 4 commits into
enoche:masterfrom
giuspillo:master

Conversation

@giuspillo

Copy link
Copy Markdown
Contributor

Hi @enoche,

thank you as always for this great job! I hope this can be interesting for you.

I've integrated the implementation for five beyond-accuracy metrics to the TopK evaluator, alongside the existing relevance-based ones (Recall, Precision, NDCG, MAP). Unlike those, these measure properties of how recommendations are distributed across the catalog, using item popularity computed from the training set.

  • GiniIndex: inequality of the recommendation-frequency distribution (0 = uniform, 1 = one item gets everything).
  • AveragePopularity: average training-set popularity of recommended items; higher = more popularity bias.
  • TailPercentage: share of recommendations coming from the long tail (least-popular items), threshold configurable via tail_ratio. Default to 0.1.
  • ItemCoverage: fraction of the catalog that appears in at least one user's top-K.
  • ShannonEntropy: entropy of the recommendation-frequency distribution; complements GiniIndex as a diversity/equity measure.

Changes

  • src/utils/metrics.py: implementations of the five metrics + item_metrics_dict registry, kept separate from the relevance-metric registry since they take item ids + popularity rather than a relevance mask. The four occourences of np.float have been replaced with np.float64 to adapt the current numpy versions.
  • src/utils/topk_evaluator.py: TopKEvaluator now computes both metric groups and merges results into metric_dict following the order metrics are declared in the config (previously accuracy metrics always came first regardless of config order).
  • src/configs/overall.yaml: all five item-distribution metrics added to the default metrics list; tail_ratio (default 0.1) added for TailPercentage as default value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant