Skip to content

Add captum, inspect, and llm as judges #21

@Shuyib

Description

@Shuyib

Feature Request

Please add captum, inspect, and llm as judge components in the project. This will allow for broader evaluation capabilities and integration with advanced model analysis tools.

Requested features to add:

  • captum for model interpretability of a encoder-decoder model
  • inspect review PII detection, toxicity and prompt injection
  • llm as a judge to review the logs incase of any issues

Motivation

Adding these judges will enhance model interpretability, debugging, and assessment by leveraging specialized tools.

Expected Outcome

  • All three judges (captum, inspect, llm) are available and can be called as part of the evaluation pipeline.
  • Documentation is updated to reflect their usage.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions