Skip to content

HuggingFace Inference API provider #484

Description

@jeremymanning

Context

The provider set is Dartmouth Chat + HuggingFace only (decided 2026-08-21; ADR 0001). HuggingFace support means the hosted Inference API, not the local transformers inference in integrations/huggingface_model.py / lazy_huggingface_model.py — those stay in the frozen layer and retire under #430.

What to build

A HuggingFaceProvider in the canonical layer (models/providers/), speaking the Inference API over HTTP with aiohttp (already a core dependency — same shape as DartmouthProvider, no new core deps, no provider extra required unless the SDK is used).

  • Text-generation against a small set of served models, selected from the live catalog rather than a hard-coded list (two rounds of hard-coded Anthropic ids were wrong; do not repeat that here).
  • HF_TOKEN from the environment or ~/.orchestrator/.env, same as other credentials.
  • Free-tier vs paid distinction must fail closed: a model the token cannot serve must raise, not silently route elsewhere. Model fallback across providers does not exist (ADR 0001).
  • Registered in the normal model registry when a credential is present, so pipelines can select the models without touching the provider directly.

Acceptance criteria

  • Contract tests (hermetic, recorded/mocked HTTP boundary is fine at this layer) covering: request shape, auth header, error mapping (401/403/402/429/5xx), model-unavailable, and the registry integration.
  • A live-huggingface job in live-tests.yml, mirroring live-dartmouth: hard-fails when HF_TOKEN is missing under ORCHESTRATOR_REQUIRE_LIVE=1, and fails when every test skips.
  • Live tests exercise at least one real served model end-to-end: generate() returns non-empty content, models.list()/discovery works, health_check() works.
  • Reasoning-model behaviour handled: a response that spends its token budget on reasoning and returns empty content must not be reported as a successful empty answer (learned from the Dartmouth work — see Verify bare model-family resolution against the live API (blocked: no Anthropic credit) #432 comments).
  • README provider table flips HuggingFace to supported only after the live job passes remotely.

Out of scope

Related: #429 (adapter-layer consolidation — this provider lands in models/providers/), #430 (retirement of the frozen layer), ADR 0001 provider policy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureRelated to toolbox design/architectureenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions