Skip to content

promptfoo.md

Latest commit

 

History

History
111 lines (83 loc) · 3.28 KB

promptfoo.md

File metadata and controls

111 lines (83 loc) · 3.28 KB
Raw

Use Promptfoo with CometAPI

Canonical docs: https://apidoc.cometapi.com/integrations/promptfoo

Use this guide to configure Promptfoo with CometAPI by setting the base URL, API key, and model or provider options. Promptfoo is a framework for prompt evaluation, regression testing, and AI red teaming. Promptfoo includes a cometapi provider that calls CometAPI through OpenAI-compatible endpoints.

Prerequisites

Configure the integration

Set your CometAPI API key

Set COMETAPI_KEY before you run evaluations:

read -rsp "CometAPI API key: " COMETAPI_KEY
printf '\n'
export COMETAPI_KEY

Choose a provider format

Promptfoo uses the following provider ID format for CometAPI:

providers:
  - cometapi:<type>:<model>

Use these provider types:

Type Use case
chat Chat completions, vision, and multimodal prompts
completion Text completion models
embedding Text embedding evaluations
image Image generation evaluations

You can also use cometapi:your-model-id for the default chat mode.

Run a CLI evaluation

The following command runs a one-prompt chat evaluation:

npx promptfoo@latest eval \
  --prompts "Write a concise release note for {{feature}}" \
  --vars feature="a unified API dashboard" \
  -r cometapi:chat:your-model-id

Create a Promptfoo config file

The following promptfooconfig.yaml evaluates the same prompt against a CometAPI model:

prompts:
  - "Classify this support request: {{message}}"

providers:
  - id: cometapi:chat:your-model-id
    config:
      temperature: 0.2
      max_tokens: 256

tests:
  - vars:
      message: "The API key works locally but fails in production."
    assert:
      - type: contains-any
        value:
          - authentication
          - configuration

Run the config file with Promptfoo:

npx promptfoo@latest eval -c promptfooconfig.yaml

Provider examples

Use a provider type that matches the model behavior that you want to evaluate:

providers:
  - cometapi:chat:your-model-id
  - cometapi:completion:your-model-id
  - cometapi:embedding:your-model-id
  - cometapi:image:your-model-id

For multimodal tests, keep the chat provider type and pass image variables through Promptfoo test variables.

Troubleshooting

Promptfoo cannot find the API key

Confirm that COMETAPI_KEY is exported in the same shell session that runs promptfoo eval.

The provider type does not match the model

Use chat for conversational and multimodal models, embedding for embedding models, and image for image generation models.

The model ID fails

Replace your-model-id with an exact model ID from the CometAPI Models page.

Related resources