Add run meta stats to artifacts by tusharshah21 · Pull Request #191 · ossf/oss-crs

tusharshah21 · 2026-05-09T18:50:30Z

Summary

Added run-level statistics capture and persistence so each run writes a meta.json under its run directory, then exposed that metadata through the artifacts command output.

This was done to satisfy the issue requirement to track and retrieve:

LLM credits used
POVs found
Seeds shared
Number of builds requested through libCRS

User Impact

User-facing change
Internal-only change

CLI/API/config/docs impact and migration:

CLI output change: artifacts JSON now includes a top-level meta object when resolving a run.
No new required flags and no command rename/removal.
Existing consumers that ignore unknown JSON fields continue to work.
Consumers with strict JSON schema validation should allow the new optional meta field.

Release Note / Changelog

I updated CHANGELOG.md ([Unreleased]) for user-facing changes
No changelog entry needed (internal-only refactor/test/chore)

Deprecation or breaking behavior:

None.
No replacement path needed.
No planned removal window.

Validation

Added/updated unit test coverage for artifacts meta output behavior.
Ran static diagnostics on changed files with no reported errors.
Attempted targeted pytest run for the updated artifacts test module, but local execution in this Windows environment is blocked during collection by a Linux-only fcntl import dependency.

Checklist

I followed Conventional Commits
I updated docs for behavior/config/CLI changes
I added/updated tests for behavior changes
I considered backward compatibility and migration impact

Closes #80

azchin

Thanks, could change the meta.json format to something like the snippet below? There's additional implementation change requests commented.

Also please sign your commits and use Conventional Commits for commit messages.

{
  "totals": {
    "artifacts": {
      "povs": 2,
      "seeds": 4,
      "patches": 1,
      "bug_candidates": 2
    },
    "llm": {
      "credits_used": 1.65
    },
    "sidecar": {
      "patch_builds": 4,
      "patch_tests": 2,
      "pov_runs": 8
    }
  },
  "crs": {
    "crs_a": {
      "artifacts": {
        "povs": 2,
        "seeds": 3,
        "patches": 1,
        "bug_candidates": 0
      },
      "llm": {
        "credits_used": 1.25
      },
      "sidecar": {
        "patch_builds": 4,
        "patch_tests": 2,
        "pov_runs": 7
      }
    }
  }
}

azchin · 2026-05-11T18:27:44Z

+                        continue
+        return round(total, 6)
+
+    def _count_build_requests_from_service_logs(self, services_dir: Path) -> int:


I also don't like the scrape of the logs for counting build requests. Please modify libCRS itself to log more systematically to a file, similar to my feedback for counting LiteLLM requests.

Also, please count the number of requests for run-pov and apply-patch-test as well. And do it per-CRS.

…bles

tusharshah21 · 2026-05-13T16:40:00Z

schema update to nested totals + per-CRS (patches, bug_candidates), structured JSONL metrics replacing log scraping, LiteLLM spend polling via /global/spend/report, and artifact counting centralized in WorkDir are all clean and cohesive enough to ship as one PR. CI green, no split needed.
Let me know if there are any other suggestion!

azchin

Added suggestions to get LiteLLM reporting working. Tracking number of artifacts and libCRS calls look good.

Again, please sign-off your commits https://github.com/ossf/oss-crs/pull/191/checks?check_run_id=75833728185.
The easiest way to do this is probably interactive rebase, and selectively reword your commits to manually add the Signed-off-by: Author Name <authoremail@example.com> for each of your changes.

azchin · 2026-05-15T04:26:09Z

    depends_on:
      oss-crs-litellm-key-gen:
-        condition: service_completed_successfully
+  condition: service_healthy


Wrong indentation, this breaks the compose.

Suggested change

condition: service_healthy

condition: service_healthy

azchin · 2026-05-15T04:30:58Z

+def get_global_spend_report() -> dict | list | None:
+    """Fetch global spend report from LiteLLM."""
+    url = f"{LITELLM_API_URL}/global/spend/report"
+    headers = {
+        "Authorization": f"Bearer {LITELLM_MASTER_KEY}",
+    }
+    try:
+        response = requests.get(url, headers=headers, timeout=30)
+        response.raise_for_status()
+        return response.json()
+    except requests.exceptions.RequestException as e:
+        print(f"Error fetching global spend report: {e}")
+        return None
+
+
+def _iter_dicts(obj):
+    if isinstance(obj, dict):
+        yield obj
+        for v in obj.values():
+            yield from _iter_dicts(v)
+    elif isinstance(obj, list):
+        for item in obj:
+            yield from _iter_dicts(item)
+
+
+def _as_float(value) -> float | None:
+    if isinstance(value, (int, float)):
+        return float(value)
+    if isinstance(value, str):
+        try:
+            return float(value)
+        except ValueError:
+            return None
+    return None
+
+
+def summarize_spend_report(report, key_requests: dict[str, dict]) -> dict:
+    """Build a stable summary with totals and per-CRS credits used."""
+    per_key: dict[str, float] = {}
+
+    if report is not None:
+        for entry in _iter_dicts(report):
+            key = None
+            for key_field in ("api_key", "key", "user_api_key", "token"):
+                raw_key = entry.get(key_field)
+                if isinstance(raw_key, str) and raw_key:
+                    key = raw_key
+                    break
+            if key is None:
+                continue
+
+            spend = None
+            for spend_field in ("spend", "total_spend", "credits_used", "cost"):
+                spend = _as_float(entry.get(spend_field))
+                if spend is not None:
+                    break
+            if spend is None:
+                continue
+            per_key[key] = per_key.get(key, 0.0) + spend
+
+    crs_summary: dict[str, dict[str, float]] = {}
+    for crs_name, info in key_requests.items():
+        api_key = str(info.get("api_key", ""))
+        crs_summary[crs_name] = {"credits_used": round(per_key.get(api_key, 0.0), 6)}
+
+    total = round(sum(v["credits_used"] for v in crs_summary.values()), 6)
+    return {
+        "totals": {"credits_used": total},
+        "crs": crs_summary,
+        "updated_at": int(time.time()),
+    }
+
+
+def write_spend_summary(summary: dict) -> None:
+    dst = SPEND_REPORT_PATH
+    tmp = f"{dst}.tmp"
+    with open(tmp, "w") as f:
+        json.dump(summary, f, indent=2, sort_keys=True)
+        f.write("\n")
+    os.replace(tmp, dst)


write_spend_summary atomic rename doesn't work because of different filesystems.
/global/spend/report uses key hashes, so using /key/info instead because we have the plaintext keys. Old impl was returning 0 for spending.

Suggested change

def get_global_spend_report() -> dict | list | None:

"""Fetch global spend report from LiteLLM."""

url = f"{LITELLM_API_URL}/global/spend/report"

headers = {

"Authorization": f"Bearer {LITELLM_MASTER_KEY}",

}

try:

response = requests.get(url, headers=headers, timeout=30)

response.raise_for_status()

return response.json()

except requests.exceptions.RequestException as e:

print(f"Error fetching global spend report: {e}")

return None

def _iter_dicts(obj):

if isinstance(obj, dict):

yield obj

for v in obj.values():

yield from _iter_dicts(v)

elif isinstance(obj, list):

for item in obj:

yield from _iter_dicts(item)

def _as_float(value) -> float | None:

if isinstance(value, (int, float)):

return float(value)

if isinstance(value, str):

try:

return float(value)

except ValueError:

return None

return None

def summarize_spend_report(report, key_requests: dict[str, dict]) -> dict:

"""Build a stable summary with totals and per-CRS credits used."""

per_key: dict[str, float] = {}

if report is not None:

for entry in _iter_dicts(report):

key = None

for key_field in ("api_key", "key", "user_api_key", "token"):

raw_key = entry.get(key_field)

if isinstance(raw_key, str) and raw_key:

key = raw_key

break

if key is None:

continue

spend = None

for spend_field in ("spend", "total_spend", "credits_used", "cost"):

spend = _as_float(entry.get(spend_field))

if spend is not None:

break

if spend is None:

continue

per_key[key] = per_key.get(key, 0.0) + spend

crs_summary: dict[str, dict[str, float]] = {}

for crs_name, info in key_requests.items():

api_key = str(info.get("api_key", ""))

crs_summary[crs_name] = {"credits_used": round(per_key.get(api_key, 0.0), 6)}

total = round(sum(v["credits_used"] for v in crs_summary.values()), 6)

return {

"totals": {"credits_used": total},

"crs": crs_summary,

"updated_at": int(time.time()),

}

def write_spend_summary(summary: dict) -> None:

dst = SPEND_REPORT_PATH

tmp = f"{dst}.tmp"

with open(tmp, "w") as f:

json.dump(summary, f, indent=2, sort_keys=True)

f.write("\n")

os.replace(tmp, dst)

def get_key_spend(api_key: str) -> float:

"""Fetch cumulative spend for a single key via /key/info."""

url = f"{LITELLM_API_URL}/key/info"

headers = {

"Authorization": f"Bearer {LITELLM_MASTER_KEY}",

}

try:

response = requests.get(

url, headers=headers, params={"key": api_key}, timeout=30

)

response.raise_for_status()

data = response.json()

info = data.get("info") or data

spend = info.get("spend")

if isinstance(spend, (int, float)):

return float(spend)

return 0.0

except requests.exceptions.RequestException as e:

print(f"Error fetching spend for key: {e}")

return 0.0

def collect_spend_summary(key_requests: dict[str, dict]) -> dict:

"""Build spend summary by querying per-key spend from LiteLLM."""

crs_summary: dict[str, dict[str, float]] = {}

total = 0.0

for crs_name, info in key_requests.items():

api_key = str(info.get("api_key", ""))

spend = get_key_spend(api_key) if api_key else 0.0

crs_summary[crs_name] = {"credits_used": round(spend, 6)}

total += spend

return {

"totals": {"credits_used": round(total, 6)},

"crs": crs_summary,

"updated_at": int(time.time()),

}

def write_spend_summary(summary: dict) -> None:

with open(SPEND_REPORT_PATH, "w") as f:

json.dump(summary, f, indent=2, sort_keys=True)

f.write("\n")

azchin · 2026-05-15T04:34:55Z

+    # Poll LiteLLM spend report and keep writing a host-recoverable summary file.
+    while not _SHUTDOWN:
+        report = get_global_spend_report()
+        summary = summarize_spend_report(report, key_requests)


Update main to use the new spend report function

Suggested change

# Poll LiteLLM spend report and keep writing a host-recoverable summary file.

while not _SHUTDOWN:

report = get_global_spend_report()

summary = summarize_spend_report(report, key_requests)

# Poll LiteLLM spend and keep writing a host-recoverable summary file.

while not _SHUTDOWN:

summary = collect_spend_summary(key_requests)

tusharshah21 added 3 commits May 10, 2026 00:18

Add run meta stats to artifacts

5b14c5c

resolve conflict

3b8ddd5

style: format run-meta files

af48d09

azchin requested changes May 11, 2026

View reviewed changes

tusharshah21 added 2 commits May 13, 2026 21:55

feat(meta): restructure run stats and add structured usage collection

c3bcc7c

fix(renderer): fallback spend report path for legacy workdir test dou…

f6c0eac

…bles

Merge branch 'main' into feat/run-meta-stats

a1c22d8

azchin requested changes May 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add run meta stats to artifacts#191

Add run meta stats to artifacts#191
tusharshah21 wants to merge 6 commits into
ossf:mainfrom
tusharshah21:feat/run-meta-stats

tusharshah21 commented May 9, 2026

Uh oh!

azchin left a comment

Uh oh!

azchin May 11, 2026

Uh oh!

tusharshah21 May 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tusharshah21 commented May 13, 2026

Uh oh!

azchin left a comment

Uh oh!

azchin May 15, 2026

Uh oh!

azchin May 15, 2026

Uh oh!

azchin May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tusharshah21 commented May 9, 2026

Summary

User Impact

Release Note / Changelog

Validation

Checklist

Uh oh!

azchin left a comment

Choose a reason for hiding this comment

Uh oh!

azchin May 11, 2026

Choose a reason for hiding this comment

Uh oh!

tusharshah21 May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tusharshah21 commented May 13, 2026

Uh oh!

azchin left a comment

Choose a reason for hiding this comment

Uh oh!

azchin May 15, 2026

Choose a reason for hiding this comment

Uh oh!

azchin May 15, 2026

Choose a reason for hiding this comment

Uh oh!

azchin May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants