Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@ jobs:
cache-dependency-path: requirements-dev.txt
- name: Install validation dependencies
run: python -m pip install -r requirements-dev.txt
- name: Run review-hardening regression tests
run: python -m unittest discover -s tests -v
- name: Validate schemas, records, and references
run: python tools/validate.py
- name: Build chronological timeline
run: python tools/build_timeline.py --format json > /tmp/modelmap-timeline.json
- name: Build 2023 year view
run: python tools/build_year.py 2023 --format markdown > /tmp/modelmap-2023.md
run: python tools/build_year.py 2023 --format json > /tmp/modelmap-2023.json
11 changes: 8 additions & 3 deletions data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,16 @@ Directories are introduced as canonical records are added; empty directories do

Initial canonical records should be YAML where hand-editability is valuable. JSON Schema files in `/schemas` define the contracts. Validation tooling may normalize YAML to JSON internally.

One canonical entity per file is preferred for reviewable diffs:
Exactly one canonical record is stored per YAML/JSON file. This is a repository invariant, not merely a preference: it keeps Git diffs, provenance, review state, and correction history scoped to one stable ID.

```text
data/techniques/grouped-query-attention.yaml
data/releases/deepseek-v3.yaml
data/sources/deepseek-v3-technical-report.yaml
data/claims/deepseek-v3-uses-mla.yaml
```

Atomic claims may be grouped by subject when that improves review ergonomics, provided every claim retains a globally unique canonical ID.
Do not store arrays of claims or other multi-record containers in canonical files. Related atomic claims may share a naming prefix or directory grouping, but each retains its own file and globally unique canonical ID.

## Identifier conventions

Expand All @@ -60,7 +61,7 @@ claim:llama-3.1-uses-gqa
relationship:llama-3.1-uses-gqa
```

IDs must remain stable if display names change.
IDs must remain stable if display names change. Entity types are bound to canonical namespaces as documented in `docs/ontology.md`.

## Null, unknown, and not applicable

Expand All @@ -74,6 +75,10 @@ Do not overload values:

Evidence references should identify a source plus the smallest useful locator available: section, table, page, config key, file/line range, release-note heading, or commit.

## Dates and timestamps

Quote ISO dates and timestamps in hand-authored YAML when practical. The shared record loader normalizes YAML-native `date` and `datetime` scalars to ISO strings before schema validation and generation, so equivalent unquoted YAML remains valid.

## Review policy

A record may move through:
Expand Down
9 changes: 9 additions & 0 deletions data/events/2023-12-08-mixtral-8x7b-weights.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
id: event:2023-12-08-mixtral-8x7b-weights
type: event
name: Mixtral 8x7B weights released via magnet link
event_type: weights_released
subject: release:mixtral-8x7b
occurred_at: '2023-12-08'
evidence:
- source:mixtral-8x7b-magnet-post
notes: The formal Mistral product announcement followed on 2023-12-11 and is represented as a separate event.
9 changes: 9 additions & 0 deletions data/events/2023-12-11-mixtral-8x7b-announced.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
id: event:2023-12-11-mixtral-8x7b-announced
type: event
name: Mixtral 8x7B formally announced by Mistral AI
event_type: model_announced
subject: release:mixtral-8x7b
occurred_at: '2023-12-11'
evidence:
- source:mixtral-8x7b-announcement
notes: Formal product announcement; the weights had already been distributed via magnet link on 2023-12-08.
8 changes: 0 additions & 8 deletions data/events/2023-12-11-mixtral-8x7b-weights.yaml

This file was deleted.

9 changes: 9 additions & 0 deletions data/sources/mixtral-8x7b-magnet-post.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
id: source:mixtral-8x7b-magnet-post
type: source
title: Mistral AI Mixtral 8x7B magnet-link post
source_type: official_announcement
url: https://x.com/MistralAI/status/1733150512395038967
publisher: Mistral AI
published_at: '2023-12-08'
retrieved_at: '2026-09-29T07:20:00Z'
notes: Original Mistral AI post distributing the Mixtral 8x7B weights via magnet link.
37 changes: 34 additions & 3 deletions docs/ontology.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,34 @@ paper:arxiv-2412-19437
benchmark:mmlu
```

Namespaces are bound to entity types. The v0.1 mapping is:

| Entity type | Namespace |
|---|---|
| model_family | `family:` |
| model_release | `release:` |
| model_variant | `variant:` |
| architecture_spec | `arch:` |
| technique | `technique:` |
| training_run | `training:` |
| implementation | `implementation:` |
| paper | `paper:` |
| source | `source:` |
| claim | `claim:` |
| relationship | `relationship:` |
| event | `event:` |
| dataset | `dataset:` |
| dataset_version | `dataset-version:` |
| benchmark | `benchmark:` |
| benchmark_version | `benchmark-version:` |
| evaluation_protocol | `eval-protocol:` |
| evaluation_run | `eval-run:` |
| organization | `org:` |
| person | `person:` |
| license | `license:` |
| hardware_system | `hardware:` |
| product_system | `product:` |

Display names and aliases are mutable metadata, not primary keys.

## 3. Claim model
Expand All @@ -86,16 +114,17 @@ A claim is the smallest independently supportable assertion.

```yaml
id: claim:example-model-uses-gqa
type: claim
subject: release:example-model
predicate: uses_technique
object: technique:grouped-query-attention
status: confirmed_primary
evidence:
- source:example-technical-report
valid_from: 2025-01-01
- source: source:example-technical-report
valid_from: '2025-01-01'
```

Claims may point to entities (`object`) or literal values (`value`), but not both simultaneously.
Claims may point to entities (`object`) or literal values (`value`), but not both simultaneously. Claim and relationship endpoints must use canonical IDs, never display names.

## 4. Evidence states

Expand Down Expand Up @@ -184,6 +213,8 @@ The UI should warn when two displayed results use materially different protocols

`occurred_at` belongs to an Event. `published_at` belongs to a Source. `retrieved_at` records when ModelMap observed a source. These dates are intentionally separate.

Quote dates in hand-authored YAML for readability and portability. The validator also normalizes YAML date/timestamp scalars to ISO strings before applying JSON Schema.

## 10. Source hierarchy

Preferred evidence order for technical claims:
Expand Down
10 changes: 8 additions & 2 deletions schemas/claim.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,19 @@
"$id": "https://modelmap.dev/schemas/claim.schema.json",
"title": "ModelMap Claim",
"type": "object",
"$defs": {
"canonicalId": {
"type": "string",
Comment thread
Alajmah marked this conversation as resolved.
Resolved
"pattern": "^(?:family|release|variant|arch|technique|training|implementation|paper|source|claim|relationship|event|dataset|dataset-version|benchmark|benchmark-version|eval-protocol|eval-run|org|person|license|hardware|product):[a-z0-9][a-z0-9._-]*$"
}
},
"required": ["id", "type", "subject", "predicate", "status", "evidence"],
"properties": {
"id": {"type": "string", "pattern": "^claim:[a-z0-9][a-z0-9._-]*$"},
"type": {"const": "claim"},
"subject": {"type": "string", "minLength": 3},
"subject": {"$ref": "#/$defs/canonicalId"},
"predicate": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"},
"object": {"type": "string", "minLength": 3},
"object": {"$ref": "#/$defs/canonicalId"},
"value": {},
"status": {
"type": "string",
Expand Down
25 changes: 25 additions & 0 deletions schemas/entity.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,30 @@
"updated_at": {"type": ["string", "null"], "format": "date-time"},
"metadata": {"type": "object", "additionalProperties": true}
},
"allOf": [
{"if": {"properties": {"type": {"const": "model_family"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^family:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "model_release"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^release:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "model_variant"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^variant:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "architecture_spec"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^arch:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "technique"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^technique:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "training_run"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^training:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "implementation"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^implementation:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "paper"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^paper:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "source"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^source:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "claim"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^claim:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "relationship"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^relationship:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "event"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^event:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "dataset"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^dataset:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "dataset_version"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^dataset-version:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "benchmark"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^benchmark:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "benchmark_version"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^benchmark-version:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "evaluation_protocol"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^eval-protocol:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "evaluation_run"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^eval-run:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "organization"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^org:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "person"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^person:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "license"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^license:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "hardware_system"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^hardware:[a-z0-9][a-z0-9._-]*$"}}}},
{"if": {"properties": {"type": {"const": "product_system"}}, "required": ["type"]}, "then": {"properties": {"id": {"pattern": "^product:[a-z0-9][a-z0-9._-]*$"}}}}
],
"additionalProperties": false
}
8 changes: 7 additions & 1 deletion schemas/event.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
"$id": "https://modelmap.dev/schemas/event.schema.json",
"title": "ModelMap Event",
"type": "object",
"$defs": {
"canonicalId": {
"type": "string",
"pattern": "^(?:family|release|variant|arch|technique|training|implementation|paper|source|claim|relationship|event|dataset|dataset-version|benchmark|benchmark-version|eval-protocol|eval-run|org|person|license|hardware|product):[a-z0-9][a-z0-9._-]*$"
}
},
"required": ["id", "type", "name", "event_type", "subject", "occurred_at", "evidence"],
"properties": {
"id": {"type": "string", "pattern": "^event:[a-z0-9][a-z0-9._-]*$"},
Expand All @@ -28,7 +34,7 @@
"organization_founded"
]
},
"subject": {"type": "string", "minLength": 3},
"subject": {"$ref": "#/$defs/canonicalId"},
"occurred_at": {"type": "string", "format": "date"},
"evidence": {
"type": "array",
Expand Down
10 changes: 8 additions & 2 deletions schemas/relationship.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,17 @@
"$id": "https://modelmap.dev/schemas/relationship.schema.json",
"title": "ModelMap Relationship",
"type": "object",
"$defs": {
"canonicalId": {
"type": "string",
"pattern": "^(?:family|release|variant|arch|technique|training|implementation|paper|source|claim|relationship|event|dataset|dataset-version|benchmark|benchmark-version|eval-protocol|eval-run|org|person|license|hardware|product):[a-z0-9][a-z0-9._-]*$"
}
},
"required": ["id", "type", "subject", "predicate", "object"],
"properties": {
"id": {"type": "string", "pattern": "^relationship:[a-z0-9][a-z0-9._-]*$"},
"type": {"const": "relationship"},
"subject": {"type": "string", "minLength": 3},
"subject": {"$ref": "#/$defs/canonicalId"},
"predicate": {
"type": "string",
"enum": [
Expand All @@ -32,7 +38,7 @@
"replaces"
]
},
"object": {"type": "string", "minLength": 3},
"object": {"$ref": "#/$defs/canonicalId"},
"claim_refs": {
"type": "array",
"items": {"type": "string", "pattern": "^claim:[a-z0-9][a-z0-9._-]*$"},
Expand Down
61 changes: 49 additions & 12 deletions schemas/year-view.schema.json
Original file line number Diff line number Diff line change
@@ -1,23 +1,60 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://modelmap.dev/schemas/year-view.schema.json",
"title": "ModelMap Year View",
"title": "ModelMap Generated Year View",
"type": "object",
"required": ["year", "generated_from", "sections"],
"$defs": {
"canonicalId": {
"type": "string",
"pattern": "^(?:family|release|variant|arch|technique|training|implementation|paper|source|claim|relationship|event|dataset|dataset-version|benchmark|benchmark-version|eval-protocol|eval-run|org|person|license|hardware|product):[a-z0-9][a-z0-9._-]*$"
},
"event": {
"type": "object",
"required": ["id", "type", "name", "event_type", "subject", "occurred_at", "evidence"],
"properties": {
"id": {"type": "string", "pattern": "^event:[a-z0-9][a-z0-9._-]*$"},
"type": {"const": "event"},
"name": {"type": "string", "minLength": 1},
"event_type": {
"type": "string",
"enum": [
"technique_introduced",
"paper_published",
"report_published",
"model_announced",
"weights_released",
"api_released",
"code_released",
"dataset_released",
"license_changed",
"checkpoint_updated",
"context_extended",
"benchmark_published",
"model_deprecated",
"training_completed",
"organization_founded"
]
},
"subject": {"$ref": "#/$defs/canonicalId"},
"occurred_at": {"type": "string", "format": "date"},
"evidence": {
"type": "array",
"minItems": 1,
"uniqueItems": true,
"items": {"type": "string", "pattern": "^source:[a-z0-9][a-z0-9._-]*$"}
},
"notes": {"type": ["string", "null"]}
},
"additionalProperties": false
}
},
"required": ["year", "generated_from", "events"],
"properties": {
"year": {"type": "integer", "minimum": 2017},
"generated_from": {"const": "canonical-events"},
"sections": {
"events": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "event_types"],
"properties": {
"name": {"type": "string", "minLength": 1},
"event_types": {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
},
"additionalProperties": false
}
"items": {"$ref": "#/$defs/event"}
}
},
"additionalProperties": false
Expand Down
Loading
Loading