diff --git a/CITATION.cff b/CITATION.cff index a3aa5ec..c216b9e 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -2,7 +2,7 @@ cff-version: 1.2.0 message: "If you use this software, please cite it as below." type: software title: "cybedtools: A Package for Reproducible Analysis of Cybersecurity Workforce and Learning Frameworks" -version: 0.4.0 +version: 0.4.1 # Zenodo concept DOI: the parent identifier that resolves to the latest # released version (currently 10.5281/zenodo.20076116, verified against # the Zenodo project's "Cite all versions" link). This DOI does not @@ -34,15 +34,15 @@ keywords: - cybersecurity education - competency mapping abstract: >- - cybedtools provides a reproducible R pipeline for ingesting, semantically - representing (via a two-tier JSON-LD schema), and querying (via SPARQL) - cybersecurity-related competency and learning-standards frameworks, - including NICE, DCWF, SFIA, ENISA ECSF, Cyber.org K-12, CSTA K-12 CS, - ACM/IEEE CSEC2017, DigComp 3.0, CyQUAL, CCSSF, and OTCCF. See - framework_summary() in the package for the current list and counts. The - cybed: vocabulary uses + cybedtools is an R package and a Python package (on PyPI), sharing one + API and a shared conformance-test suite, for building and querying a + cross-framework RDF/JSON-LD graph of cybersecurity workforce and + learning frameworks. The two-tier cybed: vocabulary uses cybed:OrganizingUnit as the cross-framework abstract for parent units and cybed:Role as a workforce-restricted subtype, with cybed:Subpoint and cybed:Example separating framework-as-specified enumerations from - pedagogical-scaffolding examples. Includes a layered data-integrity - verification rig and per-framework provenance manifests. + pedagogical-scaffolding examples. Per-framework data releases are + fetched and hash-verified with cybed_fetch(); see framework_summary() + in the package for the current list of frameworks and counts. Includes + a layered data-integrity verification rig and per-framework provenance + manifests. diff --git a/DESCRIPTION b/DESCRIPTION index 1c5ed3f..8f5fd42 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -2,20 +2,21 @@ Package: cybedtools Type: Package Title: A Package for Reproducible Analysis of Cybersecurity Workforce and Learning Frameworks -Version: 0.4.0 +Version: 0.4.1 Authors@R: person( "Ryan", "Straight", email = "ryanstraight@arizona.edu", role = c("aut", "cre", "cph"), comment = c(ORCID = "0000-0002-6251-5662") ) -Description: A reproducible pipeline for ingesting, representing, and - analyzing cybersecurity workforce competency frameworks ('NICE', - 'DCWF', 'SFIA', 'ENISA ECSF', 'CyQUAL', 'CCSSF', 'OTCCF') and - cybersecurity-related pedagogical frameworks ('Cyber.org K-12', - 'CSTA K-12 CS', 'ACM/IEEE CSEC2017', - 'DigComp 3.0'). Provides a framework-agnostic 'JSON-LD' schema - ('cybed:') and 'SPARQL' query patterns for cross-framework analysis. +Description: Builds and queries a cross-framework 'RDF'/'JSON-LD' graph + of cybersecurity workforce and learning frameworks, with a shared + 'API' and conformance tests against a companion 'Python' package + (on 'PyPI'). Per-framework data releases are fetched and + hash-verified with cybed_fetch(). Provides a framework-agnostic + 'JSON-LD' schema ('cybed:') and 'SPARQL' query patterns for + cross-framework analysis; see framework_summary() for the current + list of frameworks and counts. License: MIT + file LICENSE Encoding: UTF-8 Language: en-US diff --git a/NEWS.md b/NEWS.md index 87eae0e..c5b3597 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,16 @@ +# cybedtools 0.4.1 + +First-time-user defect fixes found against the published v0.4.0 / PyPI 0.1.0, fixed identically in R and Python. + +- **CSEC2017 held from the 2026.09.2 data release** (owner correction, on this branch before the fixes below): `docs/data-release.yml` moves `csec2017` from `shipped` to `held`, pending steward confirmation. +- **Slug alias resolution.** `cybed_license()` and `framework_similarity()` now accept either the versioned framework slug (e.g. `"nice-v2"`) or the short release-file slug (e.g. `"nice"`), matching `cybed_fetch()`'s existing behavior. The alias map is derived once, in `R/slug-alias.R` (`cybedtools/_slug_alias.py` in Python), from the shipped `framework_summary`/`framework_licenses` tables, reusing the short-slug rule `scripts/030-export-release.R`'s `release_license_row()` applies. `cybed_fetch()`'s result now carries both `framework_slug` (canonical, versioned) and `release_slug` (short) columns explicitly, rather than silently substituting one for the other. +- **`framework_similarity()` errors on an unknown `from`/`to` slug** with class `cybedtools_framework_not_found` (a specific exception in Python), listing the graph's known slugs, instead of silently returning an empty result. +- **Python `cybed_fetch()`/`load_graph()` accept a bare string** for `frameworks` as one slug, not an iterable of its characters. +- **`version` accepts a `data-v`-prefixed value** (e.g. `"data-v2026.09.2"`, matching a GitHub release tag) in both `cybed_fetch()` implementations; the leading `data-v` is stripped. +- **README and quick-start docs lead with the user path**: install, then `cybed_fetch()`/`load_graph()`, then query examples. The ingestion pipeline (`scripts/000-build.R`) moves to a separate "Rebuilding the graph from source (maintainers)" section, consistent across `README.qmd`, `concordance/start/install.qmd`, and the `getting-started` vignette. The Zenodo data-release DOI (`10.5281/zenodo.22884320`) is now named alongside the data release. +- **Concordance framework pages' footer `LICENSE`/asset links** are root-relative, fixing a 404 from any page under a subdirectory (e.g. `frameworks/`). +- `CITATION.cff` and `DESCRIPTION`'s `Description` no longer describe cybedtools as an R-only pipeline or hand-list frameworks; both now point to `framework_summary()` for the current list and note the shared R/Python API. + # cybedtools 0.4.0 ## New framework diff --git a/R/cybed-fetch.R b/R/cybed-fetch.R index 4821bf3..64994da 100644 --- a/R/cybed-fetch.R +++ b/R/cybed-fetch.R @@ -129,15 +129,32 @@ cybed_download <- function(url, destfile) { #' Already-cached files whose hash still matches the manifest are not #' re-downloaded. #' -#' @param frameworks Character vector of framework slugs to fetch (as -#' carried by [framework_summary]`$framework_slug`, e.g. `"nice-v2"`, or the release file slug, e.g. `"nice"`), or -#' `NULL` (the default) for every framework the release manifest ships. -#' @param version Character scalar release version (e.g. `"1.0.0"`), or -#' `NULL` (the default) for the data release this package version was built against, `cybed_data_release`. +#' @section Two slug vocabularies: +#' Two framework-slug vocabularies exist in this package. [framework_summary] +#' and [framework_licenses] use **versioned** slugs (`"nice-v2"`, +#' `"otccf-v1.1"`), minted into every framework node's IRI. The public data +#' release's files use **short** slugs (`"nice"`, `"otccf"`), assigned by +#' `docs/data-release.yml` and recorded in the release manifest. This +#' function, [cybed_license()], and [framework_similarity()] accept either +#' form for a `frameworks`/`slug`/`from`/`to` argument and resolve it to the +#' canonical versioned slug. The returned tibble names both forms explicitly +#' (`framework_slug` and `release_slug`) rather than silently substituting +#' one for the other. +#' +#' @param frameworks Character vector of framework slugs to fetch, either the +#' versioned form carried by [framework_summary]`$framework_slug` (e.g. +#' `"nice-v2"`) or the short release-file slug (e.g. `"nice"`; see "Two +#' slug vocabularies" above), or `NULL` (the default) for every framework +#' the release manifest ships. +#' @param version Character scalar release version, either `"2026.09.2"` or +#' `"data-v2026.09.2"` (a leading `"data-v"`, matching a GitHub release +#' tag, is stripped), or `NULL` (the default) for the data release this +#' package version was built against, `cybed_data_release`. #' @return Invisibly, a tibble with one row per fetched framework: columns -#' `framework_slug`, `path` (the cached file's local path), -#' `sha256_verified` (logical, always `TRUE` on return -- a mismatch -#' aborts instead of returning `FALSE`). +#' `framework_slug` (the canonical versioned slug, e.g. `"nice-v2"`), +#' `release_slug` (the short release-file slug, e.g. `"nice"`), `path` +#' (the cached file's local path), `sha256_verified` (logical, always +#' `TRUE` on return -- a mismatch aborts instead of returning `FALSE`). #' @family data loading #' @export #' @examples @@ -152,6 +169,7 @@ cybed_download <- function(url, destfile) { cybed_fetch <- function(frameworks = NULL, version = NULL) { base_url <- cybed_release_base_url() version <- if (is.null(version)) cybed_data_release else version + version <- sub("^data-v", "", version) manifest <- cybed_read_manifest(base_url, version) files <- manifest$files @@ -219,7 +237,8 @@ cybed_fetch <- function(frameworks = NULL, version = NULL) { } tibble::tibble( - framework_slug = slug, + framework_slug = if (is.null(entry$license_slug)) slug else entry$license_slug, + release_slug = slug, path = dest, sha256_verified = TRUE ) diff --git a/R/data.R b/R/data.R index 42f01db..0be8f10 100644 --- a/R/data.R +++ b/R/data.R @@ -11,6 +11,11 @@ #' (workforce vs pedagogy), and license are hand-curated because they #' originate outside the JSON-LD graph. #' +#' `framework_slug` is the **versioned** slug vocabulary. A second, **short** +#' vocabulary is used by the public data release's files (e.g. `"nice"` for +#' `"nice-v2"`); see the "Two slug vocabularies" section of [cybed_fetch()] +#' for the full mapping and which functions accept either form. +#' #' Several frameworks were added in v0.3.0 on steward terms: CyQUAL #' (Czech Republic, open data, attribution to CyQUAL and Masaryk #' University), CCSSF (Canada, Government of Canada copyright, used with diff --git a/R/licenses.R b/R/licenses.R index 468b710..0896e6a 100644 --- a/R/licenses.R +++ b/R/licenses.R @@ -35,9 +35,11 @@ license_table <- function() { #' says which of the two a given row carries. #' #' @param slug Character scalar, or `NULL`. Either `"cybedtools"` for the -#' package's own code, or a framework slug as carried by +#' package's own code, a framework slug as carried by #' [framework_summary]`$framework_slug` (for example `"nice-v2"`, -#' `"otccf-v1.1"`). `NULL`, the default, returns every row. +#' `"otccf-v1.1"`), or the short release-file slug (e.g. `"nice"`, +#' `"otccf"`) documented on [cybed_fetch()]. Either form resolves to the +#' same row. `NULL`, the default, returns every row. #' @return A tibble. One row per licence layer when `slug` is `NULL`, and a #' single-row tibble otherwise. #' @family licensing @@ -68,7 +70,14 @@ cybed_license <- function(slug = NULL) { ) } - row <- licenses[licenses$slug == slug, , drop = FALSE] + resolved <- if (slug %in% licenses$slug) { + slug + } else { + alias_map <- framework_slug_alias_map() + if (slug %in% names(alias_map)) unname(alias_map[[slug]]) else slug + } + + row <- licenses[licenses$slug == resolved, , drop = FALSE] if (nrow(row) == 0L) { rlang::abort( diff --git a/R/similarity-helpers.R b/R/similarity-helpers.R index 4cea793..03f9852 100644 --- a/R/similarity-helpers.R +++ b/R/similarity-helpers.R @@ -214,8 +214,12 @@ similarity_strength <- function(x) { #' @param rdf An rdf object. #' @param from,to Character scalars, the `framework_slug` values (from #' [organizing_unit_framework_bindings()]) whose organizing units are -#' compared. May be identical, to find near-duplicate units within one -#' framework. +#' compared, either as the versioned slug (`"nice-v2"`) or the short +#' release-file slug (`"nice"`; see [cybed_fetch()]). May be identical, to +#' find near-duplicate units within one framework. An unknown slug (in +#' either vocabulary, and not present in `rdf`) errors with class +#' `cybedtools_framework_not_found` rather than silently returning an +#' empty result. #' @param n Integer, matches to keep per from-unit (default `5`). #' @return A tibble with one row per (from unit, match): columns #' `from_unit`, `to_unit` (both full IRIs), `score` (numeric in `[0, 1]`), @@ -242,6 +246,10 @@ framework_similarity <- function(rdf, from, to, n = 5) { stopifnot(is.character(to), length(to) == 1L) units <- organizing_unit_framework_bindings(rdf) + known_slugs <- unique(units$framework_slug) + from <- resolve_framework_slug(from, known_slugs, arg = "from") + to <- resolve_framework_slug(to, known_slugs, arg = "to") + texts <- element_text(rdf) child_text <- unit_element_bindings(rdf) |> diff --git a/R/slug-alias.R b/R/slug-alias.R new file mode 100644 index 0000000..b023ba4 --- /dev/null +++ b/R/slug-alias.R @@ -0,0 +1,69 @@ +# R/slug-alias.R +# +# Two framework-slug vocabularies exist in this package (documented in full +# on cybed_fetch() and framework_summary()): +# +# - versioned slugs, e.g. "nice-v2", "otccf-v1.1" -- carried by +# framework_summary()$framework_slug and framework_licenses()$slug, and +# minted into every framework node's IRI in the graph. +# - release (short) slugs, e.g. "nice", "otccf" -- the names of the public +# per-framework release files, assigned by docs/data-release.yml and +# recorded in the release manifest's `slug` field (with the versioned +# form alongside it as `license_slug`). +# +# cybed_fetch() already resolves either form because it reads the release +# manifest at call time. Every OTHER public function that takes a framework +# slug (cybed_license(), framework_similarity()) has no manifest to consult, +# so this file derives the same short-slug rule +# `scripts/030-export-release.R`'s `release_license_row()` applies (strip a +# trailing version suffix; a framework that is its own edition, not a +# version of a shorter-named sibling, keeps its full slug) from the shipped +# `framework_summary`/`framework_licenses` tibbles alone, with no network or +# build-time input. This is the ONE place that map is built; both +# `cybed_license()` and `framework_similarity()` call it. + +#' Alias map from release (short) slug to canonical versioned slug +#' +#' @return A named character vector: `names()` are release/short slugs, +#' values are the canonical versioned slugs. +#' @noRd +framework_slug_alias_map <- function() { + versioned <- cybedtools::framework_summary$framework_slug + short <- sub("-v?[0-9][0-9.]*$", "", versioned) + + # csta-2026 is a framework in its own right, not an edition of csta -- + # csta-2017 already claims the "csta" short slug (mirrors the `reserved` + # handling in scripts/030-export-release.R's release_license_row()). + short[versioned == "csta-2026"] <- "csta-2026" + + stats::setNames(versioned, short) +} + +#' Resolve a framework slug (either vocabulary) against a set of known +#' (already-versioned) slugs +#' +#' @param slug Character scalar to resolve. +#' @param known Character vector of valid versioned slugs to resolve against. +#' @param arg Character scalar, the argument name to use in error messages. +#' @return The canonical versioned slug, if resolvable. +#' @noRd +resolve_framework_slug <- function(slug, known, arg = "slug") { + if (slug %in% known) { + return(slug) + } + + alias_map <- framework_slug_alias_map() + if (slug %in% names(alias_map) && unname(alias_map[[slug]]) %in% known) { + return(unname(alias_map[[slug]])) + } + + rlang::abort( + c( + "Unknown framework slug.", + "x" = paste0("`", arg, "`: '", slug, "'."), + "i" = paste0("Known slugs: ", paste(sort(known), collapse = ", "), ".") + ), + class = "cybedtools_framework_not_found", + framework_slug = slug + ) +} diff --git a/README.md b/README.md index 58c6379..b1c932e 100644 --- a/README.md +++ b/README.md @@ -892,15 +892,97 @@ vignette. Extending the schema to a new framework is covered in ## Getting started +Install the package, then fetch the public data release: hash-verified +per-framework files, downloaded once and cached locally. No source +framework text needs to be staged for this path. + +### R + +``` r +# install.packages("pak") +pak::pak("ryanstraight/cybedtools") +``` + +### Python + +``` sh +pip install cybedtools +``` + +### Fetch data and load a graph + +R: + +``` r +library(cybedtools) + +# Downloads and hash-verifies every shipped framework's release file into +# the R user cache directory, then parses them into one rdf object. +rdf <- load_graph() + +framework_summary +``` + +Python: + +``` python +from cybedtools import load_graph, framework_summary + +graph = load_graph() +framework_summary() +``` + +`load_graph()` calls `cybed_fetch()` internally (a no-op for files +already cached with a verified hash). Both accept either the versioned +framework slug (`"nice-v2"`) or the short release-file slug (`"nice"`); +see `cybed_fetch()`’s documentation for the two-vocabulary mapping. The +data release backing this package version is archived on Zenodo: +[10.5281/zenodo.22884320](https://doi.org/10.5281/zenodo.22884320). + +### Three queries + +R: + ``` r -# install.packages("remotes") -remotes::install_github("ryanstraight/cybedtools") +# 1. What's in the corpus? +framework_summary |> + dplyr::select(framework_slug, framework_name, jurisdiction, organizing_unit_count) + +# 2. Which elements does an organizing unit carry? +unit_element_bindings(rdf) |> + head(10) + +# 3. Where do two frameworks say the same thing? +framework_similarity(rdf, from = "nice", to = "ecsf", n = 3) +``` + +Python: + +``` python +# 1. What's in the corpus? +framework_summary()[["framework_slug", "framework_name", "jurisdiction", "organizing_unit_count"]] + +# 2. Which elements does an organizing unit carry? +from cybedtools import unit_element_bindings +unit_element_bindings(graph).head(10) + +# 3. Where do two frameworks say the same thing? +from cybedtools import framework_similarity +framework_similarity(graph, from_="nice", to="ecsf", n=3) ``` -The package **does not** redistribute source framework text. To run the -pipeline end-to-end, clone the repository and stage each framework’s -source file at `data/raw//` per -[`docs/framework-data-sources.md`](docs/framework-data-sources.md): +The [function +reference](https://ryanstraight.github.io/cybedtools/reference/) indexes +the full public API for both languages. + +## Rebuilding the graph from source (maintainers) + +Fetching the public release (above) is the path for using the corpus. +Rebuilding it from primary sources is a separate, maintainer-only path: +the package does not redistribute source framework text, so this +requires cloning the repository and staging each framework’s source file +at `data/raw//` per +[`docs/framework-data-sources.md`](docs/framework-data-sources.md). ``` sh git clone https://github.com/ryanstraight/cybedtools @@ -910,11 +992,11 @@ cd cybedtools Rscript scripts/000-build.R # ingestion + verification + assembly + export ``` -The +`scripts/000-build.R` is the pipeline’s entry point: it runs ingestion, +verification, JSON-LD assembly, and N-Triples export in order (see +`scripts/README.md` for the full stage list). The [`getting-started`](https://ryanstraight.github.io/cybedtools/articles/getting-started.html) -vignette walks through each stage, and the [function -reference](https://ryanstraight.github.io/cybedtools/reference/) indexes -the public API. +vignette walks through each stage. ## Citing diff --git a/README.qmd b/README.qmd index 10c572a..174fb35 100644 --- a/README.qmd +++ b/README.qmd @@ -548,12 +548,83 @@ Jurisdiction pivots, top-load NICE roles, pairwise framework comparisons, and th ## Getting started +Install the package, then fetch the public data release: hash-verified per-framework files, downloaded once and cached locally. No source framework text needs to be staged for this path. + +### R + ```r -# install.packages("remotes") -remotes::install_github("ryanstraight/cybedtools") +# install.packages("pak") +pak::pak("ryanstraight/cybedtools") ``` -The package **does not** redistribute source framework text. To run the pipeline end-to-end, clone the repository and stage each framework's source file at `data/raw//` per [`docs/framework-data-sources.md`](docs/framework-data-sources.md): +### Python + +```sh +pip install cybedtools +``` + +### Fetch data and load a graph + +R: + +```r +library(cybedtools) + +# Downloads and hash-verifies every shipped framework's release file into +# the R user cache directory, then parses them into one rdf object. +rdf <- load_graph() + +framework_summary +``` + +Python: + +```python +from cybedtools import load_graph, framework_summary + +graph = load_graph() +framework_summary() +``` + +`load_graph()` calls `cybed_fetch()` internally (a no-op for files already cached with a verified hash). Both accept either the versioned framework slug (`"nice-v2"`) or the short release-file slug (`"nice"`); see `cybed_fetch()`'s documentation for the two-vocabulary mapping. The data release backing this package version is archived on Zenodo: [10.5281/zenodo.22884320](https://doi.org/10.5281/zenodo.22884320). + +### Three queries + +R: + +```r +# 1. What's in the corpus? +framework_summary |> + dplyr::select(framework_slug, framework_name, jurisdiction, organizing_unit_count) + +# 2. Which elements does an organizing unit carry? +unit_element_bindings(rdf) |> + head(10) + +# 3. Where do two frameworks say the same thing? +framework_similarity(rdf, from = "nice", to = "ecsf", n = 3) +``` + +Python: + +```python +# 1. What's in the corpus? +framework_summary()[["framework_slug", "framework_name", "jurisdiction", "organizing_unit_count"]] + +# 2. Which elements does an organizing unit carry? +from cybedtools import unit_element_bindings +unit_element_bindings(graph).head(10) + +# 3. Where do two frameworks say the same thing? +from cybedtools import framework_similarity +framework_similarity(graph, from_="nice", to="ecsf", n=3) +``` + +The [function reference](https://ryanstraight.github.io/cybedtools/reference/) indexes the full public API for both languages. + +## Rebuilding the graph from source (maintainers) + +Fetching the public release (above) is the path for using the corpus. Rebuilding it from primary sources is a separate, maintainer-only path: the package does not redistribute source framework text, so this requires cloning the repository and staging each framework's source file at `data/raw//` per [`docs/framework-data-sources.md`](docs/framework-data-sources.md). ```sh git clone https://github.com/ryanstraight/cybedtools @@ -563,7 +634,7 @@ cd cybedtools Rscript scripts/000-build.R # ingestion + verification + assembly + export ``` -The [`getting-started`](https://ryanstraight.github.io/cybedtools/articles/getting-started.html) vignette walks through each stage, and the [function reference](https://ryanstraight.github.io/cybedtools/reference/) indexes the public API. +`scripts/000-build.R` is the pipeline's entry point: it runs ingestion, verification, JSON-LD assembly, and N-Triples export in order (see `scripts/README.md` for the full stage list). The [`getting-started`](https://ryanstraight.github.io/cybedtools/articles/getting-started.html) vignette walks through each stage. ## Citing diff --git a/concordance/_quarto.yml b/concordance/_quarto.yml index f559d04..6d6d4a2 100644 --- a/concordance/_quarto.yml +++ b/concordance/_quarto.yml @@ -80,9 +80,9 @@ website: page-footer: border: true left: | - Concordance · Content [CC BY 4.0](LICENSE.html) · Code [MIT](LICENSE.html) · [Concept DOI](https://doi.org/10.5281/zenodo.20076116) + Concordance · Content [CC BY 4.0](/LICENSE.html) · Code [MIT](/LICENSE.html) · [Concept DOI](https://doi.org/10.5281/zenodo.20076116) right: | - Built with [Quarto](https://quarto.org) & [Claude](https://claude.ai) & ❤️ + Built with [Quarto](https://quarto.org) & [Claude](https://claude.ai) & ❤️ format: html: diff --git a/concordance/start/install.qmd b/concordance/start/install.qmd index 575b380..4b4d394 100644 --- a/concordance/start/install.qmd +++ b/concordance/start/install.qmd @@ -1,80 +1,60 @@ --- title: "Quick start" -subtitle: "Install the package, stage framework data, run your first query" +subtitle: "Install the package, fetch the data, run your first query" --- -```{r} -#| label: setup -#| include: false -fl <- cybedtools::framework_licenses -permission_slugs <- c("cyqual-v1.2.0", "ccssf-2022", "otccf-v1.1") -n_granted <- sum(fl$granted[fl$slug %in% permission_slugs]) -``` - -## Install cybedtools +## Install -Install from GitHub via `remotes` or `pak`: +### R ```{.r} -# Option A: remotes -install.packages("remotes") -remotes::install_github("ryanstraight/cybedtools") - -# Option B: pak (faster on first install) install.packages("pak") -pak::pkg_install("ryanstraight/cybedtools") +pak::pak("ryanstraight/cybedtools") +``` + +### Python + +```{.bash} +pip install cybedtools ``` -## Stage the framework data +## Fetch the data and load a graph + +`cybedtools` ships no framework text in the installed package. `cybed_fetch()` downloads and hash-verifies the public per-framework data release into a local cache (never re-downloading a file whose hash still matches), and `load_graph()` calls it internally and parses the result into one graph object. Both accept either the versioned framework slug (e.g. `"nice-v2"`) or the short release-file slug (e.g. `"nice"`); see `cybed_fetch()`'s documentation for the two-vocabulary mapping. The data release backing the current package version is archived on Zenodo: [10.5281/zenodo.22884320](https://doi.org/10.5281/zenodo.22884320). + +::: {.panel-tabset} -`cybedtools` does not redistribute framework source text. Each -framework's license governs how its source data is obtained and -stored. `docs/framework-data-sources.md` documents the canonical -retrieval URL, license, and SHA256 reference checksum for each -framework in the corpus. +### R ```{.r} -# After install, see where to put each framework's source data: -system.file("doc", "framework-data-sources.md", package = "cybedtools") |> - readLines() |> head(40) -``` +library(cybedtools) -For NICE and DCWF (US Government works, not subject to US -copyright under 17 U.S.C. 105), the package -ships pointers to the canonical NIST CPRT and DoD CIO downloads. For -SFIA, ECSF, Cyber.org K-12, CSTA, CSEC2017, and DigComp, follow the -steward's redistribution policy as documented per framework. CyQUAL, -CCSSF, and OTCCF are in the corpus on their stewards' terms, -`r n_granted` by written permission and CCSSF referenced as its -steward asked, and each carries its own terms. Read the per-framework -entries before staging those three. +rdf <- load_graph() +``` -## Run the pipeline +### Python -Once source data is staged in `data/raw/`, run the pipeline scripts -in order. Each script is numbered for reproducibility: +```{.python} +from cybedtools import load_graph -```{.bash} -Rscript scripts/000-staging-check.R # confirm source data present -Rscript scripts/010-ingest-nice.R # parse NICE -> JSON-LD -Rscript scripts/010-ingest-dcwf.R # ... and so on per framework -Rscript scripts/020-assemble-jsonld.R # uniform parser pass -Rscript scripts/025-export-ntriples.R # combined RDF graph -Rscript scripts/030-load-graph.R # smoke-test SPARQL load +graph = load_graph() ``` -The combined graph lands at `data/processed/ntriples/_combined.nt`. -Every query on this site reads from that file. +::: -## First query +## First queries + +::: {.panel-tabset} + +### R ```{r} -#| label: first-query +#| label: first-query-r #| echo: true library(cybedtools) library(dplyr) -# Per-framework summary, lazy-loaded with the package +# 1. What's in the corpus? framework_summary |> select( framework_name, @@ -84,6 +64,34 @@ framework_summary |> arrange(desc(elements_per_organizing_unit_with_examples)) ``` +```{.r} +# 2. Which elements does an organizing unit carry? +unit_element_bindings(rdf) |> + head(10) + +# 3. Where do two frameworks say the same thing? +framework_similarity(rdf, from = "nice", to = "ecsf", n = 3) +``` + +### Python + +```{.python} +from cybedtools import framework_summary, unit_element_bindings, framework_similarity + +# 1. What's in the corpus? +framework_summary()[ + ["framework_name", "organizing_unit_count", "elements_per_organizing_unit_with_examples"] +].sort_values("elements_per_organizing_unit_with_examples", ascending=False) + +# 2. Which elements does an organizing unit carry? +unit_element_bindings(graph).head(10) + +# 3. Where do two frameworks say the same thing? +framework_similarity(graph, from_="nice", to="ecsf", n=3) +``` + +::: + ## Next steps - Read [the concepts](../concepts/vocabulary.qmd) page for the @@ -92,3 +100,28 @@ framework_summary |> per-framework provenance and structure. - Step through [the canonical queries](../queries/index.qmd) for worked examples. + +## Rebuilding the graph from source (maintainers) + +Fetching the public release (above) is the path for using the corpus. Rebuilding it from primary sources is a separate, maintainer-only path, run from a clone of the repository: `cybedtools` does not redistribute framework source text, so each framework's license governs how its source data is obtained and staged. `docs/framework-data-sources.md` documents the canonical retrieval URL, license, and SHA-256 reference checksum for each framework in the corpus. + +For NICE and DCWF (US Government works, not subject to US copyright under 17 U.S.C. 105), the package ships pointers to the canonical NIST CPRT and DoD CIO downloads. For SFIA, ECSF, Cyber.org K-12, CSTA, CSEC2017, and DigComp, follow the steward's redistribution policy as documented per framework. CyQUAL, CCSSF, OTCCF, SCyWF, and CyBOK are in the corpus on their stewards' terms, several by written permission, and each carries its own terms. Read the per-framework entries before staging those. + +`scripts/000-build.R` is the pipeline's entry point: it runs every stage below in order. + +```{.bash} +# After staging source data under data/raw//: +Rscript scripts/000-build.R + +# Equivalently, stage by stage: +Rscript scripts/010-ingest-nice.R # parse NICE -> tidy CSVs +Rscript scripts/010-ingest-dcwf.R # ... and so on per framework +Rscript scripts/015-verify-ingestion.R # cross-check ingestion output +Rscript scripts/020-assemble-jsonld.R # uniform JSON-LD assembly +Rscript scripts/025-export-ntriples.R # combined N-Triples graph +Rscript scripts/026-verify-graph.R # graph invariants +Rscript scripts/030-export-release.R # public per-framework data release +Rscript scripts/040-run-sparql.R # canonical query outputs +``` + +The combined graph lands at `data/processed/ntriples/_combined.nt`, and the public release (what `cybed_fetch()` downloads) lands at `data/processed/release//`. Every query on this site reads from the combined graph. diff --git a/docs/data-release.yml b/docs/data-release.yml index a637f5c..b07b6fe 100644 --- a/docs/data-release.yml +++ b/docs/data-release.yml @@ -37,6 +37,7 @@ vocabulary_iri: "https://w3id.org/cybed/ontology#" # Concept DOI of the software record. The data deposit carries its own DOI. concept_doi: "10.5281/zenodo.20076116" +data_doi: "10.5281/zenodo.22884320" # Predicates whose objects carry framework statement text. A structure_only # file drops every triple on these predicates. Derived from a predicate census @@ -68,9 +69,9 @@ long_literal_allow_predicates: # Release 2. One gzip-compressed N-Triples file per slug, named by slug. # Every framework whose policy in docs/framework-invariants.yml permits -# public redistribution now ships. csec2017 and otccf ship structure_only -# (the export's predicate-omission gate handles that automatically); every -# other slug here is full_with_attribution. ccssf moves from held to shipped +# public redistribution and whose steward has confirmed in writing ships. +# otccf ships structure_only (the export's predicate-omission gate handles +# that automatically); every other slug here is full_with_attribution. ccssf moves from held to shipped # because the Canadian Centre for Cyber Security's 2026-09-21 permission # supersedes the 2026-09-19 reference-only reading. csta-2026, scywf, and # cybok move from held to shipped because they joined the ingestion pipeline @@ -85,16 +86,15 @@ shipped: - csta - otccf - ccssf - - csec2017 - csta-2026 - scywf - cybok # Cleared by policy for a structure-only file, withheld from this release # anyway. A deposit cannot be withdrawn, so a framework ships only once its -# steward has confirmed in writing. Empty for 2026.09.2: every previously -# held framework is now confirmed or ready (see `shipped` above). -held: {} +# steward has confirmed in writing. +held: + csec2017: "Steward confirmation pending from all four copyright holders." # Refused by policy, in every release. The export stops if a slug listed here # has any policy other than local_only, and stops if a local_only slug appears diff --git a/inst/CITATION b/inst/CITATION index 4844916..dea2e7a 100644 --- a/inst/CITATION +++ b/inst/CITATION @@ -10,13 +10,13 @@ bibentry( comment = c(ORCID = "0000-0002-6251-5662")) ), year = "2026", - note = "R package version 0.4.0", + note = "R package version 0.4.1", # Zenodo concept DOI (resolves to latest version). doi = "10.5281/zenodo.20076116", url = "https://doi.org/10.5281/zenodo.20076116", textVersion = paste( "Straight, R. (2026). cybedtools: A Package for Reproducible Analysis", "of Cybersecurity Workforce and Learning Frameworks. R package version", - "0.4.0. https://doi.org/10.5281/zenodo.20076116" + "0.4.1. https://doi.org/10.5281/zenodo.20076116" ) ) diff --git a/man/cybed_fetch.Rd b/man/cybed_fetch.Rd index 535c436..44e2833 100644 --- a/man/cybed_fetch.Rd +++ b/man/cybed_fetch.Rd @@ -7,18 +7,23 @@ cybed_fetch(frameworks = NULL, version = NULL) } \arguments{ -\item{frameworks}{Character vector of framework slugs to fetch (as -carried by \link{framework_summary}\verb{$framework_slug}, e.g. \code{"nice-v2"}, or the release file slug, e.g. \code{"nice"}), or -\code{NULL} (the default) for every framework the release manifest ships.} +\item{frameworks}{Character vector of framework slugs to fetch, either the +versioned form carried by \link{framework_summary}\verb{$framework_slug} (e.g. +\code{"nice-v2"}) or the short release-file slug (e.g. \code{"nice"}; see "Two +slug vocabularies" above), or \code{NULL} (the default) for every framework +the release manifest ships.} -\item{version}{Character scalar release version (e.g. \code{"1.0.0"}), or -\code{NULL} (the default) for the data release this package version was built against, \code{cybed_data_release}.} +\item{version}{Character scalar release version, either \code{"2026.09.2"} or +\code{"data-v2026.09.2"} (a leading \code{"data-v"}, matching a GitHub release +tag, is stripped), or \code{NULL} (the default) for the data release this +package version was built against, \code{cybed_data_release}.} } \value{ Invisibly, a tibble with one row per fetched framework: columns -\code{framework_slug}, \code{path} (the cached file's local path), -\code{sha256_verified} (logical, always \code{TRUE} on return -- a mismatch -aborts instead of returning \code{FALSE}). +\code{framework_slug} (the canonical versioned slug, e.g. \code{"nice-v2"}), +\code{release_slug} (the short release-file slug, e.g. \code{"nice"}), \code{path} +(the cached file's local path), \code{sha256_verified} (logical, always +\code{TRUE} on return -- a mismatch aborts instead of returning \code{FALSE}). } \description{ \ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#experimental}{\figure{lifecycle-experimental.svg}{options: alt='[Experimental]'}}}{\strong{[Experimental]}} @@ -34,6 +39,20 @@ do not match its manifest hash is deleted and the call aborts with class Already-cached files whose hash still matches the manifest are not re-downloaded. } +\section{Two slug vocabularies}{ + +Two framework-slug vocabularies exist in this package. \link{framework_summary} +and \link{framework_licenses} use \strong{versioned} slugs (\code{"nice-v2"}, +\code{"otccf-v1.1"}), minted into every framework node's IRI. The public data +release's files use \strong{short} slugs (\code{"nice"}, \code{"otccf"}), assigned by +\code{docs/data-release.yml} and recorded in the release manifest. This +function, \code{\link[=cybed_license]{cybed_license()}}, and \code{\link[=framework_similarity]{framework_similarity()}} accept either +form for a \code{frameworks}/\code{slug}/\code{from}/\code{to} argument and resolve it to the +canonical versioned slug. The returned tibble names both forms explicitly +(\code{framework_slug} and \code{release_slug}) rather than silently substituting +one for the other. +} + \examples{ \donttest{ # Requires network access (or a `cybedtools.release_url` override diff --git a/man/cybed_license.Rd b/man/cybed_license.Rd index 4f75154..73405ef 100644 --- a/man/cybed_license.Rd +++ b/man/cybed_license.Rd @@ -8,9 +8,11 @@ cybed_license(slug = NULL) } \arguments{ \item{slug}{Character scalar, or \code{NULL}. Either \code{"cybedtools"} for the -package's own code, or a framework slug as carried by +package's own code, a framework slug as carried by \link{framework_summary}\verb{$framework_slug} (for example \code{"nice-v2"}, -\code{"otccf-v1.1"}). \code{NULL}, the default, returns every row.} +\code{"otccf-v1.1"}), or the short release-file slug (e.g. \code{"nice"}, +\code{"otccf"}) documented on \code{\link[=cybed_fetch]{cybed_fetch()}}. Either form resolves to the +same row. \code{NULL}, the default, returns every row.} } \value{ A tibble. One row per licence layer when \code{slug} is \code{NULL}, and a diff --git a/man/framework_similarity.Rd b/man/framework_similarity.Rd index e14a438..c105998 100644 --- a/man/framework_similarity.Rd +++ b/man/framework_similarity.Rd @@ -11,8 +11,12 @@ framework_similarity(rdf, from, to, n = 5) \item{from, to}{Character scalars, the \code{framework_slug} values (from \code{\link[=organizing_unit_framework_bindings]{organizing_unit_framework_bindings()}}) whose organizing units are -compared. May be identical, to find near-duplicate units within one -framework.} +compared, either as the versioned slug (\code{"nice-v2"}) or the short +release-file slug (\code{"nice"}; see \code{\link[=cybed_fetch]{cybed_fetch()}}). May be identical, to +find near-duplicate units within one framework. An unknown slug (in +either vocabulary, and not present in \code{rdf}) errors with class +\code{cybedtools_framework_not_found} rather than silently returning an +empty result.} \item{n}{Integer, matches to keep per from-unit (default \code{5}).} } diff --git a/man/framework_summary.Rd b/man/framework_summary.Rd index 55fb025..6aac010 100644 --- a/man/framework_summary.Rd +++ b/man/framework_summary.Rd @@ -98,6 +98,11 @@ via \code{data-raw/build-framework-summary.R}. Display name, framework type originate outside the JSON-LD graph. } \details{ +\code{framework_slug} is the \strong{versioned} slug vocabulary. A second, \strong{short} +vocabulary is used by the public data release's files (e.g. \code{"nice"} for +\code{"nice-v2"}); see the "Two slug vocabularies" section of \code{\link[=cybed_fetch]{cybed_fetch()}} +for the full mapping and which functions accept either form. + Several frameworks were added in v0.3.0 on steward terms: CyQUAL (Czech Republic, open data, attribution to CyQUAL and Masaryk University), CCSSF (Canada, Government of Canada copyright, used with diff --git a/man/load_graph.Rd b/man/load_graph.Rd index a8887e3..3879538 100644 --- a/man/load_graph.Rd +++ b/man/load_graph.Rd @@ -7,12 +7,16 @@ load_graph(frameworks = NULL, version = NULL) } \arguments{ -\item{frameworks}{Character vector of framework slugs to fetch (as -carried by \link{framework_summary}\verb{$framework_slug}, e.g. \code{"nice-v2"}, or the release file slug, e.g. \code{"nice"}), or -\code{NULL} (the default) for every framework the release manifest ships.} +\item{frameworks}{Character vector of framework slugs to fetch, either the +versioned form carried by \link{framework_summary}\verb{$framework_slug} (e.g. +\code{"nice-v2"}) or the short release-file slug (e.g. \code{"nice"}; see "Two +slug vocabularies" above), or \code{NULL} (the default) for every framework +the release manifest ships.} -\item{version}{Character scalar release version (e.g. \code{"1.0.0"}), or -\code{NULL} (the default) for the data release this package version was built against, \code{cybed_data_release}.} +\item{version}{Character scalar release version, either \code{"2026.09.2"} or +\code{"data-v2026.09.2"} (a leading \code{"data-v"}, matching a GitHub release +tag, is stripped), or \code{NULL} (the default) for the data release this +package version was built against, \code{cybed_data_release}.} } \value{ An rdf object (\code{\link[rdflib:rdf]{rdflib::rdf()}}) with every requested framework's diff --git a/python/pyproject.toml b/python/pyproject.toml index 247013c..fafabf9 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "cybedtools" -version = "0.1.0" +version = "0.1.1" description = "Python interface to the cybedtools cross-framework cybersecurity workforce and learning frameworks graph, mirroring the R package's API: data loading, reference tables, query helpers, similarity, and release fetching." readme = "README.md" license = "MIT" diff --git a/python/src/cybedtools/_slug_alias.py b/python/src/cybedtools/_slug_alias.py new file mode 100644 index 0000000..60e4f6e --- /dev/null +++ b/python/src/cybedtools/_slug_alias.py @@ -0,0 +1,96 @@ +"""Framework-slug alias resolution, shared by :mod:`cybedtools.data` and +:mod:`cybedtools.similarity`. + +Mirrors ``R/slug-alias.R``. Two framework-slug vocabularies exist in this +package: **versioned** slugs (``"nice-v2"``, ``"otccf-v1.1"``), carried by +``framework_summary()["framework_slug"]`` and ``framework_licenses()["slug"]`` +and minted into every framework node's IRI; and **short** release-file slugs +(``"nice"``, ``"otccf"``), assigned by ``docs/data-release.yml`` and recorded +in the release manifest's ``slug`` field (with the versioned form alongside +it as ``license_slug``). :func:`cybedtools.fetch.cybed_fetch` already +resolves either form because it reads the release manifest at call time. +Every other public function that takes a framework slug +(:func:`cybedtools.data.cybed_license`, +:func:`cybedtools.similarity.framework_similarity`) has no manifest to +consult, so this module derives the same short-slug rule +``scripts/030-export-release.R``'s ``release_license_row()`` applies (strip a +trailing version suffix; a framework that is its own edition, not a version +of a shorter-named sibling, keeps its full slug) from the shipped +``framework_summary``/``framework_licenses`` tables alone, with no network or +build-time input. +""" + +from __future__ import annotations + +import re + +__all__ = ["framework_slug_alias_map", "resolve_framework_slug"] + +_VERSION_SUFFIX_RE = re.compile(r"-v?[0-9][0-9.]*$") + + +def framework_slug_alias_map(versioned_slugs: list[str]) -> dict[str, str]: + """Alias map from release (short) slug to canonical versioned slug. + + Parameters + ---------- + versioned_slugs : list[str] + Every versioned framework slug (e.g. ``framework_summary() + ["framework_slug"]``). + + Returns + ------- + dict[str, str] + Keys are release/short slugs, values are the canonical versioned + slugs. + """ + alias: dict[str, str] = {} + for slug in versioned_slugs: + # csta-2026 is a framework in its own right, not an edition of csta + # -- csta-2017 already claims the "csta" short slug (mirrors the + # `reserved` handling in scripts/030-export-release.R's + # release_license_row()). + short = slug if slug == "csta-2026" else _VERSION_SUFFIX_RE.sub("", slug) + alias[short] = slug + return alias + + +class CybedtoolsFrameworkNotFoundError(ValueError): + """No known slug (either vocabulary) matched.""" + + +def resolve_framework_slug(slug: str, known: list[str], arg: str = "slug") -> str: + """Resolve a framework slug (either vocabulary) against known versioned slugs. + + Parameters + ---------- + slug : str + The slug to resolve, in either vocabulary. + known : list[str] + Valid, already-versioned slugs to resolve against. + arg : str + The argument name to use in the error message. + + Returns + ------- + str + The canonical versioned slug. + + Raises + ------ + CybedtoolsFrameworkNotFoundError + If `slug` cannot be resolved to a member of `known`. + """ + if slug in known: + return slug + + alias_map = framework_slug_alias_map(known) + if slug in alias_map and alias_map[slug] in known: + return alias_map[slug] + + known_sorted = ", ".join(sorted(known)) + raise CybedtoolsFrameworkNotFoundError( + f"Unknown framework slug.\n" + f"`{arg}`: '{slug}'.\n" + f"Known slugs: {known_sorted}." + ) diff --git a/python/src/cybedtools/data.py b/python/src/cybedtools/data.py index e4e54f6..6e5c5a4 100644 --- a/python/src/cybedtools/data.py +++ b/python/src/cybedtools/data.py @@ -22,6 +22,8 @@ import pandas as pd +from cybedtools._slug_alias import framework_slug_alias_map + __all__ = ["cybed_license", "framework_licenses", "framework_summary"] @@ -114,10 +116,12 @@ def cybed_license(slug: str | None = None) -> pd.DataFrame: Parameters ---------- slug : str or None - Either ``"cybedtools"`` for the package's own code, or a framework - slug as carried by ``framework_summary()["framework_slug"]`` (for - example ``"nice-v2"``, ``"otccf-v1.1"``). ``None``, the default, - returns every row. + Either ``"cybedtools"`` for the package's own code, a framework slug + as carried by ``framework_summary()["framework_slug"]`` (for example + ``"nice-v2"``, ``"otccf-v1.1"``), or the short release-file slug + (e.g. ``"nice"``, ``"otccf"``) documented on :func:`cybed_fetch`. + Either form resolves to the same row. ``None``, the default, returns + every row. Returns ------- @@ -143,7 +147,14 @@ def cybed_license(slug: str | None = None) -> pd.DataFrame: f"Received: {type(slug).__name__}." ) - row = licenses[licenses["slug"] == slug] + known = licenses["slug"].tolist() + if slug in known: + resolved = slug + else: + alias_map = framework_slug_alias_map([s for s in known if s != "cybedtools"]) + resolved = alias_map.get(slug, slug) + + row = licenses[licenses["slug"] == resolved] if row.empty: known = ", ".join(licenses["slug"].tolist()) diff --git a/python/src/cybedtools/fetch.py b/python/src/cybedtools/fetch.py index f7cefac..ae45bce 100644 --- a/python/src/cybedtools/fetch.py +++ b/python/src/cybedtools/fetch.py @@ -244,36 +244,53 @@ class FetchedFramework: """One row of a :func:`cybed_fetch` result.""" framework_slug: str + release_slug: str path: str sha256_verified: bool def cybed_fetch( - frameworks: list[str] | None = None, version: str | None = None + frameworks: str | list[str] | None = None, version: str | None = None ) -> pd.DataFrame: """Download and hash-verify per-framework release files into the user cache. Mirrors ``R/cybed-fetch.R``'s ``cybed_fetch()``. Already-cached files whose hash still matches the manifest are not re-downloaded. + Two framework-slug vocabularies exist in this package. ``framework_summary()`` + and ``framework_licenses()`` use **versioned** slugs (``"nice-v2"``, + ``"otccf-v1.1"``), minted into every framework node's IRI. The public data + release's files use **short** slugs (``"nice"``, ``"otccf"``), assigned by + ``docs/data-release.yml`` and recorded in the release manifest. This + function, :func:`cybedtools.cybed_license`, and + :func:`cybedtools.framework_similarity` accept either form for a + slug argument and resolve it to the canonical versioned slug. The + returned DataFrame names both forms explicitly (``framework_slug`` and + ``release_slug``) rather than silently substituting one for the other. + Parameters ---------- - frameworks : list[str] or None - Framework slugs to fetch (as carried by - ``framework_summary()["framework_slug"]``, e.g. ``"nice-v2"``, or the release file slug, e.g. ``"nice"``), or - ``None`` (the default) for every framework the release manifest - ships. + frameworks : str, list[str], or None + Framework slug(s) to fetch, either the versioned form carried by + ``framework_summary()["framework_slug"]`` (e.g. ``"nice-v2"``) or the + short release-file slug (e.g. ``"nice"``). A bare string is treated + as a single slug, not an iterable of characters. ``None`` (the + default) fetches every framework the release manifest ships. version : str or None - Release version (e.g. ``"1.0.0"``), or ``None`` (the default) for - the data release this package version was built against, ``DATA_RELEASE``. + Release version, either ``"2026.09.2"`` or ``"data-v2026.09.2"`` (a + leading ``"data-v"``, matching a GitHub release tag, is stripped), or + ``None`` (the default) for the data release this package version was + built against, ``DATA_RELEASE``. Returns ------- pandas.DataFrame - One row per fetched framework: columns ``framework_slug``, ``path`` - (the cached file's local path, as a string), ``sha256_verified`` - (bool, always ``True`` on return -- a mismatch raises instead of - returning ``False``). + One row per fetched framework: columns ``framework_slug`` (the + canonical versioned slug, e.g. ``"nice-v2"``), ``release_slug`` (the + short release-file slug, e.g. ``"nice"``), ``path`` (the cached + file's local path, as a string), ``sha256_verified`` (bool, always + ``True`` on return -- a mismatch raises instead of returning + ``False``). Raises ------ @@ -286,6 +303,7 @@ def cybed_fetch( """ base_url = _release_base_url() resolved_version = version if version is not None else DATA_RELEASE + resolved_version = resolved_version.removeprefix("data-v") manifest = _read_manifest(base_url, resolved_version) files = manifest["files"] @@ -296,7 +314,12 @@ def cybed_fetch( summary_to_release = { entry.get("license_slug") or entry["slug"]: entry["slug"] for entry in files } - raw = frameworks if frameworks is not None else manifest_slugs + if frameworks is None: + raw: list[str] = manifest_slugs + elif isinstance(frameworks, str): + raw = [frameworks] + else: + raw = list(frameworks) requested = [ s if s in manifest_slugs else summary_to_release.get(s, s) for s in raw ] @@ -333,12 +356,18 @@ def cybed_fetch( ) rows.append( - FetchedFramework(framework_slug=slug, path=str(dest), sha256_verified=True) + FetchedFramework( + framework_slug=entry.get("license_slug") or slug, + release_slug=slug, + path=str(dest), + sha256_verified=True, + ) ) return pd.DataFrame( { "framework_slug": [row.framework_slug for row in rows], + "release_slug": [row.release_slug for row in rows], "path": [row.path for row in rows], "sha256_verified": pd.array( [row.sha256_verified for row in rows], dtype="boolean" @@ -348,7 +377,7 @@ def cybed_fetch( def load_graph( - frameworks: list[str] | None = None, + frameworks: str | list[str] | None = None, version: str | None = None, *, backend: GraphBackend = "rdflib", diff --git a/python/src/cybedtools/similarity.py b/python/src/cybedtools/similarity.py index b7aae2a..47586d5 100644 --- a/python/src/cybedtools/similarity.py +++ b/python/src/cybedtools/similarity.py @@ -17,9 +17,10 @@ import pandas as pd +from cybedtools._slug_alias import CybedtoolsFrameworkNotFoundError, resolve_framework_slug from cybedtools.queries import element_text, organizing_unit_framework_bindings, unit_element_bindings -__all__ = ["framework_similarity"] +__all__ = ["CybedtoolsFrameworkNotFoundError", "framework_similarity"] _TOKEN_SPLIT_RE = re.compile(r"[^a-z0-9]+") @@ -183,14 +184,21 @@ def framework_similarity(graph: object, from_: str, to: str, n: int = 5) -> pd.D :func:`cybedtools.load_graph`. from_ : str The `framework_slug` whose organizing units are the rows being - matched. + matched, either the versioned slug (``"nice-v2"``) or the short + release-file slug (``"nice"``; see :func:`cybedtools.cybed_fetch`). to : str - The `framework_slug` whose organizing units are the candidates. - May equal `from_`, to find near-duplicate units within one - framework. + The `framework_slug` whose organizing units are the candidates, in + either vocabulary. May equal `from_`, to find near-duplicate units + within one framework. n : int Matches to keep per from-unit (default 5). + Raises + ------ + CybedtoolsFrameworkNotFoundError + If `from_` or `to` does not resolve (in either vocabulary) to a + framework present in `graph`. + Returns ------- pandas.DataFrame @@ -208,6 +216,10 @@ def framework_similarity(graph: object, from_: str, to: str, n: int = 5) -> pd.D msg = "`to` must be a string." raise CybedtoolsScalarInputError(msg) + known_slugs = organizing_unit_framework_bindings(graph)["framework_slug"].dropna().unique().tolist() + from_ = resolve_framework_slug(from_, known_slugs, arg="from_") + to = resolve_framework_slug(to, known_slugs, arg="to") + empty = pd.DataFrame( { "from_unit": pd.Series(dtype="object"), diff --git a/python/tests/test_data.py b/python/tests/test_data.py index c675fd8..cf6017a 100644 --- a/python/tests/test_data.py +++ b/python/tests/test_data.py @@ -46,3 +46,14 @@ def test_cybed_license_non_string_slug_raises() -> None: """A non-string, non-None slug raises CybedtoolsScalarInputError.""" with pytest.raises(CybedtoolsScalarInputError): cybed_license(123) # type: ignore[arg-type] + + +def test_cybed_license_accepts_the_short_release_slug_as_an_alias() -> None: + """cybed_license() accepts either the versioned or short release slug.""" + assert cybed_license("nice").equals(cybed_license("nice-v2")) + assert cybed_license("otccf").equals(cybed_license("otccf-v1.1")) + assert cybed_license("cybok").equals(cybed_license("cybok-v1.1.0")) + # csta-2026 is its own framework, not an edition of csta: the short + # slug "csta" resolves to csta-2017, never csta-2026. + assert cybed_license("csta").equals(cybed_license("csta-2017")) + assert not cybed_license("csta").equals(cybed_license("csta-2026")) diff --git a/python/tests/test_fetch.py b/python/tests/test_fetch.py index 27c7cbf..413414c 100644 --- a/python/tests/test_fetch.py +++ b/python/tests/test_fetch.py @@ -94,6 +94,30 @@ def test_cybed_fetch_hash_mismatch_deletes_file( assert not any(cache_root.glob(f"{bad_slug}*")) +def test_cybed_fetch_bare_string_is_one_slug_not_iterable_of_characters(mock_release_url: str) -> None: + """A bare str `frameworks` argument is one slug, not an iterable of characters.""" + result = cybed_fetch(frameworks="fixture-wf1", version="1.0.0") + assert list(result["framework_slug"]) == ["fixture-wf1"] + + +def test_cybed_fetch_result_carries_both_slug_columns(mock_release_url: str) -> None: + """The result names both the versioned and short release slug explicitly.""" + result = cybed_fetch(frameworks=["fixture-wf1"], version="1.0.0") + assert "framework_slug" in result.columns + assert "release_slug" in result.columns + # The mock manifest carries no license_slug, so both fall back to the + # same (release) slug -- still two explicit columns. + assert result["framework_slug"].iloc[0] == "fixture-wf1" + assert result["release_slug"].iloc[0] == "fixture-wf1" + + +def test_cybed_fetch_accepts_data_v_prefixed_version(mock_release_url: str) -> None: + """A `data-v`-prefixed version (matching a GitHub release tag) is accepted.""" + plain = cybed_fetch(frameworks=["fixture-wf1"], version="1.0.0") + prefixed = cybed_fetch(frameworks=["fixture-wf1"], version="data-v1.0.0") + assert plain["path"].iloc[0] == prefixed["path"].iloc[0] + + def test_load_graph_returns_populated_rdflib_graph(mock_release_url: str) -> None: """load_graph() parses the fetched framework files into one rdflib.Graph.""" import rdflib diff --git a/python/tests/test_similarity.py b/python/tests/test_similarity.py index 077c376..f96034c 100644 --- a/python/tests/test_similarity.py +++ b/python/tests/test_similarity.py @@ -4,8 +4,10 @@ import rdflib +import pytest + from cybedtools._conformance import assert_matches_golden, conformance_dir -from cybedtools.similarity import _jaccard, _tokenize, framework_similarity +from cybedtools.similarity import CybedtoolsFrameworkNotFoundError, _jaccard, _tokenize, framework_similarity def _load_fixture_graph() -> rdflib.Graph: @@ -21,12 +23,28 @@ def test_framework_similarity_matches_golden() -> None: assert_matches_golden(actual, "framework_similarity", sort_by=["from_unit", "rank"]) -def test_framework_similarity_no_units_on_either_side_is_empty() -> None: - """A slug with no organizing units returns the zero-row frame with the contract's columns.""" +def test_framework_similarity_unknown_slug_raises_never_empty() -> None: + """An unknown slug (in either vocabulary) raises rather than returning an empty result.""" graph = _load_fixture_graph() - result = framework_similarity(graph, from_="no-such-framework", to="fixture-wf2", n=5) - assert list(result.columns) == ["from_unit", "to_unit", "score", "strength", "rank"] - assert len(result) == 0 + with pytest.raises(CybedtoolsFrameworkNotFoundError): + framework_similarity(graph, from_="no-such-framework", to="fixture-wf2", n=5) + with pytest.raises(CybedtoolsFrameworkNotFoundError): + framework_similarity(graph, from_="fixture-wf1", to="no-such-framework", n=5) + + +def test_framework_similarity_accepts_short_release_slug_alias() -> None: + """The short-release-slug resolver `framework_similarity()` uses accepts either form. + + The fixture graph uses non-versioned slugs directly, so the alias + resolver itself (also exercised end-to-end in test_data.py's + cybed_license alias tests) is pinned here against a known-slugs list + carrying real versioned framework slugs. + """ + from cybedtools._slug_alias import resolve_framework_slug + + assert resolve_framework_slug("nice", ["nice-v2"]) == "nice-v2" + # csta-2026 is its own framework, not an edition of csta. + assert resolve_framework_slug("csta", ["csta-2017", "csta-2026"]) == "csta-2017" def test_framework_similarity_same_slug_on_both_sides_finds_self_matches() -> None: diff --git a/tests/testthat/test-cybed-fetch.R b/tests/testthat/test-cybed-fetch.R index fb3eefd..f79acc7 100644 --- a/tests/testthat/test-cybed-fetch.R +++ b/tests/testthat/test-cybed-fetch.R @@ -166,6 +166,36 @@ test_that("load_graph fetches and parses the requested frameworks into one graph expect_setequal(meta$framework_slug, c("fixture-wf1", "fixture-wf2")) }) +test_that("cybed_fetch's result carries both framework_slug and release_slug", { + skip_if_no_mock_release() + skip_if_not_installed("withr") + fresh_cache_dir() + + with_mock_release({ + result <- cybed_fetch(frameworks = "fixture-wf1", version = "1.0.0") + }) + + expect_true(all(c("framework_slug", "release_slug") %in% names(result))) + # The mock manifest carries no license_slug, so both columns fall back to + # the same (release) slug -- still two explicit columns, never a silent + # substitution. + expect_equal(result$framework_slug, "fixture-wf1") + expect_equal(result$release_slug, "fixture-wf1") +}) + +test_that("cybed_fetch accepts a data-v-prefixed version", { + skip_if_no_mock_release() + skip_if_not_installed("withr") + fresh_cache_dir() + + with_mock_release({ + plain <- cybed_fetch(frameworks = "fixture-wf1", version = "1.0.0") + prefixed <- cybed_fetch(frameworks = "fixture-wf1", version = "data-v1.0.0") + }) + + expect_equal(plain$path, prefixed$path) +}) + test_that("cybed_release_base_url honors the option over the environment variable and default", { old_opt <- getOption("cybedtools.release_url") old_env <- Sys.getenv("CYBEDTOOLS_RELEASE_URL", unset = NA_character_) diff --git a/tests/testthat/test-framework-licenses.R b/tests/testthat/test-framework-licenses.R index 9e5e104..23e1078 100644 --- a/tests/testthat/test-framework-licenses.R +++ b/tests/testthat/test-framework-licenses.R @@ -181,6 +181,16 @@ test_that("the scywf row names NCA, links the official page and records the gran expect_match(row$license, "derived", fixed = TRUE) }) +test_that("cybed_license accepts the short release slug as an alias for the versioned slug", { + expect_identical(cybed_license("nice"), cybed_license("nice-v2")) + expect_identical(cybed_license("otccf"), cybed_license("otccf-v1.1")) + expect_identical(cybed_license("cybok"), cybed_license("cybok-v1.1.0")) + # csta-2026 is its own framework, not an edition of csta: the short slug + # "csta" resolves to csta-2017, never csta-2026. + expect_identical(cybed_license("csta"), cybed_license("csta-2017")) + expect_false(identical(cybed_license("csta"), cybed_license("csta-2026"))) +}) + test_that("the cybok row carries the OGL and CyBOK's prescribed attribution", { row <- cybed_license("cybok-v1.1.0") expect_equal(row$public_redistribution, "full_with_attribution") diff --git a/tests/testthat/test-similarity-helpers.R b/tests/testthat/test-similarity-helpers.R index 1352af6..30b8d0f 100644 --- a/tests/testthat/test-similarity-helpers.R +++ b/tests/testthat/test-similarity-helpers.R @@ -424,3 +424,50 @@ test_that("footgun tripwire: no cur_data() and no framework-named parameters in basename(f))) } }) + +# --------------------------------------------------------------------------- +# Slug alias resolution (R/slug-alias.R), used by framework_similarity() and +# cybed_license() +# --------------------------------------------------------------------------- + +test_that("framework_slug_alias_map derives the short slug the release export uses", { + alias_map <- cybedtools:::framework_slug_alias_map() + expect_equal(unname(alias_map["nice"]), "nice-v2") + expect_equal(unname(alias_map["otccf"]), "otccf-v1.1") + expect_equal(unname(alias_map["cybok"]), "cybok-v1.1.0") + # csta-2026 is a framework in its own right, not an edition of csta. + expect_equal(unname(alias_map["csta"]), "csta-2017") + expect_equal(unname(alias_map["csta-2026"]), "csta-2026") +}) + +test_that("resolve_framework_slug accepts either vocabulary and errors on an unknown slug", { + known <- c("nice-v2", "csta-2017", "csta-2026") + expect_equal(cybedtools:::resolve_framework_slug("nice-v2", known), "nice-v2") + expect_equal(cybedtools:::resolve_framework_slug("nice", known), "nice-v2") + expect_equal(cybedtools:::resolve_framework_slug("csta", known), "csta-2017") + + expect_error( + cybedtools:::resolve_framework_slug("not-a-real-framework", known), + class = "cybedtools_framework_not_found" + ) +}) + +test_that("framework_similarity errors with a classed condition on an unknown slug, never an empty result", { + skip_if_not_installed("rdflib") + conformance_dir <- system.file("conformance", package = "cybedtools") + if (!nzchar(conformance_dir)) { + conformance_dir <- testthat::test_path("..", "..", "inst", "conformance") + } + skip_if_not(file.exists(file.path(conformance_dir, "fixture.nt")), + "conformance fixture not found") + rdf <- rdflib::rdf_parse(file.path(conformance_dir, "fixture.nt"), format = "ntriples") + + expect_error( + framework_similarity(rdf, from = "not-a-real-framework", to = "fixture-wf2"), + class = "cybedtools_framework_not_found" + ) + expect_error( + framework_similarity(rdf, from = "fixture-wf1", to = "not-a-real-framework"), + class = "cybedtools_framework_not_found" + ) +}) diff --git a/vignettes/getting-started.Rmd b/vignettes/getting-started.Rmd index 74785a0..70c1658 100644 --- a/vignettes/getting-started.Rmd +++ b/vignettes/getting-started.Rmd @@ -37,13 +37,32 @@ A single SPARQL query targeting `cybed:RoleElement` returns atomic elements acro ```{r} #| eval: false # Install from GitHub (not yet on CRAN) -# install.packages("remotes") -remotes::install_github("ryanstraight/cybedtools") +# install.packages("pak") +pak::pak("ryanstraight/cybedtools") ``` -The package depends on `rdflib` for RDF/SPARQL, `jsonlite` for JSON-LD I/O, and a small subset of the tidyverse (`dplyr`, `purrr`, `tibble`). +The package depends on `rdflib` for RDF/SPARQL, `jsonlite` for JSON-LD I/O, and a small subset of the tidyverse (`dplyr`, `purrr`, `tibble`). A native Python package with the same API is on PyPI: `pip install cybedtools`. -## Pipeline overview +## Fetch the data and load a graph + +This is the path most users want: `cybed_fetch()` downloads and hash-verifies the public per-framework data release into a local cache (a no-op for files already cached with a verified hash), and `load_graph()` calls it internally and parses the result into one graph. No staging of framework source files is required for this path -- that is only needed to *rebuild* the release from primary sources (see "Rebuilding the graph from source (maintainers)" below). + +```{r} +#| eval: false +library(cybedtools) +library(dplyr) + +rdf <- load_graph() + +framework_summary |> + arrange(desc(organizing_unit_count)) +``` + +`cybed_fetch()` and `load_graph()` accept either the versioned framework slug (e.g. `"nice-v2"`, as carried by `framework_summary$framework_slug`) or the short release-file slug (e.g. `"nice"`); see `?cybed_fetch` for the two-vocabulary mapping. The data release backing the current package version is archived on Zenodo: [10.5281/zenodo.22884320](https://doi.org/10.5281/zenodo.22884320). + +## Rebuilding the graph from source (maintainers) + +Fetching the public release (above) is the path for using the corpus. Rebuilding it from primary sources is a separate, maintainer-only path, run from a clone of the repository. The pipeline turns staged framework source files into a queryable RDF graph through five scripts. Each stage produces an artifact the next stage reads: