Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pr-title-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
# adds a label, never removes one, so a title edit won't strip a label a
# human applied for an unrelated reason (e.g. good first issue).
#
# Note: "!" here means the PR breaks osm-diffs' OUTPUT SCHEMA
# Note: "!" here means the PR breaks osmdiffs' OUTPUT SCHEMA
# (conflated.parquet), not that it breaks some public API -- see
# docs/CONTRIBUTING.md and docs/RELEASING.md's "Choosing the version
# number". It's a hint for scripts/cut-release.sh, not a substitute for the
Expand Down
22 changes: 11 additions & 11 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "osm-diffs"
name = "osmdiffs"
version = "0.8.3"
edition = "2024"
license = "MIT"
Expand Down
12 changes: 6 additions & 6 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,10 @@ RUN for bin in tippecanoe tile-join; do \


# ----------------------------------------------------------------------------
# Stage 1.3: Build and test osm-diffs binary
# Stage 1.3: Build and test osmdiffs binary
# ----------------------------------------------------------------------------

WORKDIR /usr/osm-diffs
WORKDIR /usr/osmdiffs

COPY Cargo.toml Cargo.lock build.rs rust-toolchain.toml .
COPY scripts/sbom scripts/sbom
Expand All @@ -99,7 +99,7 @@ RUN cargo test --release --locked
# Stage 1.4: Generate the SBOM for the container
# ----------------------------------------------------------------------------
#
# Runs after both binaries (tippecanoe and osm-diffs) have been built, so a
# Runs after both binaries (tippecanoe and osmdiffs) have been built, so a
# single script invocation can see the facts about both of them.

RUN sh scripts/sbom/generate-sbom.sh /artifacts/sbom.cdx.json
Expand All @@ -119,12 +119,12 @@ COPY --from=builder /usr/local/bin/tippecanoe /usr/local/bin/tippecanoe
COPY --from=builder /usr/local/bin/tile-join /usr/local/bin/tile-join

COPY --from=builder --chown=1000:1000 \
/usr/osm-diffs/target/release/osm-diffs \
/app/osm-diffs
/usr/osmdiffs/target/release/osmdiffs \
/app/osmdiffs

USER 1000

ENTRYPOINT ["/app/osm-diffs"]
ENTRYPOINT ["/app/osmdiffs"]

LABEL \
org.opencontainers.image.authors="Sascha Brawer <sascha@brawer.ch>" \
Expand Down
2 changes: 1 addition & 1 deletion deny.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ignore = [
# XML attribute/namespace handling. Both only matter for untrusted input;
# `s3` uses quick-xml solely to parse XML *responses from the S3 endpoint
# we're uploading to*, not attacker-controlled input, so this isn't
# exploitable in how osm-diffs uses it. Fix is upstream in `s3`, not
# exploitable in how osmdiffs uses it. Fix is upstream in `s3`, not
# something we can patch here (no git/path deps -- see docs/CONTRIBUTING.md
# on why): https://github.com/lvillis/s3-rs/pull/15 bumps it to >=0.41.0.
# Re-run `cargo deny check advisories` after that merges and a new `s3`
Expand Down
4 changes: 2 additions & 2 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing

Thanks for taking a look at `osm-diffs`! 👋 Contributions, questions,
Thanks for taking a look at `osmdiffs`! 👋 Contributions, questions,
and bug reports are all welcome — this project is still early, so
there’s no such thing as too small a
[Pull Request (PR)](https://docs.github.com/en/pull-requests/get-started/about-pull-requests).
Expand Down Expand Up @@ -60,7 +60,7 @@ categorized release notes — so getting the type right saves the
maintainer a manual labeling step, it’s not just a style nit.

**The `!` marker is special here.** Normally it means “breaks the public
API.” `osm-diffs` doesn’t have one client code links against — what it
API.” `osmdiffs` doesn’t have one client code links against — what it
has is an **output schema** (`conflated.parquet`), and that’s what
matters to everyone downstream. So on this project, `!` means *this PR
breaks the output schema*, per the rules in
Expand Down
4 changes: 2 additions & 2 deletions docs/LOGGING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Logging

`osm-diffs run` writes `pipeline.log` to its `--workdir`, one JSON
`osmdiffs run` writes `pipeline.log` to its `--workdir`, one JSON
object per line ([JSON Lines](https://jsonlines.org/)) —
machine-readable rather than free-form text, so a log can be grepped,
`jq`’d, or loaded into whatever analysis tool you like without first
Expand All @@ -9,7 +9,7 @@ Every record has:

- `timestamp` — RFC 3339, UTC.
- `level` — `INFO`, `WARN`, `ERROR`, …
- `target` — which module logged it (e.g. `osm_diffs::pipeline::conflate`).
- `target` — which module logged it (e.g. `osmdiffs::pipeline::conflate`).
- `message` — the human-readable text.
- `fields` — present only on records that attach structured data
(numbers, an id, …) instead of just interpolating it into the
Expand Down
4 changes: 2 additions & 2 deletions docs/PRODUCTION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Running `osm-diffs` in production
# Running `osmdiffs` in production

`osm-diffs` doesn’t run anywhere permanent yet — see
`osmdiffs` doesn’t run anywhere permanent yet — see
[`TECHNICAL_DESIGN.md`](TECHNICAL_DESIGN.md)’s “Status” section. This
document isn’t a description of an existing deployment; it’s the
operational knowledge gathered from testing on Hetzner Cloud
Expand Down
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ Want to work on the pipeline itself, or run it somewhere? Start here.
- [`TESTING.md`](TESTING.md) — how this project’s tests are organized,
what CI enforces, and how to try a change on real hardware before it
lands.
- [`LOGGING.md`](LOGGING.md) — the JSON log format `osm-diffs run`
- [`LOGGING.md`](LOGGING.md) — the JSON log format `osmdiffs run`
writes, and where weekly-run logs end up archived in S3.
- [`RELEASING.md`](RELEASING.md) — how to cut a release of `osm-diffs`.
- [`RELEASING.md`](RELEASING.md) — how to cut a release of `osmdiffs`.
- [`PRODUCTION.md`](PRODUCTION.md) — operational knowledge for running
a released container somewhere real: hardware sizing, required
configuration, serving the outputs over a CDN, what to monitor.
Expand Down
4 changes: 2 additions & 2 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Cutting a release

This document is the practical how-to for cutting a release of
`osm-diffs`. For the concepts behind *why* the process looks like this
`osmdiffs`. For the concepts behind *why* the process looks like this
(SBOM, attestations, immutable releases, ...), see
[`SUPPLY_CHAIN_SECURITY.md`](SUPPLY_CHAIN_SECURITY.md).

Expand All @@ -27,7 +27,7 @@ the pipeline’s **output schema** — not by how much code changed.
This is a different question than what [SemVer](https://semver.org/)
usually answers. Programmers normally think of SemVer in terms of API
compatibility for code that *links against* a library. Nobody links
against `osm-diffs`:
against `osmdiffs`:
downstream clients only ever consume the *data* it produces. So the
question to ask isn’t “did the code change in a breaking way,” it’s “does
this change what a client reading our output has to handle differently”:
Expand Down
14 changes: 7 additions & 7 deletions docs/SUPPLY_CHAIN_SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ process itself. This document explains the security practices we use
around ours (see [`RELEASING.md`](RELEASING.md) for the actual,
repo-specific steps). It’s written for someone who can code but hasn’t
necessarily done release engineering before. None of it is original to
`osm-diffs` — everything here is standard practice, just explained in
`osmdiffs` — everything here is standard practice, just explained in
one place.

## Containers
Expand All @@ -18,7 +18,7 @@ virtual machine, since it shares the host’s kernel instead of running
its own, while still keeping what’s inside isolated from everything
else on that host.

We publish `osm-diffs` as an [OCI image](https://opencontainers.org/) —
We publish `osmdiffs` as an [OCI image](https://opencontainers.org/) —
the open, vendor-neutral standard most container tooling implements
today — to a
[container registry](https://www.redhat.com/en/topics/cloud-native-apps/what-is-a-container-registry)
Expand All @@ -31,7 +31,7 @@ a plain-text file listing the build steps, one instruction per line.
## Minimal containers

`Containerfile` builds in two stages, and only the second one ships:
`FROM scratch`, containing nothing but the `osm-diffs` and `tippecanoe`
`FROM scratch`, containing nothing but the `osmdiffs` and `tippecanoe`
binaries. The whole build toolchain (Rust, a C compiler, `apk`, `git`,
...) stays behind in the discarded first stage. Both binaries are
statically linked against [musl](https://en.wikipedia.org/wiki/Musl),
Expand Down Expand Up @@ -64,7 +64,7 @@ for your machine.
Because we ship `FROM scratch` (see “Minimal containers” above),
there’s no `/etc/ssl` in the final image either — no OS-provided CA
trust store to fall back on for verifying a TLS server’s certificate.
Rather than adding one, `osm-diffs` embeds Mozilla’s own CA root
Rather than adding one, `osmdiffs` embeds Mozilla’s own CA root
bundle directly into the binary at compile time, via the
[`webpki-roots`](https://github.com/rustls/webpki-roots) crate (see
`main.rs`’s `build_client()`) — the same trust store
Expand All @@ -91,7 +91,7 @@ library X?” without rebuilding the software or reading its source.
## SBOM and CBOM

Our SBOM describes the container image we publish: the Rust dependency
graph of the `osm-diffs` binary, the statically linked `tippecanoe`
graph of the `osmdiffs` binary, the statically linked `tippecanoe`
binary and its libraries, build-environment details, and licenses.

Alongside it, our SBOM includes a small
Expand Down Expand Up @@ -175,7 +175,7 @@ image* was built by our workflow from our source (provenance), and

## Is this overkill for a project like this?

Fair question. `osm-diffs` is open source, processes only public data,
Fair question. `osmdiffs` is open source, processes only public data,
and handles no secrets — our actual threat model is modest compared to
a service handling user data or credentials. So no, this
isn’t strictly *necessary* the way it would be for many other projects.
Expand Down Expand Up @@ -205,7 +205,7 @@ requirement, for example under the US
and the EU
[Cyber Resilience Act](https://en.wikipedia.org/wiki/Cyber_Resilience_Act).

None of that raises `osm-diffs`’s own threat model much — and
None of that raises `osmdiffs`’s own threat model much — and
honestly, everything in this document is overkill for what a project
like this actually needs. We set it up anyway to make a different
point: this level of rigor doesn’t require a large company’s
Expand Down
2 changes: 1 addition & 1 deletion docs/TECHNICAL_DESIGN.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Technical design: `osm-diffs`
# Technical design: `osmdiffs`

Status: Work in Progress — see [“Status”](#status) at the end of this
document for what’s still ahead.
Expand Down
2 changes: 1 addition & 1 deletion scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ process these fit into.

## Testing development branches

Unrelated to how `osm-diffs` actually ships to production — this is for
Unrelated to how `osmdiffs` actually ships to production — this is for
ad hoc validation of a branch before it lands.

- [`test-on-hetzner/`](test-on-hetzner/README.md): spins
Expand Down
2 changes: 1 addition & 1 deletion scripts/cut-release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ fi
# ── scan for Conventional-Commits breaking markers since the last release ───
# This is a hint, not a rule: choosing the version is still your call (see
# "Choosing the version number" in docs/RELEASING.md). Reminder, since it's
# easy to reach for the usual meaning by reflex: on osm-diffs, "!" flags a
# easy to reach for the usual meaning by reflex: on osmdiffs, "!" flags a
# commit that breaks the OUTPUT SCHEMA (conflated.parquet), not one that
# breaks some public API -- see docs/CONTRIBUTING.md.
LAST_TAG="$(git describe --tags --abbrev=0 2>/dev/null || true)"
Expand Down
8 changes: 4 additions & 4 deletions scripts/sbom/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This directory holds the scripts that generate the Software Bill of
Materials (SBOM) — including a small Cryptographic Bill of Materials,
CBOM — for the `osm-diffs` container image. See
CBOM — for the `osmdiffs` container image. See
[`../../docs/SUPPLY_CHAIN_SECURITY.md`](../../docs/SUPPLY_CHAIN_SECURITY.md)
for what an SBOM/CBOM actually is and why we publish one; this document
only covers how it’s implemented here.
Expand Down Expand Up @@ -36,13 +36,13 @@ Linux version, compiler versions, library versions, ...) using standard
tools such as `apk`, `sed`, `awk` and `grep`. It then:

1. Runs `cargo cyclonedx` to get the Rust dependency graph for the
`osm-diffs` binary, in CycloneDX 1.5 format (the newest that
`osmdiffs` binary, in CycloneDX 1.5 format (the newest that
`cargo cyclonedx` currently supports).
2. Pipes that through [`pipeline.jq`](pipeline.jq), which upgrades it to
CycloneDX 1.7 and enriches it with build-environment metadata,
supplier/license info, the CBOM facts (a `cryptographic-asset`
component for TLS 1.3, plus `crypto:tls:*`/`crypto:catrust:*`
custom properties on the `osm-diffs` component — see that file’s own
custom properties on the `osmdiffs` component — see that file’s own
comments for why a curated CA root bundle isn’t a good fit for
CycloneDX’s `certificate` cryptographic-asset type), and the
vendored “data” components (`id-tagging-schema`,
Expand Down Expand Up @@ -83,7 +83,7 @@ Some of the facts it normally reads via Alpine’s `apk` (the exact musl,
sqlite and zlib versions statically linked into `tippecanoe`, the Alpine
version itself) aren’t available outside of that build environment. In
that case, the script inserts placeholder values, prints a warning, and
marks the SBOM as `osm-diffs:sbom:devBuild` in its metadata properties.
marks the SBOM as `osmdiffs:sbom:devBuild` in its metadata properties.
Such an SBOM is useful for checking structural or semantic validity (e.g.
with `cyclonedx-cli validate`), but must not be treated as an accurate
description of a production build.
8 changes: 4 additions & 4 deletions scripts/sbom/generate-sbom.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# SPDX-License-Identifier: MIT
#
# Generate the CycloneDX Software Bill of Materials (SBOM) for the
# osm-diffs container image: one single file, describing the osm-diffs
# osmdiffs container image: one single file, describing the osmdiffs
# binary (with its Rust dependency graph) and the two statically
# linked tippecanoe-project binaries that ship alongside it, tippecanoe
# and tile-join.
Expand All @@ -24,7 +24,7 @@
# printed. Such a placeholder-filled SBOM is fine for checking structural
# or semantic validity, but MUST NOT be treated as an accurate SBOM for a
# production build; the output is marked as such in its own metadata
# (metadata.properties: "osm-diffs:sbom:devBuild").
# (metadata.properties: "osmdiffs:sbom:devBuild").
#
# Note on the image digest: this script does not, and cannot, know the
# digest of the container image it is building -- that digest only
Expand Down Expand Up @@ -126,11 +126,11 @@ else
exit 1
fi

# ── build the osm-diffs (pipeline) SBOM fragment ─────────────────────────────
# ── build the osmdiffs (pipeline) SBOM fragment ─────────────────────────────
# cargo cyclonedx always writes next to Cargo.toml, named after the binary
# target; move it into our workdir once it's done so it can't be mistaken
# for a second, independently meaningful SBOM file.
RAW_PIPELINE="${PROJECT_ROOT}/osm-diffs_bin.cdx.json"
RAW_PIPELINE="${PROJECT_ROOT}/osmdiffs_bin.cdx.json"
rm -f "$RAW_PIPELINE" # clean up any leftover file from a previous failed run

cargo cyclonedx \
Expand Down
4 changes: 2 additions & 2 deletions scripts/sbom/merge.jq
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: MIT
#
# Assemble the final, single SBOM for the container image out of the
# osm-diffs (pipeline), tippecanoe, and tile-join component fragments.
# osmdiffs (pipeline), tippecanoe, and tile-join component fragments.
#
# Invoked as `jq -n -f merge.jq` (no stdin input; all three fragments
# are read via --slurpfile).
Expand All @@ -20,7 +20,7 @@
# serial SBOM serial number (a "urn:uuid:..." string)
# image container image name, e.g. "brawer/osmdiffs"
# timestamp build timestamp, RFC 3339
# pipeline (--slurpfile) the enriched osm-diffs SBOM fragment, as
# pipeline (--slurpfile) the enriched osmdiffs SBOM fragment, as
# produced by pipeline.jq
# tippecanoe (--slurpfile) the tippecanoe SBOM fragment, as produced
# by tippecanoe.jq
Expand Down
10 changes: 5 additions & 5 deletions scripts/sbom/pipeline.jq
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
# SPDX-License-Identifier: MIT
#
# Enrich the raw CycloneDX SBOM produced by `cargo cyclonedx` for the
# osm-diffs binary: fix up the dependency graph, attach build-environment
# osmdiffs binary: fix up the dependency graph, attach build-environment
# and supplier metadata, and add the vendored "data" components (and the
# Cryptographic Bill of Materials entries) that cargo-cyclonedx cannot see
# on its own.
#
# Input (stdin): the raw CycloneDX 1.5 document from `cargo cyclonedx`.
# Output: a CycloneDX 1.7 document, still describing only the osm-diffs
# Output: a CycloneDX 1.7 document, still describing only the osmdiffs
# application (not the final container -- that's assembled by merge.jq).
#
# Arguments (all required, passed with --arg):
Expand Down Expand Up @@ -42,8 +42,8 @@ def add_supplier:
.
end;

# Patch bom-ref of main application to read "osm-diffs-1.2.3"
# instead of "path+file:///Users/sascha/src/osm-diffs#osm-diffs".
# Patch bom-ref of main application to read "osmdiffs-1.2.3"
# instead of "path+file:///Users/sascha/src/osm-diffs#osmdiffs".
.metadata.component."bom-ref" as $orig_root_ref |
( .metadata.component.name + "-" + .metadata.component.version ) as $root_ref |
.metadata.component."bom-ref" = $root_ref |
Expand Down Expand Up @@ -273,7 +273,7 @@ else . end |

if $DEV_BUILD == "true" then
.metadata.properties = ((.metadata.properties // []) + [{
name: "osm-diffs:sbom:devBuild",
name: "osmdiffs:sbom:devBuild",
value: "true"
}])
else
Expand Down
4 changes: 2 additions & 2 deletions scripts/sbom/tippecanoe-binary.jq
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#
# Build a CycloneDX 1.7 SBOM fragment for one statically compiled
# binary from the felt/tippecanoe build that we ship alongside
# osm-diffs in the OCI container image -- tippecanoe itself, or its
# osmdiffs in the OCI container image -- tippecanoe itself, or its
# sibling tile-join (used by pipeline::tiles::join_tiles to merge
# conflated.pmtiles' overview and detail passes; see
# pipeline::conflated_tiles' module doc comment for why that split
Expand Down Expand Up @@ -79,7 +79,7 @@ def alpine_supplier: {name: "Alpine Linux", url: ["https://alpinelinux.org"]};
}]
},
properties: (if $DEV_BUILD == "true" then
[{name: "osm-diffs:sbom:devBuild", value: "true"}]
[{name: "osmdiffs:sbom:devBuild", value: "true"}]
else
[]
end)
Expand Down
Loading
Loading