Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
# toggled) showed it saves ~180s on a typical PR, since Cargo still
# re-invokes rustc for much of the dependency graph even when the
# restored target/ cache matches Cargo.lock exactly. See
# https://github.com/alltheplaces/osm-diffs/issues/597 for the numbers.
# https://github.com/brawer/osmdiffs/issues/597 for the numbers.
env:
RUSTC_WRAPPER: "sccache"
SCCACHE_GHA_ENABLED: "true"
Expand Down
4 changes: 2 additions & 2 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
FROM rust:1.98.0-alpine3.23 AS builder

ARG BUILD_TIMESTAMP
ARG IMAGE_NAME=alltheplaces/osm-diffs
ARG IMAGE_NAME=brawer/osmdiffs
ARG TIPPECANOE_VERSION=2.79.0
# Commit that the "2.79.0" tag pointed to as of 2026-08-08. Pinned by
# commit, not by tag/branch name: git tags are mutable references that
Expand Down Expand Up @@ -129,7 +129,7 @@ ENTRYPOINT ["/app/osm-diffs"]
LABEL \
org.opencontainers.image.authors="Sascha Brawer <sascha@brawer.ch>" \
org.opencontainers.image.created=$BUILD_TIMESTAMP \
org.opencontainers.image.description="Data pipeline for alltheplaces/osm-diffs" \
org.opencontainers.image.description="Data pipeline for brawer/osmdiffs" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.revision=$VCS_REF \
org.opencontainers.image.source=$VCS_URL \
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# OSM Diffs

[![Coverage Status](https://coveralls.io/repos/github/alltheplaces/osm-diffs/badge.svg?branch=main)](https://coveralls.io/github/alltheplaces/osm-diffs?branch=main)
[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/alltheplaces/osm-diffs/badge)](https://scorecard.dev/viewer/?uri=github.com/alltheplaces/osm-diffs)
[![Coverage Status](https://coveralls.io/repos/github/brawer/osmdiffs/badge.svg?branch=main)](https://coveralls.io/github/brawer/osmdiffs?branch=main)
[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/brawer/osmdiffs/badge)](https://scorecard.dev/viewer/?uri=github.com/brawer/osmdiffs)
[![Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.](https://www.repostatus.org/badges/latest/wip.svg)](https://www.repostatus.org/#wip)

Once completed, this will be a pipeline to compute weekly diffs
Expand All @@ -17,7 +17,7 @@ feed edit proposals to tools such as
[MapRoulette](https://maproulette.org/), where human users can
manually check each edit before applying it to OpenStreetMap — which
edits go where is still being worked out, see
[#696](https://github.com/alltheplaces/osm-diffs/issues/696).
[#696](https://github.com/brawer/osmdiffs/issues/696).

Just want the data this pipeline produces? See
[`docs/outputs/`](docs/outputs/) for what’s in it and how to read it.
Expand Down
20 changes: 10 additions & 10 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ there’s no such thing as too small a
[Pull Request (PR)](https://docs.github.com/en/pull-requests/get-started/about-pull-requests).

Please be kind to each other and to maintainers; see our
[Code of Conduct](https://github.com/alltheplaces/osm-diffs/blob/main/docs/CODE_OF_CONDUCT.md).
[Code of Conduct](https://github.com/brawer/osmdiffs/blob/main/docs/CODE_OF_CONDUCT.md).

## Before opening a Pull Request

Expand All @@ -26,18 +26,18 @@ cargo deny check
```

This checks the new dependency graph against
[`deny.toml`](https://github.com/alltheplaces/osm-diffs/blob/main/deny.toml)
[`deny.toml`](https://github.com/brawer/osmdiffs/blob/main/deny.toml)
— disallowed licenses, banned/duplicated crates, untrusted sources, and
known RUSTSEC/OSV advisories.

If you changed a Python script, run `uv run pytest` from `scripts/`.

After you’ve sent a Pull Request, our Continuous Integration (CI) runs a
series of checks on it — the commands above are a subset of what CI runs
([`test.yml`](https://github.com/alltheplaces/osm-diffs/blob/main/.github/workflows/test.yml),
[`cargo-deny.yml`](https://github.com/alltheplaces/osm-diffs/blob/main/.github/workflows/cargo-deny.yml)).
([`test.yml`](https://github.com/brawer/osmdiffs/blob/main/.github/workflows/test.yml),
[`cargo-deny.yml`](https://github.com/brawer/osmdiffs/blob/main/.github/workflows/cargo-deny.yml)).
Running them locally first saves a round-trip through CI — see
[`TESTING.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/TESTING.md)
[`TESTING.md`](https://github.com/brawer/osmdiffs/blob/main/docs/TESTING.md)
for what else CI checks.

## PR titles: Conventional Commits
Expand All @@ -52,10 +52,10 @@ PRs are squash-merged, and the PR title becomes the commit message on

`type` is one of `feat`, `fix`, `docs`, `style`, `refactor`, `perf`,
`test`, `build`, `ci`, or `chore`. A bot
([`pr-title-lint.yml`](https://github.com/alltheplaces/osm-diffs/blob/main/.github/workflows/pr-title-lint.yml))
([`pr-title-lint.yml`](https://github.com/brawer/osmdiffs/blob/main/.github/workflows/pr-title-lint.yml))
checks this on every PR and applies an `enhancement`/`bug`/`breaking-change`
label from it, which feeds
[`.github/release.yml`](https://github.com/alltheplaces/osm-diffs/blob/main/.github/release.yml)’s
[`.github/release.yml`](https://github.com/brawer/osmdiffs/blob/main/.github/release.yml)’s
categorized release notes — so getting the type right saves the
maintainer a manual labeling step, it’s not just a style nit.

Expand All @@ -64,7 +64,7 @@ API.” `osm-diffs` doesn’t have one client code links against — what it
has is an **output schema** (`conflated.parquet`), and that’s what
matters to everyone downstream. So on this project, `!` means *this PR
breaks the output schema*, per the rules in
[`RELEASING.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/RELEASING.md#choosing-the-version-number)
[`RELEASING.md`](https://github.com/brawer/osmdiffs/blob/main/docs/RELEASING.md#choosing-the-version-number)
— not that some function signature changed. A `refactor!:` or `chore!:`
that happens to rename a Parquet column is exactly as `!` as a `feat!:`
that removes one; a `feat:` CLI flag that doesn’t touch the schema isn’t
Expand All @@ -73,10 +73,10 @@ floor, but the actual version is still your call, per `RELEASING.md`.

## Where to go next

- [`docs/TESTING.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/TESTING.md)
- [`docs/TESTING.md`](https://github.com/brawer/osmdiffs/blob/main/docs/TESTING.md)
— how this project’s tests are organized, and how to try a change on
real full-planet data before it lands.
- [`docs/RELEASING.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/RELEASING.md)
- [`docs/RELEASING.md`](https://github.com/brawer/osmdiffs/blob/main/docs/RELEASING.md)
— how a release actually ships, for once your change has landed on
`main`.

Expand Down
4 changes: 2 additions & 2 deletions docs/LOGGING.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,12 @@ thing. Two concrete examples of what this data enables, both written by
[Claude Code](https://claude.com/claude-code) straight from a run’s
logs, with no other tooling built for the purpose:

- [alltheplaces/osm-diffs#665, comment](https://github.com/alltheplaces/osm-diffs/pull/665#issuecomment-5303068423) —
- [brawer/osmdiffs#665, comment](https://github.com/brawer/osmdiffs/pull/665#issuecomment-5303068423) —
a full memory/disk/timing analysis of a full-planet run on
memory-constrained hardware, validating the design assumption behind
`OsmFeatureIndex` (relying on the OS page cache instead of an
explicit decode cache).
- [alltheplaces/osm-diffs#636](https://github.com/alltheplaces/osm-diffs/issues/636) —
- [brawer/osmdiffs#636](https://github.com/brawer/osmdiffs/issues/636) —
a survey of OpenStreetMap data-quality issues, found by combing
through `pipeline.log`’s `could not build geometry` warnings and
cross-referencing the flagged features against the live OSM API.
Expand Down
14 changes: 7 additions & 7 deletions docs/PRODUCTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
document isn’t a description of an existing deployment; it’s the
operational knowledge gathered from testing on Hetzner Cloud
([`scripts/test-on-hetzner`](../scripts/test-on-hetzner/README.md),
[#711](https://github.com/alltheplaces/osm-diffs/issues/711),
[#722](https://github.com/alltheplaces/osm-diffs/issues/722)), written
[#711](https://github.com/brawer/osmdiffs/issues/711),
[#722](https://github.com/brawer/osmdiffs/issues/722)), written
down now so whoever sets up the actual deployment doesn’t have to
re-establish it from scratch.

Expand Down Expand Up @@ -47,25 +47,25 @@ generous limit.
to **~143GB** once `import_osm` finishes and its temporary files are
cleaned up — consistent across two independent runs on different
hardware (this sweep, and the earlier
[#665](https://github.com/alltheplaces/osm-diffs/issues/665) shakedown,
[#665](https://github.com/brawer/osmdiffs/issues/665) shakedown,
which saw ~174GB peak / ~148GB settled). **220–250GB** gives reasonable
headroom over that peak without being wildly oversized; the 400GB
default `scripts/test-on-hetzner/cloud_test.py` uses for testing is
itself generous margin, not a sizing recommendation.

## Container invocation

The published image (`ghcr.io/alltheplaces/osm-diffs:vX.Y.Z` — pin an
The published image (`ghcr.io/brawer/osmdiffs:vX.Y.Z` — pin an
exact tag; see
[the releases page](https://github.com/alltheplaces/osm-diffs/releases)
[the releases page](https://github.com/brawer/osmdiffs/releases)
for the current version and its notes) runs as:

```sh
podman run --rm --read-only \
--memory=8g --cpus=6 \
-v /path/to/workdir:/workdir \
--env-file s3.env \
ghcr.io/alltheplaces/osm-diffs:vX.Y.Z \
ghcr.io/brawer/osmdiffs:vX.Y.Z \
run --workdir /workdir --run_id "$RUN_ID"
```

Expand Down Expand Up @@ -249,7 +249,7 @@ get started:
- **Step timings drifting**: `import_osm`’s own sub-steps
(`import_osm.fetch`/`.open`/`.prune`/`.assemble`/`.index`, logged
individually as of
[#761](https://github.com/alltheplaces/osm-diffs/pull/761)) are worth
[#761](https://github.com/brawer/osmdiffs/pull/761)) are worth
watching over time as the planet grows — a slow drift is expected;
a sudden jump on an otherwise-unchanged config is worth investigating
the way this document’s own `--mem-limit` sweep did.
Expand Down
8 changes: 4 additions & 4 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ two checks described in
done for real rather than assumed. This is exactly what was done by hand
to confirm v0.6.9, the first release cut with `cut-release.sh` — see the
comment trail on
[alltheplaces/osm-diffs#562](https://github.com/alltheplaces/osm-diffs/pull/562)
[brawer/osmdiffs#562](https://github.com/brawer/osmdiffs/pull/562)
for that walkthrough, which is what `verify-release.sh` automates.

## Rules
Expand Down Expand Up @@ -219,8 +219,8 @@ for that walkthrough, which is what `verify-release.sh` automates.
- A few low-priority, deliberately-deferred items are tracked separately
and don’t block anything: automated freshness checks for vendored
dependencies
([#555](https://github.com/alltheplaces/osm-diffs/issues/555)), moving
([#555](https://github.com/brawer/osmdiffs/issues/555)), moving
`cargo-cyclonedx` off Alpine’s edge repo once it’s available in stable
([#556](https://github.com/alltheplaces/osm-diffs/issues/556)), and
([#556](https://github.com/brawer/osmdiffs/issues/556)), and
watching for an emerging standard on index-level SBOMs for multi-arch
images ([#589](https://github.com/alltheplaces/osm-diffs/issues/589)).
images ([#589](https://github.com/brawer/osmdiffs/issues/589)).
20 changes: 10 additions & 10 deletions docs/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,41 +6,41 @@ report it privately.
We prefer the [GitHub mechanism for privately reporting a
vulnerability](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability#privately-reporting-a-security-vulnerability):
under the [repository’s security
tab](https://github.com/alltheplaces/osm-diffs/security), click
tab](https://github.com/brawer/osmdiffs/security), click
“Report a vulnerability” to open the advisory form.

For how we secure the release process itself — build provenance,
SBOMs, signed attestations — see
[`SUPPLY_CHAIN_SECURITY.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/SUPPLY_CHAIN_SECURITY.md).
[`SUPPLY_CHAIN_SECURITY.md`](https://github.com/brawer/osmdiffs/blob/main/docs/SUPPLY_CHAIN_SECURITY.md).
Our container has no OS at all, so there’s no OS trust store to verify
outbound TLS connections against either — see that document’s
[“Certificate
trust”](https://github.com/alltheplaces/osm-diffs/blob/main/docs/SUPPLY_CHAIN_SECURITY.md#certificate-trust)
trust”](https://github.com/brawer/osmdiffs/blob/main/docs/SUPPLY_CHAIN_SECURITY.md#certificate-trust)
section for what we do instead.

Every change also goes through
[SAST](https://en.wikipedia.org/wiki/Static_application_security_testing)
via [GitHub CodeQL](https://codeql.github.com/), enforced by branch
protection on `main` — see
[`TESTING.md`](https://github.com/alltheplaces/osm-diffs/blob/main/docs/TESTING.md)
[`TESTING.md`](https://github.com/brawer/osmdiffs/blob/main/docs/TESTING.md)
for details. We publish an [OSSF
Scorecard](https://scorecard.dev/viewer/?uri=github.com/alltheplaces/osm-diffs)
Scorecard](https://scorecard.dev/viewer/?uri=github.com/brawer/osmdiffs)
analysis on every push to `main` (badge in the
[README](https://github.com/alltheplaces/osm-diffs#readme)); its
[README](https://github.com/brawer/osmdiffs#readme)); its
“Vulnerabilities” check reports known, already-public advisories in
our dependencies (RUSTSEC/OSV) — the same ones
[Dependabot](https://github.com/alltheplaces/osm-diffs/network/updates)
[Dependabot](https://github.com/brawer/osmdiffs/network/updates)
already opens PRs for. A nonzero score there isn’t a new finding and
doesn’t need private disclosure; check [open pull
requests](https://github.com/alltheplaces/osm-diffs/pulls) for the fix
requests](https://github.com/brawer/osmdiffs/pulls) for the fix
already in progress.

On top of that,
[`cargo-deny.yml`](https://github.com/alltheplaces/osm-diffs/blob/main/.github/workflows/cargo-deny.yml)
[`cargo-deny.yml`](https://github.com/brawer/osmdiffs/blob/main/.github/workflows/cargo-deny.yml)
gates PRs and runs weekly, via
[`cargo-deny`](https://github.com/EmbarkStudios/cargo-deny) against
the rules in
[`deny.toml`](https://github.com/alltheplaces/osm-diffs/blob/main/deny.toml) —
[`deny.toml`](https://github.com/brawer/osmdiffs/blob/main/deny.toml) —
dependency licenses, banned/duplicated crates, untrusted sources, and
known advisories alike. An advisory that’s already known, already
tracked, and not fixable from this repo (e.g. pinned deep in a
Expand Down
4 changes: 2 additions & 2 deletions docs/SUPPLY_CHAIN_SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ reusable workflow,
that file and
[`verify-release.sh`](../scripts/verify-release.sh) for the mechanism,
and
[alltheplaces/osm-diffs#608](https://github.com/alltheplaces/osm-diffs/pull/608)
[brawer/osmdiffs#608](https://github.com/brawer/osmdiffs/pull/608)
for this having been checked against a real release, not just assumed.

We use GitHub’s native
Expand All @@ -151,7 +151,7 @@ per-architecture image gets its own SBOM and provenance attestation,
while the top-level manifest list gets only provenance — it has no
software content of its own to describe. This matches current
mainstream practice; see
[alltheplaces/osm-diffs#589](https://github.com/alltheplaces/osm-diffs/issues/589)
[brawer/osmdiffs#589](https://github.com/brawer/osmdiffs/issues/589)
for where it might evolve.

## Immutable releases
Expand Down
Loading
Loading