Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Keep the build context small and reproducible. The image installs deps fresh
# (pip, npm), so none of these belong in it.
.git
.github
.gitlab-ci.yml
.idea
.pytest_cache
.DS_Store
**/__pycache__
**/*.pyc
**/*.egg-info
**/venv
**/.venv
**/node_modules
singularity
spike
conductor
docs
*.sif
data
logs
75 changes: 75 additions & 0 deletions .github/workflows/build_docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Build Docker image

on:
push:
branches: ["develop"]
tags: ["v*"]
workflow_dispatch:

concurrency: docker

env:
REGISTRY: ghcr.io
REGISTRY_IMAGE: educelab/preppy

permissions:
contents: read
packages: write

jobs:
build:
name: Build and push
runs-on: ubuntu-latest
steps:
- name: Clean up disk space
run: |
# The multi-arch build + toolchain is large; reclaim runner space.
df -h /
sudo rm -rf /usr/lib/jvm
sudo rm -rf /usr/local/.ghcup
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/share/swift
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
df -h /

- name: Checkout
uses: actions/checkout@v4

- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.REGISTRY_IMAGE }}
# `latest` tracks the most recent release tag; `edge` tracks develop.
flavor: latest=false
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=edge

- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.REGISTRY_IMAGE }}:buildcache
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.REGISTRY_IMAGE }}:buildcache,mode=max
94 changes: 94 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: CI

on:
push:
branches: ["develop"]
tags: ["v*"]
pull_request:
workflow_dispatch:

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
# ===========================================================================
# Python package (preppy/)
# ---------------------------------------------------------------------------
# Runs the suite across supported interpreters. ImageMagick + the `preview`
# extra (trimesh/pyrender, with OSMesa for offscreen GL) are installed so the
# texture-normalization and rendered-thumbnail paths actually execute and
# catch regressions here rather than self-skipping. The heavier geometry
# toolchain (ktx/gltfpack/node) and pymeshlab remain the manual `integration`
# job's job; those tests still self-skip here. The tool-availability skips are
# for bare local-dev machines, not CI.
# ===========================================================================
test:
name: py ${{ matrix.python-version }}
runs-on: ubuntu-latest
env:
# Headless offscreen GL backend so the pyrender model-preview tests run
# instead of skipping on a missing GL context.
PYOPENGL_PLATFORM: osmesa
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install system deps (ImageMagick + offscreen GL)
run: |
sudo apt-get update
sudo apt-get install -y imagemagick libosmesa6 libgl1

- name: Install
run: |
python -m pip install --upgrade pip setuptools wheel
pip install '.[test,preview]'

- name: Run tests
run: python -m pytest tests/

- name: Smoke test
run: preppy -h

# Full end-to-end run over the sample object, exercising the real toolchain.
# Heavy and environment-sensitive, so it is opt-in (manual dispatch) and
# non-blocking (continue-on-error); whatever tools install successfully are
# exercised, the rest self-skip. See README / singularity/preppy.def for the
# authoritative install.
integration:
name: integration (full toolchain)
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v4

- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"

- name: Install external toolchain
run: |
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
sudo apt-get update
sudo apt-get install -y imagemagick nodejs
npm install -g gltfpack
# KTX-Software (`ktx create`) — install per README for full KTX2 tests;
# absent, the KTX2/embed integration tests skip cleanly.

- name: Install package + embed helper deps
run: |
pip install '.[test,validate,preview]'
npm install --prefix preppy/node

- name: Run tests
run: python -m pytest tests/
52 changes: 0 additions & 52 deletions .gitlab-ci.yml

This file was deleted.

18 changes: 9 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

DRI Voyager Preppy turns source OBJs + textures into web-ready 3D assets for the custom `<dri-viewer>` web component. Each imaged object has several **variants** (RGB, IR, PGS, …); the pipeline emits **one self-contained `.glb` per variant** (meshopt-compressed geometry with its KTX2 texture(s) embedded) plus a viewer-native per-object **`manifest.json`** and an optional top-level `index.json`.

This replaced the original Smithsonian-Voyager path (`.svx.json` scene descriptors + `items.json`, via `obj2gltf` + `gltf-pipeline`). That legacy path survives only as the deprecated `voyager-obj2glb` tool (`convert.py`).
This replaced the original Smithsonian-Voyager path (`.svx.json` scene descriptors + `items.json`, via `obj2gltf` + `gltf-pipeline`). That legacy path survives only as the deprecated `preppy-obj2glb` tool (`convert.py`).

## External dependencies (not pip-installable)

The pipeline shells out to CLI tools that must be on `PATH` (see README for install; `tools.py` detects them, `voyager-check-tools` reports status). npm-installed CLIs are invoked as `<name>.cmd` on Windows (`platform.system()` check in `tools.py`).
The pipeline shells out to CLI tools that must be on `PATH` (see README for install; `tools.py` detects them, `preppy-check-tools` reports status). npm-installed CLIs are invoked as `<name>.cmd` on Windows (`platform.system()` check in `tools.py`).

- **ImageMagick** (`magick`/`mogrify`) — normalize textures to 8-bit sRGB; crop thumbnails.
- **`ktx`** (KTX-Software **≥ v5**, `ktx create` — `toktx` was removed in v5) — KTX2/Basis encoding.
- **`gltfpack`** (meshoptimizer) — OBJ → decimated, meshopt-compressed geometry glb.
- **`node`** (20+) + the bundled `@gltf-transform/core` helper (`preppy/node/embed.mjs`) — embeds KTX2 into the geometry glb. Install its deps once: `npm install --prefix preppy/node`.
- Legacy only: `obj2gltf` + `gltf-pipeline` (for `voyager-obj2glb`).
- Legacy only: `obj2gltf` + `gltf-pipeline` (for `preppy-obj2glb`).
- Optional: `pymeshlab` (`.[validate]`) for the Hausdorff decimation gate.
- Optional: `trimesh` + `pyrender` (`.[preview]`) for the rendered model-preview thumbnail (`preview.py`). Needs an offscreen GL backend; when absent the thumbnail falls back to a texture center-crop.

Expand All @@ -28,13 +28,13 @@ Python deps (`natsort`, `Pillow`, `numpy`, `scipy`, `tqdm`) install via `pip ins
pip install -e '.[validate,test]' # editable install + optional pymeshlab/pytest
npm install --prefix preppy/node # KTX2 embed helper deps (once)

voyager-preppy -i config.json -o out/ # batch: variants → self-contained glbs + manifest.json + index.json
voyager-check-tools # report external toolchain status
voyager-obj2glb -i mesh.obj -o mesh.glb # LEGACY single OBJ → Draco GLB (deprecated)
voyager-merge-items a.json b.json -o merged.json # LEGACY items.json merge (deprecated)
preppy -i config.json -o out/ # batch: variants → self-contained glbs + manifest.json + index.json
preppy-check-tools # report external toolchain status
preppy-obj2glb -i mesh.obj -o mesh.glb # LEGACY single OBJ → Draco GLB (deprecated)
preppy-merge-items a.json b.json -o merged.json # LEGACY items.json merge (deprecated)
```

There **is** a test suite now (`tests/`, pytest): `python -m pytest tests/`. Tests that need the external tools (or pymeshlab) skip cleanly when they're absent, so a bare run still covers the pure logic. CI (`.gitlab-ci.yml`) runs the suite across Python 3.11–3.13 plus the `voyager-preppy -h` smoke test, with a manual, non-blocking `integration` job that exercises the full external toolchain. The `singularity/dri-voyager-preppy.def` bundles all deps for reproducible/HPC runs.
There **is** a test suite now (`tests/`, pytest): `python -m pytest tests/`. Tests that need the external tools (or pymeshlab) skip cleanly when they're absent, so a bare run still covers the pure logic. CI (`.github/workflows/ci.yml`, GitHub Actions) runs the suite across Python 3.11–3.13 plus the `preppy -h` smoke test, with a manual, non-blocking `integration` job that exercises the full external toolchain. A second workflow (`.github/workflows/build_docker.yml`) builds and publishes the multi-arch Docker image to `ghcr.io/educelab/preppy`. The `Dockerfile` (and `singularity/preppy.def`) bundle all deps for reproducible/container/HPC runs.

## Architecture

Expand All @@ -52,7 +52,7 @@ Console entrypoints in `preppy/apps/` are thin argparse CLIs over the library mo

### Input config format

The `voyager-preppy` input JSON is a flat array of **objects**, validated by `templates/prep-models.schema.json` (examples: `prep-models-example.json`, `mvs-example.json`):
The `preppy` input JSON is a flat array of **objects**, validated by `templates/prep-models.schema.json` (examples: `prep-models-example.json`, `mvs-example.json`):

- An **object** needs `id`, `title`, and a `variants` array. Optional `prefix` (output folder/file prefix; defaults to `id`), `titles`, `inventory`, `description`, `credit`, `date`, `units` (default `cm`), `nodataFill`.
- A **variant** needs `suffix` (stable key: names the file + is the manifest variant `id`) and `obj`. Optional `label`, `default`, `nodataFill` (resolved variant ?? object ?? CLI), and per-variant `credit`/`date`/`method`/`description`. Textures are resolved transitively from the OBJ's `map_Kd`. Relative `obj` paths resolve against `--data-root` (default CWD), not the config file's location.
Expand Down
88 changes: 88 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# syntax=docker/dockerfile:1
#
# Preppy delivery-pipeline image. Bundles the full external toolchain
# (ImageMagick, KTX-Software, gltfpack, Node) + the Python package with the
# `preview` extra so the default thumbnail is a rendered model preview.
#
# Derived from singularity/preppy.def. Multi-arch (linux/amd64, linux/arm64):
# the KTX-Software .deb is selected per-arch via the buildx-provided TARGETARCH.
ARG BASE_IMAGE=python:3.11-slim
FROM ${BASE_IMAGE}

LABEL org.opencontainers.image.title="Preppy"
LABEL org.opencontainers.image.description="Mesh preparation for DRI Voyager: source OBJs + textures -> web-ready self-contained glb per variant + manifest."
LABEL org.opencontainers.image.authors="Seth Parker <c.seth.parker@uky.edu>"
LABEL org.opencontainers.image.source="https://github.com/educelab/preppy"
LABEL org.opencontainers.image.licenses="GPL-3.0-or-later"

# TARGETARCH is injected by Docker buildx (amd64 | arm64).
ARG TARGETARCH

ENV DEBIAN_FRONTEND=noninteractive \
# Headless offscreen GL backend for the pyrender model-preview thumbnail.
# (libosmesa6 is installed below; if GL fails on a host the thumbnail falls
# back to a texture crop.)
PYOPENGL_PLATFORM=osmesa \
PIP_NO_CACHE_DIR=1

# System dependencies:
# imagemagick -> texture normalization (magick/mogrify)
# libosmesa6, libgl1 -> offscreen GL for the rendered model-preview thumbnail
# curl/git/gcc/g++/make -> fetch installers + build any sdist-only wheels
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
curl \
gcc \
g++ \
git \
make \
tzdata \
imagemagick \
libosmesa6 \
libgl1 \
&& rm -rf /var/lib/apt/lists/*

# Node.js (24 LTS) for gltfpack and the KTX2 embed helper.
RUN curl -fsSL https://deb.nodesource.com/setup_24.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/* \
&& node -v

# KTX-Software (provides `ktx create`; `toktx` is not used). The .def targeted a
# v5 that is not yet released as a final tag, so pin the latest stable that ships
# `ktx create` for both amd64 and arm64. Bump when v5.0.0 ships.
# Assets: https://github.com/KhronosGroup/KTX-Software/releases
ARG KTX_VERSION=4.4.2
RUN set -eux; \
case "${TARGETARCH}" in \
amd64) ktx_arch="x86_64" ;; \
arm64) ktx_arch="arm64" ;; \
*) echo "unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
esac; \
curl -fsSL -o /tmp/ktx.deb \
"https://github.com/KhronosGroup/KTX-Software/releases/download/v${KTX_VERSION}/KTX-Software-${KTX_VERSION}-Linux-${ktx_arch}.deb"; \
apt-get update; \
apt-get install -y /tmp/ktx.deb; \
rm -f /tmp/ktx.deb; \
rm -rf /var/lib/apt/lists/*; \
ktx --version

# Node CLI tools: gltfpack (geometry). obj2gltf/gltf-pipeline are the deprecated
# legacy path, kept until it is removed.
RUN npm install -g gltfpack obj2gltf gltf-pipeline

WORKDIR /usr/local/educelab/preppy
COPY . .

# Install the embed helper's npm deps (@gltf-transform/core + meshoptimizer) and
# the Python package with the `preview` extra (trimesh + pyrender). The install
# is editable so the embed helper resolves node_modules relative to the package
# source (assemble.NODE_DIR) — the same reason the .def uses --editable.
RUN npm install --prefix preppy/node \
&& python3 -m pip install --upgrade pip wheel setuptools \
&& python3 -m pip install --editable '.[preview]'

# No restrictive ENTRYPOINT: any console script (preppy, preppy-check-tools,
# preppy-obj2glb, preppy-merge-items) can be used as the command. Bare
# `docker run <image>` prints the pipeline's help.
CMD ["preppy", "-h"]
Loading
Loading