Skip to content

Port the translators to the moveapps R package - #14

Merged
cleemansen merged 21 commits into
mainfrom
feature/port-to-moveapps-package
Oct 7, 2026
Merged

cleemansen merged 21 commits into
mainfrom
feature/port-to-moveapps-package

Conversation

@cleemansen

@cleemansen cleemansen commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Trello 1402 — port both translator Apps to the current R co-pilot, which ships the SDK as the moveapps R package.
Design and measurements: docs/superpowers/specs/2026-09-30-port-to-moveapps-package-design.md; task plan next to it.

Why

Since R co-pilot v4 the SDK is a package and the image has no src/, so the App Dockerfile broke at mv ../src.
Nothing but the platform ever built that Dockerfile, which is how the break stayed invisible.

What

  • Port. rds_2_csv.R / csv_2_rds.R call moveapps (still no move1). The conversion moved, unchanged, into
    testable functions in r/link.R. r/src/ is gone. r/renv.lock is re-snapshotted for R 4.6.1 / renv 1.2.4 /
    moveapps 1.0.3 — seven direct packages, no testthat.
  • One Dockerfile, both Apps. FROM the current R co-pilot; conda is copied from the python co-pilot's image
    (/opt/conda checksummed identical to condaforge/miniforge3:26.7.2-0) instead of a Miniforge installer fetched
    from "latest". ARG DIRECTION has no default: a forgotten edit fails the build instead of building the other App.
  • Python lock. The App and the python tests install python/conda-linux-64.lock (explicit, 161 packages, plain
    conda) instead of resolving environment.yml at build time, so an App version gets exactly what CI tested.
    python/lock.sh resolves it from environment.yml, which stays open; refresh it before a release. The weekly run
    tests a fresh resolution and warns when the lock is behind; CI fails while the lock was not made from the current
    environment.yml.
  • Tests.
    • python: 10 → 17 tests, including a contract test against golden CSV pairs.
    • R: 0 → 56 testthat expectations (unit tests per function, the same goldens).
    • CI job app: builds the real Dockerfile for both directions on a public stand-in for the private co-pilot
      (docker buildx bake), runs coherence checks of the version pins, the R tests and a round trip
      R → python → R of 12 cases inside the images, and requires a build without a valid DIRECTION to fail at the check.
    • The goldens and round-trip expectations were produced by the v2.2.0 production images; every later commit
      proves itself by reproducing them.
  • Defects found by the new tests, fixed one commit each (each commit's diff of the pins is its evidence):
    1. An empty TrajectoryCollection failed with "No objects to concatenate" — it now becomes the NULL output the README
      promises.
    2. R wrote wall-clock times and python re-localised them: two fixes in the repeated autumn hour collided
      (AmbiguousTimeError). link.csv now carries UTC instants; UTC data — all Movebank data — stay byte-identical.
    3. R parsed python's times with %S and dropped fractions: up to 0.999 s off. Now %OS.
    4. R's %OS3 truncates: a time stored just below its millisecond lost it on the way to python (10 of 75
      sub-second fixes in the stork sample, 1 ms early). Rounded now; no golden changes.
    5. movingpandas drops single-fix tracks silently. The App log now names every dropped track.
    6. python's True/False came back as character; logical columns now come back as logical.

Not obvious from the diff

  • The start script is picked in RUN, not by COPY ${DIRECTION}.sh. BuildKit resolves a COPY source before the
    preceding RUN check executes, so a build without DIRECTION failed on "/.sh": not found — and a grep for the
    check's message still matched, because the log quotes the command. The CI grep is anchored to the message at the
    line end; both positive builds give 0 matches, both negative ones 2.
  • ~/.cache/conda is created as root. The co-pilot's install -d … $HOME/.cache/R leaves ~/.cache owned by
    root, and conda env create fails on its notices cache. The stand-in reproduces the co-pilot, so CI sees the same.
  • Lock repositories are CRAN. The snapshot was taken with P3M binaries (minutes instead of hours, emulated);
    renv then recorded the P3M URL per package. Normalised to CRAN; the co-pilot's repo override decides at restore.
  • R time assertions compare absolute differences. expect_equal's tolerance is relative: at 1.4e9 s its
    default forgives ~20 s, so a pin of "v2.2.0 truncates sub-seconds" passed with and without the truncation.

Proofs (local, Apple Silicon, amd64 emulated)

  • Refactoring: this branch's R files inside the v2.2.0 production images reproduce every golden and pin (no diff).
  • Port and every fix: both Apps built on the real co-pilots reproduce the pins the public stand-in measures
    (no diff) — CI and production agree.
  • Downstream: every new output.rds (11 cases) is read by the v2.2.0 R → python App, exit 0 — R Apps that were not
    rebuilt keep working.
  • DIRECTION: missing and invalid are refused on the real co-pilot, with the message.
  • Mutations: a changed pin, a changed Miniforge tag, a changed R expectation and an environment.yml edit without a
    new lock each fail the bake suite; a stale lock shows up in the drift report.

After merge

Tag v3.0.0 - a major release: downstream Apps see logical columns as logical instead of character, times
with their fractions of a second, and NULL instead of a failure for an empty TrajectoryCollection. The python lock
is from 2026-10-01; refresh it first if the weekly run reports it behind. Then one App version per translator. Its Dockerfile is this Dockerfile with exactly one line changed:

  • move2_loc to MovingPandas: ARG DIRECTION → ARG DIRECTION=r2python
  • MovingPandas to move2_loc: ARG DIRECTION → ARG DIRECTION=python2r

The app job's first cold run took 8m36s; its timeout-minutes is 20.

cleemansen and others added 21 commits September 30, 2026 10:04
The translators are pinned to the old R co-pilot: their R side sources
the SDK's src/ files, which the moveapps package era removed, so a
build on the current co-pilot fails. The spec also covers what scoping
turned up: an untested R side, an App Dockerfile nothing but the
platform ever builds, and a conda that differs from every python App.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Running every input through the production images replaced three
suspected defects with five confirmed ones: an empty collection fails
the App, DST ambiguity fails it, sub-second times are truncated,
single-fix tracks vanish, and logical columns come back as text. The
suspected tz-aware index is dead code, since movingpandas drops the
zone itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No App version on the platform has ever pulled a base from outside the
couchbits GitLab registry; kaniko holds no Docker Hub credentials and
no mirror. The python co-pilot carries the same /opt/conda, untouched,
in the registry the platform already reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourteen tasks: pin v2.2.0 from the production images, refactor and
port against those pins, add the CI job, fix the five measured defects
one commit each. Push, pull request and App versions wait for an
explicit go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The eight move2_loc inputs came from an older cut of the Template's
data: same observations, but more event columns and another track id
column than Movebank delivers today. Taken byte for byte from
Template_R_Function_App e00af0d. The List file stays, it is the only
input with list columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2.2.0 production images, pulled from the platform registry,
produced every golden and every expectation here - so what later
steps must reproduce is literally what runs in production. The
scripts that did it are committed alongside and are reused to prove
the refactoring, the port and each fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The goldens from v2.2.0 now bind python: R's CSV pair goes in, and
what python hands back must match production byte for byte. Adds the
first tests of the meta.csv and link.csv python writes, and of reading
meta.csv and building the collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both entry scripts were a single top-level tryCatch, so nothing on the
R side could be tested. link.R holds the conversion as functions, the
entry scripts keep only the SDK glue. Running this checkout's R files
inside the v2.2.0 production images reproduces every golden and every
round-trip expectation, so the move changes nothing. Includes the
first R tests: unit tests per function and the contract goldens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since R co-pilot v4 the SDK is the moveapps package and the image has
no src/, so the old build failed at mv ../src. The entry scripts now
call the package; its readRdsInput needs no move1, so the patched copy
goes too. The lock moves to R 4.6.1.

conda comes from the python co-pilot's image instead of a Miniforge
installer fetched from "latest", so the python half resolves like every
python App. One Dockerfile builds both Apps: DIRECTION has no default,
so a forgotten edit fails the build instead of building the other App.
The start script is picked in RUN: BuildKit resolved a COPY of
${DIRECTION}.sh before the check ran and failed on "/.sh: not found".

Built on the real co-pilots, both Apps reproduce every v2.2.0 golden
and round-trip expectation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The App Dockerfile was built by nothing but the platform, which is how
the mv ../src break stayed invisible. CI now builds the real Dockerfile
for both directions, on a public stand-in for the private co-pilot, and
runs the R unit tests, the round trip and a coherence check of the
version pins inside the images. A build without a valid DIRECTION must
fail, at the check: the grep matches the message the check prints, not
the command the log quotes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README promises a NULL object, but the python half failed with
"No objects to concatenate", and the R half would not have stored an
output for zero rows either. Python now writes empty files, and R
stores the NULL result: the empty file the next R App rejects with
code 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R wrote wall-clock times and python re-localised them, so two fixes an
hour apart in the repeated autumn hour became the same local time and
the App failed with AmbiguousTimeError. link.csv now carries UTC;
meta.csv still names the zone and python converts. UTC data - all real
Movebank data - stay byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python writes fractions of a second, R parsed them with %S and dropped
them: 75 fixes of the stork sample came back up to 0.999 s early. The
1 ms left is R's own %OS3, which truncates on the way to python: 10 of
the 75 are stored just below their millisecond.

The time assertions of the R tests compare absolute differences now:
expect_equal's tolerance is relative, and at 1.4e9 seconds it forgave
20 s - the old pin could not tell truncated times from exact ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A track with a single fix cannot become a trajectory, and movingpandas
drops it without a word: the List sample goes in with 4 tracks and
comes back with 1. The format cannot keep them, so the App log now
names every dropped track and its number of fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python writes booleans as True/False, which read.csv leaves as text, so
every logical column came back as character: 91 in the samples, and 8
list columns of the List sample that R flattens to logical on the way
out. The other class changes of the CSV bridge stay as they are and
stay pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
%OS3 truncates, so a time stored just below its millisecond lost it on
the way to python: 10 of the 75 sub-second fixes of the stork sample
came back 1 ms early, in v2.2.0 as well. Half a millisecond added
before formatting makes the truncation round. No golden changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bake overrides a variable from the environment variable of the same
name, and GitHub's runners export CONDA=/usr/share/miniconda: the App
job passed that path as the conda image and failed on "invalid
reference format". The variable is CONDA_IMAGE now; the Dockerfile's
ARG CONDA stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first cold run took 8m36s without any cache. Twice that, rounded
up, cuts off a hanging build within 20 minutes instead of 90.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The R half was locked, the python half resolved anew at every build:
the platform built an App version from whatever conda-forge offered
that day, never the environment CI had tested. The App and the python
tests now install python/conda-linux-64.lock, an explicit package list
python/lock.sh resolves from environment.yml - plain conda, no extra
tool. environment.yml stays deliberately open; the lock is refreshed
before a release.

The weekly run tests environment.yml resolved fresh, as before, and
warns when the lock is behind it. test/coherence.R fails while the
lock was not made from the current environment.yml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings v2.2.1 (#15, Trello #1419) onto v3: v3.0.0-RC1 locks
movingpandas 0.23.0 and wrote the same pickles without locations.

transform_to_pickle.py merged cleanly; the dropped-track warning
still reads `data`, which keeps `coords_x`/`coords_y`. The test
conflict was two additions side by side. `pytz` stays out - v3
removed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cleemansen
cleemansen requested a review from annescharf October 6, 2026 13:16
@cleemansen
cleemansen merged commit 178b850 into main Oct 7, 2026
2 checks passed
@cleemansen
cleemansen deleted the feature/port-to-moveapps-package branch October 7, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants