Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 46 additions & 19 deletions claude-notes/plans/2026-06-24-fma-torso-bodyparts3d-splat.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,22 +62,49 @@ Notes:
brush blobbed the detail into a "Warhol" look; 0.0025 at 810x1080 restores the
ribcage/vertebrae). Frames re-rendered.

## Follow-ups (proposed, next PR)

The splat is currently isotropic spheres (no orientation) — too big = blobs,
too small = disconnected dots. The real upgrade, in one pass over the meshes:

- [ ] **Anisotropic surface-fit gaussians** ("connect the dots"): read OBJ
*faces* (the bake currently drops them) -> per-vertex normals -> orient
each gaussian flat-to-surface (`scale[3]` tangent-wide / normal-thin,
`quat` from normal). splat3d's `Gaussian3D` already supports scale+quat;
the three.js page needs a real splat shader (oriented quads). This is the
"muscle memory of the nodes" — each gaussian inherits its shape from the
structure it came from. NOT voxels (those are discrete/volumetric; these
are continuous surfaces).
- [ ] **Third "map FMA" view**: bake a per-gaussian FMA structure id + legend
(idx -> FMA concept / name / colour) into SPL1, then pick-to-label in 3D
and sync selection with the /fma-style partonomy graph. Realises the
osint-cad-splat thesis: graph and splat are one node at one address, two
payloads. Own page (/torso-map) vs folding labels into /torso-live: TBD
with user.
## Follow-up PR — anisotropic + GUID-tag + map (branch claude/torso-anisotropic-map)

- [x] **SPL2 format** (supersedes SPL1): hdr 40B [`SPL2`|count|node_count|radius|
bbox]; body 21B [pos 3f | normal 3i8 | rgb 3u8 | opacity u8 | node_row u16].
Helix-orderable + residual-ready (the codec PR slots in here).
- [x] **Anisotropic surface-fit gaussians** ("connect the dots"): bake reads OBJ
`vn` (BodyParts3D ships normals — free, no face traversal); render driver
orients each gaussian flat-to-surface (`scale=[t,t,thin]`, `quat` aligns
local-z to the normal). Tangent 0.004 connects within a structure while
rib gaps stay visible. NOT voxels (continuous surfaces, not a discrete grid).
- [x] **Per-node SoA + O(1) switch** (the GUID/value-tenant backbone): bake emits
`torso.nodes.json` — one row per FMA structure (178 rows, 91 own meshes):
fma id, name, depth, HHTL tier-ranks, colour, gaussian RANGE (start+count),
OBJ-geometry tenant (centroid + bbox + FJ handles). Each gaussian carries
its node_row. Consumers build the switch (row -> node) once -> O(1) tenant
reads. Position = real BodyParts3D coordinate; identity = the FMA node.
- [x] **/torso-map page**: click a gaussian -> node_row -> node SoA -> FMA label +
partonomy breadcrumb; structure list (graph -> splat) highlights gaussians.
Realises the osint-cad-splat thesis: graph and splat, one node at one address.
- [x] tsc clean. Browser pick-interaction not exercised here (raycast-on-Points
logic is standard; geometry verified via the CPU frames).

## Helix-anchor codec — MEASURED (branch claude/torso-helix-codec)

`tools/spl_codec.py` encodes SPL2 -> SPL3 and round-trips it. The x265-for-
gaussians design, mapped to signals already in SPL2 + the node SoA:
helix = 3D Morton (Z-order) of position = identity/GUID order (locality-preserving)
anchor = FMA node (SoA centroid + per-node colour) = the I-frame, random-access
motion = gaussian offset from its node anchor (the motion vector)
residual = helix-ordered zig-zag delta of (motion, normal)
colour = ANCHOR-PREDICTED -> 0 per-gaussian bytes (a 178-entry node palette)

Measured on the real torso (231,515 gaussians):
- SPL2 21.0 B/g -> SPL3 7.47 B/g => **2.8x smaller** (zlib entropy stand-in)
- colour: **exact, 887 B total** for ALL colour (crisp by construction, no bleed)
- position round-trip RMSE **0.00001** (16-bit quant, effectively lossless)
- node_row RLE 35 KB / 231K gaussians (structures contiguous in helix order)
- stream split: motion 1.02 MB, normal 671 KB (the optimization target -> octahedral
+ range coder), rows 35 KB, palette 887 B

Validates the design before wiring it into the render. Next increments:
- [ ] octahedral normals + range coder (the 671 KB normal stream)
- [ ] decode SPL3 at cockpit load; anisotropic/edge-aware reconstruction
(node_row-bounded + normal-oriented = crisp colours in the render)
- [ ] animation: deform node anchors -> motion-skinned gaussians follow
(Motion-Blender GS; the partonomy is the rig)
Binary file modified cockpit/public/torso-frames/torso_000.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_001.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_002.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_003.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_004.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_005.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_006.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_007.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_008.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_009.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_010.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_011.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_012.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_013.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_014.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_015.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_016.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_017.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_018.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified cockpit/public/torso-frames/torso_019.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
43 changes: 12 additions & 31 deletions cockpit/public/torso.manifest.json
Original file line number Diff line number Diff line change
@@ -1,42 +1,23 @@
{
"source": "BodyParts3D 4.0 (DBCLS) part-of OBJ, decimated 99%",
"license": "CC-BY 4.0",
"attribution": "BodyParts3D, (c) The Database Center for Life Science licensed under CC Attribution 4.0 International",
"source": "BodyParts3D 4.0 (DBCLS) part-of OBJ, decimated 99%, with vn normals",
"license": "CC-BY 4.0 (site) / CC-BY-SA 2.1 JP (2013 files)",
"attribution": "BodyParts3D, (c) The Database Center for Life Science. Current site licence: CC-BY 4.0; 2013 mesh files embed CC-BY-SA 2.1 Japan.",
"format": "SPL2 (anisotropic + node-row tags); node SoA in torso.nodes.json",
"root_fma": "FMA7181",
"root_name": "trunk",
"concepts": 102,
"concepts": 178,
"meshes": 577,
"vertices_total": 693959,
"gaussians": 231320,
"radius": 0.0045,
"gaussians": 231515,
"radius": 0.0035,
"bbox_min": [
-0.5499420475739409,
-0.2690260109315338,
-0.5499102767007714,
-0.269514341019139,
-0.9999999999999999
],
"bbox_max": [
0.5499420475739409,
0.2690260109315338,
0.5499102767007714,
0.269514341019139,
1.0000000000000002
],
"regions": [
{
"fma": "FMA259209",
"name": "thoracic segment of trunk"
},
{
"fma": "FMA259211",
"name": "abdominal segment of trunk"
},
{
"fma": "FMA10427",
"name": "body wall"
},
{
"fma": "FMA9579",
"name": "perineum"
}
],
"generated_by": "crates/osint-bake/tools/bake_torso_splat.py",
"format": "SPL1: hdr[magic4|count u32|radius f32|bbox_min 3f|bbox_max 3f]; body count*[pos 3f|rgb 3u8|opacity u8]; little-endian"
"owners": 91
}
1 change: 1 addition & 0 deletions cockpit/public/torso.nodes.json

Large diffs are not rendered by default.

Binary file modified cockpit/public/torso.splat
Binary file not shown.
272 changes: 272 additions & 0 deletions cockpit/src/TorsoMap.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
// FMA torso · map view — the splat AS the GUID substrate.
//
// Renders the same torso.splat (SPL2: real BodyParts3D geometry, per-vertex
// normal, per-gaussian node-row tag) but treats it as the value-tenant SoA it
// is: click a gaussian -> its node row -> O(1) into torso.nodes.json (the node
// SoA) -> the FMA structure's identity (concept id, name, partonomy path,
// gaussian range). Click a structure in the list -> its gaussians highlight.
// Geometry, graph, and splat are three tenants of one identity, switch-selected.
//
// Geometry: BodyParts3D, (c) The Database Center for Life Science (CC-BY 4.0 /
// CC-BY-SA 2.1 JP). Attribution shown in-view.
import { useEffect, useMemo, useRef, useState } from 'react';
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

interface NodeRow {
row: number;
fma: string;
name: string;
depth: number;
parent: number | null;
tiers: number[];
rgb: [number, number, number];
g_start: number;
g_count: number;
centroid: [number, number, number] | null;
}
interface NodesDoc { attribution: string; root: string; nodes: NodeRow[] }

interface Spl2 {
count: number;
positions: Float32Array;
colors: Float32Array; // 0..1 rgb
rows: Float32Array; // node_row per gaussian
}

function decodeSpl2(buf: ArrayBuffer): Spl2 {
const dv = new DataView(buf);
const magic = String.fromCharCode(dv.getUint8(0), dv.getUint8(1), dv.getUint8(2), dv.getUint8(3));
if (magic !== 'SPL2') throw new Error(`bad magic "${magic}" (expected SPL2)`);
const count = dv.getUint32(4, true);
const off = 40;
const positions = new Float32Array(count * 3);
const colors = new Float32Array(count * 3);
const rows = new Float32Array(count);
for (let i = 0; i < count; i++) {
const b = off + i * 21;
// upright: (x,y,z) -> (x,z,-y)
positions[i * 3] = dv.getFloat32(b, true);
positions[i * 3 + 1] = dv.getFloat32(b + 8, true);
positions[i * 3 + 2] = -dv.getFloat32(b + 4, true);
colors[i * 3] = dv.getUint8(b + 15) / 255;
colors[i * 3 + 1] = dv.getUint8(b + 16) / 255;
colors[i * 3 + 2] = dv.getUint8(b + 17) / 255;
rows[i] = dv.getUint16(b + 19, true);
}
return { count, positions, colors, rows };
}

const VERT = `
attribute vec3 aColor;
attribute float aRow;
uniform float uSelected;
uniform float uSize;
varying vec3 vColor;
varying float vDim;
void main() {
vColor = aColor;
vDim = (uSelected < 0.0 || abs(aRow - uSelected) < 0.5) ? 1.0 : 0.16;
vec4 mv = modelViewMatrix * vec4(position, 1.0);
gl_Position = projectionMatrix * mv;
gl_PointSize = uSize * (1.0 / -mv.z);
}`;
const FRAG = `
precision mediump float;
varying vec3 vColor;
varying float vDim;
void main() {
vec2 c = gl_PointCoord - 0.5;
if (dot(c, c) > 0.25) discard;
float a = smoothstep(0.25, 0.05, dot(c, c));
gl_FragColor = vec4(vColor * vDim, a);
}`;

function mount(
container: HTMLDivElement,
splat: Spl2,
onPick: (row: number) => void,
apiRef: { current: { select: (row: number) => void } | null },
): () => void {
let w = container.clientWidth || window.innerWidth;
let h = container.clientHeight || window.innerHeight;
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x0a0e17);
const camera = new THREE.PerspectiveCamera(45, w / h, 0.01, 100);
camera.position.set(0, 0.05, 3.0);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(w, h);
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
container.appendChild(renderer.domElement);

const geom = new THREE.BufferGeometry();
geom.setAttribute('position', new THREE.BufferAttribute(splat.positions, 3));
geom.setAttribute('aColor', new THREE.BufferAttribute(splat.colors, 3));
geom.setAttribute('aRow', new THREE.BufferAttribute(splat.rows, 1));
const mat = new THREE.ShaderMaterial({
vertexShader: VERT,
fragmentShader: FRAG,
uniforms: { uSelected: { value: -1 }, uSize: { value: 4.2 } },
transparent: true,
depthWrite: true,
});
const points = new THREE.Points(geom, mat);
scene.add(points);

const controls = new OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
controls.dampingFactor = 0.08;
controls.target.set(0, 0, 0);
controls.minDistance = 0.6;
controls.maxDistance = 12;

const ray = new THREE.Raycaster();
ray.params.Points = { threshold: 0.02 };
const ndc = new THREE.Vector2();
let downAt = { x: 0, y: 0 };
renderer.domElement.addEventListener('pointerdown', (e) => { downAt = { x: e.clientX, y: e.clientY }; });
renderer.domElement.addEventListener('pointerup', (e) => {
if (Math.hypot(e.clientX - downAt.x, e.clientY - downAt.y) > 5) return; // drag, not click
const r = renderer.domElement.getBoundingClientRect();
ndc.x = ((e.clientX - r.left) / r.width) * 2 - 1;
ndc.y = -((e.clientY - r.top) / r.height) * 2 + 1;
ray.setFromCamera(ndc, camera);
const hit = ray.intersectObject(points, false);
if (hit.length && hit[0].index != null) onPick(splat.rows[hit[0].index]);
});

apiRef.current = { select: (row: number) => { mat.uniforms.uSelected.value = row; } };

let raf = 0;
const tick = () => { raf = requestAnimationFrame(tick); controls.update(); renderer.render(scene, camera); };
tick();
const onResize = () => {
w = container.clientWidth || window.innerWidth; h = container.clientHeight || window.innerHeight;
camera.aspect = w / h; camera.updateProjectionMatrix(); renderer.setSize(w, h);
};
const ro = new ResizeObserver(onResize); ro.observe(container);
return () => {
cancelAnimationFrame(raf); ro.disconnect(); controls.dispose();
geom.dispose(); mat.dispose(); renderer.dispose();
if (renderer.domElement.parentNode === container) container.removeChild(renderer.domElement);
};
}

export function TorsoMap() {
const ref = useRef<HTMLDivElement>(null);
const apiRef = useRef<{ select: (row: number) => void } | null>(null);
const [splat, setSplat] = useState<Spl2 | null>(null);
const [doc, setDoc] = useState<NodesDoc | null>(null);
const [sel, setSel] = useState<number | null>(null);
const [error, setError] = useState<string | null>(null);

useEffect(() => {
let cancelled = false;
fetch('/torso.splat')
.then((r) => { if (!r.ok) throw new Error(`HTTP ${r.status} torso.splat`); return r.arrayBuffer(); })
.then((b) => !cancelled && setSplat(decodeSpl2(b)))
.catch((e) => !cancelled && setError(String(e)));
fetch('/torso.nodes.json')
.then((r) => (r.ok ? r.json() : null))
.then((d) => !cancelled && d && setDoc(d as NodesDoc))
.catch(() => {});
return () => { cancelled = true; };
}, []);

useEffect(() => {
const c = ref.current;
if (!c || !splat) return;
return mount(c, splat, (row) => setSel(row), apiRef);
}, [splat]);

useEffect(() => { apiRef.current?.select(sel ?? -1); }, [sel]);

// the "switch": row -> node, built once (O(1) lookup)
const byRow = useMemo(() => {
const m = new Map<number, NodeRow>();
doc?.nodes.forEach((n) => m.set(n.row, n));
return m;
}, [doc]);

// partonomy breadcrumb (walk parent links) for the selected node
const path = useMemo(() => {
if (sel == null) return [];
const out: NodeRow[] = [];
let cur: number | null = sel;
for (let i = 0; i < 24 && cur != null; i++) {
const n = byRow.get(cur);
if (!n) break;
out.push(n);
cur = n.parent;
}
return out.reverse();
}, [sel, byRow]);

const owners = useMemo(
() => (doc?.nodes ?? []).filter((n) => n.g_count > 0).sort((a, b) => b.g_count - a.g_count),
[doc],
);

const selNode = sel != null ? byRow.get(sel) : undefined;
const css = (rgb: [number, number, number]) => `rgb(${rgb[0]},${rgb[1]},${rgb[2]})`;

return (
<div style={{ position: 'fixed', inset: 0, background: '#0a0e17', overflow: 'hidden', color: '#cdd9e5', font: '12px ui-monospace, monospace' }}>
<div ref={ref} style={{ position: 'absolute', inset: 0 }} />

<div style={{ position: 'absolute', top: 12, left: 16, pointerEvents: 'none' }}>
<div style={{ fontSize: 15, color: '#fff' }}>FMA torso · map</div>
<div style={{ opacity: 0.7 }}>
click a structure — splat ↔ FMA, one node at one address
{doc ? ` · ${owners.length} structures` : ''}
</div>
</div>

{error && (
<div style={{ position: 'absolute', top: '46%', width: '100%', textAlign: 'center', color: '#ff8095' }}>{error}</div>
)}

{/* selected structure + partonomy path */}
{selNode && (
<div style={{ position: 'absolute', top: 12, right: 16, width: 290, background: '#0e1422cc', border: '1px solid #20304a', borderRadius: 6, padding: '10px 12px' }}>
<div style={{ display: 'flex', alignItems: 'center', gap: 8 }}>
<span style={{ width: 11, height: 11, borderRadius: 3, background: css(selNode.rgb) }} />
<span style={{ color: '#fff', fontSize: 14 }}>{selNode.name}</span>
</div>
<div style={{ opacity: 0.7, marginTop: 4 }}>
{selNode.fma} · {selNode.g_count.toLocaleString()} gaussians · depth {selNode.depth}
</div>
<div style={{ opacity: 0.55, marginTop: 6, fontSize: 11 }}>HHTL tiers [{selNode.tiers.join(', ')}]</div>
<div style={{ marginTop: 8, borderTop: '1px solid #1b2740', paddingTop: 6, opacity: 0.85 }}>
{path.map((n, i) => (
<div key={n.row} style={{ paddingLeft: i * 8, color: n.row === sel ? '#fff' : '#8aa0bb', cursor: 'pointer' }} onClick={() => setSel(n.row)}>
{i > 0 ? '└ ' : ''}{n.name}
</div>
))}
</div>
</div>
)}

{/* structure list (graph -> splat) */}
<div style={{ position: 'absolute', bottom: 30, left: 16, maxHeight: '42%', width: 250, overflowY: 'auto', background: '#0e1422aa', border: '1px solid #1b2740', borderRadius: 6, padding: 6 }}>
{owners.map((n) => (
<div
key={n.row}
onClick={() => setSel(n.row)}
style={{ display: 'flex', alignItems: 'center', gap: 7, padding: '2px 4px', borderRadius: 3, cursor: 'pointer', background: n.row === sel ? '#1d2b40' : 'transparent', color: n.row === sel ? '#fff' : '#9fb2c9' }}
>
<span style={{ width: 9, height: 9, borderRadius: 2, background: css(n.rgb), flex: '0 0 auto' }} />
<span style={{ overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>{n.name}</span>
</div>
))}
</div>

<div style={{ position: 'absolute', top: 14, right: 320, font: '12px ui-monospace, monospace' }}>
<a href="/torso-live" style={{ color: '#7fa6c4', textDecoration: 'none' }}>live orbit →</a>
</div>
<div style={{ position: 'absolute', bottom: 8, left: 16, color: '#5a6b7e', fontSize: 10, pointerEvents: 'none' }}>
{doc?.attribution ?? 'BodyParts3D, (c) The Database Center for Life Science (CC-BY 4.0 / CC-BY-SA 2.1 JP)'}
</div>
</div>
);
}
Loading