Question: best approach to rendering a NIFTI volume in world space

[kmannislands]

Hello, thanks for the excellent library!

I'm setting out to render voxel data from Nifti's in a similar fashion to what you do in your volume3DNavigatorTimeCurve.html example (Axial/Sagittal/Coronal planes rendered in three js). The issue is I'm indexing to other geometries that are already affined to world space.

I can see some handling o NIFTI's ijk to world affine in that it's clearly read from the header in your NIFTI decoder and stashed into metadata, but I can't seem to see the metadata being consumed elsewhere.

I've tried modifying the vertex shader in your example to incorporate another Matrix4 uniform, which I call worldToIjkCentered. I've created this affine by logic like:

function computeWorldToIjkCentered(affine: Matrix4, dims: [number, number, number]): Matrix4 {
    const clone = invertAffine(affine.clone());

    const [i, j, k] = dims;

    const translation = new THREE.Vector3(i, j, k).multiplyScalar(-0.5);

    const affineToIJK = new THREE.Matrix4().makeTranslation(translation.x, translation.y, translation.z);

    const affineToWorld = affineToIJK.multiply(clone);

    return affineToWorld;
}

where invertAffine does what you probably expect and I call the method with a three Matrix4 version of the NIFTI header affine and the ijk dimensions.

I'm applying the new affine in the vertex shader like:

precision highp float;
varying vec2 vUv;
varying vec4 worldCoord;

uniform mat4 worldToIjkCentered;

void main() {
  vUv = uv;
  vec4 mvPosition = modelViewMatrix * vec4(position, 1.0);
  gl_Position = projectionMatrix * mvPosition;

  // this line changed
  vec4 normalizedPosition = worldToIjkCentered * vec4(position, 1.0);
  worldCoord = modelMatrix * vec4(normalizedPosition.x, normalizedPosition.y, normalizedPosition.z, 1.0);
}

Which, when combined with applying the ijkToWorld affine from the NIFTI header to the Plane Geometries works well for the Coronal plane but has some unanticipated results (to me at least) on the other planes. Reading through the vertex shader, I assume this issue is related to some of the assumptions there so my plan is to tackle that next.

I understand that my struggles are more with Three DataTextures and glsl shaders than with Pixpipe, but I figured I would ask: is there a more straight forward way to go about this using PixPipe? Any thoughts or advice for how to go about this?

I'm happy to contribute an example back to pixpipe once I've got this sorted.

Thanks in advance, and congrats on what seems to be your first GitHub issue from a user!