Conversation
We can use GLTFExporter to directly export the mesh (alongside the associated texture if any) to a GLB file. I'm not certain if revokeObjectURL needs to be called from setTimeout; calling it right after click() works too but I'm worried this may result in a race condition, and StackOverflow says setTimeout(0) is safer as it sequences revokeObjectURL after an event generated by click().
We are hitting significant performance issues with retexturing flow due to ray casts and material evaluation. These can be improved in a variety of ways but the only thing that keeps this stage very fast seems to be a full conversion to the GPU. This change implements the initial scaffolding and UV rasterization with a gutter; we inflate each triangle in UV space a bit and rasterize a mesh into the RT. The RT is then copied into a regular DataTexture, as GLB export will not work with RT left as is; this step could be improved in the future.
We use three-mesh-bvh support for BVH access on GPU to directly ray cast or run closest point queries against the original geometry on GPU. As a result, we get face id (which we'll need to index original geometry with to get attributes) and barycentrics. Barycentrics are swizzled in the closest-point implementation, so we need to swizzle them back. For now the shader outputs the debug color based on a triangle id hash.
In retexture.js, we now read original attributes using FloatAttributeTexture that three-mesh-bvh helpfully provides. Initially we will only need UV/colors, but normals will be required in the future so we might as well propagate them. A caveat wrt normals is that we skin vertex positions using getVertexPosition, but the normals don't have corresponding functionality in three.js so they will be left in bind pose. They will however be transformed by the world matrix at least, and remeshing skinned meshes ideally requires reskinning anyway.
To get access to all materials in the source model simultaneously from our shader, we need them available as GPU resources. WebGL (and WebGPU) does not support binding arrays; because of this, to access textures we need to merge them into one big texture array, with the same resolution for each texture. Materials are easier and can be represented as a 2D texture with each row specifying a fixed layout for material parameters. To avoid accidental memory use we limit the material texture array to 1GB and downscale all textures until they all fit. The shader replicates most of the current JS logic as is to reach approximate visual parity. For simplicity, we only store one UV transform (based on .map) and drop rotation. Assets with rotation are very rare; and three.js shaders only transform 2 UVs per draw call, which we don't handle properly for AO anyway - so it's easier to just store one simplified transform for now.
We are now at parity with the JS-evaluated materials, as we apply AO and emissive textures including the relevant factors. Note that three.js uses UV1 to sample AO maps; for now we simplify and assume UV0 is equivalent to UV1 for assets with AO, to avoid carrying another UV set and transform matrix everywhere.
To fully replace the original resampling pipeline we need to transfer vertex colors on the GPU as well. Fortunately, almost the entire pipeline can stay as is; we need to render a different mesh into a temporary texture and copy the resulting colors from the texture. For simplicity, we mostly replicate the original scheme but drop weighting from triangle centroids; this could be restored in the future but isn't strictly necessary for now. The samples are rendered as points, each point ends up doing the same ray cast / material eval. It's not super efficient due to quad overshade but we usually have comparatively few points and the bottlenecks here are mostly CPU side.
We now rely exclusively on new GPU implementation in repaint.js; so we no longer need the separate mode and any of the old code. Also default texture resolution to 1024 as we can handle much larger images with no problems, and remove sloppy simplification as it didn't prove to be useful here.
Subsequent attempts can use the existing cache, which we clear when loading a different file; revert doesn't change the source model and thus can keep the cache alive. Building these is fast for simpler models but can get expensive for models with many large textures.
For normal map baking to work, we need defined source and target tangent spaces. We use MeshoptTangents to generate both, although for source tangents we preserve existing tangents if they are defined, which is important to maintain the original look. The same shader then uses MRT to output the normal map data which is reprojected from the original (perturbed, if possible) normal into the target tangent space. For simplicity we reindex mesh after tangent generation instead of using more complex vertex splitting; this reindexing is now external to unwrapMesh/generateTangents and has to be done manually.
This should finally conclude the new version of the remeshing demo; we now optionally bake roughness/metalness into a separate texture, instead of fusing these into color. The integration was very straightforward as we were already evaluating all of this; bakeMaterial needed to be renamed to bakeShader though for clarity. Also remove 4096 resolution added earlier; for now it looks like we are more limited by the 1024 atlas rescaling and this doesn't buy us anything other than longer bakes.
|
Profile for the backpack remesh above (with cached paint data; the first remesh pays ~170ms extra to build and upload the cached resources); times are in msec: 
Generally speaking the bottleneck tends to be in UV unwrapping now; I tried to look a little bit into whether unwrapping can be faster by tweaking existing settings but didn't get very far. It's reasonable on smaller mesh resolutions but pretty much requires simplification to be in the pipeline - unwrapping non-simplified remeshed geometry is quite slow in comparison. The resulting UV charts could be better although unwrapping is a super difficult problem, and also is made worse because the remeshed geometry here occasionally has "fins" (two-sided triangles that connect two other pieces of geometry together) which is an artifact of remeshing I haven't figured out how to cleanly fix yet. Definitely some future work remains before the results are devoid of artifacts. 
|
39b9bff to
bb6e11a
Compare
We were trying to compute join points between original edges offset by the gutter width, but if the original triangle is long and thin, this creates a join point very far from the original triangle, which then results in a long and thin line that writes over other texels. Instead we apply a more conservative offset, that pushes each point away from the opposite edge by the gutter width. This doesn't result in a 1px thin border but it covers some of the gaps, and ideally we'd need an infill pass anyway. Also switch to normalSeamWeight=0 when generating UV charts; this reduces the chart count significantly without adverse effects.
The new gutter expansion strategy does not result in perfect coverage that mitigates bilinear filtering enough. Additionally, leaving target pixels with alpha 0 results in issues with mip filtering and block compression, although the latter technically could be solved if the compressor is alpha-aware. To mitigate all these we now use an infill pass; the original texture gets a mip chain after the initial render, and we find the first mip level in the chain that has non-zero resulting alpha. This is using bilinear filtering within each mip level, but because the original data is trivially premultiplied (alpha=1/0), we can use this to un-premultiply the result and recover the original value. This is more or less free as it runs on the GPU after the initial texture fill, so we leave it enabled by default.
|
Ok, hopefully this is the final piece. The original version had an issue with chart growth that I was hoping would not be visible in practice but it was: the code that grew triangles used an unstable edge offset, that guaranteed a 1px distance between offset edge and original edge - but for very thin triangles, this resulted in pretty much arbitrary elongation which then rasterized excess pixels all over other charts, resulting in occasional line of pixel artifacts. This is in addition to other issues with a large offset - we do not currently differentiate between internal and boundary edges of the chart so we technically expand all triangles inward too, which is suboptimal as well. This is possible to fix geometrically with an extra layer of triangles but a bit cumbersome; ideally we'd use conservative rasterization but naturally neither WebGL nor WebGPU expose it. So for now I opted for a much less aggressive expansion of the original triangles, which by itself doesn't fully cover the bilinear footprint, followed by an infill pass - we generate mipmaps of the rasterized texture and then replace every pixel with alpha=0 with a different pixel that we find by checking the increasing mip levels with linear filtering. The original data here is trivially premultiplied (alpha=1 or 0), so the generated mips are too, and once we find a filtered pixel with alpha > 0 we un-premultiply it. This results in a nice and cheap neighbor infill while staying within WebGL capabilities and not using any complex hierarchical schemes; as a bonus the textures should compress better now, and are opaque. This does defeat an ability to preserve alpha cutout surfaces but I've already spent 4 days on this rewrite and I am very done 😓 maybe I'll add proper transparency support in the future. For now this seems to work at the cost of even more code and generates nicely filled textures like this: 
|
Skinned objects used bind pose normals/tangents; three.js didn't provide any way to transform them until 0.184, so we update to latest to be able to use applyBoneTransform with Vector4. To keep the code uniform, we now apply skinning transform to positions manually as well.
This accelerates processing for cases when remeshing is called without repaint, at the cost of a longer than usual first rebake: BVH construction can be slow! This also gives us easier control over BVH parameters in the future.
1 participant
This is pretty much a complete rewrite of the repainting portion of the remeshing demo :) I got a little carried away...
The original demo was quite slow when generating high resolution textures. Most of the cost was in JS raycasts; I tried a few different options for how to accelerate these (rewrite casting function to iterative traversal; tweak three-mesh-bvh construction; replace three-mesh-bvh with tinybvh compiled to JS; use WebWorkers). Ultimately it was all pretty cumbersome, and the only option that got reasonable tracing times involved tinybvh compiled to JS and used from WebWorkers, which required someone to maintain tinybvh.js and that wasn't going to be me. Plus even once tracing cost was removed, material evaluation was not free and computing normal maps added even more time.
So I decided to bite the bullet and move repainting entirely to WebGL. Thankfully three-mesh-bvh was a huge help here, as it already provided full support and shader code for tracing the BVH, including closest-point sampling, and also had helpful utilities for exposing vertex data as textures.
To be able to do material sampling, not just ray casts, materials need to get bound into the shader in a "bindless" manner. Neither WebGPU nor WebGL support bindless, so all textures referenced by all materials need to get rescaled to the same size and copied into a texture array. We limit the size of the resulting array to 1024x1024 layers (and 1 GB total); this limits the quality a little bit for some assets but makes this practical. If WebGPU gains true bindless support in the future this could get ported to WebGPU instead, but for now WebGL is enough.
With all that, ray casting and sampling becomes very fast and is practical to run at high resolutions like 2048. Which then made it fairly easy to integrate normal map and material (metallic-roughness) map baking too. For normal map baking, we need source and destination tangents, and we use MeshoptTangents module to generate these now.
Vertex color repaint works using the same approach, with an intermediate texture that is copied to vertex color stream on the CPU.
The results are quite faithful with the right settings; e.g. this is a remesh of a backpack mesh with normal/material map baking, voxel resolution 128, thickening, simplification enabled and regularization disabled:
The demo now also has an 'exportFile' button which produces a GLB file with all textures bundled, courtesy of three.js GLTFExporter.
The JS code used averaging of pixels along triangle edges; to keep this PR a little smaller we don't do that (it requires a separate intermediate texture with RGBA16F format and a separate resolve shader). There might also be occasional issues with the ray casting code as the thickness needs to be tuned better for some meshes to work well, and I'm not sure if casting is watertight either.