Repository navigation
Added model, training and data preparation features - #149
Conversation
… for 100m patches
leavauchier
changed the title
|
|
||
| # Check that using the new version of the code returns | ||
| # the same model structure with the default parameters as the older version | ||
| def test_pyg_randlanet_model_structure(): | ||
| num_features = 9 | ||
| num_classes = 6 | ||
| decimation = 4 | ||
| num_neighbors = 16 | ||
|
|
||
| model1 = PyGRandLANet( | ||
| num_features, | ||
| num_classes, | ||
| decimation=decimation, | ||
| num_neighbors=num_neighbors, | ||
| ) | ||
|
|
||
| model2 = PyGRandLANet( | ||
| num_features, | ||
| num_classes, | ||
| decimation=decimation, | ||
| num_neighbors=num_neighbors, | ||
| dims=[32, 128, 256, 512], | ||
| ) | ||
|
|
||
| # The default dims are (32, 128, 256, 512), so passing them explicitly must | ||
| # yield the same per-block input/output channel dimensions. | ||
| for block_name in ("block1", "block2", "block3", "block4"): | ||
| block1 = getattr(model1, block_name) | ||
| block2 = getattr(model2, block_name) | ||
| assert block1.d_in == block2.d_in | ||
| assert block1.d_out == block2.d_out | ||
|
|
||
| # The first linear layer (fc0) maps features to the bottleneck dimension. | ||
| assert model1.fc0.in_features == model2.fc0.in_features | ||
| assert model1.fc0.out_features == model2.fc0.out_features | ||
|
|
||
| # Explicitly check the expected default embedding dimensions. | ||
| assert model1.block1.d_out == 32 | ||
| assert model1.block2.d_out == 128 | ||
| assert model1.block3.d_out == 256 | ||
| assert model1.block4.d_out == 512 | ||
|
|
||
| # model2 | ||
| assert model2.block1.d_out == 32 | ||
| assert model2.block2.d_out == 128 | ||
| assert model2.block3.d_out == 256 | ||
| assert model2.block4.d_out == 512 | ||
|
|
||
|
|
There was a problem hiding this comment.
nitpick: not sure this is interesting to keep this check on both models (one is enough as you mentioned that checking that the older version still works is done with test_run_predict)
There was a problem hiding this comment.
both Fractal model and LargeDataset_100K_... will be embedded in the docker image, it does not seem very heavy compared to the Docker image weight, but is it what you meant?
There was a problem hiding this comment.
The models can be embedded in the docker image with some caveats. If the model changes (without changing the code) , regenerating the docker image is compulsory if we want to update the model. I have actually some concerns about the docker instance size ( 2 go or higher) as the patch size is now larger for the new model.
There was a problem hiding this comment.
Yep, this github repo does not intend to store trained model (public models should rather be published to hugging face), but the previously embedded model is here to provide an example and let people try myria3d without having to train a model on their own
Same concerns about the patch size (we'd need to experiment on that)
| ) | ||
|
|
||
| # DICT of batch size given subtile_width for externalized-embeddings model. | ||
| # Adjust these to match your externalized-embeddings assets. |
There was a problem hiding this comment.
pas sûre de comprendre l'intérêt de ce dict, l'info est pas dans le yaml ?
There was a problem hiding this comment.
oui l'info est dans le yaml, j'ai modifié pour lire directement depuis le config file
| @@ -4,14 +4,48 @@ | |||
| import os.path as osp | |||
There was a problem hiding this comment.
Pour tester la possibilité de changer le batch size et vérifier que ça marche bien, est-ce que ça te parait pertinent de faire un 2e toy dataset avec une autre taille de subtile ? (dans l'absolu ça pourrait même être n'importe quoi d'autre que 50 ou 100)
| f"--config-name={EXTERNAL_EMBEDDINGS_CONFIG_NAME}", | ||
| "+model.neural_net_hparams.dims=[48,192,384,768]", | ||
| f"--config-name={config_file.stem}", # config file name without the ".yaml" extension | ||
| #"+model.neural_net_hparams.dims=[48,192,384,768]", # dims in yaml config file, so no need to override here |
| for ckpt_file in Path(EXTERNAL_EMBEDDINGS_CONFIG_DIR).glob("*.ckpt"): | ||
| # try to find yaml with ckpt prefix and -inference-Myria3DV*.yaml suffix using glob | ||
| config_file = list(Path(EXTERNAL_EMBEDDINGS_CONFIG_DIR).glob(ckpt_file.stem + "-inference-Myria3DV*.yaml")) | ||
| if config_file: |
There was a problem hiding this comment.
sorry I forgot something: it would be better to raise an error when a ckpt file does not have a matching yaml file to prevent silently not running tests on checkpoints from trained_assets
MR: added model, training and data preparation features