The current CI is minimal and doesn't fully test the notebooks
Got some more help from @AGoose on this topic
There are some approaches you might take to test a notebook without the user seeing the tests:
- Create a set of "blueprint" notebooks that contain test cells. Add "test" cell tags to these test cells, and then use
nbformat to generate a production-ready notebook without these cells using the TagRemovePreprocessor
- Create a set of "test" notebooks, which execute the production notebooks in the current namespace (e.g. using the
%run magic)
Concerning how to test statistical outputs, well, that's another problem entirely, and I suppose one approach is to either parametrise your test mechanism to allow a higher number of samples when running your tests, or to perhaps identify the distribution on the various quantities you observe and check that your result lies within that window. I suppose you might already be doing that!
The current CI is minimal and doesn't fully test the notebooks
Got some more help from @AGoose on this topic
There are some approaches you might take to test a notebook without the user seeing the tests:
nbformatto generate a production-ready notebook without these cells using theTagRemovePreprocessor%runmagic)Concerning how to test statistical outputs, well, that's another problem entirely, and I suppose one approach is to either parametrise your test mechanism to allow a higher number of samples when running your tests, or to perhaps identify the distribution on the various quantities you observe and check that your result lies within that window. I suppose you might already be doing that!