Specifically presented as taking the output of `distributed-downloader` and then does all these things to _create_ the TreeOfLife dataset. - [x] Rename repo: TreeOfLife-toolbox - [x] Rename under `src`: `TOL-toolbox` - [ ] Reorganize under `TOL-toolbox` to have submodules based on function (e.g., `filtering`, `data-transfer`) with sub-submodules (e.g., `research-filtering`) - [ ] Add from TreeOfLife-dev as submodules in `src/TOL-toolbox`: - [ ] webdataset creation - [ ] metadata transfer - [ ] lookup table creator - [ ] Add from TreeOfLife-dev at root-level `scripts/` directory all lookup table exclude creators (many of these are [here](https://github.com/Imageomics/TreeOfLife-dev/tree/main/src/processing)): - [ ] PDQ hash process - [ ] Image type separator (museum, camera trap, and citizen science categories) - [ ] Museum Specimen processing: label vs specimen filter - [ ] camera trap processing: megadetector & reducer - [ ] citizen science image processing - [ ] face detection - [ ] Add from TreeOfLife-dev at root-level `notebooks/` directory all lookup table exclude creators: - [ ] clustering and other EDA determinations - [ ] Add root-level `data/` folder with embeddings for museum support sets (relates to #22). This will be done following the merge of PRs #1 through #15.
Specifically presented as taking the output of
distributed-downloaderand then does all these things to create the TreeOfLife dataset.src:TOL-toolboxTOL-toolboxto have submodules based on function (e.g.,filtering,data-transfer) with sub-submodules (e.g.,research-filtering)src/TOL-toolbox:scripts/directory all lookup table exclude creators (many of these are here):notebooks/directory all lookup table exclude creators:data/folder with embeddings for museum support sets (relates to Add processing tools for exclusion sets (as methods) #22).This will be done following the merge of PRs #1 through #15.