Skip to content

Concrete standardization examples - #39

Open
nehatk17 wants to merge 5 commits into
mainfrom
update-docs
Open

nehatk17 wants to merge 5 commits into
mainfrom
update-docs

Conversation

@nehatk17

@nehatk17 nehatk17 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Adding in some examples for standardizing 'raw' data inspired by real BBQS teams.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🚀 Preview for this pull request is published at

https://aplbrain.github.io/bbqs-ember-docs-previews/pr-preview/pr-39/

Built from 6a59a53 to gh-pages in aplbrain/bbqs-ember-docs-previews. Updates on every push; not removed when the pull request closes.

@erikjohnson24 erikjohnson24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Had some small suggestions you can accept/reject.

Do you want to link to any existing standardization scripts or ember-dandi datasets here?

Comment thread docs/user-guide/data-standardization.md Outdated

```
myExperimentSession/
├── folder1/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the folders here? different days/times? different subjects? probably subjects?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these folders just represent an arbitrary way to split up a long recording by time. so it is all the same subject and session. I've added some clarification in a new commit.

Comment thread docs/user-guide/data-standardization.md Resolved
Comment thread docs/user-guide/data-standardization.md Resolved
Comment thread docs/user-guide/data-standardization.md Resolved
Each of the audio and video files in the folder1 in myExperiment hold data that is continuous in time with the files of folder2, which is continuous with folder3 and so and and so forth. Altogether, folders 1 - 5 constitute a single recording session. This means that all the data should be contained inside a single NWB file.
Each of the audio and video files in the `split1` folder in myExperiment hold data that is continuous in time with the files of `split2`, which is continuous with `split3` and so and and so forth. Altogether, folders `split1` - `split5` constitute a single recording session. Best practices indicate that all the data should be contained inside a single NWB file.

To make it easier for a secondary user to parse the data, then, we will concatenate all video files together, all audio files together, and all pose data together within the single NWB file. The concatenated video file will be referenced externally, and audio files will be represented with `ndx-sound`, an NWB extension for audio data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To make it easier for a secondary user to parse the data, then, we will concatenate all video files together, all audio files together, and all pose data together within the single NWB file. The concatenated video file will be referenced externally, and audio files will be represented with `ndx-sound`, an NWB extension for audio data.
To make it easier for a secondary user to parse the data, we will concatenate all audio and pose files into a single NWB file that also contains structured links to the video. We will also concatenate the video files, which will be referenced externally (see [Best Practice: External Mode for details](https://nwbinspector.readthedocs.io/en/dev/best_practices/image_series.html#use-external-mode-for-videos-of-animals)). Audio files will be represented with `ndx-sound`, an NWB extension for audio data.

Suggested rephrasings and note explaining why videos have to be stored outside NWB



Each of the audio and video files in the folder1 in myExperiment hold data that is continuous in time with the files of folder2, which is continuous with folder3 and so and and so forth. Altogether, folders 1 - 5 constitute a single recording session. This means that all the data should be contained inside a single NWB file.
Each of the audio and video files in the `split1` folder in myExperiment hold data that is continuous in time with the files of `split2`, which is continuous with `split3` and so and and so forth. Altogether, folders `split1` - `split5` constitute a single recording session. Best practices indicate that all the data should be contained inside a single NWB file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Each of the audio and video files in the `split1` folder in myExperiment hold data that is continuous in time with the files of `split2`, which is continuous with `split3` and so and and so forth. Altogether, folders `split1` - `split5` constitute a single recording session. Best practices indicate that all the data should be contained inside a single NWB file.
Each of the audio and video files in the `split1` folder in myExperiment hold data that is continuous in time with the files of `split2`, which is continuous with `split3` and so forth. Altogether, folders `split1` - `split5` constitute a single recording session. Best practices indicate that all the data should be contained inside a single NWB file.

### Example 2 - Audio, video and pose-tracking in BIDS

<!-- Now let's suppose we want to instead use BIDS, which would allow us to maintain the original way we had split up the experiment session. We can also keep the original data formats for audio and video files.
Now let's suppose we want to instead use BIDS, which would allow us to keep the original data formats for audio and video files. The SLEAP Data would need to be exported into a csv and transformed according to the `motion` modality within BIDS.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Now let's suppose we want to instead use BIDS, which would allow us to keep the original data formats for audio and video files. The SLEAP Data would need to be exported into a csv and transformed according to the `motion` modality within BIDS.
Now let's suppose we want to instead use BIDS, which would allow us to keep the original data formats for audio and video files. The SLEAP Data would need to be exported into a `.tsv` table and transformed according to the `motion` modality within BIDS.

Comment on lines +162 to +165
└── motion/
├── sub-01_ses-01_task-free_tracksys-camera_motion.tsv
├── sub-01_ses-01_task-free_tracksys-camera_motion.json
└── sub-01_ses-01_task-free_tracksys-camera_channels.tsv

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I would actually contend that for Pose Estimation specifically, this is not BIDS Motion as sourcedata but rather is a derivative (mostly because it can be rerun, rederived, and will change across models and versions)

Compared to a source-time motion tracker (such as marker-based) which is what that BIDS modality is really meant for


where `audio.wav` and `video.mp4` are the concatenated audio and video respectively from folders 1 - 5. The `_motion.tsv` contains time-series information of tracked body-part coordinates from the complete recording. `_channels.tsv` describes the tracked coordinate channels such as nose X, nose Y, etc.

Alternatively, NWB could be used within the BIDS structure by exporting the .slp files to NWB and placing the file inside the `derivatives/` folder.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


### Example 3 - Multiple animals interacting

Expanding on Examples 1 and 2, now there are two animals in the cage. For this example, the NWB extension for handling data that represents multiple subjects at the same time should be used. Thus, the audio and video, which now capture two animals at once, will be placed into `ndx-multisubjects`, while the individual pose-tracking information will be placed into each animal's individual NWB file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Expanding on Examples 1 and 2, now there are two animals in the cage. For this example, the NWB extension for handling data that represents multiple subjects at the same time should be used. Thus, the audio and video, which now capture two animals at once, will be placed into `ndx-multisubjects`, while the individual pose-tracking information will be placed into each animal's individual NWB file.
Expanding on Examples 1 and 2, now there are two animals in the field of view. For this example, the NWB extension for handling data that represents multiple subjects at the same time should be used. Thus, the audio and video, which now capture two animals at once, will be placed into `ndx-multisubjects`, while the individual pose-tracking information will be placed into each animal's individual NWB file.

To use slightly less triggering wording to the public eye

└── RightAnkle_Z
```

This data is transformed into appropriate containers within a single NWB file, producing a standardized result that can be loaded with other tools and processing scripts:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This data is transformed into appropriate containers within a single NWB file, producing a standardized result that can be loaded with other tools and processing scripts:
This data is transformed into appropriate neurodata types within a single NWB file, producing a standardized result that can be loaded with other tools and processing scripts:

to avoid any potential confusion around the wording of 'container' [which is far too overloaded these days...]

Comment thread docs/index.md
5. You can publish your dataset, making it publicly available and creating a DOI.

!!! tip
If you are on a BBQS team, please reach out to us to get this workflow started at [help@emberarchive.org](mailto:help@emberarchive.org).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you are on a BBQS team, please reach out to us to get this workflow started at [help@emberarchive.org](mailto:help@emberarchive.org).
If you are on a BBQS team, please reach out to us to get this workflow started at [help@emberarchive.org](mailto:help@emberarchive.org). Our staff is available to help through every step of the process!

IDK any other more emphatic wordings to get the point across?

@CodyCBakerPhD

Copy link
Copy Markdown
Contributor

Overall looks good - left my thoughts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants