Skip to content

Zhang ferret hyperscanning recordings skipped over missing, misnamed or misplaced files #62

Open

Description

@CodyCBakerPhD

The nightly ingest of Zhang's ferret hyperscanning data (incoming 000480, standardized 000547) skips the recordings below. All paths are under sourcedata/raw/Ferret_Hyperscanning/ in 000480. Each needs a fix in the source data, or a decision to leave it out.

Waiting on merged ephys files (8 recordings)

These are reported as pending and retried every night. The converter expects each animal's ephys at <recording>.rec/<animal>HS<n>/<recording>_<animal>HS<n>_merged.rec.

Recording Folder Problem
0235HS3_0237HS4_20260320_102109 0235-0237/03.20.2026 The 0237HS4 file is named …_0237HS4merged.rec, with no _ before merged
0235HS3_0237HS4_20260325_124806 0235-0237/03.25.2026 The file in 0235HS3/ is named …_0236HS3_merged.rec, with animal 0236 instead of 0235
0235HS3_0237HS4_20260330_150816 0235-0237/03.30.2026 Both merged files sit at the top of the .rec folder, not in 0235HS3/ and 0237HS4/
0235HS6_0237HS6_20260425_132118 0235-0237/04.25.2026 The 0235HS5/ folder and both merged file names say 0235HS5, while the recording name says 0235HS6
0236HS3_0237HS4_20260331_101851 0236-0237/03.31.2026 No headstage folders or merged files, only the base station .rec
0236HS3_0237HS4_20260331_110624 0236-0237/03.31.2026 No headstage folders or merged files
0236HS3_0237HS4_20260401_130419 0236-0237/04.01.2026 No headstage folders or merged files
0236HS5_0237HS6_20260408_155939 0236-0237/04.08.2026 The 0237HS6 folder is named 0237HS8

Never picked up

These are skipped with only a log line, so they don't show as pending.

What Where Count
Session1/Session2 or Session_1/Session_2 subfolders. The two checked each hold a recording and its Videos/ 0235-0236: 03.18, 04.23. 0235-0237: 03.19, 04.23, 04.28, 05.07, 05.13, 05.20. 0236-0237: 03.18, 04.23, 04.28 22 folders
Calibration videos named CamNN-Calibration-<time>-NNNN.avi. The converter only reads CamNN-cal-<time>-NNNN.avi 0235-0236: 03.06, 03.08, 03.09, 03.10, 03.13, 03.16. 0235-0237: 03.08, 03.13. 0236-0237: 03.06, 03.08, 03.13, 03.16 60 files
A recording directly in the pair folder, not in a date folder 0235-0236/0235HS4_0236HS3_20260327_125659.rec 1

Recording names the converter does not recognize:

Recording Problem
02355HS6_0236HS5_20260408_134217 Animal ID 02355, likely 0235
0235HS3_0237HS4_Session1_20260308_142653 Extra Session1
0235HS4_0236HS3_Session1_20260306_154144 Extra Session1
0236HS3_0237HS4_Session1_20260306_114321 Extra Session1
0236HS3_0237HS4_Session1_20260308_130347 Extra Session1
0235HS3_0237HS4_sess1_20260318_111532 Extra sess1
0236HS3_0237HS4_RecordingTest1_20260223_155333 A test recording, maybe left out on purpose

Also left out, probably on purpose: 0235-0236/03.10.2026/0236HS3_PMC_rawdownsample.set and the README.txt files in 0236-0237/03.06.2026 and 0236-0237/03.08.2026.

Questions for the lab

  • Can the merged files in the first table be renamed or moved to the expected layout? Can the missing ones for 03.31 and 04.01 be uploaded?
  • Is 0235HS6_0237HS6_20260425_132118 recorded on headstage 5 or 6?
  • Should the Session subfolders and the oddly named recordings be converted? If so, they can be renamed and moved to the usual layout, or the converter can learn these layouts.
  • Should RecordingTest1 stay out?
  • Are the CamNN-Calibration-… files calibration videos? If so, the converter can accept that spelling.

Activity

  1. CodyCBakerPhD commented on Nov 19, 2026

    @CodyCBakerPhD
    MemberAuthor

    Update from #63, not merged yet. Once it merges, the converter reads:

    • the Session1/Session_1 subfolders, the extra Session1, sess1, Sess1 or Merge in recording names, the missing _ before merged, and the CamNN-Calibration-… videos
    • merged files loose in the .rec folder, as on 03.30
    • five mislabeled recordings, through a corrections table in labs/zhang/ferret-hyperscanning/code/config.yaml

    Each correction was checked against the merged files' headers. The merge copies the recording's start time into both merged files and records the serial of the headstage that recorded each. Those serials are 03103 00106 for HS3, 00105 for HS4, 00127 for HS5 and 00130 for HS6.

    That converts 107 of the 111 recordings on EMBER. 0235-0236/0235HS4_0236HS3_20260327_125659.rec is no longer on EMBER. With #63, the nightly download also deletes leftovers like it from the runner.

    Still needed from the lab

    • Upload the merged files for 0236-0237/03.31.2026/0236HS3_0237HS4_20260331_101851, 0236-0237/03.31.2026/0236HS3_0237HS4_20260331_110624 and 0236-0237/04.01.2026/0236HS3_0237HS4_20260401_130419. Each has only the base station .rec, which holds no ephys.

    Renames that would make the names match the contents

    These recordings convert through the corrections table, so none of these renames blocks the ingest. Once a file is renamed to the standard name, the converter uses that name instead of its correction. A rename has to cover every name listed for its recording. Otherwise the recording goes back to pending.

    Recording Rename Evidence
    0235-0237/03.25.2026/0235HS3_0237HS4_20260325_124806 0235HS3/…_0236HS3_merged.rec to …_0235HS3_merged.rec The file has HS3's serial. 0236 has no recording that day
    0235-0237/04.25.2026/0235HS6_0237HS6_20260425_132118 The recording folder and its base station .rec to 0235HS5_0237HS6_20260425_132118 The 0235HS5/ folder and both merged names say HS5, and 0235's file has HS5's serial
    0236-0237/04.08.2026/0236HS5_0237HS6_20260408_155939 Folder 0237HS8/ to 0237HS6/ The file in it is named …_0237HS6_merged.rec and has HS6's serial. 0237 wore HS8 in the 14:39 recording that day
    0235-0236/04.08.2026/02355HS6_0236HS5_20260408_134217 02355 to 0235 in the recording folder, its base station .rec and both merged files The pair folder, the 0235HS6/ folder and its file say 0235, and the serial matches HS6
    0236-0237/03.08.2026/0236HS3_0237HS4_Session1_20260308_130347 0236HS3/…_HS30236_merged.rec to …_0236HS3_merged.rec, and 0237HS4/…_130347_merged.rec to …_130347_0237HS4_merged.rec Each file's serial matches the folder it sits in
    0235-0237/03.30.2026/0235HS3_0237HS4_20260330_150816 Move each merged file into 0235HS3/ or 0237HS4/ Names and serials are already right

    Questions

    1. March headstage serials. From April on, every merged file's serial matches its HS number, all 138 files. Six files in four March recordings don't:

    Recording Serials
    0235-0236/03.19.2026/0235HS4_0236HS3_20260319_130321 Both files report HS4. They are the same size to the byte but have different contents
    0235-0237/03.20.2026/0235HS3_0237HS4_20260320_102109 Both files report HS4
    0236-0237/03.19.2026/0236HS3_0237HS4_20260319_143327 Swapped. The 0236HS3 file reports HS4 and the 0237HS4 file reports HS3. Already published
    0235-0237/03.18.2026/0235HS3_0237HS4_sess1_20260318_111532 Swapped the same way. Converts once #63 merges

    Two files can't both come from HS4, so the serial field is sometimes wrong. That's why these recordings are converted under their names. Were the loggers swapped at the merge for the two swapped recordings? If so, the two animals' ephys is swapped in their NWB files.

    2. Short partner files. A session's two merged files are usually within 1% of each other in size (85 of 107 sessions). In these six, one animal's file is far smaller. Did that animal's logger stop early? Should these sessions be marked?

    Recording Short file Partner's file
    0235-0237/04.15.2026/0235HS7_0237HS8_20260415_164755 0235, 0.002 GB 3.49 GB
    0235-0237/05.19.2026/0235HS5_0237HS6_20260519_103735 0235, 0.13 GB 6.48 GB
    0235-0237/04.17.2026/0235HS7_0237HS8_20260417_134302 0237, 0.34 GB 3.30 GB
    0235-0236/04.09.2026/0235HS6_0236HS5_20260409_151033 0235, 0.71 GB 3.34 GB
    0236-0237/03.08.2026/0236HS3_0237HS4_Session1_20260308_130347 0237, 0.66 GB 2.54 GB
    0236-0237/04.28.2026/Session_1/0236HS7_0237HS8_20260428_131255 0237, 1.68 GB 4.34 GB

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions