After modifications to handle many corner cases (haxp data, split scans due to clock breaks, etc), the current data linking script is very slow on a filesystem with low IOPS. Linking a single scan can be O(1s) which is a huge bottleneck at this stage. The slowdown is likely due to many file stat operations, and repeated use of "find" command. We should find a way to simplify the heuristics to work solely on a single directory listing whenever possible which can be much faster.
After modifications to handle many corner cases (haxp data, split scans due to clock breaks, etc), the current data linking script is very slow on a filesystem with low IOPS. Linking a single scan can be O(1s) which is a huge bottleneck at this stage. The slowdown is likely due to many file stat operations, and repeated use of "find" command. We should find a way to simplify the heuristics to work solely on a single directory listing whenever possible which can be much faster.