Repository navigation
Compression handling improvements - #41
Merged
Merged
Conversation
- Add synthetic unsorted MDF4 fixture generator (50MB, 4 channel groups) - Add benchmark group for unsorted data reading performance - Optimize read_all_channels_unsorted_from_bytes with Vec-indexed lookup instead of HashMap for O(1) access with better cache locality - ~20% performance improvement on large unsorted files Co-authored-by: Claude <noreply@anthropic.com>
- Add synthetic VLSD/VLSC fixture generator (~30MB, 100K variable-length records) - Add benchmarks: unsorted_vlsd_30mb, mdf4_vlsc_5kb - Optimize read_all_channels_unsorted_from_bytes with pre-built CG lookup Vec replacing HashMap get_mut() calls in hot loop (~47% faster on large files) Co-authored-by: Claude <noreply@anthropic.com>
- Fix needless_late_init warnings in mdfinfo3.rs and mdfreader4.rs - Fix for_kv_map warning in mdfinfo3.rs - Allow unstable_name_collisions for binrw::BinReaderExt::read_le() Co-authored-by: Claude <noreply@anthropic.com>
- Add bench_conversion: measures convert_all_channels() performance - Add bench_export: measures export to Parquet, CSV, and compressed Parquet - Add read_load_convert helper for full pipeline benchmarks Co-authored-by: Claude <noreply@anthropic.com>
- Parallelize Parquet and CSV exports across CGs using rayon - Add arrow-csv dependency (evaluated but ArrayFormatter was faster for single CG) - CSV export: ~367ms (slight improvement from parallelization) - Parquet export: ~168ms (unchanged, I/O bound) Co-authored-by: Claude <noreply@anthropic.com>
- Avoid finish_cloned() in text_to_value_calculation and text_to_text_calculation by iterating directly over values_slice() and offsets_slice() - Reuse String buffer across records in string decoding (UTF-8, UTF-16, SBC) instead of allocating a new String for each record These optimizations reduce memory allocations for files with text channels and string conversions, improving performance for text-heavy MDF4 files. Co-authored-by: Claude <noreply@anthropic.com>
… errors - Remove cn_type==4 from fixed-length read paths; sync channels now expose their ATBLOCK reference instead of reading garbage numeric data. - Add public getters for CN precision/limits/ranges and CG path separator. - Replace unimplemented! with bail! for unsupported array channel types. Contributor: Anthropic Claude
…nalInfo - Decode event signal channel data using template EVBLOCK sync type - Return &Ev4Block directly instead of redundant EventSignalInfo - Add MDF3 range/limit/sampling-rate getters for API parity - Fallback range/limit computation from loaded channel data - Remove is_mlsd_channel from public API (internal detail) - Keep is_sync_channel and sync_type as useful channel semantics - Add Python bindings for new getters and decode_event_records
… in Rust/Python/C APIs - Add Mdf delegation methods: list_file_history, get_file_history_blocks, list_events, get_event_blocks, get_event_block, list_channel_hierarchy, get_channel_hierarchy_blocks, get_channel_hierarchy_block, list_source_information, get_source_information_blocks, list_attachments, get_attachment_blocks, get_attachment_block, get_attachment_embedded_data, get_sample_reduction_blocks - Include File History section in Python __repr__ for MDF4 files - Add C API functions: get_file_history_count, get_file_history_time_ns, get_event_count, get_event_type, get_channel_hierarchy_count, get_source_information_count - Keep &mut self on list_* methods to preserve lazy XML parsing via get_md_comment(); remove flawed get_md_comment_ref() bypass - Apply cargo clippy --fix suggestions
…Changed flate2 backend to zlib-rs for better performance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release development branch to main.
Changes included:
This includes all commits from development since main.