ChangeLog

----------------------------------------------------------------------------

DTLMod (0.6) not released yet (target: September 2026)

Improvements:
  - End-of-stream signaling. Once the last publisher closes an engine, a
    subscriber that requests a transaction which was never produced is now
    released with a new EndOfStreamException instead of blocking forever
    (mirroring the StepStatus::EndOfStream returned by ADIOS2's BeginStep).
    This lets a consumer terminate when its upstream stops. Implemented for
    both the Staging and File engines, for any number of subscribers. Exposed
    in the Python bindings as EndOfStreamException.
  - New ReductionMethod::get_fidelity(var, transaction_id) returning the
    fraction of the original information retained after reduction, in [0, 1].
    Decimation reports the retained-element fraction; compression derives a
    fidelity from the accuracy bound (shape and reduced size are uninformative
    for it), so subscribers can reason about data quality, not just volume.
    Exposed in the Python bindings as ReductionMethod.get_fidelity().
  - New Engine::cancel_pending_activities() and Engine::drain() helpers, now
    used by every transaction teardown path of the Staging and File engines,
    so that cancelling and emptying an ActivitySet is done in one place
  - Improved test coverage
    - Regression test for the cancellation of a Staging transaction with four
      publishers, checking that no activity is destroyed while still running
    - Regression test for the cancellation of a File transaction with two
      pending writes, checking that both are canceled
    - Both tests need at least two activities in the set to be meaningful, a
      case the existing single-publisher cancellation tests could not cover
    - The test suite now aborts at startup if it loaded a libdtlmod other than
      the one just built, which LD_LIBRARY_PATH can silently cause

Bug Fixes:
  - Transaction cancellation only canceled part of the in-flight activities.
    Activity::cancel() mutates the ActivitySet being iterated over, in two
    different ways: it is a simcall, hence a scheduling point at which a
    resumed actor can clear the set (StagingEngine), and it fires the
    on_this_completion callback that erases the activity from its set
    (FileEngine). Indexing the set therefore skipped roughly every other
    activity. The skipped ones kept running, and the subsequent clear()
    dropped their last handle, leaving an ActivityImpl alive in the kernel,
    still consuming link or disk bandwidth for the rest of the simulation.
    Multi-publisher Staging scenarios could also deadlock
  - Transaction teardown after a cancellation emptied the ActivitySets without
    cancelling what was left in them, in StagingEngine::end_sub_transaction(),
    StagingEngine::begin_pub_transaction() and FileEngine::sub_read(), whereas
    pub_close() and sub_close() did cancel first
  - pub_close() and sub_close() canceled the leftover activities
    unconditionally, flipping the state of those that had already completed
    and firing their completion callbacks a second time

----------------------------------------------------------------------------

DTLMod (0.5) May 25, 2026

Improvements:
  - Per-transaction compression ratio variability is now active
    - CompressionReductionMethod::get_reduced_variable_global_size() and
      get_reduced_variable_local_size() now accept an optional transaction_id
      parameter and call get_effective_ratio(transaction_id) to compute the
      actual ratio, which may be perturbed around the nominal value when
      ratio_variability > 0
    - Engine::put() passes the current transaction id to both calls, so the
      simulated transfer size varies realistically from transaction to transaction
      when ratio_variability is set
    - The same transaction_id always produces the same effective ratio
      (deterministic hash of variable name and transaction id), making
      simulations reproducible
    - ReductionMethod base-class virtuals and all overrides (Compression,
      Decimation) updated accordingly; Decimation ignores the parameter as its
      reduced sizes are transaction-independent
    - Python bindings updated: get_reduced_variable_global_size() and
      get_reduced_variable_local_size() accept an optional transaction_id
      keyword argument (default 0)
  - Transaction cancellation support
    - New Engine::cancel_transaction(transaction_id) allows an external actor
      to cancel a specific in-flight transaction, unblocking both its
      publishers and subscribers
    - Publishers and subscribers receive a TransactionCanceledException;
      catching it allows them to survive the cancellation and continue to
      subsequent transactions
    - Works with both File and Staging engines across any number of publishers
      and subscribers
  - Memory efficiency: progressive eviction of metadata transaction entries
    - Once all subscribers have consumed a transaction, its entries are evicted
      from the in-memory Metadata::transaction_infos_ map
    - When metadata export is enabled and publishers and subscribers coexist
      (file streaming), evicted entries are progressively flushed to
      per-variable temporary files; the final metadata file is assembled at
      pub_close() from those files and any remaining in-memory entries,
      preserving the existing format and transaction count
    - When the stream is opened by subscribers only after all publishers have
      closed (sequential scenario), memory-only eviction is performed (the
      metadata file has already been written by pub_close())
    - Memory footprint of the File engine now grows as O(N_pub) instead of
      O(N_pub × N_transactions) for long-running concurrent streaming workloads
  - Improved test coverage
    - Comprehensive C++ and Python test suite for transaction cancellation,
      covering cancellation while waiting for a subscriber, while waiting for
      a publisher, and mid-transaction for both File and Staging engines
    - Tests for the 3-parameter form of DTL::add_stream
    - Tests for propagation of reduction parameters to subscribers for both
      compression and decimation methods

Bug Fixes:
  - SZ compression model: the ratio formula incorrectly used -log10(accuracy)
    instead of accuracy^beta, inverting the accuracy-to-ratio relationship;
    constants recalibrated to give ratio ≈ 7 at accuracy = 1e-3 and ratio ≈ 2
    at accuracy = 1e-6
  - CompressionReductionMethod did not register the subscriber-side variable
    during inquire_variable, preventing Engine::get() from simulating
    decompression costs
  - DecimationReductionMethod did not propagate its parameterization to
    subscribers, preventing them from querying reduced variable sizes after
    inquire_variable
  - Transaction cancellation in StagingEngine: CancelException and
    NetworkFailureException thrown during pub_close and sub_close were not
    caught when a cancellation was in progress, leaving in-flight activities
    in an inconsistent state
  - StagingEngine::begin_sub_transaction did not decrement
    num_subscribers_starting_ when a transaction was canceled, causing an
    imbalance in subsequent synchronization
  - CompressionReductionMethod::ParameterizedCompression stored a raw pointer
    to the associated Variable; replaced with the variable name string to
    eliminate potential dangling references

API Changes:
  - New Engine method:
    - Engine::cancel_transaction(unsigned int transaction_id) cancels a
      specific transaction performed by this engine. This method must be
      called from an external actor not involved in the transaction. It raises
      a TransactionCanceledException that publishers and subscribers must catch
      to survive the cancellation and continue to subsequent transactions. If
      both sides have already moved past the given transaction ID, the call is
      a no-op to avoid accidentally canceling a subsequent transaction.
  - New Stream helper methods:
    - Stream::get_engine_type() and Stream::get_transport_method() respectively
      return the enum value of the engine type and transport method for a
      Stream.
    - As a consequence, in the Python bindings, the Stream.engine_type and
      Stream.transport_method readonly properties that return the engine type
      and transport method as a string have been renamed to
      Stream.engine_type_str and Stream.transport_method_str; the existing
      properties now return the enum values.
    - Stream::get_reduction_method(name) retrieves a reduction method
      associated to the stream by name, returning std::nullopt if not found.
  - New ReductionMethod virtual method:
    - ReductionMethod::propagate_for_subscriber(publisher_var, subscriber_var)
      copies the publisher-side parameterization to the subscriber variable;
      the default implementation is a no-op; both DecimationReductionMethod
      and CompressionReductionMethod provide overrides called by
      Stream::inquire_variable.
  - Behavior modifications:
    - DTL::add_stream can now take an Engine::Type and a Transport::Method as
      optional parameters (both defaulting to Undefined if not specified).
    - Reduction methods can now be declared in the DTL configuration file via
      a "reduction_methods" array in the stream definition.

----------------------------------------------------------------------------

DTLMod (0.4) February 16, 2026

Major improvements:
  - Data Reduction framework
    - New abstract ReductionMethod interface for extensible data reduction strategies
    - Decimation method: spatial subsampling with per-dimension stride, optional
      interpolation (linear, quadratic, cubic), configurable computational cost,
      and support for both publisher-side and subscriber-side application
    - Compression method: size reduction with preserved shape, supporting three
      compressor profiles (fixed ratio, SZ-inspired, and ZFP-inspired models),
      separate compression and decompression costs, and per-transaction ratio
      variability
    - Publisher-side reduction state is propagated to subscribers through
      inquire_variable, enabling detection and prevention of conflicting
      double reductions
    - Reduction operations work with both File and Staging engines
  - New documentation pages for the reduction feature (Reduction, Decimation,
    and Compression) and updated Main Concepts page
  - Improved test coverage
    - Comprehensive C++ and Python tests for both reduction methods
    - Coverage of error handling, parameter validation, re-parameterization,
      and publisher-subscriber workflows for each reduction method
    - New test for subscriber-first arrival pattern in File engine
  - Code quality and CI improvements
    - Improved coverage reporting with SonarQube and CodeFactor integration
    - Added subscriber synchronization barrier in File engine
    - Removed unnecessary defensive guards and race condition checks that
      cannot occur under SimGrid's maestro orchestration

API Changes:
  - New Stream method:
    - Stream::define_reduction_method(name) creates a named reduction method
      ("decimation" or "compression") for the stream
  - New Variable methods and properties:
    - Variable::set_reduction_operation(method, parameters) applies a reduction
      with key-value parameters to a variable
    - Variable::is_reduced(), is_reduced_by_publisher(), is_reduced_by_subscriber()
      query the reduction state
    - Variable::get_reduction_method() retrieves the applied ReductionMethod
  - New ReductionMethod class with query methods:
    - get_reduced_variable_global_size(), get_reduced_variable_local_size()
    - get_reduced_variable_shape()
    - get_flop_amount_to_reduce_variable(), get_flop_amount_to_decompress_variable()
  - Full Python bindings for all reduction operations, including nine new
    exception types for parameter validation errors
  - Engine::put() now automatically simulates reduction cost and transports the
    reduced data size when the variable is reduced
  - Engine::get() now automatically simulates decompression cost after receiving
    compressed data

----------------------------------------------------------------------------

DTLMod (0.3) January 19, 2026

Major improvements:
  - Enhanced code quality with comprehensive refactoring 
    - Improved modern C++17 usage throughout the codebase
    - Reduced code complexity and technical debt
  - Memory safety and robustness improvements
    - Fixed critical memory safety issue in DTL connection management
    - Fixed race condition in Engine creation
    - Plugged memory leaks
  - Major refactoring with improved memory management, encapsulation, and move-only semantics
  - Improved CI/CD infrastructure
    - Added Valgrind and sanitizer checks (AddressSanitizer, UndefinedBehaviorSanitizer)
    - Enhanced test coverage and validation
  - Comprehensive proofread of the documentation

API Changes:
  - DTL::get_stream_by_name_or_null() renamed to DTL::get_stream_by_name()
    Returns std::optional<std::shared_ptr<Stream>> instead of raw pointer
    Python: dtl.get_stream_by_name() now returns None if stream not found
  - Metadata management transferred from Engine to Stream
    Stream::get_metadata_file_name() replaces Engine::get_metadata_file_name()
    Python: stream.metadata_file_name replaces engine.metadata_file_name
  - Stream method chaining improvements
    set_engine_type() and set_transport_method() now return Stream& instead of Stream*
    Enables cleaner fluent interface: stream.set_engine_type(...).set_transport_method(...)
  - DTL::create() now accepts std::string_view and has empty default parameter
  - String parameters changed to std::string_view for better performance:
    - DTL::add_stream(), DTL::get_stream_by_name()
    - Stream and Variable constructors and methods
  - get_engine_type_str() and get_transport_method_str() now return std::optional<const char*>

----------------------------------------------------------------------------

DTLMod (0.2) November 11, 2025

  - New comprehensive documentation
  - Full python bindings
  - Improved code coverage
  - Reduced technical debt
  - Fixed deadlocks in File engine
  - Update to SimGrid 4.1 and FSMod 0.4
  - API:
    - Make metadata export configurable at stream level.
    - Have a simpler version of Engine::put() that automatically uses the
      local size of the Variable.
    - Stream::remove_variable now throws an UnknownVariableException rather
      than returning a bool

----------------------------------------------------------------------------

DTLMod (0.1) July 4, 2025

First Release

----------------------------------------------------------------------------