Skip to content

[improvement](agg) Add passthrough local exchange before streaming ag…#65791

Closed
zclllyybb wants to merge 1 commit into
apache:opt_perf_4.1from
zclllyybb:codex/streaming-agg-passthrough-opt-perf-4.1
Closed

[improvement](agg) Add passthrough local exchange before streaming ag…#65791
zclllyybb wants to merge 1 commit into
apache:opt_perf_4.1from
zclllyybb:codex/streaming-agg-passthrough-opt-perf-4.1

Conversation

@zclllyybb

Copy link
Copy Markdown
Contributor

…gregation

Problem: non-final streaming aggregation can consume rows directly from its upstream pipeline, even when separating the pipelines with a local passthrough exchange is required to match the intended Repeat-to-streaming-aggregation execution shape. opt_perf_4.1 already carries the thrift field but does not expose or consume it end to end.

Solution: expose enable_local_exchange_before_agg as a default-enabled session variable and propagate it to TQueryOptions. StreamingAggOperatorX requests PASSTHROUGH only for non-final streaming aggregation. The existing hash-join passthrough priority and explicit shuffled_agg_ids HASH_SHUFFLE override remain intact; disabling the variable, or receiving an unset field from an older FE, retains the original distribution behavior.

Validation: ./build.sh --fe; ./build.sh --be -j 48; ./run-fe-ut.sh --run org.apache.doris.qe.VariableMgrTest; ./run-be-ut.sh --run --filter=StreamingAggOperatorTest.require_passthrough_local_exchange_before_non_finalize_agg -j 90. On an isolated FE/BE, an uncached GROUPING SETS query with agg_phase=2 produced REPEAT -> LOCAL_EXCHANGE(PASSTHROUGH) -> STREAMING_AGGREGATION by default; setting the variable false removed that local exchange while retaining the streaming aggregation and result rows.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

…gregation

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65579

Problem Summary: opt_perf_4.1 exposes enable_local_exchange_before_agg, but its default non-final streaming-aggregation path requests HASH_SHUFFLE whenever grouping keys are present. A streaming aggregation needs a pipeline boundary before aggregation without repartitioning rows, so Repeat-to-streaming-aggregation plans do not receive the intended passthrough local exchange.

### Release note

Non-final streaming aggregation requests a passthrough local exchange by default when enable_local_exchange_before_agg is enabled. The setting false retains the existing base distribution behavior.

### Check List (For Author)

- Test: BE and FE unit tests; isolated-cluster profile validation

- Behavior changed: Yes, streaming aggregation uses passthrough rather than hash local exchange for this default-enabled setting.

- Does this need documentation: No
@zclllyybb
zclllyybb force-pushed the codex/streaming-agg-passthrough-opt-perf-4.1 branch from a8a90db to 8508fcf Compare July 19, 2026 09:37
@zclllyybb zclllyybb closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants