perf: select one message per run in SQL with DISTINCT ON#38578
perf: select one message per run in SQL with DISTINCT ON#38578weijun-xia wants to merge 1 commit into
Conversation
Follow-up to langgenius#38359. The advanced-chat workflow run list loaded every Message matching the page's run ids and then de-duplicated to one per run in Python via setdefault, whose kept row depended on the database's row return order. Select a single message per run directly in SQL using DISTINCT ON (workflow_run_id), ordered by created_at desc then id desc, so extra Message rows are not fetched when a run has more than one message, and the aggregation rule (most recent message per run) is explicit and stable instead of order-dependent. Adds a test asserting the de-duplication is pushed into SQL.
Pyrefly Type Coverage
|
|
@wylswz following up on your review suggestion from #38359. Before finalizing, I'd like to confirm one design decision with you: this PR makes the "one message per run" selection explicit as the latest message per run: SELECT DISTINCT ON (workflow_run_id) ...
WHERE app_id = :app_id AND workflow_run_id IN (:run_ids)
ORDER BY workflow_run_id, created_at DESC, id DESCI went with Could you confirm that "latest message per run" is the aggregation rule you'd want here? If you'd rather use "earliest message" ( |
Summary
Follow-up to #38359, addressing the reviewer suggestion:
Before
get_paginate_advanced_chat_workflow_runsloaded every Message matching the page's run ids (workflow_run_id IN (...)) and de-duplicated to one per run in Python viasetdefault, so:After
Select a single message per run directly in SQL:
created_atdesc,iddesc as a tie-break), rather than order-dependent.Message.idis a UUID, so "lowest-id" is not time-ordered;created_atis used instead (there is a composite(created_at, id)index that supports this ordering).Notes
DISTINCT ONis PostgreSQL-specific, matching dify's supported database.Testing
test_get_paginate_advanced_chat_workflow_runs_dedups_one_message_per_run_in_sql: compiles the query for the PostgreSQL dialect and asserts it usesDISTINCT ONwith the explicitcreated_at DESCordering (de-duplication happens in SQL, not Python).ruff checkandruff formatare clean.