Skip to content

coroutine: generate less code for each co_await - #322

Merged
travisdowns merged 2 commits into
redpanda-data:v26.3.xfrom
travisdowns:td-coawait-less-code-v26.3.x
Oct 22, 2026
Merged

travisdowns merged 2 commits into
redpanda-data:v26.3.xfrom
travisdowns:td-coawait-less-code-v26.3.x

Conversation

@travisdowns

@travisdowns travisdowns commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Cherry-pick of upstream scylladb#3775 (587e3b9), applied cleanly.

Also picks the upstream commit just before it, 8ffe7c4 ("test: file_utils: align DMA write sizes correctly"). Without it file_utils_test fails on current CI runners with EINVAL from O_DIRECT writes in test_tmp_file, test_tmp_dir and test_tmp_dir_with_path.

Every co_await of a future in a coroutine inlines the code of the future's awaiter, so a large application with many co_awaits compiles a lot of copies of it. ScyllaDB, for example, has about 7,000 co_awaits. This patch removes from it a check whose result is always the same, and code that never runs. This makes the generated code a bit smaller, compiles a bit faster, and executes fewer instructions for each co_await. The improvement is small: ScyllaDB's code gets 0.5-0.7% smaller (up to 2% in coroutine-heavy source files), it compiles 0.3% faster, and coroutine_perf's co_await benchmarks execute 1-9% fewer instructions. Nothing changes in the behavior of co_await, and nothing becomes harder to read.

The three changes are:

  1. awaiter::await_resume() called future::get(), which first calls wait() - in case get() is called in a seastar::thread, where the future may not be available yet - and then takes the result. But await_resume() is only called when the future is available: either await_ready() just returned true, or the coroutine was resumed by the future's promise, or (when a preemption check suspended the coroutine although the future was available) by the scheduler. So the check in wait() and the call to future_base::do_wait() behind it were dead code at every co_await. This patch makes await_resume() take the result directly, using get_available_state_ref().take() - exactly what get() does after wait(). The awaiter needs a friend declaration in future to call that private function.

  2. future_state_base::rethrow_exception() always throws (it calls std::rethrow_exception()), but wasn't declared [[noreturn]]. Because it is defined in future.cc, not in a header, the compiler couldn't see that it doesn't return - in any build mode (except perhaps when link-time optimization sees its definition, after the code was generated) - so it generated, at every co_await, code for the case that it returns: continuing to take the result after the exception was rethrown. This patch declares both overloads [[noreturn]], which removes that code. That code never ran: rethrow_exception() never returns.

  3. internal::get_need_preempt_var(), called by need_preempt() at every co_await (in await_ready()), wasn't declared noexcept. When Seastar is built as a shared library (SEASTAR_BUILD_SHARED_LIBS, e.g., in ScyllaDB's dev and debug modes) this function is defined out of line, so the compiler can't see that it doesn't throw, and because need_preempt() is noexcept, it generated at every call an exception landing pad that calls std::terminate(). This patch declares the function noexcept, which it is: it only returns a reference to a thread_local pointer. This removes the landing pads. When Seastar is built as a static library the function is inline, the compiler already knew it doesn't throw, and this change makes no difference.

Run-time effect:

The first change removes a load and a never-taken branch from every co_await; the other two only remove code that is never executed. The number of allocations and tasks doesn't change. We ran tests/perf/coroutine_perf in release mode, on one core, three times before and after this patch. The benchmarks that consist of nothing but co_awaits of futures execute fewer instructions per iteration (median of 3 runs), and also take fewer cycles (best of 3):

                         instructions          cycles
                        before  after        before  after
   coro_ready            233.1  212.1 (-9.0%)  40.9   39.1
   coro_ready_nested3    625.2  594.1 (-5.0%) 114.4  108.6
   coro_ready_nested5   1017.2  976.2 (-4.0%) 202.0  196.1
   coro_after_yield      925.2  895.2 (-3.2%) 202.6  193.8
   coro_per_field_10    2036.4 2006.4 (-1.5%) 400.4  400.5
   ready                 190.0  187.0 (-1.6%)  33.3   34.0
   nested_ready_chain    754.2  747.1 (-0.9%) 136.7  139.0

Benchmarks that use other awaiters (generators, try_future) or continuations were unaffected, within a few cycles of noise. In real code, a co_await is surrounded by much more work than in these benchmarks, so the run-time improvement is correspondingly smaller.

Code size and compilation:

We measured with ScyllaDB's build. In dev mode (-O2, Seastar as a shared library), the text of the scylla executable went down by 0.74% (587 KB, from 79.62 to 79.03 MB), and that of libseastar.so by 0.27%. Compiling ScyllaDB's 45 heaviest source files, the generated code went down by 0.6% and the number of instructions the compiler executed by 0.3%; files with many co_awaits gained the most, e.g., db/schema_applier.cc 1.8% less code and 0.8% fewer compiler instructions, service/storage_service.cc 1.5% and 0.9%. The compiler's peak memory use didn't change.

In release mode (-O3, Seastar as a static library, so the third change doesn't apply), we compiled a sample of 24 source files - 12 with many co_awaits and 12 chosen at random - without a PGO profile: their code went down by 0.5% (from 0.2% to 1.0% for each of the files with many co_awaits), and the compiler's instructions by 0.3%. (With a PGO profile recorded before this patch, the coroutines' code is bigger until the profile is recorded again - as with any change to the code that a profile was recorded with.)

Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrow optimizations preserve readiness, result consumption, and exception propagation, with no unresolved issues identified.

No open issues 🎉

What changed in this PR

This PR reduces code generated for future co_await operations by removing redundant waiting checks and making existing exception guarantees visible to the compiler, while preserving coroutine behavior.

Changes:

  • Extract results directly from ready futures in both awaiter variants.
  • Mark exception-rethrowing functions [[noreturn]].
  • Mark the preemption-monitor accessor noexcept in shared and static builds.
File Description
src/​core/​reactor.cc Adds noexcept to the shared-build preemption accessor definition.
include/​seastar/​core/​preempt.hh Adds matching noexcept guarantees to both accessor variants.
include/​seastar/​core/​future.hh Adds [[noreturn]] annotations and grants awaiters access to future state.
include/​seastar/​core/​coroutine.hh Bypasses redundant waiting when consuming ready future results.

Review effort: Balanced

Muhammad-Ali-Farhan and others added 2 commits October 22, 2026 09:48
memory_dma_alignment() describes the alignment requirement for DMA buffer addresses, not write lengths. On systems where statx reports a smaller memory alignment than the disk write alignment, using it alone to size DMA writes can result in EINVAL from O_DIRECT writes.

Use the larger of the memory and disk-write alignments as the write-size quantum. Both are powers of two, so generated sizes remain valid for temporary_buffer::aligned() and satisfy the disk-write length requirement, while buffer addresses continue to use memory_dma_alignment().

The full dev test suite passes with 95/95 tests.

Signed-off-by: Muhammad Ali Farhan <moh.alifarhan@gmail.com>
(cherry picked from commit 8ffe7c4)
Every co_await of a future in a coroutine inlines the code of the
future's awaiter, so a large application with many co_awaits compiles a
lot of copies of it. ScyllaDB, for example, has about 7,000 co_awaits.
This patch removes from it a check whose result is always the same, and
code that never runs. This makes the generated code a bit smaller,
compiles a bit faster, and executes fewer instructions for each
co_await. The improvement is small: ScyllaDB's code gets 0.5-0.7%
smaller (up to 2% in coroutine-heavy source files), it compiles 0.3%
faster, and coroutine_perf's co_await benchmarks execute 1-9% fewer
instructions. Nothing changes in the behavior of co_await, and nothing
becomes harder to read.

The three changes are:

1. awaiter::await_resume() called future::get(), which first calls
   wait() - in case get() is called in a seastar::thread, where the
   future may not be available yet - and then takes the result. But
   await_resume() is only called when the future is available: either
   await_ready() just returned true, or the coroutine was resumed by
   the future's promise, or (when a preemption check suspended the
   coroutine although the future was available) by the scheduler. So
   the check in wait() and the call to future_base::do_wait() behind it
   were dead code at every co_await. This patch makes await_resume()
   take the result directly, using get_available_state_ref().take() -
   exactly what get() does after wait(). The awaiter needs a friend
   declaration in future to call that private function.

2. future_state_base::rethrow_exception() always throws (it calls
   std::rethrow_exception()), but wasn't declared [[noreturn]]. Because
   it is defined in future.cc, not in a header, the compiler couldn't
   see that it doesn't return - in any build mode (except perhaps when
   link-time optimization sees its definition, after the code was
   generated) - so it generated, at every co_await, code for the case
   that it returns: continuing to take the result after the exception
   was rethrown. This patch declares both overloads [[noreturn]], which
   removes that code. That code never ran: rethrow_exception() never
   returns.

3. internal::get_need_preempt_var(), called by need_preempt() at every
   co_await (in await_ready()), wasn't declared noexcept. When Seastar
   is built as a shared library (SEASTAR_BUILD_SHARED_LIBS, e.g., in
   ScyllaDB's dev and debug modes) this function is defined out of line,
   so the compiler can't see that it doesn't throw, and because
   need_preempt() is noexcept, it generated at every call an exception
   landing pad that calls std::terminate(). This patch declares the
   function noexcept, which it is: it only returns a reference to a
   thread_local pointer. This removes the landing pads. When Seastar is
   built as a static library the function is inline, the compiler
   already knew it doesn't throw, and this change makes no difference.

Run-time effect:

The first change removes a load and a never-taken branch from every
co_await; the other two only remove code that is never executed. The
number of allocations and tasks doesn't change. We ran
tests/perf/coroutine_perf in release mode, on one core, three times
before and after this patch. The benchmarks that consist of nothing but
co_awaits of futures execute fewer instructions per iteration (median of
3 runs), and also take fewer cycles (best of 3):

                         instructions          cycles
                        before  after        before  after
   coro_ready            233.1  212.1 (-9.0%)  40.9   39.1
   coro_ready_nested3    625.2  594.1 (-5.0%) 114.4  108.6
   coro_ready_nested5   1017.2  976.2 (-4.0%) 202.0  196.1
   coro_after_yield      925.2  895.2 (-3.2%) 202.6  193.8
   coro_per_field_10    2036.4 2006.4 (-1.5%) 400.4  400.5
   ready                 190.0  187.0 (-1.6%)  33.3   34.0
   nested_ready_chain    754.2  747.1 (-0.9%) 136.7  139.0

Benchmarks that use other awaiters (generators, try_future) or
continuations were unaffected, within a few cycles of noise. In real
code, a co_await is surrounded by much more work than in these
benchmarks, so the run-time improvement is correspondingly smaller.

Code size and compilation:

We measured with ScyllaDB's build. In dev mode (-O2, Seastar as a shared
library), the text of the scylla executable went down by 0.74% (587 KB,
from 79.62 to 79.03 MB), and that of libseastar.so by 0.27%. Compiling
ScyllaDB's 45 heaviest source files, the generated code went down by
0.6% and the number of instructions the compiler executed by 0.3%;
files with many co_awaits gained the most, e.g., db/schema_applier.cc
1.8% less code and 0.8% fewer compiler instructions,
service/storage_service.cc 1.5% and 0.9%. The compiler's peak memory
use didn't change.

In release mode (-O3, Seastar as a static library, so the third change
doesn't apply), we compiled a sample of 24 source files - 12 with many
co_awaits and 12 chosen at random - without a PGO profile: their code
went down by 0.5% (from 0.2% to 1.0% for each of the files with many
co_awaits), and the compiler's instructions by 0.3%. (With a PGO profile
recorded before this patch, the coroutines' code is bigger until the
profile is recorded again - as with any change to the code that a
profile was recorded with.)

(cherry picked from commit 587e3b9)
@travisdowns
travisdowns force-pushed the td-coawait-less-code-v26.3.x branch from 82fe25e to 6bc39be Compare October 22, 2026 12:48
@travisdowns
travisdowns merged commit ace1374 into redpanda-data:v26.3.x Oct 22, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants