Repository navigation
coroutine: generate less code for each co_await - #322
Merged
travisdowns merged 2 commits intoOct 22, 2026
Merged
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The narrow optimizations preserve readiness, result consumption, and exception propagation, with no unresolved issues identified.
No open issues 🎉
What changed in this PR
This PR reduces code generated for future co_await operations by removing redundant waiting checks and making existing exception guarantees visible to the compiler, while preserving coroutine behavior.
Changes:
- Extract results directly from ready futures in both awaiter variants.
- Mark exception-rethrowing functions
[[noreturn]]. - Mark the preemption-monitor accessor
noexceptin shared and static builds.
| File | Description |
|---|---|
| src/core/reactor.cc | Adds noexcept to the shared-build preemption accessor definition. |
| include/seastar/core/preempt.hh | Adds matching noexcept guarantees to both accessor variants. |
| include/seastar/core/future.hh | Adds [[noreturn]] annotations and grants awaiters access to future state. |
| include/seastar/core/coroutine.hh | Bypasses redundant waiting when consuming ready future results. |
Review effort: Balanced
memory_dma_alignment() describes the alignment requirement for DMA buffer addresses, not write lengths. On systems where statx reports a smaller memory alignment than the disk write alignment, using it alone to size DMA writes can result in EINVAL from O_DIRECT writes. Use the larger of the memory and disk-write alignments as the write-size quantum. Both are powers of two, so generated sizes remain valid for temporary_buffer::aligned() and satisfy the disk-write length requirement, while buffer addresses continue to use memory_dma_alignment(). The full dev test suite passes with 95/95 tests. Signed-off-by: Muhammad Ali Farhan <moh.alifarhan@gmail.com> (cherry picked from commit 8ffe7c4)
Every co_await of a future in a coroutine inlines the code of the
future's awaiter, so a large application with many co_awaits compiles a
lot of copies of it. ScyllaDB, for example, has about 7,000 co_awaits.
This patch removes from it a check whose result is always the same, and
code that never runs. This makes the generated code a bit smaller,
compiles a bit faster, and executes fewer instructions for each
co_await. The improvement is small: ScyllaDB's code gets 0.5-0.7%
smaller (up to 2% in coroutine-heavy source files), it compiles 0.3%
faster, and coroutine_perf's co_await benchmarks execute 1-9% fewer
instructions. Nothing changes in the behavior of co_await, and nothing
becomes harder to read.
The three changes are:
1. awaiter::await_resume() called future::get(), which first calls
wait() - in case get() is called in a seastar::thread, where the
future may not be available yet - and then takes the result. But
await_resume() is only called when the future is available: either
await_ready() just returned true, or the coroutine was resumed by
the future's promise, or (when a preemption check suspended the
coroutine although the future was available) by the scheduler. So
the check in wait() and the call to future_base::do_wait() behind it
were dead code at every co_await. This patch makes await_resume()
take the result directly, using get_available_state_ref().take() -
exactly what get() does after wait(). The awaiter needs a friend
declaration in future to call that private function.
2. future_state_base::rethrow_exception() always throws (it calls
std::rethrow_exception()), but wasn't declared [[noreturn]]. Because
it is defined in future.cc, not in a header, the compiler couldn't
see that it doesn't return - in any build mode (except perhaps when
link-time optimization sees its definition, after the code was
generated) - so it generated, at every co_await, code for the case
that it returns: continuing to take the result after the exception
was rethrown. This patch declares both overloads [[noreturn]], which
removes that code. That code never ran: rethrow_exception() never
returns.
3. internal::get_need_preempt_var(), called by need_preempt() at every
co_await (in await_ready()), wasn't declared noexcept. When Seastar
is built as a shared library (SEASTAR_BUILD_SHARED_LIBS, e.g., in
ScyllaDB's dev and debug modes) this function is defined out of line,
so the compiler can't see that it doesn't throw, and because
need_preempt() is noexcept, it generated at every call an exception
landing pad that calls std::terminate(). This patch declares the
function noexcept, which it is: it only returns a reference to a
thread_local pointer. This removes the landing pads. When Seastar is
built as a static library the function is inline, the compiler
already knew it doesn't throw, and this change makes no difference.
Run-time effect:
The first change removes a load and a never-taken branch from every
co_await; the other two only remove code that is never executed. The
number of allocations and tasks doesn't change. We ran
tests/perf/coroutine_perf in release mode, on one core, three times
before and after this patch. The benchmarks that consist of nothing but
co_awaits of futures execute fewer instructions per iteration (median of
3 runs), and also take fewer cycles (best of 3):
instructions cycles
before after before after
coro_ready 233.1 212.1 (-9.0%) 40.9 39.1
coro_ready_nested3 625.2 594.1 (-5.0%) 114.4 108.6
coro_ready_nested5 1017.2 976.2 (-4.0%) 202.0 196.1
coro_after_yield 925.2 895.2 (-3.2%) 202.6 193.8
coro_per_field_10 2036.4 2006.4 (-1.5%) 400.4 400.5
ready 190.0 187.0 (-1.6%) 33.3 34.0
nested_ready_chain 754.2 747.1 (-0.9%) 136.7 139.0
Benchmarks that use other awaiters (generators, try_future) or
continuations were unaffected, within a few cycles of noise. In real
code, a co_await is surrounded by much more work than in these
benchmarks, so the run-time improvement is correspondingly smaller.
Code size and compilation:
We measured with ScyllaDB's build. In dev mode (-O2, Seastar as a shared
library), the text of the scylla executable went down by 0.74% (587 KB,
from 79.62 to 79.03 MB), and that of libseastar.so by 0.27%. Compiling
ScyllaDB's 45 heaviest source files, the generated code went down by
0.6% and the number of instructions the compiler executed by 0.3%;
files with many co_awaits gained the most, e.g., db/schema_applier.cc
1.8% less code and 0.8% fewer compiler instructions,
service/storage_service.cc 1.5% and 0.9%. The compiler's peak memory
use didn't change.
In release mode (-O3, Seastar as a static library, so the third change
doesn't apply), we compiled a sample of 24 source files - 12 with many
co_awaits and 12 chosen at random - without a PGO profile: their code
went down by 0.5% (from 0.2% to 1.0% for each of the files with many
co_awaits), and the compiler's instructions by 0.3%. (With a PGO profile
recorded before this patch, the coroutines' code is bigger until the
profile is recorded again - as with any change to the code that a
profile was recorded with.)
(cherry picked from commit 587e3b9)
travisdowns
force-pushed
the
td-coawait-less-code-v26.3.x
branch
from
October 22, 2026 12:48
82fe25e to
6bc39be
Compare
StephanDollberg
approved these changes
Oct 22, 2026
StephanDollberg
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of upstream scylladb#3775 (587e3b9), applied cleanly.
Also picks the upstream commit just before it, 8ffe7c4 ("test: file_utils: align DMA write sizes correctly"). Without it file_utils_test fails on current CI runners with EINVAL from O_DIRECT writes in test_tmp_file, test_tmp_dir and test_tmp_dir_with_path.
Every co_await of a future in a coroutine inlines the code of the future's awaiter, so a large application with many co_awaits compiles a lot of copies of it. ScyllaDB, for example, has about 7,000 co_awaits. This patch removes from it a check whose result is always the same, and code that never runs. This makes the generated code a bit smaller, compiles a bit faster, and executes fewer instructions for each co_await. The improvement is small: ScyllaDB's code gets 0.5-0.7% smaller (up to 2% in coroutine-heavy source files), it compiles 0.3% faster, and coroutine_perf's co_await benchmarks execute 1-9% fewer instructions. Nothing changes in the behavior of co_await, and nothing becomes harder to read.
The three changes are:
awaiter::await_resume() called future::get(), which first calls wait() - in case get() is called in a seastar::thread, where the future may not be available yet - and then takes the result. But await_resume() is only called when the future is available: either await_ready() just returned true, or the coroutine was resumed by the future's promise, or (when a preemption check suspended the coroutine although the future was available) by the scheduler. So the check in wait() and the call to future_base::do_wait() behind it were dead code at every co_await. This patch makes await_resume() take the result directly, using get_available_state_ref().take() - exactly what get() does after wait(). The awaiter needs a friend declaration in future to call that private function.
future_state_base::rethrow_exception() always throws (it calls std::rethrow_exception()), but wasn't declared [[noreturn]]. Because it is defined in future.cc, not in a header, the compiler couldn't see that it doesn't return - in any build mode (except perhaps when link-time optimization sees its definition, after the code was generated) - so it generated, at every co_await, code for the case that it returns: continuing to take the result after the exception was rethrown. This patch declares both overloads [[noreturn]], which removes that code. That code never ran: rethrow_exception() never returns.
internal::get_need_preempt_var(), called by need_preempt() at every co_await (in await_ready()), wasn't declared noexcept. When Seastar is built as a shared library (SEASTAR_BUILD_SHARED_LIBS, e.g., in ScyllaDB's dev and debug modes) this function is defined out of line, so the compiler can't see that it doesn't throw, and because need_preempt() is noexcept, it generated at every call an exception landing pad that calls std::terminate(). This patch declares the function noexcept, which it is: it only returns a reference to a thread_local pointer. This removes the landing pads. When Seastar is built as a static library the function is inline, the compiler already knew it doesn't throw, and this change makes no difference.
Run-time effect:
The first change removes a load and a never-taken branch from every co_await; the other two only remove code that is never executed. The number of allocations and tasks doesn't change. We ran tests/perf/coroutine_perf in release mode, on one core, three times before and after this patch. The benchmarks that consist of nothing but co_awaits of futures execute fewer instructions per iteration (median of 3 runs), and also take fewer cycles (best of 3):
Benchmarks that use other awaiters (generators, try_future) or continuations were unaffected, within a few cycles of noise. In real code, a co_await is surrounded by much more work than in these benchmarks, so the run-time improvement is correspondingly smaller.
Code size and compilation:
We measured with ScyllaDB's build. In dev mode (-O2, Seastar as a shared library), the text of the scylla executable went down by 0.74% (587 KB, from 79.62 to 79.03 MB), and that of libseastar.so by 0.27%. Compiling ScyllaDB's 45 heaviest source files, the generated code went down by 0.6% and the number of instructions the compiler executed by 0.3%; files with many co_awaits gained the most, e.g., db/schema_applier.cc 1.8% less code and 0.8% fewer compiler instructions, service/storage_service.cc 1.5% and 0.9%. The compiler's peak memory use didn't change.
In release mode (-O3, Seastar as a static library, so the third change doesn't apply), we compiled a sample of 24 source files - 12 with many co_awaits and 12 chosen at random - without a PGO profile: their code went down by 0.5% (from 0.2% to 1.0% for each of the files with many co_awaits), and the compiler's instructions by 0.3%. (With a PGO profile recorded before this patch, the coroutines' code is bigger until the profile is recorded again - as with any change to the code that a profile was recorded with.)