Demonstrate FrontierInterest::IfCapability#687
Draft
frankmcsherry wants to merge 4 commits intomaster-nextfrom
Draft
Demonstrate FrontierInterest::IfCapability#687frankmcsherry wants to merge 4 commits intomaster-nextfrom
FrontierInterest::IfCapability#687frankmcsherry wants to merge 4 commits intomaster-nextfrom
Conversation
Member
Author
|
One thought while reading through the diff is that it should be possible, and is probably worth it, to factor the core operator logic into something parameterized by a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR explores an alternate form of
arrange_corethat produces a trace handle that .. 1. cannot be shared across dataflows, but 2. only needs to be activated on frontier transitions if it holds a capability. At the moment, this is presented as a new method that needs to be opted in to, so .. potentially not great risk to merge.Both
TraceAgentandTraceWriterare refactored to extract the shared queue that communicates batches and progress statements, wrapped aroundInnertypes that do everything else. Without the shared queue, you have anArranged<G,Tr>that can be used in the local dataflow, just without animportmethod to bring it in to other dataflows.There is a new example
event_driven.rsborrowed from timely dataflow, in which we constructddataflows each of one input, a region containing anarrangeoperator andkfilter operators, with a probe at the end outside the region. The experiment repeatedly adds an update to one input, and ticks all inputs.Using the existing
arrange_corewhich has theAlwaysfrontier interest:Using the dataflow-local operator with the same arguments:
The "speed-up" takes the iteration time from ~360ms to 4ms.