Skip to content

Investigate whether we can use auto-commit in arroyo #414

Description

@untitaker

rdkafka has two options related to auto-committing:

  • enable.auto.commit -- commit staged offsets automatically
  • enable.auto.commit.store -- auto-stage offsets that have been read

we could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using enable.auto.commit to write them out periodically. this way we would not have to explicitly debounce commits.

arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.

i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions