Skip to content

feat(streaming): add AWS Kinesis streaming provider - #8967

Open
dpbevin wants to merge 2 commits into
dotnet:mainfrom
dpbevin:dev/bevo/kinesis
Open

feat(streaming): add AWS Kinesis streaming provider#8967
dpbevin wants to merge 2 commits into
dotnet:mainfrom
dpbevin:dev/bevo/kinesis

Conversation

@dpbevin

@dpbevin dpbevin commented May 1, 2024

Copy link
Copy Markdown
Contributor

Relates to #8966

Adds a persistent stream provider for AWS Kinesis using the AWS .NET SDK directly, without requiring the JVM-based Kinesis Client Library. The provider includes client and silo configuration, shard discovery and topology monitoring, durable iterator recovery, provider-correlated tests, and Ministack-backed CI coverage.

This PR is organized as two commits: the exact grain-backed checkpointer commit reviewed in #10345, followed by the Kinesis provider commit. The PRs cannot form a native GitHub stack because their head branches live in different forks.

@scalalang2

scalalang2 commented May 15, 2024

Copy link
Copy Markdown
Contributor

this is what I've been waiting for. thanks :)

Comment thread src/Orleans.Streaming/Checkpointers/StreamCheckpointerGrain.cs
@dpbevin
dpbevin force-pushed the dev/bevo/kinesis branch 3 times, most recently from 30ea2c9 to be54597 Compare June 2, 2024 01:28
@dpbevin

dpbevin commented Jun 2, 2024

Copy link
Copy Markdown
Contributor Author

CI build failing due to missing package even though the package can be selected from that feed locally.

Unable to find package AWSSDK.Kinesis. No packages exist with this id in source(s): dotnet-public

@scalalang2

scalalang2 commented Jun 2, 2024

Copy link
Copy Markdown
Contributor

CI build failing due to missing package even though the package can be selected from that feed locally.

Unable to find package AWSSDK.Kinesis. No packages exist with this id in source(s): dotnet-public

@ReubenBond
Could you please check this out?

@idanbenyehoshua

Copy link
Copy Markdown

Just wanted to check in and see if there’s been any progress on this. It’s something I’ve been hoping to have for a while.
Currently, I'm running on AWS and using SQS. However, I have another project where Kinesis would be a better fit.

@jamescarter-le

Copy link
Copy Markdown
Contributor

I can see some recent activity on this PR, I'm currently using SQS, and a custom PR for SQS FIFO.

If someone is able to provide a list of outstanding tasks on this PR, I can provide resource to get this over the line.

@ADringer

Copy link
Copy Markdown

Has this been parked? For my project Kinesis integration would be very useful

@ReubenBond ReubenBond changed the title Initial cut of AWS Kinesis streaming feat(streaming): add AWS Kinesis streaming provider Aug 5, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3f5060d4-b229-476c-bdf3-9923ad459bb2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3f5060d4-b229-476c-bdf3-9923ad459bb2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants