Add S3 module with utilities

[dlindhol]

I've started making some utilities for working with S3 buckets, initially to support an S3 granule list adapter (coming soon). I decided to play with both the "regular" and the effectful approach. I have a number of TODOs for issues that we can discuss.

The testing also raises some questions. The first time I ran with localstack, it did take about a minute for it to start up. Subsequent runs were much faster. Not sure how it managed that. I wonder how it would play out in CI. I'd hate to slow down the tests too much. Note that I also tested with real buckets, breaking my "rule" for external dependencies.

[dlindhol]

Error in CI already...

[lindholc]

If you're only using delay, you only need Sync.

[dlindhol]

I was playing type tetris to get as generic of an F as possible. I wasn't sure if I needed to go for Async here when making an AsyncClient. I guess making is separate from the client-ing so this could be Sync. Anything more generic?

[lindholc]

I think there is some confusion on S3Client. S3Client is not pure, it's just a synchronous (read: blocking, since it needs to do network stuff) S3 client.

[dlindhol]

I was going for a simple way to get a client for those who don't want to deal with effects. I suppose that has no place here.

[dlindhol]

Do you see any value keeping an effectful synchronous client constructor?

[lindholc]

I dislike the "bag of methods" util objects.

I think it would be better to define a trait and wrap an S3 client implementation:

trait S3Client[F[_]] {
  def getKeys(...): Stream[F, String]
}

object S3Client {
  def fromAsyncClient[F[_]: Sync](client: S3AsyncClient): Resource[F, S3Client] =
    new S3Client {
      override def getKeys(...): Stream[F, String] = ???
    }
}

This has the benefit of being able to make a no-op implementation that just returns a list of canned values for testing. With S3Utils.getKeys you're forced to use a real S3 client.

[lindholc]

Could try s3proxy instead.