New Datastore Interfaces

[whyrusleeping]

The interfaces in go-datastore have been needing updating for a while. This came up in #28 so i decided to start the conversation up. Lets collect all these ideas together here so we can move forward with this.

Interface additions i've been looking for involve streaming interfaces:

type StreamingDatastore interface {
    // GetR returns a reader to read the value directly from the source (sans buffering)
    GetR(k ds.Key) (io.Reader, error)

    // PutW returns a new put writer that can be used to stream a value 
    // into the datastore
    PutW() (WritePutter, error)
}

// WritePutter is used to stream values into the datastore. Since the caller may not 
// know the key they want to store this value as until the write is complete (content 
// addressing) the key is set after the write is complete
type WritePutter interface{
    io.Writer
    FinishWithKey(k ds.Key) error
}

Also, for queries and maybe 'gets' in general, it would be nice to have a readonly view interface. Many storage backends (bolt in particular) allow you to read and operate on the data within the datastore without making a copy (i.e. using the datastores memory) within a closure. This would be really useful for things like 'load this protobuf object from disk' where we don't need our own copy of the bytes.

// A datastore implementing ReadClosurer allows passing a closure to operate
// on a readonly view of the data in question
type ReadClosurer interface {
    GetFunc(k ds.Key, func(i interface{}) error) error
}

Alternatively, this could be done with an output parameter and interfaces:

type OutParamer interface{
    GetOut(k ds.Key, out DSUnmarshaler) error
}

type DSUnmarshaler interface {
  DSUnmarshal(val []byte) error
}

This would behave similarly to the json.Unmarshaler interfaces.

[kevina]

@whyrusleeping

// PutW returns a new put writer that can be used to stream a value 
// into the datastore
PutW() (io.WriteCloser, error)

Did you mean to return WritePutter here?

[whyrusleeping]

@kevina ah, yeah. my bad

[Stebalien]

Can we also stop lying and just use byte slices/readers instead of interface{}?

[whyrusleeping]

That would be nice, yes. Acceptings []byte, io.Reader and io.WriterTo would be great (with the last one being the most efficient)