-
Notifications
You must be signed in to change notification settings - Fork 18
Document the streaming attachment transport (JS SDK) #563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -74,7 +74,7 @@ The **Attachment Table** is a local-only table that stores metadata about each f | |||||
| **Metadata stored:** | ||||||
| - `id` - Unique attachment identifier (UUID) | ||||||
| - `filename` - File name with extension (e.g., `photo-123.jpg`) | ||||||
| - `localUri` - Path to file in local storage | ||||||
| - `localUri` - Reference to the file in local storage. The format is platform-specific: a file path on native platforms and Node.js, or an internal `indexeddb://` reference on web | ||||||
| - `size` - File size in bytes | ||||||
| - `mediaType` - MIME type (e.g., `image/jpeg`) | ||||||
| - `state` - Current sync state (see states above) | ||||||
|
|
@@ -96,6 +96,8 @@ The **Remote Storage Adapter** is an interface you implement to connect PowerSyn | |||||
| - `downloadFile(attachment)` - Download file from cloud storage | ||||||
| - `deleteFile(attachment)` - Delete file from cloud storage | ||||||
|
|
||||||
| In the JavaScript/TypeScript SDK, apps that transfer large files can replace this adapter with a streaming transport; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). | ||||||
|
|
||||||
| **Common pattern:** | ||||||
| For security reasons, client-side implementations should use **signed URLs** | ||||||
| 1. Request a signed upload/download URL from your backend | ||||||
|
|
@@ -115,14 +117,16 @@ The **Local Storage Adapter** handles file persistence on the device. PowerSync | |||||
| - `fileExists(path)` - Check if file exists | ||||||
| - `getLocalUri(filename)` - Get full path for a filename | ||||||
|
|
||||||
| In the JavaScript/TypeScript SDK, adapters that can relocate a file without loading it into memory implement the `StreamingLocalStorageAdapter` subinterface, which adds `moveFile(sourceUri, targetUri)`. Configuring the queue with a streaming-capable adapter enables [`saveFileFromUri`](#upload-an-attachment). The Node.js, Expo, and React Native FS adapters are streaming-capable; the web IndexedDB adapter is not. | ||||||
|
|
||||||
| **Built-in adapters:** | ||||||
| - **IndexedDB** - For web browsers (`IndexDBFileSystemStorageAdapter`) | ||||||
| - **Node.js Filesystem** - For Node/Electron (`NodeFileSystemAdapter`) | ||||||
| - **React Native** - For React Native with Expo or bare React Native we have a dedicated package [(`@powersync/attachments-storage-react-native`)](https://github.com/powersync-ja/powersync-js/tree/main/packages/attachments-storage-react-native) | ||||||
| - **Native mobile storage** - For Flutter, Kotlin, Swift | ||||||
|
|
||||||
| <Warning> | ||||||
| The React Native local storage adapter requires Expo 54 or later. | ||||||
| The React Native local storage adapter requires Expo 54 or later. The Expo streaming transport requires Expo 56 or later. | ||||||
| </Warning> | ||||||
|
|
||||||
| ### Attachment Queue | ||||||
|
|
@@ -135,6 +139,8 @@ The **Attachment Queue** is the orchestrator that manages the entire attachment | |||||
| - **Performs cleanup** - Removes archived files that are no longer needed | ||||||
| - **Verifies integrity** - Checks local files exist and repairs inconsistencies | ||||||
|
|
||||||
| In the JavaScript/TypeScript SDK, remote transfers buffer each file through JS memory by default. Apps that handle large files can swap in a streaming transport adapter; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's also only mention this once (it's also mentioned under attachment table, I don't think it belongs there). |
||||||
|
|
||||||
| **Watched Attachments pattern:** | ||||||
| The queue needs to know which attachments exist in your data model. The `watchAttachments` function you provide monitors your data model and returns a list of attachment IDs that your app references. The queue compares this list with its internal attachment table to determine: | ||||||
| - **New attachments** - Download them | ||||||
|
|
@@ -653,6 +659,9 @@ const attachmentQueue = new AttachmentQueue({ | |||||
| db: db, // PowerSync database instance | ||||||
| localStorage, | ||||||
| remoteStorage, | ||||||
| // Or a transportAdapter in place of remoteStorage; see "Transferring | ||||||
| // Large Files Without Buffering" under Advanced Topics. | ||||||
| // transportAdapter, | ||||||
|
|
||||||
| // Define which attachments exist in your data model | ||||||
| watchAttachments: (onUpdate) => { | ||||||
|
|
@@ -1359,6 +1368,24 @@ async function uploadProfilePhoto(imageBlob: Blob, userId: string) { | |||||
| // 3. Update user record in same transaction | ||||||
| // 4. Automatically upload file in background | ||||||
| // 5. Update state to SYNCED when complete | ||||||
|
|
||||||
| // For files already on disk (e.g. a captured video or audio recording), | ||||||
| // saveFileFromUri queues the upload without reading the file into memory. | ||||||
| // Requires a streaming-capable local storage adapter (StreamingLocalStorageAdapter: | ||||||
| // Node.js, Expo, or React Native FS; not available on web). | ||||||
| async function attachRecording(localUri: string, recordingId: string) { | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we should add a second snippet for this (to have one with a blob and one streaming example) instead of combining it into one? |
||||||
| return attachmentQueue.saveFileFromUri({ | ||||||
| localUri, // path to the existing file | ||||||
| fileExtension: 'm4a', | ||||||
| mediaType: 'audio/mp4', | ||||||
| updateHook: async (tx, attachment) => { | ||||||
| await tx.execute( | ||||||
| 'UPDATE recordings SET audio_id = ? WHERE id = ?', | ||||||
| [attachment.id, recordingId] | ||||||
| ); | ||||||
| } | ||||||
| }); | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ```dart Flutter | ||||||
|
|
@@ -1488,6 +1515,8 @@ The `updateHook` parameter is the recommended way to link attachments to your da | |||||
| <CodeGroup> | ||||||
|
|
||||||
| ```typescript JavaScript/TypeScript | ||||||
| import { AttachmentState } from '@powersync/web'; | ||||||
|
|
||||||
| // Downloads happen automatically when watchAttachments references a file | ||||||
|
|
||||||
| async function getProfilePhotoUri(userId: string): Promise<string | null> { | ||||||
|
|
@@ -1509,42 +1538,61 @@ async function getProfilePhotoUri(userId: string): Promise<string | null> { | |||||
| return null; | ||||||
| } | ||||||
|
|
||||||
| if (attachment.state === 'SYNCED' && attachment.local_uri) { | ||||||
| if (attachment.state === AttachmentState.SYNCED && attachment.local_uri) { | ||||||
| return attachment.local_uri; | ||||||
| } | ||||||
|
|
||||||
| return null; | ||||||
| } | ||||||
|
|
||||||
| // Example: Display image in React with watch query | ||||||
| // Example: display the image in React on web. On web, local_uri is an | ||||||
| // internal indexeddb:// reference, so read the bytes through the local | ||||||
| // storage adapter and convert them to an object URL. On React Native and | ||||||
| // Node.js, local_uri is a real file path and can be used directly | ||||||
| // (e.g. <Image source={{ uri: localUri }} /> in React Native). | ||||||
| function ProfilePhoto({ userId }: { userId: string }) { | ||||||
| const [photoUri, setPhotoUri] = useState<string | null>(null); | ||||||
| const [photoUrl, setPhotoUrl] = useState<string | null>(null); | ||||||
|
|
||||||
| useEffect(() => { | ||||||
| let objectUrl: string | null = null; | ||||||
|
|
||||||
| const watch = db.watch( | ||||||
| `SELECT a.local_uri, a.state | ||||||
| `SELECT a.local_uri, a.media_type, a.state | ||||||
| FROM users u | ||||||
| LEFT JOIN attachments a ON a.id = u.photo_id | ||||||
| WHERE u.id = ?`, | ||||||
| [userId], | ||||||
| { | ||||||
| onResult: (result) => { | ||||||
| onResult: async (result) => { | ||||||
| const row = result.rows?._array[0]; | ||||||
| if (row?.state === 'SYNCED' && row?.local_uri) { | ||||||
| setPhotoUri(row.local_uri); | ||||||
| if (row?.state === AttachmentState.SYNCED && row?.local_uri) { | ||||||
| const buffer = await localStorage.readFile(row.local_uri); | ||||||
| const nextUrl = URL.createObjectURL( | ||||||
| new Blob([buffer], { type: row.media_type ?? 'image/jpeg' }) | ||||||
| ); | ||||||
| if (objectUrl) { | ||||||
| URL.revokeObjectURL(objectUrl); | ||||||
| } | ||||||
| objectUrl = nextUrl; | ||||||
| setPhotoUrl(nextUrl); | ||||||
| } | ||||||
| } | ||||||
| } | ||||||
| ); | ||||||
|
|
||||||
| return () => watch.close(); | ||||||
| return () => { | ||||||
| watch.close(); | ||||||
| if (objectUrl) { | ||||||
| URL.revokeObjectURL(objectUrl); | ||||||
| } | ||||||
| }; | ||||||
| }, [userId]); | ||||||
|
|
||||||
| if (!photoUri) { | ||||||
| if (!photoUrl) { | ||||||
| return <div>Loading photo...</div>; | ||||||
| } | ||||||
|
|
||||||
| return <img src={photoUri} alt="Profile" />; | ||||||
| return <img src={photoUrl} alt="Profile" />; | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
|
|
@@ -1788,6 +1836,10 @@ internal sealed class PhotoState | |||||
|
|
||||||
| </CodeGroup> | ||||||
|
|
||||||
| <Note> | ||||||
|
simolus3 marked this conversation as resolved.
|
||||||
| Web SDK only: `local_uri` is an internal `indexeddb://` reference rather than a URL the browser can load. Passing it directly to an `<img src>` fails with `net::ERR_UNKNOWN_URL_SCHEME`. Read the file through the local storage adapter and convert it to an object URL first, as shown in the JavaScript/TypeScript example above. Native SDKs return a real file path that can be used directly. | ||||||
| </Note> | ||||||
|
|
||||||
| ### Delete an Attachment | ||||||
|
|
||||||
| <CodeGroup> | ||||||
|
|
@@ -2151,6 +2203,64 @@ var queue = new AttachmentQueue(new AttachmentQueueOptions | |||||
| ``` | ||||||
| </CodeGroup> | ||||||
|
|
||||||
| ### Transferring Large Files Without Buffering | ||||||
|
khawarizmus marked this conversation as resolved.
|
||||||
|
|
||||||
| This section applies to the JavaScript/TypeScript SDK only. In the Dart and Kotlin SDKs, the remote storage interface is already stream-based (`Stream`/`Flow`), so transfers can avoid buffering. The Swift SDK currently receives files as `Data` and has no streaming equivalent yet. | ||||||
|
|
||||||
| By default, the queue transfers files by buffering them through JS memory: the entire file is read into an `ArrayBuffer` before it is handed to the remote storage adapter, and the reverse for downloads. This works well for small files but limits the practical attachment size, particularly in React Native, where a large video can exhaust the JS heap on lower-end devices. | ||||||
|
|
||||||
| To stream instead, configure the queue with a transport adapter in place of the remote storage adapter (you provide one or the other, not both; TypeScript enforces this). A transport owns all remote operations through three methods: | ||||||
|
|
||||||
| - `upload(attachment)` - Transfer the file at `attachment.localUri` to remote storage | ||||||
| - `download(attachment)` - Fetch the remote file into `attachment.localUri` (the queue assigns the destination path before the call) | ||||||
| - `delete(attachment)` - Remove the file from remote storage | ||||||
|
|
||||||
| The streaming-capable local storage adapters each create a ready-made transport through their `createTransportAdapter` method, so you don't implement these methods yourself: | ||||||
|
|
||||||
| - `ExpoFileSystemStorageAdapter` (`@powersync/attachments-storage-react-native`) - The transport streams with Expo's native `File.upload`/`File.downloadFileAsync`. Using the transport requires Expo 56 or later; using only the storage adapter requires Expo 54 | ||||||
| - `ReactNativeFileSystemStorageAdapter` (`@powersync/attachments-storage-react-native`) - The transport streams with `uploadFiles`/`downloadFile` from `@dr.pogodin/react-native-fs`, uploading as a raw binary `PUT` by default | ||||||
| - `NodeFileSystemAdapter` (`@powersync/node`) - The transport streams with `fetch` and Node.js filesystem streams | ||||||
|
|
||||||
| All three take the same options. `resolveUpload` and `resolveDownload` map an attachment to the HTTP request that transfers its bytes, typically a signed URL from your backend. `deleteFile` performs the remote delete, which is a plain remote call rather than a byte transfer. | ||||||
|
|
||||||
| <Note> | ||||||
| The transport API requires `@powersync/web` v2.2.0, `@powersync/react-native` v2.0.3, or `@powersync/node` v0.21.0 or later. React Native also requires `@powersync/attachments-storage-react-native` v0.1.0 or later. | ||||||
| </Note> | ||||||
|
|
||||||
| ```typescript | ||||||
| import { AttachmentQueue } from '@powersync/react-native'; | ||||||
| import { ExpoFileSystemStorageAdapter } from '@powersync/attachments-storage-react-native'; | ||||||
|
|
||||||
| const localStorage = new ExpoFileSystemStorageAdapter(); | ||||||
|
|
||||||
| // Streams bytes natively and owns upload/download/delete. No remoteStorage needed. | ||||||
| const transportAdapter = localStorage.createTransportAdapter({ | ||||||
| resolveUpload: async (attachment) => ({ | ||||||
| url: await getSignedUploadUrl(attachment.filename), // from your backend | ||||||
| mimeType: attachment.mediaType ?? 'application/octet-stream' | ||||||
| }), | ||||||
| resolveDownload: async (attachment) => ({ | ||||||
| url: await getSignedDownloadUrl(attachment.filename) | ||||||
| }), | ||||||
| deleteFile: async (attachment) => { | ||||||
| await deleteFromStorage(attachment.filename); // your backend or storage SDK call | ||||||
| } | ||||||
| }); | ||||||
|
|
||||||
| const attachmentQueue = new AttachmentQueue({ | ||||||
| db, | ||||||
| localStorage, | ||||||
| transportAdapter, // owns all remote operations; used in place of remoteStorage | ||||||
| watchAttachments: (onUpdate) => { | ||||||
| // Same as in Initialize Attachment Queue | ||||||
| } | ||||||
| }); | ||||||
|
|
||||||
| await attachmentQueue.startSync(); | ||||||
| ``` | ||||||
|
|
||||||
| For files your app produces on disk (camera captures, recordings, exports), combine a native transport with [`saveFileFromUri`](#upload-an-attachment). The file moves into managed storage and uploads without ever being read into memory; `saveFile` would read it into an `ArrayBuffer` just to write it back to disk. | ||||||
|
|
||||||
| ### Custom Storage Adapters | ||||||
|
|
||||||
| The following is an example of how to implement a custom storage adapter for IPFS: | ||||||
|
|
@@ -2355,6 +2465,43 @@ public sealed class IPFSStorageAdapter(HttpClient http) : IRemoteStorageAdapter | |||||
|
|
||||||
| </CodeGroup> | ||||||
|
|
||||||
| ### Custom Transport Adapters | ||||||
|
|
||||||
| In the JavaScript/TypeScript SDK, you can also implement [`AttachmentTransportAdapter`](#transferring-large-files-without-buffering) yourself. A custom remote storage adapter can already customize where bytes go, but it always receives the file as a full in-memory buffer. A transport works from the file's path instead, which enables: | ||||||
|
|
||||||
| - **Buffer-free transfers** - Let a native package transfer directly between the file system and the network, bypassing JS entirely, as the built-in transports do | ||||||
| - **Resumable transfers** - The queue retries a failed operation by calling the transport again on the next sync interval. A transport built on a resumable protocol such as [tus](https://tus.io) or S3 multipart upload can continue from the last confirmed offset instead of restarting from zero. Downloads can resume a partial file with HTTP `Range` requests | ||||||
| - **Encryption** - Encrypt files before upload and decrypt them after download for end-to-end encrypted attachments, without holding the whole file in memory | ||||||
|
|
||||||
| ```typescript | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Instead of this empty snippet, linking to the relevant interface in our tsdoc reference might be better. |
||||||
| import { | ||||||
| AttachmentRecord, | ||||||
| AttachmentTransportAdapter, | ||||||
| LocatedAttachmentRecord | ||||||
| } from '@powersync/web'; | ||||||
|
|
||||||
| class ResumableTransportAdapter implements AttachmentTransportAdapter { | ||||||
| async upload(attachment: LocatedAttachmentRecord): Promise<void> { | ||||||
| // attachment.localUri points at the source file. Transfer it to remote | ||||||
| // storage, e.g. in chunks that resume from the last confirmed offset | ||||||
| // if a previous attempt was interrupted. | ||||||
| } | ||||||
|
|
||||||
| async download(attachment: LocatedAttachmentRecord): Promise<void> { | ||||||
| // attachment.localUri is the destination path, assigned by the queue. | ||||||
| // Fetch the remote file into it. | ||||||
| } | ||||||
|
|
||||||
| async delete(attachment: AttachmentRecord): Promise<void> { | ||||||
| // Remove the file from remote storage. | ||||||
| } | ||||||
| } | ||||||
|
khawarizmus marked this conversation as resolved.
|
||||||
| ``` | ||||||
|
|
||||||
| For a working reference, see the built-in [`NodeFileSystemTransportAdapter`](https://github.com/powersync-ja/powersync-js/blob/main/packages/node/src/attachments/NodeFileSystemTransportAdapter.ts), which streams with `fetch` and Node.js filesystem streams. | ||||||
|
|
||||||
| Throwing from any method marks the operation as failed; the queue retries it on the next sync interval, subject to your [error handler](#error-handling). | ||||||
|
Comment on lines
+2502
to
+2503
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO this is obvious enough that it doesn't need to be mentioned.
Suggested change
|
||||||
|
|
||||||
| ### Verification and Recovery | ||||||
|
|
||||||
| `verifyAttachments()` is always called internally during `startSync()`. | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we calling it TypeScript SDK anywhere else?
This also still makes it sound like all other SDKs necessarily buffer the file, when that is not true for Kotlin and Dart.