Skip to content

Commit 8c7a2f1

Browse files
icecrasher321icecrasher321
authored
fix(uploads): require an explicit byte ceiling on workspace-file downloads (#6985)
* fix(uploads): require an explicit byte ceiling on workspace-file downloads Workspace files are admitted at 5 GB because they stream straight to object storage, but a tool that pulls one back to hand it to a third party buffers the whole thing in the shared app process. maxBytes was optional on every download helper, so 51 call sites had silently inherited "unbounded". Make maxBytes required on all five entry points so a new call site cannot inherit it again, and give each existing site a ceiling: the destination's own documented limit where the route already declared one, otherwise the 100 MB this codebase already uses for buffered work. Multi-attachment routes were the worse case — Gmail, Outlook, SendGrid and SMTP downloaded every attachment via Promise.all and only summed the sizes once they were all resident, so their pre-check on declared sizes protected nothing. Add downloadServableFilesWithinBudget, which walks the list against a shrinking budget, and use the same running budget in Slack, Jira, Discord and Quiver. * fix(uploads): bound Sim-page asset inlining before the bytes are resident The ceiling on the rendered page checked the finished document, by which point renderSimPageDocumentWithAssets had already downloaded every referenced image concurrently with no per-download limit and base64-inlined them — so the allocation the check exists to prevent had already happened. Pick the inline set from recorded sizes before fetching anything, against a per-document budget as well as the existing per-image one, and give each download its own ceiling in case a row understates its object. An image that does not fit keeps its URL reference, exactly as an oversized one already did. * improvement(uploads): charge the page-render budget by delivered bytes Planning the inline set from recorded sizes left the aggregate ceiling resting on metadata being accurate, and needed a paragraph explaining why that was safe. Downloading one image at a time and subtracting what each download actually returned needs no such argument: the budget cannot be exceeded whatever a row says, and the peak is one image rather than the sum of them. Also drop two rough edges the first pass left behind — an SFTP total-size check that became unreachable once the download carried the remaining budget, and a Dataverse error helper whose optional size argument existed only to paper over one caller that had not passed it.
1 parent 28cfa43 commit 8c7a2f1

54 files changed

Lines changed: 862 additions & 308 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apps/sim/app/api/tools/agiloft/attach/route.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ import { getValidationErrorMessage, parseRequest } from '@/lib/api/server'
66
import { checkInternalAuth } from '@/lib/auth/hybrid'
77
import { secureFetchWithPinnedIP } from '@/lib/core/security/input-validation.server'
88
import { generateRequestId } from '@/lib/core/utils/request'
9+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
910
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
11+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
1012
import type { RawFileInput } from '@/lib/uploads/utils/file-schemas'
1113
import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils'
1214
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
@@ -75,13 +77,18 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
7577

7678
let fileBuffer: Buffer
7779
try {
78-
const servable = await downloadServableFileFromStorage(userFile, requestId, logger)
80+
const servable = await downloadServableFileFromStorage(userFile, requestId, logger, {
81+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
82+
})
7983
fileBuffer = servable.buffer
8084
} catch (error) {
8185
const notReady = docNotReadyResponse(error)
8286
if (notReady) return notReady
8387
logger.error(`[${requestId}] Failed to download file from storage:`, error)
84-
return NextResponse.json({ success: false, error: toError(error).message }, { status: 500 })
88+
return NextResponse.json(
89+
{ success: false, error: toError(error).message },
90+
{ status: isPayloadSizeLimitError(error) ? 413 : 500 }
91+
)
8592
}
8693

8794
const resolvedFileName = data.fileName || userFile.name || 'attachment'

‎apps/sim/app/api/tools/box/upload/route.ts‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@ import { boxUploadContract } from '@/lib/api/contracts/storage-transfer'
55
import { parseRequest } from '@/lib/api/server'
66
import { checkInternalAuth } from '@/lib/auth/hybrid'
77
import { generateRequestId } from '@/lib/core/utils/request'
8+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
89
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
10+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
911
import { processFilesToUserFiles, type RawFileInput } from '@/lib/uploads/utils/file-utils'
1012
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
1113
import { docNotReadyResponse } from '@/lib/uploads/utils/servable-file-response'
@@ -55,14 +57,16 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
5557
const denied = await assertToolFileAccess(userFile.key, authResult.userId, requestId, logger)
5658
if (denied) return denied
5759
try {
58-
const result = await downloadServableFileFromStorage(userFile, requestId, logger)
60+
const result = await downloadServableFileFromStorage(userFile, requestId, logger, {
61+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
62+
})
5963
fileBuffer = result.buffer
6064
} catch (error) {
6165
const notReady = docNotReadyResponse(error)
6266
if (notReady) return notReady
6367
return NextResponse.json(
6468
{ success: false, error: getErrorMessage(error, 'Failed to download file') },
65-
{ status: 500 }
69+
{ status: isPayloadSizeLimitError(error) ? 413 : 500 }
6670
)
6771
}
6872
fileName = validatedData.fileName || userFile.name

‎apps/sim/app/api/tools/brex/upload-receipt/route.test.ts‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ vi.mock('@/app/api/files/authorization', () => ({
2727
assertToolFileAccess: mockAssertToolFileAccess,
2828
}))
2929

30+
import { PayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
3031
import { POST } from '@/app/api/tools/brex/upload-receipt/route'
3132

3233
const mockFetch = vi.fn()
@@ -194,11 +195,25 @@ describe('POST /api/tools/brex/upload-receipt', () => {
194195
expect(inputValidationMockFns.mockSecureFetchWithPinnedIP).not.toHaveBeenCalled()
195196
})
196197

198+
it('asks the downloader for at most 50 MB', async () => {
199+
await POST(createMockRequest('POST', baseBody))
200+
201+
expect(mockDownloadFileFromStorage).toHaveBeenCalledWith(
202+
expect.anything(),
203+
expect.any(String),
204+
expect.anything(),
205+
{ maxBytes: 50 * 1024 * 1024 }
206+
)
207+
})
208+
197209
it('rejects files over the 50 MB limit', async () => {
198-
mockDownloadFileFromStorage.mockResolvedValueOnce({
199-
buffer: Buffer.alloc(50 * 1024 * 1024 + 1),
200-
contentType: 'application/pdf',
201-
})
210+
mockDownloadFileFromStorage.mockRejectedValueOnce(
211+
new PayloadSizeLimitError({
212+
label: 'storage file download',
213+
maxBytes: 50 * 1024 * 1024,
214+
observedBytes: 50 * 1024 * 1024 + 1,
215+
})
216+
)
202217

203218
const response = await POST(createMockRequest('POST', baseBody))
204219
expect(response.status).toBe(400)

‎apps/sim/app/api/tools/brex/upload-receipt/route.ts‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import {
99
validateUrlWithDNS,
1010
} from '@/lib/core/security/input-validation.server'
1111
import { generateRequestId } from '@/lib/core/utils/request'
12+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
1213
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
1314
import { processFilesToUserFiles, type RawFileInput } from '@/lib/uploads/utils/file-utils'
1415
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
@@ -51,23 +52,25 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
5152

5253
let fileBuffer: Buffer
5354
try {
54-
const resolved = await downloadServableFileFromStorage(userFile, requestId, logger)
55+
const resolved = await downloadServableFileFromStorage(userFile, requestId, logger, {
56+
maxBytes: MAX_RECEIPT_SIZE_BYTES,
57+
})
5558
fileBuffer = resolved.buffer
5659
} catch (error) {
5760
const notReady = docNotReadyResponse(error)
5861
if (notReady) return notReady
62+
if (isPayloadSizeLimitError(error)) {
63+
return NextResponse.json(
64+
{ success: false, error: 'Receipt file exceeds the 50 MB limit' },
65+
{ status: 400 }
66+
)
67+
}
5968
logger.error(`[${requestId}] Failed to download receipt file:`, error)
6069
return NextResponse.json(
6170
{ success: false, error: getErrorMessage(error, 'Unknown error') },
6271
{ status: 500 }
6372
)
6473
}
65-
if (fileBuffer.length > MAX_RECEIPT_SIZE_BYTES) {
66-
return NextResponse.json(
67-
{ success: false, error: 'Receipt file exceeds the 50 MB limit' },
68-
{ status: 400 }
69-
)
70-
}
7174

7275
const effectiveReceiptName = receiptName || userFile.name
7376
const endpoint = expenseId

‎apps/sim/app/api/tools/confluence/upload-attachment/route.ts‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@ import { confluenceUploadAttachmentContract } from '@/lib/api/contracts/selector
55
import { parseRequest } from '@/lib/api/server'
66
import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
77
import { validateAlphanumericId, validateJiraCloudId } from '@/lib/core/security/input-validation'
8+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
89
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
10+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
911
import { processSingleFileToUserFile, type RawFileInput } from '@/lib/uploads/utils/file-utils'
1012
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
1113
import { docNotReadyResponse } from '@/lib/uploads/utils/servable-file-response'
@@ -94,7 +96,14 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
9496
let fileBuffer: Buffer
9597
let resolvedContentType: string
9698
try {
97-
const servable = await downloadServableFileFromStorage(userFile, 'confluence-upload', logger)
99+
const servable = await downloadServableFileFromStorage(
100+
userFile,
101+
'confluence-upload',
102+
logger,
103+
{
104+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
105+
}
106+
)
98107
fileBuffer = servable.buffer
99108
resolvedContentType = servable.contentType
100109
} catch (error) {
@@ -105,7 +114,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
105114
{
106115
error: `Failed to download file: ${getErrorMessage(error, 'Unknown error')}`,
107116
},
108-
{ status: 500 }
117+
{ status: isPayloadSizeLimitError(error) ? 413 : 500 }
109118
)
110119
}
111120

‎apps/sim/app/api/tools/daytona/upload/route.ts‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { daytonaUploadFileContract } from '@/lib/api/contracts/tools/daytona'
55
import { parseRequest } from '@/lib/api/server'
66
import { checkInternalAuth } from '@/lib/auth/hybrid'
77
import { generateRequestId } from '@/lib/core/utils/request'
8+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
89
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
910
import { processFilesToUserFiles, type RawFileInput } from '@/lib/uploads/utils/file-utils'
1011
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
@@ -62,11 +63,19 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
6263

6364
logger.info(`[${requestId}] Downloading file: ${userFile.name} (${userFile.size} bytes)`)
6465
try {
65-
const servable = await downloadServableFileFromStorage(userFile, requestId, logger)
66+
const servable = await downloadServableFileFromStorage(userFile, requestId, logger, {
67+
maxBytes: MAX_UPLOAD_SIZE_BYTES,
68+
})
6669
fileBuffer = servable.buffer
6770
} catch (error) {
6871
const notReady = docNotReadyResponse(error)
6972
if (notReady) return notReady
73+
if (isPayloadSizeLimitError(error)) {
74+
return NextResponse.json(
75+
{ success: false, error: 'File exceeds upload limit of 100MB' },
76+
{ status: 400 }
77+
)
78+
}
7079
logger.error(`[${requestId}] Failed to download file from storage:`, error)
7180
return NextResponse.json(
7281
{ success: false, error: getErrorMessage(error, 'Failed to download file') },

‎apps/sim/app/api/tools/discord/send-message/route.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,11 @@ import { parseRequest } from '@/lib/api/server'
66
import { checkInternalAuth } from '@/lib/auth/hybrid'
77
import { validateNumericId } from '@/lib/core/security/input-validation'
88
import { generateRequestId } from '@/lib/core/utils/request'
9+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
910
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
11+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
1012
import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils'
11-
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
13+
import { downloadServableFilesWithinBudget } from '@/lib/uploads/utils/file-utils.server'
1214
import { docNotReadyResponse } from '@/lib/uploads/utils/servable-file-response'
1315
import { assertToolFileAccess } from '@/app/api/files/authorization'
1416

@@ -146,12 +148,10 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
146148

147149
let resolved: Array<{ buffer: Buffer; contentType: string }>
148150
try {
149-
resolved = await Promise.all(
150-
userFiles.map(async (file, i) => {
151-
logger.info(`[${requestId}] Downloading file ${i}: ${file.name}`)
152-
return await downloadServableFileFromStorage(file, requestId, logger)
153-
})
154-
)
151+
resolved = await downloadServableFilesWithinBudget(userFiles, requestId, logger, {
152+
totalMaxBytes: MAX_BUFFERED_TRANSFER_BYTES,
153+
label: 'Total attachment size',
154+
})
155155
} catch (error) {
156156
const notReady = docNotReadyResponse(error)
157157
if (notReady) return notReady
@@ -161,7 +161,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
161161
success: false,
162162
error: `Failed to download attachment: ${getErrorMessage(error, 'Unknown error')}`,
163163
},
164-
{ status: 500 }
164+
{ status: isPayloadSizeLimitError(error) ? 413 : 500 }
165165
)
166166
}
167167

‎apps/sim/app/api/tools/dropbox/upload/route.ts‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,10 @@ import { dropboxUploadContract } from '@/lib/api/contracts/storage-transfer'
55
import { parseRequest } from '@/lib/api/server'
66
import { checkInternalAuth } from '@/lib/auth/hybrid'
77
import { generateRequestId } from '@/lib/core/utils/request'
8+
import { isPayloadSizeLimitError } from '@/lib/core/utils/stream-limits'
89
import { httpHeaderSafeJson } from '@/lib/core/utils/validation'
910
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
11+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
1012
import { processFilesToUserFiles, type RawFileInput } from '@/lib/uploads/utils/file-utils'
1113
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
1214
import { docNotReadyResponse } from '@/lib/uploads/utils/servable-file-response'
@@ -58,14 +60,16 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
5860
const denied = await assertToolFileAccess(userFile.key, authResult.userId, requestId, logger)
5961
if (denied) return denied
6062
try {
61-
const result = await downloadServableFileFromStorage(userFile, requestId, logger)
63+
const result = await downloadServableFileFromStorage(userFile, requestId, logger, {
64+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
65+
})
6266
fileBuffer = result.buffer
6367
} catch (error) {
6468
const notReady = docNotReadyResponse(error)
6569
if (notReady) return notReady
6670
return NextResponse.json(
6771
{ success: false, error: getErrorMessage(error, 'Failed to download file') },
68-
{ status: 500 }
72+
{ status: isPayloadSizeLimitError(error) ? 413 : 500 }
6973
)
7074
}
7175
fileName = userFile.name

‎apps/sim/app/api/tools/elevenlabs/audio/route.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import {
2020
isModelSafeWorkspaceFileKey,
2121
MODEL_UNSAFE_WORKSPACE_FILE_ERROR_MESSAGE,
2222
} from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance'
23+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
2324
import { getMimeTypeFromExtension } from '@/lib/uploads/utils/file-utils'
2425
import { downloadFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
2526
import { assertToolFileAccess } from '@/app/api/files/authorization'
@@ -153,7 +154,9 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
153154
{ status: 400 }
154155
)
155156
}
156-
const buffer = await downloadFileFromStorage(file, requestId, logger)
157+
const buffer = await downloadFileFromStorage(file, requestId, logger, {
158+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
159+
})
157160
const ext = file.name.split('.').pop()?.toLowerCase() || ''
158161
source = {
159162
buffer,

‎apps/sim/app/api/tools/firecrawl/parse/route.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import {
1111
isModelSafeWorkspaceFileKey,
1212
MODEL_UNSAFE_WORKSPACE_FILE_ERROR_MESSAGE,
1313
} from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance'
14+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
1415
import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils'
1516
import { downloadServableFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
1617
import { docNotReadyResponse } from '@/lib/uploads/utils/servable-file-response'
@@ -85,7 +86,10 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
8586
const { buffer, contentType } = await downloadServableFileFromStorage(
8687
userFile,
8788
requestId,
88-
logger
89+
logger,
90+
{
91+
maxBytes: MAX_BUFFERED_TRANSFER_BYTES,
92+
}
8993
)
9094

9195
const formData = new FormData()

0 commit comments

Comments
 (0)