Skip to content

dexisback/frusta

BranchesTags

Repository files navigation

Frusta

______              _        
|  ___|            | |       
| |_ _ __ _   _ ___| |_ __ _ 
|  _| '__| | | / __| __/ _` |
| | | |  | |_| \__ \ || (_| |
\_| |_|   \__,_|___/\__\__,_|

manual single-node chunking engine | s3-style upload semantics | resumable + idempotent

Frusta is a manual, S3-style chunk upload engine built on Node.js + Express + Prisma. It accepts out-of-order and parallel chunk uploads, supports resume, enforces idempotency, and merges to a final artifact using atomic file operations.

What It Solves

  • Upload large files reliably over unstable networks.
  • Resume interrupted uploads without re-sending completed chunks.
  • Prevent duplicate chunk corruption under retries/races.
  • Keep merge and cleanup deterministic with explicit upload session states.

Core Features

  • Chunked upload lifecycle: INITIATED -> UPLOADING -> COMPLETING -> COMPLETED | FAILED
  • Out-of-order chunk acceptance
  • Parallel chunk upload support
  • Resume support via uploaded-chunk introspection endpoint
  • Duplicate chunk protection:
    • Filesystem guard (chunkN existence check)
    • DB uniqueness (@@unique([uploadSessionId, chunkIndex]))
    • createMany(..., skipDuplicates: true) idempotency
  • Safe write path:
    • Chunk write to .part then atomic rename to chunkN
    • Final merge to <uploadId>-<fileName>.part then atomic rename to final file
  • Automatic temp cleanup after successful merge
  • Failure rollback in chunk ingest path (if DB write fails after file write, chunk file is removed)
  • Zod request validation for body/query/params
  • Centralized error and not-found middleware
  • Request logging middleware with latency printouts
  • Test-time in-memory Prisma fallback for fast deterministic integration testing

Architecture

System Diagram

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e1f5fe', 'primaryTextColor': '#01579b', 'primaryBorderColor': '#0288d1', 'lineColor': '#0288d1', 'secondaryColor': '#e8f5e9', 'tertiaryColor': '#fff3e0'}}}%%
graph TB
    subgraph "Client"
        C1[Browser / Mobile App / k6]
    end

    subgraph "HTTP Layer"
        A1[Express App<br/>src/app.ts]
        A2[Upload Router<br/>src/modules/uploads/uploads.routes.ts]
    end

    subgraph "Cross-Cutting Middleware"
        M1[requestLogger]
        M2[express.json]
        M3[notFound]
        M4[errorHandler]
    end

    subgraph "Controller Layer<br/>src/modules/uploads/uploads.controller.ts"
        CTRL1[initiateController]
        CTRL2[chunkController]
        CTRL3[statusController]
        CTRL4[completedController]
    end

    subgraph "Validation Layer<br/>src/modules/uploads/uploads.schema.ts"
        V1[Zod: incomingSandesha]
        V2[Zod: chunkQuery]
        V3[Zod: completedSandesha]
        V4[Zod: statusParams]
    end

    subgraph "Service Layer<br/>src/modules/uploads/uploads.service.ts"
        S1[prepareUploadDir]
        S2[storeChunk<br/>stream pipeline + atomic rename]
        S3[deleteChunk<br/>rollback cleanup]
        S4[mergeChunks<br/>ordered stream merge + atomic rename]
    end

    subgraph "Data Layer<br/>src/db/prisma.ts"
        D1[Prisma Client]
        D2[(PostgreSQL)]
        D3[(In-Memory Adapter<br/>Test Fallback)]
    end

    subgraph "File System Storage"
        F1[uploads/temp/&lt;uploadId&gt;/chunkN]
        F2[uploads/final/&lt;uploadId&gt;-&lt;fileName&gt;]
    end

    C1 -->|POST /uploads/initiate| A1
    C1 -->|POST /uploads/chunk| A1
    C1 -->|GET /uploads/:id/status| A1
    C1 -->|POST /uploads/complete| A1

    A1 --> M2
    M2 --> M1
    M1 --> A2
    A2 --> CTRL1
    A2 --> CTRL2
    A2 --> CTRL3
    A2 --> CTRL4

    CTRL1 --> V1
    CTRL2 --> V2
    CTRL4 --> V3
    CTRL3 --> V4

    CTRL1 -->|create uploadSession| D1
    CTRL1 --> S1
    CTRL2 -->|findUnique uploadSession| D1
    CTRL2 -->|update status UPLOADING| D1
    CTRL2 --> S2
    CTRL2 -->|createMany skipDuplicates| D1
    CTRL2 -->|count uploadedChunks| D1
    CTRL3 -->|findUnique + findMany| D1
    CTRL4 -->|findUnique uploadSession| D1
    CTRL4 -->|count chunks vs totalChunks| D1
    CTRL4 -->|update COMPLETING| D1
    CTRL4 --> S4
    CTRL4 -->|update COMPLETED/FAILED| D1

    S1 --> F1
    S2 -->|write .part → rename chunkN| F1
    S3 -->|rm chunkN / .part| F1
    S4 -->|read chunk0..N| F1
    S4 -->|write .part → rename final| F2
    S4 -->|rm temp dir| F1

    D1 --> D2
    D1 -.-> D3

    A2 --> M3
    M3 --> M4
    M4 --> C1
Loading

Upload Session State Machine

stateDiagram-v2
    [*] --> INITIATED : POST /initiate
    INITIATED --> UPLOADING : first chunk arrives
    UPLOADING --> UPLOADING : chunk N stored
    UPLOADING --> COMPLETING : POST /complete<br/>all chunks verified
    COMPLETING --> COMPLETED : merge + atomic rename success
    COMPLETING --> FAILED : merge error / exception
    UPLOADING --> FAILED : unhandled error
Loading

Data Flow Highlights

  1. Initiate: Client → Router → initiateController → Zod validation → Prisma (uploadSession.create) → prepareUploadDir → Temp directory created.
  2. Chunk Ingest: Client → Router → chunkController → Zod validation → Prisma (uploadSession.findUnique + update) → storeChunk (stream pipeline, atomic .part → chunkN rename) → Prisma (uploadChunk.createMany with skipDuplicates) → On DB failure, deleteChunk rolls back the file.
  3. Status / Resume: Client → Router → statusController → Prisma (uploadSession.findUnique + uploadChunk.findMany) → Returns uploadedChunks[] array.
  4. Complete / Merge: Client → Router → completedController → Zod validation → Prisma (count verification) → Status COMPLETING → mergeChunks (ordered stream pipeline into .part, atomic rename to final, recursive temp cleanup) → Prisma status COMPLETED (or FAILED on error).

Upload Flow

  1. POST /uploads/initiate creates an upload session row and a temp directory for uploadId.
  2. POST /uploads/chunk stores raw chunk bytes (application/octet-stream) by chunkIndex.
  3. Clients can upload chunks in any order, retry safely, and in parallel.
  4. GET /uploads/:uploadId/status returns uploadedChunks[] so clients resume only missing indexes.
  5. POST /uploads/complete verifies uploadedChunks === totalChunks, merges in index order, atomically renames final artifact, then deletes temp chunk directory.

API

Base path: /uploads

1) Initiate Upload

  • POST /uploads/initiate
  • Body:
{
  "fileName": "video.mp4",
  "fileSize": 734003200,
  "totalChunks": 128
}
  • Success: 201
{
  "success": true,
  "statusCode": 201,
  "message": "upload session initiated",
  "data": { "uploadId": "uuid" }
}

2) Upload Chunk

  • POST /uploads/chunk?uploadId=<uuid>&chunkIndex=<int>
  • Headers: Content-Type: application/octet-stream
  • Body: raw chunk bytes
  • Success: 200
{
  "success": true,
  "statusCode": 200,
  "message": "chunk stored",
  "data": { "uploadedChunks": 42, "totalChunks": 128 }
}

3) Upload Status (Resume API)

  • GET /uploads/:uploadId/status
  • Success: 200
{
  "success": true,
  "statusCode": 200,
  "message": "upload status fetched",
  "data": {
    "uploadId": "uuid",
    "status": "UPLOADING",
    "uploadedChunks": [0, 1, 2, 5, 6],
    "totalChunks": 128
  }
}

4) Complete Upload

  • POST /uploads/complete
  • Body:
{
  "uploadId": "uuid"
}
  • Success: 200
  • Idempotent behavior: if already completed, returns 200 with "upload already completed".

Storage and Merge Semantics

  • Root: UPLOAD_ROOT (default: uploads)
  • Temp chunks: uploads/temp/<uploadId>/chunk<index>
  • Final artifact: uploads/final/<uploadId>-<fileName>
  • Chunk and final files are first written as .part, then atomically renamed.
  • On merge success: temp chunk directory is recursively removed.

Data Model (Prisma)

uploadSession

  • id (UUID, PK)
  • fileName
  • fileSize (BigInt)
  • totalChunks
  • status (INITIATED | UPLOADING | COMPLETING | COMPLETED | FAILED)
  • mergeStartedAt, mergeCompletedAt
  • createdAt, updatedAt
  • Index: status

uploadChunk

  • id (UUID, PK)
  • uploadSessionId (FK -> uploadSession, onDelete: Cascade)
  • chunkIndex
  • size
  • createdAt
  • Unique constraint: (uploadSessionId, chunkIndex)
  • Index: uploadSessionId

Project Structure

src/
  app.ts                          # express app wiring + middleware + routes
  server.ts                       # process entrypoint
  config/env.ts                   # zod env validation
  db/prisma.ts                    # prisma client + in-memory test fallback
  middleware/
    requestLogger.middleware.ts
    notFound.middleware.ts
    errorHandler.middleware.ts
  modules/uploads/
    uploads.routes.ts             # upload route map
    uploads.schema.ts             # zod contracts
    uploads.controller.ts         # request orchestration
    uploads.service.ts            # fs/stream chunk + merge engine
    uploads.constants.ts          # status constants + storage paths
    uploads.types.ts
  utils/
    apiError.ts
    apiResponse.ts
    asyncHandler.ts

prisma/
  schema.prisma
  migrations/

tests/
  integration/upload.integration.test.ts
  unit/upload.controller.unit.test.ts
  unit/upload.service.unit.test.ts
  unit/upload.schema.unit.test.ts

benchmarks/
  k6/
    upload-flow.js
    api.js
    checks.js
    options.js
    payload.js
    run.sh

Local Setup

  1. Install dependencies:
npm install
  1. Configure environment:
# .env
NODE_ENV=development
PORT=3000
DATABASE_URL=postgresql://<user>:<pass>@<host>:5432/<db>
# optional:
# UPLOAD_ROOT=uploads
  1. Apply DB schema:
npx prisma migrate deploy
  1. Run:
npm run dev

Environment Variables

  • NODE_ENV: development | test | production
  • PORT: HTTP port (default 3000)
  • DATABASE_URL: required unless using in-memory test mode
  • UPLOAD_ROOT (optional): storage root (default uploads)
  • FRUSTA_TEST_USE_REAL_DB (optional): when true in test mode, bypasses in-memory DB fallback

NPM Scripts

  • npm run dev: run server in watch mode via tsx
  • npm run test: run full Vitest suite
  • npm run typecheck: TypeScript type-check
  • npm run build: build to dist/
  • npm run start: run compiled server
  • npm run bench:k6:upload: run k6 flow with env-driven params
  • npm run bench:k6:upload:local: convenience local benchmark profile

Testing

Run:

npm test

Current snapshot (run on March 12, 2026):

  • 29/29 tests passing
  • Unit: 23 (controller + service + schema)
  • Integration: 6 (full HTTP upload flow)

Notes:

  • In NODE_ENV=test, DB defaults to in-memory Prisma adapter unless FRUSTA_TEST_USE_REAL_DB=true.
  • Integration tests cover upload initiation, chunk ingest, out-of-order behavior, status/resume API, complete/merge path, and invalid index rejection.

Benchmark (k6) Snapshot

Script: benchmarks/k6/upload-flow.js

Run profile used:

  • Date: March 12, 2026
  • BASE_URL=http://127.0.0.1:3000
  • VUS=5
  • DURATION=15s
  • CHUNK_SIZE=8
  • SLEEP_SECONDS=0
  • Server mode: NODE_ENV=test (in-memory DB path)

Observed results:

  • Checks pass rate: 100% (126/126)
  • HTTP failure rate: 0.00%
  • Total HTTP requests: 54
  • Throughput: 1.86 req/s
  • http_req_duration avg: 1664.49 ms
  • http_req_duration p90: 2081.15 ms
  • http_req_duration p95: 2654.46 ms
  • Completed upload iterations: 6

Threshold status for this run:

  • http_req_failed < 1%: pass
  • checks > 99%: pass
  • http_req_duration p95 < 2500ms: narrowly missed (2654.46ms)

Why This Design Holds Up

  • Stream-based writes (pipeline) reduce memory pressure and give cleaner backpressure handling.
  • Atomic renames protect against partial file visibility.
  • Session/chunk split in DB keeps metadata clean and queryable.
  • Idempotent chunk semantics make retries safe under network noise and client duplication.
  • Controller/service separation keeps the upload engine modular and easy to evolve.

License

This project is licensed under the ISC License. See LICENSE.

About

S3-inspired object store. single-node, parallel chunk pipeline

Resources

Readme

License

ISC license
Activity

Stars

13 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages