Skip to content

Stream large settlement CSV exports instead of building the full string in memoryΒ #87

Description

@Jagadeeshftw

πŸ“Œ Description

Per the README, GET /api/v1/settlements?format=csv "ignores pagination and exports every matching, sorted row" β€” for a large settlement history, toCsv in src/utils/csv.ts currently builds the entire CSV string in memory before sending it, which doesn't scale and blocks the event loop during serialization.

🧩 Requirements and context

  • Stream the CSV response row-by-row (e.g. via res.write per chunk) instead of materializing the full string.
  • Preserve the exact same escaping behavior expected from the CSV-escaping fix elsewhere in this batch.
  • Keep response headers (Content-Type, filename) identical to the current export.

πŸ› οΈ Suggested execution

  • Add a streaming variant of the CSV writer in src/utils/csv.ts (or a new csvStream.ts) used by routes/settlements.ts.
  • Add a test simulating a large settlement set and asserting the response streams incrementally (not just correctness of final content).
  • Leave the anchors CSV export on the simple in-memory path unless it also proves large in practice.

βœ… Acceptance criteria

  • Large settlement CSV exports stream instead of blocking on a single large string build.
  • Output content is byte-identical to the previous in-memory implementation for the same data.
  • Test coverage for the streaming path.

πŸ”’ Security notes

Reduces memory-exhaustion/DoS risk from a very large unpaginated export.

πŸ“‹ Guidelines

  • Minimum 95% test coverage
  • Clear documentation
  • Timeframe: 96 hours

Metadata

Metadata

Assignees

Labels

GrantFox OSSGrantFox open-source programMaybe RewardedGrantFox: potentially rewarded contributionOfficial Campaign | FWC26GrantFox official campaign issueenhancementNew feature or improvementperformancePerformance / caching

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions