fix(backend): prevent DoS by limiting scraper fetch payload size

[Vedhant26]

What changed

• Added safeFetchText helper in �ackend/utils/ragUtilities.js.
• Modified scrapeWebpage and scrapeTitle to use safeFetchText instead of blindly buffering (await fetch(url)).text() into memory.
• The new fetch logic enforces a 5MB maximum payload size and validates that the Content-Type is a text-based document.

Why

Calling etch(url).text() on user-provided URLs without size constraints creates an unconstrained resource consumption vulnerability (DoS). If a malicious actor inputs a URL that serves an infinite data stream or a massive multi-gigabyte file, the Node.js process attempts to buffer the entire payload into a single string. This rapidly exhausts the heap memory limit, crashing the backend API via an Out Of Memory (OOM) Panic. This fix ensures that oversized payloads are rejected and aborted before they can crash the server.

How to test

Run the backend:
�ash id="t1x9ab" pnpm run dev
Expected result:

• The RAG scraper works normally for regular documentation pages.
• Attempting to ingest a massive file (e.g., an ISO or a mock 10GB stream) throws a Payload too large. Exceeded 5MB limit. error and safely aborts the operation without crashing the API.

Screenshots (if UI change)

N/A

Related issue

Closes #94

---

Pre-Submission Checklist

• Branch named following GSSoC convention
• Changes committed with meaningful commit message
• Changes pushed to remote fork
• Relevant tests executed successfully
• Code follows project contribution guidelines

[Vedhant26]

@avishek0769 I am ready for any constructuve criticism :)

[avishek0769]

@Vedhant26 One suggestion: instead of throwing an error when the 5 MB limit is exceeded, return the data collected up to that limit. That way, the function still provides useful output rather than failing entirely once the limit is reached.

[avishek0769]

@Vedhant26 Any update on this PR?

[Vedhant26]

Working in it , I'll reach out soon

[Vedhant26]

.

[avishek0769]

@Vedhant26 Resolve the conflicts