This Next.js 14 application allows users to upload documents (.pdf, .docx, .txt) and uses a Large Language Model (LLM) to extract the primary insured entity name. The extracted name is then matched against a predefined list of insureds to find the best match and its internal ID.
- File Upload: User drags/drops or selects PDF, DOCX, or TXT files.
- Queue & Status: Files appear in a list with an "uploaded" or "queued" status.
- Client-Side Parsing: Each file's text content is extracted in the browser.
- API Call for Extraction: The extracted text is sent to a Next.js API route.
- LLM Processing: The API route calls an LLM (Groq/OpenAI) to identify the primary insured name.
- Name Normalization: Both extracted and predefined insured names are normalized (lowercase, no punctuation/suffixes).
- Entity Matching: The LLM's result is compared against a predefined list using string similarity.
- Result Update: The UI updates with the extracted name, matched ID, and confidence score.
- Manual Match Option: If confidence is low (<80%), the user can manually select a match.
- Error Display: Any errors during parsing, LLM call, or matching are shown in the UI.
Live Demo: (Add link if you deploy it, e.g., Vercel, Netlify) Screenshots: (Consider adding a GIF or a couple of screenshots of the UI in action)
(Replace placeholder-for-your-screenshot.png with an actual image after taking one)
- Drag-and-Drop File Uploader:
- Accepts
.pdf,.docx, and.txtfiles. - Supports uploading multiple files at once.
- Displays an upload list with file names and processing status (uploaded, parsing, extracting, matching, done, error).
- Accepts
- LLM-Powered Extraction:
- Calls an LLM (configurable: OpenAI GPT series, Groq Llama/Mixtral, or a stub) to parse document text.
- Extracts the primary insured entity name.
- LLM prompts are configurable in
lib/config.ts.
- Entity Matching:
- Matches the LLM-extracted name against a hard-coded list of insureds (
lib/insuredsData.ts). - Normalization logic handles case, punctuation, and common corporate suffixes (Inc, LLC, Corp).
- Uses Levenshtein distance-based string similarity (
string-similaritylibrary).
- Matches the LLM-extracted name against a hard-coded list of insureds (
- Result Display:
- For each file:
- File name
- Extracted insured name
- Matched internal ID
- Confidence score (%)
- If no match ≥ 80% similarity, shows "No match" with an option to manually select from the insured list.
- For each file:
- Error Handling: Gracefully handles and surfaces network, parsing, and LLM API errors.
- No Persistence: No database, ORM, or auth. All data is in-memory and resets on refresh.
- Framework: Next.js 14 (App Router)
- Language: TypeScript
- UI: React 18
- Styling: Plain CSS (inline styles and global CSS)
- File Parsing:
pdfjs-dist(for .pdf)mammoth(for .docx)
- String Similarity:
string-similarity - LLM Integration:
openaiNode.js SDK (compatible with OpenAI and Groq) - Testing: Jest (or Vitest - specify if you used it) for unit testing matching utilities.
-
LLM API Keys:
- The application can use OpenAI or Groq APIs.
- Create a
.env.localfile in the project root:# For OpenAI (Optional) OPENAI_API_KEY=your_openai_api_key_here # For Groq (Recommended for speed/free tier) GROQ_API_KEY=your_groq_api_key_here
- Refer to
.env.examplefor the structure. - If no API key is provided for the configured LLM, the application will use a stubbed LLM response.
-
LLM Provider & Model Selection:
- Modify
lib/config.tsto choose the active LLM provider and model:// lib/config.ts export const ACTIVE_LLM_PROVIDER: LlmProvider = process.env.GROQ_API_KEY ? 'groq' : (process.env.OPENAI_API_KEY ? 'openai' : 'stub'); export const OPENAI_MODEL = 'gpt-3.5-turbo-0125'; export const GROQ_MODEL = 'llama3-8b-8192'; // Or 'mixtral-8x7b-32768'
- Modify
-
LLM Prompts:
- System and user prompts for the LLM are defined in
lib/config.ts:LLM_SYSTEM_PROMPTLLM_USER_PROMPT_TEMPLATE
- Customize these prompts to improve extraction accuracy based on your document types.
- System and user prompts for the LLM are defined in
-
Clone the repository:
git clone https://github.com/your-username/llm-doc-parser.git cd llm-doc-parser -
Install dependencies:
npm install # or # yarn install
-
Set up environment variables:
- Copy
.env.exampleto.env.local. - Add your API keys to
.env.local(see Configuration section).
- Copy
-
Copy PDF.js worker (if not already automated):
- The PDF parsing library (
pdfjs-dist) requires a worker file. Thepackage.jsonincludes a script to copy it. - If you encounter issues, ensure
pdf.worker.min.jsis in the/publicdirectory. You can run the copy script manually:(Ensure the path in thenpm run copy-pdf-worker
copy-pdf-workerscript inpackage.jsonmatches yourpdfjs-distinstallation if you update the library.)
- The PDF parsing library (
-
Run the development server:
npm run dev
The application will be available at
http://localhost:3000. -
Populate Sample Claims:
- Add your sample
.pdf,.docx, and.txtfiles to the/sample-claimsdirectory to have documents ready for testing.
- Add your sample
##🧪 Running Tests
- Unit tests for the name matching utility (
lib/match.ts) are included. - To run tests:
Or for watch mode:
npm testnpm run test:watch
app/ ├── api/extract-entity/route.ts # Serverless function for LLM calls ├── page.tsx # Main UI and application logic └── globals.css # Global styles components/ ├── Dropzone.tsx # File drag-and-drop component └── ResultsTable.tsx # Component to display processing results lib/ ├── config.ts # LLM prompts, model configs, provider selection ├── insuredsData.ts # Hard-coded insureds list ├── match.ts # String similarity and normalization logic ├── match.test.ts # Tests for matching logic └── parser.ts # File text extraction logic (pdf, docx, txt) public/ ├── pdf.worker.min.js # PDF.js worker file (copied via script) └── (other static assets) sample-claims/ └── (Place your sample .pdf, .docx, .txt files here) .env.example # Example environment variables .gitignore jest.config.js # Jest configuration (if using Jest) jest.setup.js # Jest setup (if using Jest) next.config.js package.json README.md tsconfig.json
- More sophisticated text chunking for very large documents.
- Web Workers for client-side parsing to improve UI responsiveness.
- Option to edit/confirm the LLM's extracted name before matching.
- Support for more file types (e.g., .odt, image-based PDFs with OCR).
- Batch processing progress indicators.
- More advanced confidence scoring or explanation for matches.
- Inspired by the need for efficient document processing.
- Uses fantastic open-source libraries like Next.js, React, pdfjs-dist, mammoth, and string-similarity.
Happy Extracting! 🎉