Skip to content

About

app allows users to upload documents and uses a Large Language Model (LLM) to extract the primary insured entity name

Resources

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

LLM Document Insured Extractor 📄🤖

This Next.js 14 application allows users to upload documents (.pdf, .docx, .txt) and uses a Large Language Model (LLM) to extract the primary insured entity name. The extracted name is then matched against a predefined list of insureds to find the best match and its internal ID.

Code Summary:

  1. File Upload: User drags/drops or selects PDF, DOCX, or TXT files.
  2. Queue & Status: Files appear in a list with an "uploaded" or "queued" status.
  3. Client-Side Parsing: Each file's text content is extracted in the browser.
  4. API Call for Extraction: The extracted text is sent to a Next.js API route.
  5. LLM Processing: The API route calls an LLM (Groq/OpenAI) to identify the primary insured name.
  6. Name Normalization: Both extracted and predefined insured names are normalized (lowercase, no punctuation/suffixes).
  7. Entity Matching: The LLM's result is compared against a predefined list using string similarity.
  8. Result Update: The UI updates with the extracted name, matched ID, and confidence score.
  9. Manual Match Option: If confidence is low (<80%), the user can manually select a match.
  10. Error Display: Any errors during parsing, LLM call, or matching are shown in the UI.

Live Demo: (Add link if you deploy it, e.g., Vercel, Netlify) Screenshots: (Consider adding a GIF or a couple of screenshots of the UI in action)

Screenshot of LLM Document Insured Extractor UI (Replace placeholder-for-your-screenshot.png with an actual image after taking one)

✨ Features

  • Drag-and-Drop File Uploader:
    • Accepts .pdf, .docx, and .txt files.
    • Supports uploading multiple files at once.
    • Displays an upload list with file names and processing status (uploaded, parsing, extracting, matching, done, error).
  • LLM-Powered Extraction:
    • Calls an LLM (configurable: OpenAI GPT series, Groq Llama/Mixtral, or a stub) to parse document text.
    • Extracts the primary insured entity name.
    • LLM prompts are configurable in lib/config.ts.
  • Entity Matching:
    • Matches the LLM-extracted name against a hard-coded list of insureds (lib/insuredsData.ts).
    • Normalization logic handles case, punctuation, and common corporate suffixes (Inc, LLC, Corp).
    • Uses Levenshtein distance-based string similarity (string-similarity library).
  • Result Display:
    • For each file:
      • File name
      • Extracted insured name
      • Matched internal ID
      • Confidence score (%)
    • If no match ≥ 80% similarity, shows "No match" with an option to manually select from the insured list.
  • Error Handling: Gracefully handles and surfaces network, parsing, and LLM API errors.
  • No Persistence: No database, ORM, or auth. All data is in-memory and resets on refresh.

🛠️ Tech Stack

  • Framework: Next.js 14 (App Router)
  • Language: TypeScript
  • UI: React 18
  • Styling: Plain CSS (inline styles and global CSS)
  • File Parsing:
    • pdfjs-dist (for .pdf)
    • mammoth (for .docx)
  • String Similarity: string-similarity
  • LLM Integration: openai Node.js SDK (compatible with OpenAI and Groq)
  • Testing: Jest (or Vitest - specify if you used it) for unit testing matching utilities.

⚙️ Configuration

  1. LLM API Keys:

    • The application can use OpenAI or Groq APIs.
    • Create a .env.local file in the project root:
      # For OpenAI (Optional)
      OPENAI_API_KEY=your_openai_api_key_here
      
      # For Groq (Recommended for speed/free tier)
      GROQ_API_KEY=your_groq_api_key_here
    • Refer to .env.example for the structure.
    • If no API key is provided for the configured LLM, the application will use a stubbed LLM response.
  2. LLM Provider & Model Selection:

    • Modify lib/config.ts to choose the active LLM provider and model:
      // lib/config.ts
      export const ACTIVE_LLM_PROVIDER: LlmProvider = process.env.GROQ_API_KEY ? 'groq' : (process.env.OPENAI_API_KEY ? 'openai' : 'stub');
      export const OPENAI_MODEL = 'gpt-3.5-turbo-0125';
      export const GROQ_MODEL = 'llama3-8b-8192'; // Or 'mixtral-8x7b-32768'
  3. LLM Prompts:

    • System and user prompts for the LLM are defined in lib/config.ts:
      • LLM_SYSTEM_PROMPT
      • LLM_USER_PROMPT_TEMPLATE
    • Customize these prompts to improve extraction accuracy based on your document types.

🚀 Getting Started

  1. Clone the repository:

    git clone https://github.com/your-username/llm-doc-parser.git
    cd llm-doc-parser
  2. Install dependencies:

    npm install
    # or
    # yarn install
  3. Set up environment variables:

    • Copy .env.example to .env.local.
    • Add your API keys to .env.local (see Configuration section).
  4. Copy PDF.js worker (if not already automated):

    • The PDF parsing library (pdfjs-dist) requires a worker file. The package.json includes a script to copy it.
    • If you encounter issues, ensure pdf.worker.min.js is in the /public directory. You can run the copy script manually:
      npm run copy-pdf-worker
      (Ensure the path in the copy-pdf-worker script in package.json matches your pdfjs-dist installation if you update the library.)
  5. Run the development server:

    npm run dev

    The application will be available at http://localhost:3000.

  6. Populate Sample Claims:

    • Add your sample .pdf, .docx, and .txt files to the /sample-claims directory to have documents ready for testing.

##🧪 Running Tests

  • Unit tests for the name matching utility (lib/match.ts) are included.
  • To run tests:
    npm test
    Or for watch mode:
    npm run test:watch

📁 Project Structure

app/ ├── api/extract-entity/route.ts # Serverless function for LLM calls ├── page.tsx # Main UI and application logic └── globals.css # Global styles components/ ├── Dropzone.tsx # File drag-and-drop component └── ResultsTable.tsx # Component to display processing results lib/ ├── config.ts # LLM prompts, model configs, provider selection ├── insuredsData.ts # Hard-coded insureds list ├── match.ts # String similarity and normalization logic ├── match.test.ts # Tests for matching logic └── parser.ts # File text extraction logic (pdf, docx, txt) public/ ├── pdf.worker.min.js # PDF.js worker file (copied via script) └── (other static assets) sample-claims/ └── (Place your sample .pdf, .docx, .txt files here) .env.example # Example environment variables .gitignore jest.config.js # Jest configuration (if using Jest) jest.setup.js # Jest setup (if using Jest) next.config.js package.json README.md tsconfig.json

💡 Potential Future Enhancements

  • More sophisticated text chunking for very large documents.
  • Web Workers for client-side parsing to improve UI responsiveness.
  • Option to edit/confirm the LLM's extracted name before matching.
  • Support for more file types (e.g., .odt, image-based PDFs with OCR).
  • Batch processing progress indicators.
  • More advanced confidence scoring or explanation for matches.

🙏 Acknowledgements

  • Inspired by the need for efficient document processing.
  • Uses fantastic open-source libraries like Next.js, React, pdfjs-dist, mammoth, and string-similarity.

Happy Extracting! 🎉

About

app allows users to upload documents and uses a Large Language Model (LLM) to extract the primary insured entity name

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages