DataFlow Ingestion Engine

A distributed, async data ingestion and workload management backend built with Java Spring Boot

---

🚀 Live Demo — Try It Right Now

The engine is deployed at:

https://dataflow-ingestion-engine.onrender.com

⚠️ Free tier note: Render's free instances spin down after 15 minutes of inactivity. The first request may take 50–60 seconds to wake the server. Subsequent requests are fast. If you see a delay, just wait — the server is cold-starting.

Quick smoke test — verify the server is alive:

curl https://dataflow-ingestion-engine.onrender.com/actuator/health

Expected:

{ "status": "UP", "components": { "db": { "status": "UP" } } }

Note: Do NOT visit the root URL (/) in a browser — it will return a 500 error because there is no homepage. All endpoints are under /api/v1/ and /actuator/. This is expected and correct behaviour for a REST API backend.

---

Table of Contents

1. Project Overview
2. Architecture Diagram
3. Core Features
4. Design Patterns Used
5. Database Schema
6. API Endpoints Reference
7. Prerequisites and Setup
8. Running the Project Locally
9. Testing the Live API — In Order
10. Deployment Guide — Render + Aiven
11. Concurrency and Race Condition Prevention
12. Interview Talking Points

---

Project Overview

Analytics teams face a common challenge: reliably ingesting massive datasets without overwhelming the system or losing records. This engine solves that with:

• Zero data loss — every record is tracked, validated, and logged
• Async processing — HTTP requests return in milliseconds; heavy work runs on background threads
• Backpressure — rate limiting and queue caps prevent crashes under 1,000+ concurrent submissions
• Full data lineage — complete audit trail of every job's lifecycle, stored in MySQL

Who This Is Built For

• Analytics engineers who need stable, governed data pipelines
• Teams moving from manual CSV uploads to automated, observable ingestion
• Architects who need to demonstrate operational efficiency

---

Architecture Diagram

CLIENT (Postman / Frontend)
        |
        |  POST /api/v1/jobs (JSON body)
        v
EMBEDDED TOMCAT SERVER (port 10000 on Render / 8080 locally)
  └── Spring DispatcherServlet
        |
        v
CONTROLLER LAYER
  ├── JobController        (request validation, rate limit headers, client IP)
  └── MetricsController   (/job-efficiency, /system-health, /jobs-over-time)
        |
        v
SERVICE LAYER
  ├── JobService           (submit, get status, cancel)
  ├── MetricsService       (KPI queries, health check, time-range)
  ├── RateLimiterService   (ConcurrentMap, AtomicInteger, sliding window)
  └── JobProcessingService (@Async — fire-and-forget)
        |  Runs on background ThreadPool (DataFlow-Worker-N)
        |  • Parse via Strategy Pattern
        |  • Validate records
        |  • Update DB with progress
        |  • Publish events via Observer Pattern
        |
        +------------------+------------------+
        |                  |                  |
        v                  v                  v
STRATEGY LAYER      OBSERVER LAYER      THREAD POOL (AsyncConfig)
CsvStrategy         JobCompletionLogger  Core:  5 threads
JsonStrategy        (writes audit logs)  Max:  20 threads
(via Factory)       JobNotification      Queue: 500 jobs
                    (console/email/slack)
        |
        v
AIVEN MYSQL DATABASE (cloud-hosted, SSL required)
  ├── ingestion_jobs  (id, job_name, data_type, status, client_id,
  │                    total_records, processed_records, failed_records,
  │                    submitted_at, started_at, completed_at)
  └── job_logs        (id, job_id FK, log_level, message, stage,
                       record_count, created_at)

---

Core Features

1. Asynchronous Job Management

• Clients submit jobs via REST API and immediately receive a Job ID (response in under 50ms)
• Heavy processing runs on a dedicated background thread pool
• Clients poll GET /api/v1/jobs/{id} to check progress

2. Workload Throttling

• Rate limiting: Max 100 requests/minute per client (configurable)
• Thread pool backpressure: Queue capacity of 500 jobs; excess triggers CallerRunsPolicy
• Standard HTTP headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

3. Data Governance and Lineage

• Every job event (PENDING → RUNNING → COMPLETED) is logged with a timestamp
• Per-record validation failures are logged with the specific reason
• Full audit trail queryable via GET /api/v1/jobs/{id}/logs

4. Analytics Aggregation Endpoints

• GET /api/v1/metrics/job-efficiency — KPIs for an executive dashboard
• GET /api/v1/metrics/system-health — JVM, queue depth, health status
• GET /api/v1/metrics/jobs-over-time — time-windowed trend data

---

Design Patterns Used

Strategy Pattern — Parser Selection

IngestionStrategy  (Interface)
    + parse(content)
    + validate(record)
         |
         +--- CsvParserStrategy   (OpenCSV, RFC 4180 safe)
         +--- JsonParserStrategy  (Jackson, Array + NDJSON)

Selected by:
IngestionStrategyFactory
    Map<DataType, Strategy>

Why? Adding a new format (XML, Parquet) requires zero changes to existing code — just implement the interface and register it. This is the Open/Closed Principle.

---

Observer Pattern — Event Notification

JobEventPublisher (Subject)
    List<JobEventListener>
         |
         |  notifies all
         +--- JobCompletionLogger   (priority=10, writes to DB)
         +--- JobNotificationObserver (priority=50, console/email/Slack)
         +--- (Future: MetricsUpdater, etc. — add without any code changes)

Why? When a job completes, multiple things need to happen (audit log, notification, metrics update). The Observer pattern lets us add new reactions without modifying the core processing code.

---

Database Schema

-- TABLE: ingestion_jobs  (main job tracking table)
CREATE TABLE ingestion_jobs (
    id                BIGINT AUTO_INCREMENT PRIMARY KEY,
    job_name          VARCHAR(255) NOT NULL,
    data_type         ENUM('CSV', 'JSON') NOT NULL,
    status            ENUM('PENDING','RUNNING','COMPLETED','FAILED','CANCELLED') NOT NULL,
    client_id         VARCHAR(100),
    total_records     INT DEFAULT 0,
    processed_records INT DEFAULT 0,
    failed_records    INT DEFAULT 0,
    error_message     TEXT,
    priority          INT DEFAULT 1,
    submitted_at      DATETIME NOT NULL,
    started_at        DATETIME,
    completed_at      DATETIME,
    last_updated_at   DATETIME,

    INDEX idx_status       (status),
    INDEX idx_submitted_at (submitted_at),
    INDEX idx_client_id    (client_id)
);

-- TABLE: job_logs  (data lineage / audit trail)
CREATE TABLE job_logs (
    id           BIGINT AUTO_INCREMENT PRIMARY KEY,
    job_id       BIGINT NOT NULL,
    log_level    ENUM('INFO','WARN','ERROR','DEBUG') NOT NULL,
    message      TEXT NOT NULL,
    stage        VARCHAR(50),
    record_count INT,
    created_at   DATETIME NOT NULL,

    FOREIGN KEY (job_id) REFERENCES ingestion_jobs(id) ON DELETE CASCADE,
    INDEX idx_job_id_created (job_id, created_at)
);

Job Lifecycle State Machine

PENDING  -->  RUNNING  -->  COMPLETED  (terminal)
                    \
                     -->  FAILED      (terminal)

From PENDING only:
PENDING  -->  CANCELLED               (terminal)

---

API Endpoints Reference

Job Management /api/v1/jobs

Method	Endpoint	Description	HTTP Status
POST	/api/v1/jobs	Submit a new ingestion job	202 Accepted
GET	/api/v1/jobs	List all jobs (optional ?status=PENDING)	200 OK
GET	/api/v1/jobs/{id}	Get specific job status	200 OK
DELETE	/api/v1/jobs/{id}	Cancel a pending job	200 OK
GET	/api/v1/jobs/{id}/logs	Get audit trail for a job	200 OK

Metrics /api/v1/metrics

Method	Endpoint	Description
GET	/api/v1/metrics/job-efficiency	KPIs: success rate, throughput
GET	/api/v1/metrics/system-health	Queue depth, JVM, health status
GET	/api/v1/metrics/jobs-over-time	Time-windowed job summary
GET	/api/v1/metrics/summary	Combined dashboard snapshot

Spring Actuator (built-in monitoring)

Endpoint	Description
GET /actuator/health	DB connectivity, app status
GET /actuator/metrics	JVM metrics (memory, threads, HTTP)

---

Prerequisites and Setup

1. Java 17 (JDK)

# Check if installed:
java -version
# Expected: openjdk version "17.x.x"

# Install if missing:
# Windows: https://adoptium.net/
# Mac:     brew install openjdk@17
# Ubuntu:  sudo apt install openjdk-17-jdk

2. Apache Maven

# Check if installed:
mvn -version
# Expected: Apache Maven 3.x.x

# Install if missing:
# Windows: https://maven.apache.org/download.cgi
# Mac:     brew install maven
# Ubuntu:  sudo apt install maven

3. MySQL 8.0

# Check if installed:
mysql --version

# Install if missing:
# Windows: https://dev.mysql.com/downloads/installer/
# Mac:     brew install mysql && brew services start mysql
# Ubuntu:  sudo apt install mysql-server && sudo systemctl start mysql

---

Running the Project Locally

Step 1 — Create the MySQL Database

# Open MySQL shell:
mysql -u root -p

# Inside the shell:
CREATE DATABASE dataflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
SHOW DATABASES;
EXIT;

Step 2 — Clone or Download the Project

git clone https://github.com/sahastraWin/dataflow-ingestion-engine.git
cd dataflow-ingestion-engine/dataflow-engine

# Verify structure:
ls
# Should show: pom.xml  src/  logs/

Step 3 — Configure the Database Connection

Open src/main/resources/application.properties and update:

spring.datasource.url=jdbc:mysql://localhost:3306/dataflow_db?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC
spring.datasource.username=root
spring.datasource.password=YOUR_PASSWORD_HERE

If your MySQL root password is empty, use: spring.datasource.password=

Step 4 — Build the Project

mvn clean install -DskipTests
# Expected output: [INFO] BUILD SUCCESS

-DskipTests skips tests during first build. Run mvn test separately.

Step 5 — Run the Application

mvn spring-boot:run

You should see in the console:

╔══════════════════════════════════════════════════╗
║     DataFlow Ingestion Engine - STARTED          ║
║     API Base: http://localhost:8080/api/v1        ║
║     Health:   http://localhost:8080/actuator/health║
╚══════════════════════════════════════════════════╝

Step 6 — Verify It's Running

GET http://localhost:8080/actuator/health

Expected response:

{
  "status": "UP",
  "components": {
    "db": { "status": "UP" },
    "diskSpace": { "status": "UP" }
  }
}

---

Testing the Live API — In Order

Use Postman or curl. Replace BASE_URL with:

• Live (Render): https://dataflow-ingestion-engine.onrender.com
• Local: http://localhost:8080

Run the tests in this exact order — each step uses the jobId returned by the previous one.

---

Step 1 — Health Check (confirm server is alive)

GET {{BASE_URL}}/actuator/health

Expected: { "status": "UP", "components": { "db": { "status": "UP" } } }

If db.status is DOWN, the database connection has failed. Check the environment variables on Render.

---

Step 2 — Submit a CSV Job

POST {{BASE_URL}}/api/v1/jobs
Content-Type: application/json

{
  "jobName": "Q4-Sales-CSV-Test",
  "dataType": "CSV",
  "rawContent": "transaction_id,product_id,customer_id,price,quantity,category\nTXN001,PROD_A,CUST_001,29.99,2,Electronics\nTXN002,PROD_B,CUST_002,15.00,1,Books\nTXN003,PROD_C,CUST_003,49.99,3,Clothing\nTXN004,PROD_D,CUST_004,bad_price,1,Toys",
  "priority": 1
}

Expected response (202 Accepted):

{
  "jobId": 1,
  "jobName": "Q4-Sales-CSV-Test",
  "status": "PENDING",
  "message": "Job accepted and queued for processing. Use the jobId to track status.",
  "submittedAt": "2024-01-15T10:30:00"
}

📌 Save the jobId — you'll use it in all subsequent steps.

---

Step 3 — Poll Job Status

GET {{BASE_URL}}/api/v1/jobs/1

Replace 1 with your actual jobId. Poll this endpoint until status becomes COMPLETED or FAILED.

Expected response:

{
  "jobId": 1,
  "jobName": "Q4-Sales-CSV-Test",
  "status": "COMPLETED",
  "totalRecords": 4,
  "processedRecords": 3,
  "failedRecords": 1,
  "successRate": 75.0,
  "durationSeconds": 2
}

---

Step 4 — View Audit Trail / Data Lineage

GET {{BASE_URL}}/api/v1/jobs/1/logs

This shows every event that happened during the job's lifecycle — parsing, validation failures, completion.

Expected response:

[
  { "logLevel": "INFO",  "message": "Job submitted and queued", "stage": "LIFECYCLE" },
  { "logLevel": "INFO",  "message": "Parsing complete: 4 records detected in CSV payload", "stage": "PARSING" },
  { "logLevel": "WARN",  "message": "Record 4 validation failed: Price is not a valid number: bad_price", "stage": "VALIDATION" },
  { "logLevel": "INFO",  "message": "Job completed: 3/4 records processed (75.0% success)", "stage": "LIFECYCLE" }
]

Optional: get only the last N log entries:

GET {{BASE_URL}}/api/v1/jobs/1/logs?limit=5

---

Step 5 — Submit a JSON Job

POST {{BASE_URL}}/api/v1/jobs
Content-Type: application/json

{
  "jobName": "Product-Inventory-JSON",
  "dataType": "JSON",
  "rawContent": "[{\"transaction_id\":\"T001\",\"product_id\":\"P001\",\"customer_id\":\"C001\",\"price\":29.99,\"quantity\":5},{\"transaction_id\":\"T002\",\"product_id\":\"P002\",\"customer_id\":\"C002\",\"price\":15.00,\"quantity\":2}]"
}

Repeat Steps 3 and 4 with the new jobId.

---

Step 6 — Submit a Third Job (for cancellation demo)

POST {{BASE_URL}}/api/v1/jobs
Content-Type: application/json

{
  "jobName": "Job-To-Cancel",
  "dataType": "CSV",
  "rawContent": "id,name\n1,Alice\n2,Bob",
  "priority": 1
}

---

Step 7 — Cancel the Job (while PENDING)

DELETE {{BASE_URL}}/api/v1/jobs/{id}

Replace {id} with the jobId from Step 6. This only works if the job is still in PENDING state. If it already moved to RUNNING, you'll get a 409 Conflict.

---

Step 8 — List All Jobs

GET {{BASE_URL}}/api/v1/jobs

Optional filters:

GET {{BASE_URL}}/api/v1/jobs?status=COMPLETED
GET {{BASE_URL}}/api/v1/jobs?status=FAILED
GET {{BASE_URL}}/api/v1/jobs?status=PENDING

---

Step 9 — View Analytics Dashboard

GET {{BASE_URL}}/api/v1/metrics/job-efficiency

Expected response:

{
  "totalJobs": 3,
  "completedJobs": 2,
  "failedJobs": 0,
  "pendingJobs": 0,
  "overallSuccessRate": 100.0,
  "averageProcessingRatePerSecond": 2500.5,
  "totalRecordsProcessed": 10420
}

---

Step 10 — System Health Metrics

GET {{BASE_URL}}/api/v1/metrics/system-health

Expected response:

{
  "healthStatus": "HEALTHY",
  "queueDepth": 0,
  "activeJobs": 0,
  "jvmUsedMemoryMb": 128,
  "jvmMaxMemoryMb": 512,
  "jvmMemoryUsagePercent": 25.0,
  "activeThreadCount": 18
}

---

Step 11 — Jobs Over Time (Trend Data)

GET {{BASE_URL}}/api/v1/metrics/jobs-over-time

---

Step 12 — Trigger Rate Limiting (optional stress test)

Submit 15+ requests quickly using Postman Collection Runner (set iterations to 15 on Step 2). Around request 101 you will receive:

HTTP 429 Too Many Requests
{
  "status": 429,
  "error": "RATE_LIMIT_EXCEEDED",
  "message": "Rate limit exceeded. Limit: 100/min. Resets in 45s."
}

---

Deployment Guide — Render + Aiven

This project is deployed using Render (free web service) + Aiven (managed MySQL). The combination gives you a production-grade SSL-secured cloud setup.

---

Infrastructure Overview

Component	Platform	Plan
App hosting	Render	Free Web Service
MySQL database	Aiven	Free trial ($300 credit, 30 days)
SSL	Both platforms	Included automatically

---

Step 1 — Set Up Aiven MySQL

1. Go to https://aiven.io and sign up (free trial, $300 credits)

2. Click Create service → choose MySQL

3. Select the free/smallest tier and your preferred region

4. Wait ~2 minutes for the service to reach Running state

5. On the service Overview page, note down the connection details:

   Field	Where to find it
   Host	Overview → Connection information → Host
   Port	Overview → Connection information → Port
   Database name	defaultdb (default)
   Username	avnadmin (default)
   Password	Overview → Connection information → Password (click reveal)

6. Download the CA certificate — click Show next to "CA certificate", then download it. This is needed for SSL verification.

SSL is required by Aiven. Your JDBC URL must include useSSL=true&requireSSL=true.

The full JDBC URL format:

jdbc:mysql://<HOST>:<PORT>/defaultdb?useSSL=true&requireSSL=true&serverTimezone=UTC

Example:

jdbc:mysql://mysql-afb3de5-sahastrajeet-f8d3.g.aivencloud.com:17713/defaultdb?useSSL=true&requireSSL=true&serverTimezone=UTC

---

Step 2 — Configure application.properties for Aiven

In src/main/resources/application.properties, the database section should read:

spring.datasource.url=jdbc:mysql://<YOUR_AIVEN_HOST>:<PORT>/defaultdb?useSSL=true&requireSSL=true&serverTimezone=UTC
spring.datasource.username=avnadmin
spring.datasource.password=${DB_PASSWORD}
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

The password is injected at runtime via the DB_PASSWORD environment variable — never hardcode it in the file.

Commit and push:

git add src/main/resources/application.properties
git commit -m "Update datasource config for Aiven MySQL"
git push origin main

---

Step 3 — Deploy on Render

1. Go to https://render.com and sign up (free, GitHub login works)

2. Click New + → Web Service

3. Connect your GitHub repository: sahastraWin/dataflow-ingestion-engine

4. Set the branch to main

5. Render will auto-detect the Dockerfile in dataflow-engine/ — confirm it

6. Configure the service:

   Setting	Value
   Name	dataflow-ingestion-engine
   Region	Any (pick closest to you)
   Branch	main
   Runtime	Docker
   Instance Type	Free

7. Under Environment Variables, add:

   Key	Value
   DB_PASSWORD	Your Aiven password (from Step 1)
   PORT	10000 (Render's default free port)

   The DB_URL and DB_USERNAME are already set in application.properties if you committed them in Step 2. If you prefer to keep them out of the file, add them as env vars too:

   Key	Value
   DB_URL	jdbc:mysql://<HOST>:<PORT>/defaultdb?useSSL=true&requireSSL=true&serverTimezone=UTC
   DB_USERNAME	avnadmin
   DB_PASSWORD	<your aiven password>

8. Click Create Web Service

Render will build the Docker image and deploy. In ~3–5 minutes your app will be live at:

https://dataflow-ingestion-engine.onrender.com

---

Step 4 — Verify the Deployment

curl https://dataflow-ingestion-engine.onrender.com/actuator/health

Expected:

{ "status": "UP", "components": { "db": { "status": "UP" } } }

If db.status is DOWN:

• Double-check the DB_PASSWORD environment variable on Render
• Verify the Aiven service is in Running state (not paused)
• Confirm the host and port in application.properties match the Aiven overview

---

Step 5 — Re-deploying After Code Changes

Render auto-deploys on every push to main:

git add .
git commit -m "your change"
git push origin main

Or trigger a manual deploy from the Render dashboard → Manual Deploy button.

---

Why Render + Aiven?

Feature	Detail
Cost	Render free tier is always-free for web services
Sleep policy	Free tier spins down after 15 min inactivity; first request wakes it (50s delay)
Database	Aiven free trial gives $300 credit (30 days); enough for extended demos
SSL	Aiven enforces SSL by default — production-grade security
Build	Auto-builds from GitHub on every push via Docker
HTTPS	Free HTTPS on Render included automatically

---

Option B — Run Locally with Docker

cd dataflow-engine
mvn clean package -DskipTests
docker build -t dataflow-engine .
docker run -p 8080:8080 \
  -e DB_PASSWORD="your_password" \
  dataflow-engine

---

Option C — Share the JAR File

cd dataflow-engine
mvn clean package -DskipTests
# JAR is created at: target/engine-1.0.0.jar

Anyone with Java 17 can run it:

java -jar engine-1.0.0.jar

---

Concurrency and Race Condition Prevention

The Problem: Multiple Threads Updating the Same Job

Thread A reads:  processedRecords = 500
Thread B reads:  processedRecords = 500   <-- stale read
Thread A writes: processedRecords = 501
Thread B writes: processedRecords = 501   <-- lost increment (should be 502)

Solution: One Thread Per Job

Job Queue
  Job #1  -->  Thread 1
  Job #2  -->  Thread 2
  Job #3  -->  Thread 3
  Job #4  -->  Thread 4

Each job is dequeued exactly once. One worker thread owns one job for its entire lifetime. Two threads never process the same job simultaneously. This eliminates counter race conditions entirely.

Database Consistency via @Transactional

@Transactional
public void markJobAsCompleted(IngestionJob job) {
    job.setStatus(COMPLETED);
    job.setCompletedAt(LocalDateTime.now());
    jobRepository.save(job);      // if this fails,
    eventPublisher.publish(job);  // DB rolls back automatically
}

Thread-Safe Rate Limiting

// ConcurrentHashMap: segment-level locking, no global lock
ConcurrentHashMap<String, RequestWindow> clientWindows;

// AtomicInteger: single CPU instruction (CAS), no synchronized block
AtomicInteger requestCount;
requestCount.incrementAndGet();

---

Interview Talking Points

"Why did you build this?"

I noticed the focus on operational efficiency and workload management. To understand the challenges an analytics team faces — ingesting massive datasets reliably without overwhelming infrastructure — I built a backend engine that directly addresses those problems.

On the Thread Pool Design:

Instead of creating a new thread per job (which would create thousands of threads under load and crash the JVM), I implemented a ThreadPoolTaskExecutor with 5 core threads, 20 maximum, and a 500-job queue. When the queue fills, CallerRunsPolicy kicks in — the HTTP request thread processes the job itself, which naturally slows inbound submissions. This is backpressure: the system communicates its saturation by slowing down the inbound flow.

On Preventing Race Conditions:

Our architecture guarantees that only one worker thread ever processes a single job. The job queue acts as a distribution mechanism — once a job ID is dequeued, no other thread can claim it. Additionally, all database mutations go through @Transactional boundaries, ensuring atomicity even if the JVM crashes mid-operation.

On the Strategy Pattern:

The Strategy Pattern lets us support multiple data formats without ever modifying the processing pipeline. Adding Parquet support tomorrow means implementing one interface and registering it — zero changes to the existing code. This is the Open/Closed Principle in action.

On Data Governance:

Every state transition is persisted to the job_logs table with a timestamp, stage name, and record count. This creates a queryable lineage trail. If a job from 6 months ago processed incorrectly, we can replay its exact log to understand what validation rules failed and why. In regulated industries — finance, healthcare — this kind of audit trail is legally required.

---

Project Structure

dataflow-engine/
├── pom.xml
├── Dockerfile
├── src/
│   ├── main/
│   │   ├── java/com/dataflow/engine/
│   │   │   ├── DataFlowEngineApplication.java
│   │   │   ├── config/
│   │   │   │   └── AsyncConfig.java
│   │   │   ├── controller/
│   │   │   │   ├── JobController.java
│   │   │   │   └── MetricsController.java
│   │   │   ├── dto/
│   │   │   ├── exception/
│   │   │   ├── model/
│   │   │   │   ├── IngestionJob.java
│   │   │   │   ├── JobLog.java
│   │   │   │   ├── JobStatus.java
│   │   │   │   └── DataType.java
│   │   │   ├── observer/
│   │   │   │   ├── JobEventListener.java
│   │   │   │   ├── JobEventPublisher.java
│   │   │   │   ├── JobCompletionLogger.java
│   │   │   │   └── JobNotificationObserver.java
│   │   │   ├── repository/
│   │   │   │   ├── IngestionJobRepository.java
│   │   │   │   └── JobLogRepository.java
│   │   │   ├── scheduler/
│   │   │   │   └── JobCleanupScheduler.java
│   │   │   └── service/
│   │   │       ├── JobService.java
│   │   │       ├── JobProcessingService.java
│   │   │       ├── MetricsService.java
│   │   │       ├── RateLimiterService.java
│   │   │       └── strategy/
│   │   │           ├── IngestionStrategy.java
│   │   │           ├── CsvParserStrategy.java
│   │   │           ├── JsonParserStrategy.java
│   │   │           └── IngestionStrategyFactory.java
│   │   └── resources/
│   │       └── application.properties
│   └── test/
│       └── java/com/dataflow/engine/
│           └── CsvParserStrategyTest.java
└── logs/
    └── dataflow-engine.log

---

Configuration Reference

Property	Default	Description
server.port	${PORT:8080}	HTTP port (Render injects PORT=10000)
dataflow.async.core-pool-size	5	Always-alive worker threads
dataflow.async.max-pool-size	20	Max threads under burst
dataflow.async.queue-capacity	500	Max queued jobs before backpressure
dataflow.rate-limit.max-requests-per-minute	100	Rate limit per client
spring.jpa.hibernate.ddl-auto	update	Schema management (use validate in prod)

---

Built With

• Java 17 — Records, text blocks, switch expressions
• Spring Boot 3.2.5 — Web, Data JPA, Validation, Actuator, Scheduling
• MySQL 8.4 (Aiven) — Cloud-hosted relational store with SSL and indexed queries
• HikariCP — High-performance JDBC connection pool
• OpenCSV — RFC 4180 compliant CSV parsing
• Jackson — JSON serialization/deserialization
• Lombok — Boilerplate elimination
• JUnit 5 + AssertJ — Unit testing
• Render — Free cloud app hosting with Docker support
• Aiven — Managed cloud MySQL with SSL

---

Built to demonstrate production-grade backend architecture principles: async processing, concurrency safety, data governance, and operational observability.