OnDeviceRAG (Android)

A fully offline RAG (Retrieval-Augmented Generation) app. Add a PDF; text extraction, chunking, embedding, vector search and answer generation all run on the device. No server, no API key, no account, no telemetry — the only network traffic is the one-time model download.

This is the Kotlin + Jetpack Compose port of the SwiftUI app LocalVectorDB, built along Google's app architecture guidance (Now in Android): MVVM + Clean Architecture, Hilt, Room, Navigation 3, Kotlin Coroutines/Flow.

---

Table of contents

• How it works
• Requirements
• Getting started
• Model catalog
• Module structure
• Architecture notes
• Configuration
• Tests and quality
• Privacy
• Known limitations
• Roadmap
• Troubleshooting

---

How it works

PDF (content:// URI)
   │  PdfTextExtractor — PdfBox-Android; per-page text, de-hyphenation, whitespace normalization
   ▼
List<ExtractedPage>
   │  TextChunker — sentence boundaries (BreakIterator), overlap, sized to the embedder
   ▼
List<TextChunk>
   │  Embedder — multilingual-e5-small (ONNX Runtime, int8) · fallback: LexicalEmbedder (FNV-1a trigrams)
   ▼
Vectors ──► Room (chunk text + page + little-endian BLOB + which model produced it)
                 │
                 ▼  on launch / when the embedder changes / after every ingest and delete
             VectorIndex — one flat row-major FloatArray in memory, swapped as an immutable snapshot
                 │
   question ─────┤  matrix · query vector = cosine similarities (vectors are L2-normalized)
                 ▼
             top-5 chunks ──► system prompt (rules) + user message (quotes + question)
                                  ──► LFM2-1.2B-RAG (LEAP SDK) ──► token stream + [1][2] citations

Because every vector is L2-normalized, the dot product is the cosine similarity: one multiply-add loop scores thousands of chunks in milliseconds, no ANN index needed.

Requirements

Android Studio	Narwhal 3 or newer (AGP 9.4)
JDK	17+ (gradle/gradle-daemon-jvm.properties asks for JDK 25 for the daemon; Foojay downloads it)
minSdk / targetSdk	31 / 37 — the LEAP SDK declares minSdkVersion=31
ABI	arm64-v8a only (the LEAP SDK ships natives for this ABI only)
Device	A real arm64 device with ≥ 6 GB RAM recommended (LLM ~800 MB + embedder ~130 MB)
Network	First use only: LLM 804 MB, embedding model 129 MB

Getting started

git clone <repo-url> OnDeviceRAG
cd OnDeviceRAG
./gradlew assembleDebug          # build
./gradlew test                   # unit tests (incl. Robolectric)
./gradlew ktlintCheck :app:lintDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk

In the app:

1. Documents tab: the embedding model downloads (with a percentage), then Add PDF and pick a file. Extraction → vector computation (progress bar) → saving are shown; the document appears as "N pages · M chunks".
2. Chat tab: Load downloads the LFM2 weights (% · MB) and loads them into memory. Ask a question; the retrieved sources are listed under "N sources" before the first token arrives.

Model catalog

Role	Model	Size	Source	Local path
LLM	LFM2-1.2B-RAG / Q5_K_M (GGUF)	804 MB	LEAP catalog (leap.liquid.ai → Hugging Face)	no_backup/leap_models/
Embedding	intfloat/multilingual-e5-small, model_qint8_avx512_vnni.onnx + tokenizer.json	113 + 16 MB	Hugging Face	no_backup/embedding_models/
Fallback embedding	Lexical.charTrigram.v1 (FNV-1a, 2048 dims)	—	compiled in	—

LEAP catalog names were verified against https://leap.liquid.ai/api/edge-sdk/model-manifest?model_name=…&quantization_method=…&platform=Android; LFM2-1.2B (base) and Q4_K_M (~697 MB) exist as well. LFM2-1.2B-RAG is tuned to answer from supplied context and hallucinates noticeably less than the base model.

Why this embedding model: docs/adr/0001-embedding-model.md. LLM integration decisions: docs/adr/0002-llm-integration.md.

Module structure

app/                    Activity, NavDisplay (Navigation 3), bottom navigation, Hilt Application, Timber
build-logic/            convention plugins (android.library/.compose/.feature, hilt, room, jvm.library)
core/
  model/                pure Kotlin domain models and sealed error types
  common/               dispatcher qualifiers, @ApplicationScope, Logger interface
  database/             Room: DocumentEntity, ChunkEntity (BLOB vector), DAOs, VectorCodec
  ingestion/            PdfTextExtractor (PdfBox), TextNormalizer, TextChunker
  embedding/            Embedder interface, LexicalEmbedder, ONNX/E5 embedder + Kotlin Unigram tokenizer, EmbedderProvider
  vectorsearch/         VectorIndex (pure Kotlin)
  llm/                  LlmService (LEAP wrapper), LlmModelConfig, RagPromptBuilder
  data/                 DocumentRepository, IngestionManager, IndexSynchronizer, RagPipeline, use cases
  designsystem/         theme, colors
  ui/                   shared composables (StatusBanner, ProgressCard, EmptyState, UiMessage)
  testing/              fakes of the interfaces, MainDispatcherRule
feature/
  documents/            DocumentsRoute / Screen / ViewModel / UiState / Action
  chat/                 ChatRoute / Screen / ViewModel / UiState / Action
docs/adr/               architecture decision records

Dependencies always point inwards: feature → core:data → core:{database,ingestion,embedding,vectorsearch,llm} → core:model. Feature modules never depend on each other; the UI layer never sees Room/LEAP/PdfBox/ONNX types (concrete classes are internal and bound in Hilt modules).

Architecture notes

The decisions that are easy to get wrong:

• Vectors are stored as little-endian BLOBs (VectorCodec). The JVM default is big-endian, so the byte order is pinned explicitly; a database copied between devices decodes identically.
• The embedder identity is part of the document record. Vectors from different models do not share a space; VectorIndex.rebuild filters by the active model and reports staleCount. The UI shows those chunks with an orange warning ("delete and re-add").
• The lexical fallback uses FNV-1a, not hashCode(). Vectors are written to disk, so bucket assignment must be stable across processes and releases. Vectors are interchangeable with the iOS app (LexicalEmbedderTest checks golden values). Lowercasing uses the Turkish locale so that İ/ı stay distinct in Turkish documents.
• Room is the single source of truth; VectorIndex is a derived view. IndexSynchronizer rebuilds it when the embedder becomes ready/fallback and after every ingest or delete. A rebuild swaps an immutable snapshot, so a concurrent search never sees a half-built matrix.
• Ingestion and model downloads run in @ApplicationScope and survive rotation. Answer generation runs in viewModelScope; the user can Stop it and the partial text is kept.
• Every question opens a new LEAP conversation. In RAG each question carries its own context; previous quotes must not leak into the next answer.
• Quotes go in the user message, not the system prompt. Per its model card, LFM2-1.2B-RAG expects "Use the following context to answer questions:"; on device it ignored context placed in the system prompt entirely (fluent answers unrelated to the document). Rules stay in the system prompt; quotes + question form the user turn.
• The LEAP thread count is pinned by hand. The SDK's CpuThreadAdvisor derives its limit from the JVM heap and chose a single decoding thread on an 8 GB phone (prefill took minutes). The core LeapDownloader is used with GenerationTimeParameters(numberOfDecodingThreads = 2..4); the manifest's temperature: 0 is preserved.
• minimumScore = 0.01. Mean-pooled embeddings are anisotropic; an absolute threshold cannot separate relevant from irrelevant, so filtering is left to the LLM, which is told to say the answer is not in the document. The tiny value only drops chunks the lexical fallback scored at exactly 0.
• Model files live under noBackupFilesDir. Hundreds of megabytes, always re-downloadable; excluded from Auto Backup and device transfer. The Room database is backed up (data_extraction_rules.xml).
• Error boundary: data sources turn exceptions into Result.failure(<sealed domain error>); ViewModels unwrap the Result; the UI only renders state and the copy from strings.xml.

Configuration

What	Where	Default
LLM name / quantization	core/llm/LlmModelConfig	LFM2-1.2B-RAG / Q5_K_M
Decoding threads	LeapLlmService.decodingThreads()	availableProcessors/2, clamped to 2..4
Retrieved quotes (top-K)	RagPipeline.TOP_K	5
Similarity threshold	RagPipeline.MINIMUM_SCORE	0.01
Chunk size	Embedder.recommendedChunkCharacters	E5: 1000, Lexical: 600 characters
Chunk overlap	TextChunker	20 % of the chunk size
Embedding batch size	DefaultIngestionManager.BATCH_SIZE	16
Embedding model files	core/embedding/onnx/E5ModelSpec	URL + size + SHA-256
Sentence/word locale	TextChunker, LexicalEmbedder	tr (documents are expected to be Turkish)

Tests and quality

./gradlew test                    # 88 unit tests (JUnit 4, kotlinx-coroutines-test, Turbine, Robolectric)
./gradlew ktlintCheck             # ktlint_official (.editorconfig); ktlintFormat fixes
./gradlew :app:lintDebug          # Android Lint, NewApi at error level
E5_TOKENIZER_JSON=/path/tokenizer.json ./gradlew :core:embedding:testDebugUnitTest --tests '*Golden*'

No mocking library; fakes of the interfaces live in core/testing. Coverage:

• TextChunkerTest (overlap, page boundary, overlong sentence, empty input), TextNormalizerTest
• VectorCodecTest (round-trip, endianness, empty array), RagDatabaseTest (cascade delete, projection)
• VectorIndexTest (ranking, topK, minimumScore, stale count, dimension mismatch)
• LexicalEmbedderTest (determinism, ‖v‖≈1, FNV-1a golden values, inflected words)
• UnigramTokenizerTest, TokenizerJsonParserTest, MeanPoolingTest, E5TokenizerGoldenTest (byte-for-byte match with HF tokenizers on the real vocabulary; skipped unless the file path is set)
• RagPromptBuilderTest, RagPipelineTest (no-result / normal path), DefaultIngestionManagerTest
• DocumentsViewModelTest, ChatViewModelTest (canAsk conditions, streaming, cancel, errors)

Privacy

• Documents, extracted text and vectors never leave the device; they live in the app's Room database.
• The only network traffic is the one-time model downloads (LEAP catalog / Hugging Face).
• No analytics, telemetry, accounts or API keys; INTERNET is the only permission requested.

Known limitations

• LLM language quality. LFM2's official language list is English, Arabic, Chinese, French, German, Japanese, Korean, Portuguese and Spanish. Answers in other languages (e.g. Turkish) stay grounded in the document but can read clumsily; questions in a supported language get cleaner answers. The prompt asks the model to answer in the language of the question.
• Time to first token. With a 1.2B model and a ~1500-token prompt, a mid-range SoC (Helio G85) needs 40–75 s before the first token. Stronger devices are much faster; lowering TOP_K shortens the prefill.
• No OCR. Scanned PDFs without a text layer are rejected with an explicit error.
• PDF only. No DOCX, EPUB or plain text.
• Brute-force search; the whole index lives in RAM. Fast for thousands of chunks; hundreds of thousands would need an ANN index.
• Re-indexing is manual. After an embedding model change, old documents must be deleted and re-added (the UI says so).
• No conversation history. Every question is independent.
• arm64-v8a only. On an x86_64 emulator the APK installs via ARM translation; if the LEAP natives fail to load this is shown as LlmStatus.Failed, and embedding/search keep working.
• Tokenizer normalization uses Java NFKC instead of SentencePiece's nmt_nfkc (see ADR 0001).

Roadmap

• Hybrid retrieval: combine semantic and lexical scores.
• OCR for scanned PDFs (ML Kit Text Recognition).
• More formats (plain text, Markdown, DOCX, EPUB).
• Conversation history with query rewriting for follow-ups.
• Persistent, incrementally updated index instead of a full rebuild.
• ANN index (HNSW) for hundreds of thousands of chunks.
• Language-selectable UI and pipeline locale.
• A stronger multilingual LLM option once available in the catalog.

Troubleshooting

Symptom	Cause / fix
Documents shows "Using the lexical fallback embedder"	The embedding model could not be downloaded/loaded; the reason is in the banner. Fix connectivity and tap "Retry semantic model".
"N chunks were produced with a different embedding model"	They were vectorized by another model; delete and re-add the document.
Chat shows "The model cannot run on this device"	Non-arm64 ABI; the LEAP natives are missing. Use a real device.
"Model download failed: …"	Network error or a name missing from the catalog; check LlmModelConfig.
The answer takes very long	Prefill; see Known limitations. Stop cancels.
"This PDF has no text layer"	Scanned document; OCR is not implemented yet.
"This PDF is encrypted"	The owner password forbids text extraction.

License

MIT. Third-party components keep their own licenses: the LEAP SDK and LFM2 weights are governed by LiquidAI's terms; multilingual-e5-small (MIT), PdfBox-Android (Apache 2.0), ONNX Runtime (MIT).