Teman Isyarat — Android App

Teman Isyarat (Indonesian for "Sign Friend") is an on-device, real-time Indonesian Sign Language (BISINDO) recognition app built with Flutter and Kotlin. It captures hand and pose landmarks via CameraX and MediaPipe, runs inference through a custom TFLite model, and displays predictions — all fully offline.

---

Table of Contents

• Overview
• Features
• Architecture
• Project Structure
• Tech Stack
• Getting Started
• Building
• Project Ecosystem
• License

---

Overview

The app is the mobile endpoint of a three-part system:

1. lm/ — MediaPipe landmark extraction pipeline from raw video
2. model/ — GRU-based neural network training & TFLite export
3. android/ (this repo) — Flutter app wrapping the exported TFLite model for on-device inference

It recognizes 20 BISINDO vocabulary words (Central Java dialect) through live camera feed, without requiring internet connectivity.

Category	Words
Pronouns	Aku, Kamu, Dia
Common	Salam, Terima Kasih, Maaf, Nama
Time	Hari Ini, Besok
Color	Merah, Kuning
Family	Ayah, Ibu
Count	Satu, Dua, Tiga
Other	Teman, Buku
Fruit	Apel, Pisang

Features

• Real-time camera translation — Live CameraX preview with on-screen prediction overlay
• On-device ML — All inference runs locally via TensorFlow Lite; no network required
• MediaPipe landmark tracking — Pose (9 upper-body keypoints) + hands (2 × 21 landmarks)
• Temporal smoothing — Majority-vote over a 10-frame history window for stable output
• Circular frame buffer — 125-frame native FloatArray buffer feeding the TFLite model
• Camera switch — Toggle between front and rear cameras
• Translation history — Persisted locally via history.txt using path_provider
• Skeleton overlay — Live canvas rendering of detected hand and pose landmarks

Architecture

┌────────────────────────────────────────────────────────────┐
│                    Flutter / Dart (UI)                     │
│  ┌─────────┐  ┌──────────────┐  ┌──────────┐  ┌─────────┐  │
│  │ Home    │  │ Translate    │  │ Artikel  │  │Settings │  │
│  │ Page    │  │ Page         │  │ Page     │  │ Page    │  │
│  └─────────┘  └──────┬───────┘  └──────────┘  └─────────┘  │
│                      │ MethodChannel                       │
├──────────────────────┼─────────────────────────────────────┤
│              Android PlatformView (Kotlin)                 │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  CameraX (PreviewView)  ◄──►  MediaPipe Landmarker   │  │
│  │                              (Pose + Hand)           │  │
│  │                                       │              │  │
│  │  ┌────────────────────────────────────┘              │  │
│  │  │  assembleFrame() → 51 landmarks × 3 = 153-dim     │  │
│  │  │                                                   │  │
│  │  │  Circular Buffer (125 frames) ──► TFLite Interp.  │  │
│  │  │                        Softmax + Temporal Smooth. │  │
│  │  │                        ──► MethodChannel callback │  │
│  │  └───────────────────────────────────────────────────┘  │
│  └──────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────┘

Data Flow

1. CameraX frames are fed to MediaPipe Hand + Pose Landmarker
2. 51 landmarks (9 pose + 21 left hand + 21 right hand) × 3 (x, y, z) are assembled per frame
3. Landmarks are pushed into a circular 125-frame FloatArray buffer
4. When full, the buffer is sent to TFLite Interpreter.run() with input shape [1, 125, 153]
5. Raw logits (20 classes) go through softmax → confidence threshold (0.7) → majority-vote temporal smoothing (10-frame window)
6. The predicted label is sent back to Dart via MethodChannel and rendered in the UI

Project Structure

temanisyarat/
├── lib/
│   ├── main.dart                         # App entry, MainPage, navigation, articles, settings
│   ├── pages/
│   │   └── translate/
│   │       ├── translate_page.dart        # Camera + sign translation UI
│   │       ├── translate_controller.dart  # Business logic, channel bridge, history
│   │       └── widgets/
│   │           └── scanning_dots.dart     # Animated scanning indicator
│   └── services/
│       └── history_service.dart           # Local file persistence for predictions
├── android/app/src/main/kotlin/com/example/android/
│   ├── handlandmarker/
│   │   ├── HandLandmarkerPlugin.kt        # Flutter plugin registration (PlatformViewFactory)
│   │   ├── HandLandmarkerView.kt          # PlatformView: CameraX + landmark buffer + TFLite
│   │   ├── HandLandmarkerHelper.kt        # MediaPipe Hand/Pose Landmarker wrapper
│   │   └── HandLandmarkerOverlay.kt       # Canvas skeleton overlay
│   └── MainActivity.kt                   # FlutterActivity entry
├── android/app/src/main/assets/models/
│   └── model_raw.tflite                   # Trained TFLite classification model
├── android/                              # Android native project root
├── assets/icon/logo.png                  # App launcher icon
├── pubspec.yaml                          # Flutter dependencies & config
└── .tool-versions                        # Tool version pinning

Tech Stack

UI Layer (Dart / Flutter)

Component	Technology
Framework	Flutter 3.41 / Dart
State Management	StatefulWidget + setState
Navigation	Navigator.push / Bottom Navigation
Persistence	path_provider (file I/O)
Permissions	permission_handler (camera)

Native Layer (Kotlin / Android)

Component	Technology
Camera	CameraX (view, lifecycle, camera2) 1.4.2
Landmark Detection	MediaPipe Tasks Vision 0.10.29 (Pose + Hand)
ML Runtime	TensorFlow Lite 2.14.0 + select TF ops
Platform Bridge	Flutter MethodChannel + PlatformView
Min SDK	24
Kotlin	2.3.21
Gradle	9.5.1

ML Model

Property	Value
Input shape	[1, 125, 153]
Output shape	[1, 20] (logits)
Classes	20 BISINDO words
Architecture	GRU + 1D Conv + TempAttention
Model size	~2.6 MB (TFLite FP16)
Temporal smoothing	10-frame majority vote
Confidence threshold	0.7

Getting Started

Prerequisites

• Flutter 3.41+ (see .tool-versions)
• JDK 17+ (OpenJDK 26 recommended)
• Android SDK (compileSdk from Flutter Gradle plugin)
• Android device or emulator (API 24+)

Setup

# Clone the repository
git clone https://github.com/temanisyarat/android.git
cd android

# Install Flutter dependencies
flutter pub get

# Run on connected device
flutter run

The TFLite model is bundled in android/app/src/main/assets/models/model_raw.tflite and is copied to the device cache directory on first launch.

Lint & Test

flutter analyze        # Static analysis
flutter test           # Run widget tests

Building

# Debug APK
flutter build apk --debug

# Release APK
flutter build apk --release

# App Bundle
flutter build appbundle --release

# iOS (macOS only)
flutter build ios --release

Note: Release signing currently uses the debug configuration. Configure a proper release keystore before publishing.

Project Ecosystem

This app is part of the Teman Isyarat monorepo, which includes:

Repository	Purpose
lm/	MediaPipe landmark extraction pipeline — converts raw BISINDO videos to .npz landmark arrays
model/	GRU-based neural network training, evaluation, and TFLite export
android/ (you are here)	Flutter + Kotlin Android app for on-device real-time recognition
manager/	Obsidian vault with ADRs, specs, sprint tracking, and team documentation

Data Flow Across Repositories

Video (raw MP4)  ──►  lm/  ──►  .npz landmarks  ──►  model/  ──►  .tflite  ──►  android/  ──►  Live predictions
                         (MediaPipe extract)           (Train GRU)              (On-device inference)

License

This project is developed for academic purposes under the Hibah Jarprak program at Universitas Sebelas Maret (UNS), in partnership with GERKATIN Solo.

---

Built with Flutter, MediaPipe, and TensorFlow Lite for Indonesian Sign Language (BISINDO) recognition.