Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,13 @@ INTENT → SPEC 순으로 그쪽이 이깁니다.
| 세션 API | 부분 (결함) | 오류를 `raise` 대신 `return`, 404 대신 500, 생성 시 모델 이름 미검증 |
| Android · 데스크톱 · 웹 클라이언트 | 구현 | `app/build.gradle.kts` |
| iOS | 부분 | Xcode 스크립트가 없는 `:composeApp` 모듈을 호출 |
| 모델 선택 | 부분 | 클라이언트에 하드코딩, 서버 목록 미사용 |
| 모델 선택, 서버 주소 | 구현, 테스트 있음 | 서버의 `GET /api/models` 를 읽음 (S2.5). 서버 주소 파싱과 API 키 포함 (S2.2). Android 는 주소 설정 화면이 아직 없음 |
| 대화 기록 | 부분 | 메모리에만 |
| 설정 화면, 도메인 계층(Clean Architecture) | 예정 | 빈 파일 |
| 온디바이스 추론 · 오프라인 · 동기화 | 예정 | 코드 없음 |
| 네이티브 데스크톱 (GraalVM) | 구현 (Windows x64), 미실행 | #71. 장기적으로 compose-multiplatform-extended 가 관리 |
| Kotlin 테스트 | **없음** | `1 + 2 == 3` 하나뿐 |
| 네이티브 데스크톱 (GraalVM) | 예정 (SPEC S3.4) | Windows x64 빌드 경로는 develop 에 있으나(#71) 네이티브 빌드는 실행해 본 적이 없다. 장기적으로 compose-multiplatform-extended 가 관리 |
| Kotlin 테스트 | 부분 | 서버 주소, 모델 목록 파싱, 종료 코드 매핑은 테스트 있음(CI). 화면과 뷰모델은 테스트 없음 |
| GPU(CUDA) 추론 | 예정 (M4) | 엔진 기본값은 `device="cpu"`. torchnative CUDA 는 실행해 본 적이 없다 |

## 3. 구조

Expand Down
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Most AI chat apps are a thin window onto someone else's server. Gemstone starts

Today that means a Kotlin [Compose Multiplatform](https://www.jetbrains.com/compose-multiplatform/)
client for Android, iOS, desktop and the web, talking to a Python server you run yourself on your
own GPU. Next, the same Python model code moves *into* the app through
own machine (the engine runs on the CPU for now; GPU support is planned). Next, the same Python model code moves *into* the app through
[Python Multiplatform](https://github.com/thisisthepy/python-multiplatform), so chatting needs no
separate server and no network at all.

Expand All @@ -40,7 +40,7 @@ separate server and no network at all.
- 🚀 **Streaming chat** — tokens arrive over a WebSocket as they are generated and render as Markdown on the fly.
- 🧠 **Visible reasoning** — the model's `<think>` block appears as a collapsible panel with elapsed time.
- 🔌 **Tool calling** — weather, public holidays, exchange rates, a calculator and web search, run server-side in parallel and fed back to the model.
- 📦 **Open-weight models on one engine** — Qwen 3 0.6B served by a single transformers-based engine that runs on PyTorch or on [torchnative](https://github.com/thisisthepy/torchnative). Gemstone is becoming a local Ollama replacement: concurrent requests (continuous batching), OpenAI- and Ollama-compatible APIs and 8-bit weights are planned for November 2026.
- 📦 **Open-weight models on one engine** — Qwen 3 0.6B served by a single transformers-based engine that runs on PyTorch or on [torchnative](https://github.com/thisisthepy/torchnative). Gemstone is becoming a local Ollama replacement: concurrent requests (continuous batching with a paged KV cache) and OpenAI- and Ollama-compatible APIs work today; 4-bit weights and GGUF are planned.
- 🖥️ **Native desktop feel** — a JetBrains Jewel decorated window, with Dmg / Msi / Deb installers.

## 🚀 Quick start
Expand Down Expand Up @@ -152,16 +152,22 @@ Gemstone is an early, working prototype. The honest state of each piece:
| Area | State |
|---|---|
| Streaming chat, reasoning display, tool calling | ✅ Working |
| One transformers engine, continuous batching, paged KV cache (on CPU by default) | ✅ Working on PyTorch; 🟡 not yet verified on torchnative |
| Several resident models, eviction, `keep_alive` | ✅ Working |
| OpenAI-compatible API (`/v1`, tool calls passed through) | 🟡 Partial — no `n` > 1, logprobs or embeddings |
| Ollama-compatible API (chat, generate, tags, show, pull, delete, ps, copy, create, embed) | 🟡 Partial — `push` returns 501; no Modelfile, template or blobs |
| Security defaults (loopback bind, origin checks, API key) | ✅ Working |
| Android, desktop and web clients | ✅ Working |
| Model choice and server address in the client | ✅ Models are read from the server; Android has no address setting yet |
| iOS client | 🟡 Framework targets configured; the Xcode build script needs fixing |
| Model choice in the client | 🟡 Hard-coded list, not yet read from the server |
| Chat history | 🟡 In memory only |
| Automated tests | 🟡 Python tests (including a real model) and Kotlin tests for the network layer run in CI; the UI is untested |
| Native desktop executable (GraalVM, no JVM) | ⏳ Planned — a Windows x64 build path exists but has not been run |
| GPU (CUDA) inference | ⏳ Planned |
| 4-bit weights, GGUF | ⏳ Planned |
| Settings screen | ⏳ Planned |
| On-device inference via Python Multiplatform | ⏳ Planned |
| Offline mode, cross-device sync | ⏳ Planned |
| OpenAI-compatible API | ⏳ Planned |
| Native desktop executable (GraalVM, no JVM) | ⏳ In progress |
| Automated tests | ⏳ Not yet — the first priority |

## 📖 Documentation

Expand Down
20 changes: 10 additions & 10 deletions docs/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,14 @@ a status:
OpenAI-compatible API (S1.10), the Ollama-compatible API with its model management (S1.14,
S1.15), the residency shared by all three, with several resident models and derived models
(S1.14, `test_multi_resident.py`), the tool-result cache (S1.6) and the backend
removal (S1.8). Everything else has **no
behavioural test** yet: the only Kotlin test,
`app/src/commonTest/kotlin/gemstone/ComposeAppCommonTest.kt`, asserts `1 + 2 == 3`. So
`implemented` without a named test means *read in code*, never *verified by a test*.
removal (S1.8), plus batching, queue limits, first-use download, errors, tool robustness, security
and client compatibility (each item names its module). Real-model tests (marker `real_model`) run in
CI. Kotlin tests in `app/src/commonTest` cover the network layer (`ModelCatalogTest`,
`ServerAddressTest`, `ChatCloseEventTest`); the UI has none. `implemented` without a named test means
*read in code*, never *verified by a test*.

Status was established by reading the `develop` branch at commit `2be0e37` (2026-10-02). S1.11–S1.16
and S3.4 also record the maintainer's decisions of the same day ([`serving/engine.md`](serving/engine.md)).
Status was first established by reading `develop` at commit `2be0e37` (2026-10-02) and is updated
with every change since. S1.11–S1.16 and S3.4 also record the maintainer's decisions of that day ([`serving/engine.md`](serving/engine.md)).

---

Expand Down Expand Up @@ -274,7 +275,8 @@ Decided (2026-10-02):
Implemented (#84), `api/src/main/engine.py`:

- `Engine(model_id, dtype=, device=, quantization=, chat_template=, ...)` loads a causal LM with
`from_pretrained`. `quantization="q8_0"` goes through torchnative's `TorchnativeConfig`.
`from_pretrained`. `quantization="q8_0"` goes through torchnative's `TorchnativeConfig`; this is the
November weight format but is unverified until torchnative's #15 gate measures q8_0 with batching.
- Calling it with chat messages streams the reply as text chunks. Temperature 0 is greedy and
equals `transformers` `generate`; `seed` makes sampling reproducible.
- Concurrent callers are batched (S1.12). A request the batch cannot serve takes the exclusive
Expand Down Expand Up @@ -307,14 +309,12 @@ timeout and its 503 / 1013 mapping), `api/tests/test_engine.py`, `api/tests/test
SmolLM2-135M. Still open for M1: the same tests on torchnative (cpu, mps) once torchnative TN-M1
lands (2026-10-24).

Proposal, pending confirmation: the engine is transformers 5.x continuous batching with a paged KV
The 2026-10-02 proposal, now implemented (#85, #91) and awaiting verification on torchnative: the engine is transformers 5.x continuous batching with a paged KV
cache, kernels live in torchnative, and the engine uses only the public `torch` / `transformers`
API so a GPU server can run the same code on upstream PyTorch. Constraint: custom kernels cannot be
registered from Gemstone, because `torch.library` registrations are no-ops on torchnative. They
must be torchnative operators. See [`serving/engine.md`](serving/engine.md).

No code on `develop`. Test: none.

### S1.12 Continuous batching — `partial` · G5

Concurrent requests to a loaded model are scheduled into one running batch. A request joins and
Expand Down
Loading
Loading