Vulkan Exploration Project

This project explores Vulkan GPU acceleration for TinyLLM training on the B170D device (Linux-based, ARMv8-A).

Overview

• Device: B170D (Linux-based, ARMv8-A)
• Framework: TinyLLM (GGUF-compatible)
• Goal: Implement Vulkan-based GPU acceleration and optimize inference performance on edge devices.

Key Components

• Vulkan Exploration: Benchmarks and experiments with Vulkan shaders for TinyLLM inference
• 4-bit Quantization: Integrated 4-bit quantization (INT4) for reduced memory footprint
• GGUF Export: Conversion utilities to produce optimized GGUF models for TinyLLM

Quick Start

cd ~/tinyllm
python3 explore_vulkan.py \
  --owner=ember-agentic \
  --repo=tinyllm \
  --icd-path=/data/data/com.termux/files/home/.openclaw/vulkan/ \
  --token=ghp_xxx

Output

• Generates big_causal_19ep.q4.gguf (Q4_0 quantized model, ~4 MB)
• Provides reference F32 GGUF for comparison

Repository

• README: https://github.com/ember-agentic/tinyllm
• State: Under development — Vulkan benchmarks and 4-bit quantization integration

Dependencies

• openclaw (installed via Node.js alias)
• GitHub SSH access configured
• Vulkan ICD files (path specified via VK_ICD_PATH)

Usage

Local Run

cd ~/tinyllm && python3 explore_vulkan.py \
  --owner=ember-agentic \
  --repo=tinyllm \
  --icd-path=/data/data/com.termux/files/home/.openclaw/vulkan/ \
  --token=ghp_xxx

Crond Schedule

0 * * * * cd /data/data/com.termux/files/home/tinyllm && \
  python3 explore_vulkan.py >> trajectories/cron_log.txt 2>&1

Logs & Trajectories

• All exploration steps and GitHub interactions are logged to: trajectories/explore_vulkan_<timestamp>.jsonl
• Trajectories are stored in the ~/tinyllm/trajectories/ directory.