This guide shows a full CodeFinetuner run on the TinyUSB examples. The result is a local autocomplete model that has learned the structure on how to use the TinyUSB API.
The run was done on a RunPod pod with a single NVIDIA RTX 4090 (24 GB).
- Why the Examples Folder
- Setup
- Data Preparation
- Configuration
- Run the Pipeline
- Training Results
- Evaluation Results
- Example Completions
- Use the Model in VS Code
TinyUSB has two kinds of code:
src/— the implementation of the stack.examples/— the usage of the stack.
We want a model that calls the API correctly, so only examples/ is used as raw_data_path.
Start a RunPod pod with an RTX 4090 and the CodeFinetuner GPU image:
ghcr.io/cuolm/codefinetuner:gpu
The container starts an SSH service automatically. See the RunPod Setup Guide for pod creation and SSH access, and the NVIDIA GPU / CUDA Compatibility Guide if training falls back to CPU.
Check that the GPU is visible:
nvidia-smiClone TinyUSB into the data directory of your workspace:
mkdir -p data
cd data
git clone https://github.com/hathach/tinyusb.git
cd ..The split is done by the pipeline (split_mode: "auto", 80/10/10).
Create config/codefinetuner_config.yaml:
# shared across all stages
globals: &globals
workspace_path: null # null: defaults to current working directory (CWD)
model_name: "unsloth/Qwen2.5-Coder-3B"
fim_prefix_token: "<|fim_prefix|>"
fim_middle_token: "<|fim_middle|>"
fim_suffix_token: "<|fim_suffix|>"
fim_pad_token: "<|fim_pad|>"
eos_token: "<|endoftext|>"
label_pad_token_id: -100
max_token_sequence_length: 2048
data_language: "c"
data_extensions: [".c", ".h"]
use_unsloth: true
# preprocess stage
preprocess:
<<: *globals
split_mode: "auto"
train_ratio: 0.8 # ignored when split mode manual
eval_ratio: 0.1 # ignored when split mode manual
test_ratio: 0.1 # ignored when split mode manual
max_code_blocks_ast_depth: 8
min_middle_tokens_length: 8
max_middle_tokens_length: 256
fim_examples_per_subblock_ratio: 1.0
rand_to_ast_fim_examples_ratio: 0.0
rand_examples_min_prefix_suffix_tokens_length: 8
tokenizer_batch_size: 32
raw_data_path: data/tinyusb/examples
tree_sitter_parser_path: null # uses tree_sitter_language_pack parser
tree_sitter_definitions_path: null # uses package internal definitions
rng_seed: 0
# finetune stage
finetune:
<<: *globals
model_attn_implementation: "sdpa"
lora_r: 32
lora_alpha: 64
lora_dropout: 0.0
lora_bias: "none"
lora_target_modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "down_proj", "up_proj"]
selected_checkpoint_strategy: "best" # "best" or "last"
trainer_resume_from_checkpoint: null # null->fresh start, "last"->take last checkpoint
trainer_clear_checkpoint_dir: false
trainer_num_train_epochs: 1
trainer_per_device_train_batch_size: 8
trainer_per_device_eval_batch_size: 8
trainer_gradient_accumulation_steps: 4
trainer_learning_rate: 5e-5
trainer_weight_decay: 0.01
trainer_max_grad_norm: 1.0
trainer_lr_scheduler_type: "cosine"
trainer_warmup_steps: 20
trainer_gradient_checkpointing: true
trainer_logging_steps: 10
trainer_eval_strategy: "steps"
trainer_eval_steps: 20
trainer_save_strategy: "steps"
trainer_save_steps: 20
trainer_logging_strategy: "steps"
dataset_shuffle_buffer_size: 50000
dataset_shuffle_seed: 0
# evaluate stage
evaluate:
<<: *globals
benchmark_sample_size: 500
benchmark_shuffle_buffer_size: 10000000
benchmark_shuffle_seed: 42
generation_checkpoint: "pipeline" # "pipeline" -> selected checkpoint from finetune stage
generation_batch_size: 5
generation_max_new_tokens: 128
generation_do_sample: false
generation_temperature: 0.1 # only active if generation_do_sample: true
generation_top_p: 0.99 # only active if generation_do_sample: true
codebleu_ngram_weight: 0.1
codebleu_weighted_ngram_weight: 0.1
codebleu_syntax_ast_weight: 0.4
codebleu_dataflow_weight: 0.4
sentencebleu_ngram_weight_1: 0.25
sentencebleu_ngram_weight_2: 0.25
sentencebleu_ngram_weight_3: 0.25
sentencebleu_ngram_weight_4: 0.25
line_match_number_of_lines: 2
plot_only: false
benchmark_use_existing_dataset: false
# convert stage
convert:
<<: *globals
# mlflow tracking (optional)
tracking:
<<: *globals
use_mlflow: true
mlflow_tracking_path: "/workspace/mlflow" # null -> <workspace_path>/outputs/mlflow
mlflow_experiment_name: "codefinetuner"
mlflow_model_logging_strategy: "adapter" # "none", "adapter", "gguf", or "all"As a tool:
codefinetuner --config="config/codefinetuner_config.yaml"Or, from source:
uv run codefinetuner --config="config/codefinetuner_config.yaml"Follow the run in MLflow:
uv run mlflow ui --backend-store-uri sqlite:///workspace/mlflow/mlflow.dbOne epoch is 135 steps with an effective batch size of 32 (8 × 4 gradient accumulation).
| Step | Train Loss | Eval Loss |
|---|---|---|
| 20 | 1.028 | 1.128 |
| 40 | 0.757 | 0.833 |
| 60 | 0.496 | 0.707 |
| 80 | 0.384 | 0.692 |
| 100 | 0.328 | 0.689 |
| 120 | 0.282 | 0.687 |
| 135 | – | 0.687 |
The eval loss is flat at ~0.687 from step 60 on, while the train loss keeps falling. One epoch is enough.
The evaluate stage compares the base model (unsloth/Qwen2.5-Coder-3B) with the fine-tuned LoRA model on 500 held-out FIM examples.
| Metric | Base | LoRA | Δ | Change | N |
|---|---|---|---|---|---|
| SentenceBLEU ↑ | 0.267 | 0.509 | +0.242 | +90.8 % | 500 |
| CodeBLEU ↑ | 0.430 | 0.608 | +0.178 | +41.5 % | 252 |
| Exact Match ↑ | 0.070 | 0.306 | +0.236 | +337.1 % | 500 |
| Line Match ↑ | 0.070 | 0.344 | +0.274 | +391.4 % | 500 |
| Edit Similarity ↑ | 0.418 | 0.631 | +0.213 | +50.9 % | 500 |
| Perplexity ↓ | 3.949 | 2.947 | −1.002 | −25.4 % | 500 |
Exact match rises from 7 % to 31 %. CodeBLEU is only computed on the 252 samples that parse as valid C on their own.
Most points in the scatter plot are above the y = x line, with a large group at 1.0. Some points are below it, so the adapter is better on average, not on every sample.
All examples come from the held-out benchmark.
// reference
for (uint8_t i = 1; i <= CFG_TUH_DEVICE_MAX; i++) {
if (tuh_msc_mounted(i)) {
dev_addr = i;
break;
}
}// base
for (uint8_t i = 0; i < TUSB_MSC_DEVICE_MAX; i++) {
if (tuh_msc_is_mounted(i)) {
dev_addr = i + 1;
break;
}
}// lora
for (uint8_t i = 1; i <= CFG_TUH_DEVICE_MAX; i++) {
if (tuh_msc_mounted(i)) {
dev_addr = i;
break;
}
}codebleu 0.603 -> 1.000, exact_match 0 -> 1
tuh_msc_is_mounted() and TUSB_MSC_DEVICE_MAX do not exist in TinyUSB.
// reference
_desc_str[0] = (uint16_t) ((TUSB_DESC_STRING << 8) | (2 * chr_count + 2));// base
_desc_str[0] = (uint16_t) (2 + chr_count);// lora
_desc_str[0] = (uint16_t) ((TUSB_DESC_STRING << 8) | (2 * chr_count + 2));codebleu 0.260 -> 1.000, perplexity 1.544 -> 1.001
// reference
tusb_init(BOARD_TUD_RHPORT, &dev_init);// base
tud_init(&dev_init);// lora
tusb_init(BOARD_TUD_RHPORT, &dev_init);perplexity 2.762 -> 1.040
// reference
{
blink_interval_ms = tud_mounted() ? BLINK_MOUNTED : BLINK_NOT_MOUNTED;
}// base
{
if (tud_composite_enabled()) {
tud_composite_set_state(TUD_COMPOSITE_STATE_POWERUP);
} else {
tud_set_state(TUD_STATE_POWERUP);
}
}// lora
{
blink_interval_ms = tud_mounted() ? BLINK_MOUNTED : BLINK_NOT_MOUNTED;
}codebleu 0.519 -> 1.000, perplexity 6.029 -> 1.021
Note: It is difficult to tell if the model has overfitted too much on the training data. The real testing and usefulness are only visible when using the model for autocomplete tasks, because tools such as llama.vscode and llama.vim do not sample deterministically, they use top_k, top_p, and temperature values, unlike the benchmark's greedy decoding.
The convert stage writes the GGUF model to:
outputs/convert/results/Qwen2.5-Coder-3B-lora-merged.gguf
Download it from the pod, serve it with llama.cpp and point llama.vscode at it. See the inference-vscode guide for the full setup.