Model Export

Open-source implementation of the unified multi-format LLM export and benchmarking pipeline described in our research.

---

Research Paper

SOLO-Export: Unified CLI for Multi-Format Export with Post-Export Benchmarking for Optimized LLM Deployment

Samarth Shukla, Dhruv Diddi, Zeeshaan Mohammed — TechRxiv, October 2025

Read the paper: https://doi.org/10.36227/techrxiv.176072115.56696333/v1

Deploying large language models across heterogeneous environments requires format-specific conversion, precision tuning, and consistent evaluation — tasks that are often fragmented across multiple tools. The paper introduces a unified, configuration-driven pipeline for exporting causal LLMs to multiple deployment backends with integrated post-export benchmarking.

Key findings

Result	Detail
GPU latency	FP16 exports on TensorRT achieved up to 3.2× lower latency vs. baseline FP32
Model compression	INT8 quantization reduced storage by >60% with minimal perplexity impact
Benchmarking	Standardized evaluation on Penn Treebank — latency, token-level accuracy, and perplexity
Deployment coverage	Single pipeline for ONNX, TorchScript, Hugging Face, TensorRT, and TFLite across CPU and CUDA

This repository (model-export) is the maintained open-source codebase for that system.

---

Overview

Model Export is a Python toolkit for exporting causal language models from Hugging Face into multiple optimized deployment formats, applying FP16/INT8 quantization where supported, and benchmarking every artifact with a consistent evaluation harness.

Hugging Face model  →  Export (ONNX / TS / TRT / TFLite / HF)  →  Quantize (FP16 / INT8)  →  Evaluate  →  metrics.json

Use it via CLI for local workflows or via the FastAPI REST API for programmatic and service-based export jobs.

---

Features

Export formats

Format	Backend	Best for
ONNX	ONNX Runtime	Cross-platform serving, broad runtime support
TorchScript	PyTorch	On-premises and cloud PyTorch deployments
Hugging Face	Transformers	Hub-compatible artifact packaging
TensorRT	NVIDIA TensorRT	GPU-optimized inference (CUDA required)
TFLite	TensorFlow Lite	Edge and mobile CPU deployments

Quantization

• FP16 — GPU acceleration with ~50% storage reduction
• INT8 — Dynamic post-training quantization for CPU/edge targets (~75% compression)

Device targets

• CPU — ONNX Runtime, TorchScript, TFLite
• CUDA — ONNX, TorchScript, TensorRT (with automatic GPU availability checks)
• MPS — configurable for Apple Silicon workflows

Evaluation metrics

Each export can be benchmarked with:

• Inference latency — warm-up passes, repeated batches, mean/median/P95 aggregation
• Token-level accuracy
• Perplexity — evaluated against Penn Treebank (paper) or a custom JSONL dataset

Results are written to metrics.json alongside each exported artifact.

---

Tech stack

Layer	Technologies
Language	Python ≥ 3.8
Deep learning	PyTorch, Transformers
Export runtimes	ONNX / ONNX Runtime, TorchScript, TensorRT, TensorFlow Lite
Quantization	FP16 casting, INT8 dynamic quantization (ONNX Runtime PTQ)
API	FastAPI, Uvicorn, Pydantic
Configuration	YAML-driven workflow (config/export.yaml)

---

Architecture

model_exporter/
├── cli/              # Per-format CLI entry points
├── exporters/        # ONNX, TorchScript, HF, TensorRT, TFLite export logic
├── quantization/     # FP16 and INT8 quantization helpers
├── evaluation/       # Format-specific evaluators (latency, accuracy, perplexity)
├── api/              # FastAPI routes for export and eval jobs
└── utils/            # Config loader, logger, config manager

How it works

1. Configuration — config/export.yaml defines the base model, export formats, quantization schemes, device targets, and evaluation settings.
2. Model selection — Interactive CLI updates the config with GPT-2, TinyLlama, DialoGPT, or other Hugging Face causal LM checkpoints.
3. Export — Each exporter loads the model once, wraps it for the target runtime (e.g. ONNX dynamic axes for variable batch/sequence length), and writes artifacts to a uniform directory tree under output/.
4. Quantization — FP16 and INT8 variants are generated per format where the backend supports it.
5. Evaluation — The evaluation harness dispatches to format-specific evaluators and saves metrics.json next to each export.
6. API layer — FastAPI exposes the same export/eval capabilities over HTTP with async job tracking (see README_API.md).

---

Installation

git clone https://github.com/samarthshukla6/Model_Export.git
cd Model_Export
pip install -e .

Or install directly:

pip install model-exporter

Optional dependencies

Some exporters require additional runtime libraries:

• ONNX — onnx, onnxruntime
• TensorRT — NVIDIA TensorRT SDK + CUDA
• TFLite — tensorflow or ai-edge-litert

---

Quick start

1. Select a model

model-export-select-model

Updates config/export.yaml with your chosen Hugging Face checkpoint.

2. Configure exports

Edit config/export.yaml:

model:
  base: gpt2

export:
  formats:
    - onnx
    - torchscript
    - hf
    - tensorrt
    - tflite
  quantization:
    - fp16
    - int8

devices:
  - cpu
  - cuda

output_dir: output

evaluation:
  dataset: config/eval_dataset.jsonl
  metrics:
    - accuracy
    - latency
    - perplexity

3. Run exports

# All formats (via main CLI)
model-export --config config/export.yaml

# Individual formats
model-export-onnx --config config/export.yaml
model-export-ts --config config/export.yaml
model-export-hf --config config/export.yaml
model-export-trt --config config/export.yaml
model-export-tflite --config config/export.yaml

# Evaluation only
model-export-eval --config config/export.yaml

4. Run the API server

python run_api.py
# → http://localhost:8000/docs

See README_API.md for full endpoint documentation.

---

CLI reference

Command	Description
model-export	Run full export + evaluation pipeline
model-export-select-model	Interactively set the base Hugging Face model
model-export-onnx	Export to ONNX (dynamic axes, FP16/INT8)
model-export-ts	Export to TorchScript
model-export-hf	Export Hugging Face–compatible artifacts
model-export-trt	Export to TensorRT (GPU)
model-export-tflite	Export to TensorFlow Lite (CPU)
model-export-eval	Run post-export benchmarking
model-export-api	Launch the FastAPI server

---

Output structure

Exports are organized by model, precision, format, and device:

output/
├── gpt2_fp16_onnx_cpu/
│   ├── model.onnx
│   └── metrics.json
├── gpt2_int8_ts_cuda/
│   ├── model.pt
│   └── metrics.json
└── ...

Each metrics.json contains latency statistics, accuracy, and perplexity for direct cross-format comparison — the same standardized layout used in the paper's experiments.

---

Citation

If you use this tool or build on the research, please cite:

@article{shukla2025soloexport,
  title   = {SOLO-Export: Unified CLI for Multi-Format Export with Post-Export Benchmarking for Optimized LLM Deployment},
  author  = {Shukla, Samarth and Diddi, Dhruv and Mohammed, Zeeshaan},
  journal = {TechRxiv},
  year    = {2025},
  doi     = {10.36227/techrxiv.176072115.56696333}
}

---

License

See repository license for terms of use.

Contributing

Issues and pull requests are welcome at github.com/samarthshukla6/Model_Export.