AI / ML research · 2025
LLM Optimization for Edge Deployment
Compressed LLaMA-3.1-8B by 73.6% (15GB → 3.95GB) with a 273% CPU decode speedup — running on a Raspberry Pi 4.
- PyTorch
- llama.cpp
- GGUF
- CUDA
- Quantization
73.6%model size reduction (15GB → 3.95GB)
273%CPU decode speedup (1.16 → 4.33 tok/s)
The problem
Run a capable 8B LLM on commodity edge hardware — a Raspberry Pi 4, with no GPU.
My role
Researcher & engineer (MS project).
What I built
Compressed LLaMA-3.1-8B with Taylor-based structured pruning, SmoothQuant outlier suppression, and Fisher-information-guided mixed-precision quantization (Q4/Q5/Q6) with sensitivity-aware bit-width allocation across all 32 transformer layers, using imatrix calibration on WikiText-2.
Architecture & stack
- Taylor structured pruning (MLP params 8.03B → 6.34B, 30%)
- SmoothQuant outlier suppression
- Fisher-guided mixed-precision Q4/Q5/Q6 quantization (per-layer bit allocation)
- imatrix calibration on WikiText-2; GGUF export via llama.cpp
- Deployed on Raspberry Pi 4 (~4.2GB memory footprint)