Event NameSecond Event

Build Your Pocket Brain:
Tiny On-Device LLMs via
MediaPipe on Android

Your Name
Your Role at Company
hero render
Agenda
01
Why On-Device?
02
Shrinking the Model
Now
03
Running on Android
04
Fine Tuning with LoRA
05
Production Considerations
3
live demos

Is MediaPipe LLM Inference truly
Write Once, Run Anywhere?

Shrinking the Model: Quantization
  • Convert model weights & activations from high-precision formats (like 32-bit floating point) to low-precision types (like 8-bit integers).
  • Reducing bit precision lowers memory and energy costs while speeding up arithmetic, enabling efficient model execution on edge devices.
diagram / code / screenshot
FP32 → INT8 Quantization
4×
smaller weights
32-bit floats → 8-bit integers
75%
less memory
per parameter
0
network calls
inference stays on device
Wrapper Classes
  • Class from external Library which is not in our control
import androidx.compose.runtime.Immutable @Immutabledata class UserListWrapper( val user: User) // class from external librarydata class User(val id: Int, val name: String)
Stable
Primitives · String · data class with val
SkippableRestartable
UnStable
Interfaces · List · var properties
Recomposes
vs
Gemma 3n 2B
🏃 Running Gemma Models Directly On Android App
Demo 1
Key Takeaways
Keep inference on device: faster, private, offline
Quantize before you ship: smaller, faster, cheaper
Use LoRA to specialise a tiny model for your domain
Thankyou.
Do you have any questions?
Scan to connect with me 😎
← → to navigate · F for fullscreen