If you’ve been building something that touches on-device ML — image models, speech, or now GenAI — you’ve almost certainly run into TensorFlow Lite at some point. Google quietly renamed and re-architected it into LiteRT, and the rebrand comes with enough new capability that it’s worth taking seriously, not just as a name change.
What it actually is
LiteRT is Google’s framework for on-device ML and GenAI deployment — it covers conversion, runtime, and optimization in a single stack. It’s the direct successor to TensorFlow Lite, but the scope has grown well beyond “run a tflite model on a phone.”
What’s new that matters
- Compiled Model API — This is the headline change. Instead of manually wiring up delegates for each accelerator, the Compiled Model API handles automatic accelerator selection, true async execution, and efficient I/O buffer management for you. According to Google, this is now the recommended path for new native execution work in Kotlin and C++.
- Unified NPU acceleration — NPU support across different chipset makers (Qualcomm, MediaTek, and others) is now a single consistent API, graduated from early access and open to all users. Before this it was fragmented by vendor.
- GPU acceleration via ML Drift — new buffer interoperability aimed specifically at reducing latency for GenAI inference, not just classical ML.
- Tensor API in C++ — a lightweight, tensor-centric library for graph authoring, if you need to manipulate tensors directly instead of going through the full model pipeline.
- LiteRT CLI — tooling aimed at agentic coding workflows, letting AI agents interact more directly with the LiteRT build/deploy loop.
The ecosystem around it
LiteRT isn’t a solo project — it’s the foundation layer of a small constellation of Google repos:
- LiteRT-LM — the orchestration layer specific to running LLMs (Gemma and others) on edge devices; this is what powers things like Google AI Edge Gallery underneath.
- litert-torch — converts PyTorch models directly to
.tfliteformat, with a separate path via Generative API for re-authoring LLMs. Worth noting: the PyTorch converter is in Beta, and the Generative API is explicitly in Alpha — not everything here is production-stable yet. - ai-edge-quantizer — post-training quantization to shrink models for resource-constrained deployment.
- litert-samples — official sample apps, including a new ASR (automatic speech recognition) sample and image segmentation examples that show the Interpreter API and Compiled Model API side by side.
Getting started — two realistic paths
If you’re new to on-device ML, Google’s guide walks you through Android Studio with a real-time segmentation app step by step. If you’re already comfortable in the space, the fastest path is to grab a pre-trained .tflite model from Kaggle Models and integrate it directly via the standard runtime.
The caveat that needs saying out loud
Some of the performance benchmarks Google shows here — like the multi-token prediction work in LiteRT-LM that delivers more than double decode speed — are Google’s own benchmarks, not independently verified. Treat them as a starting point for your own testing, not as a guarantee.
Why this matters beyond the API list
The bigger story is cost and control: running inference on-device via NPU/GPU offload and quantization means fewer round trips to cloud inference, lower latency, and no data leaving the device. For teams weighing the cost of cloud AI compute against edge deployment, LiteRT is the most complete answer Google has right now to “can we just run this locally instead?”