InternVL3.5-1B — LiteRT-LM (on-device Vision-Language Model)

OpenGVLab/InternVL3_5-1B converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models, and the same runtime that runs litert-community/FastVLM-0.5B).

InternVL3.5-1B is a compact vision-language model: an InternViT vision encoder + pixel-shuffle + MLP projector feeding a Qwen3-0.6B language decoder (the newer Qwen3 backbone is what distinguishes it from the InternVL3-2B build, which used Qwen2.5-1.5B). This bundle runs it through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully on-device.


File	`InternVL3_5-1B.litertlm` (~0.82 GB)
Vision	InternViT encoder + pixel-shuffle + MLP projector, int8 weights — single 448×448 image → 256 image tokens
Decoder	Qwen3-0.6B, int4 weights (symmetric, blockwise-32 + OCTAV optimal-clipping); input embedding INT8 (externalized section)
Compute	integer
Context (KV cache)	2048
Image input	resized to 448×448 (ImageNet normalization is baked into the vision encoder)
Base model	OpenGVLab/InternVL3_5-1B (Apache-2.0)

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/InternVL3_5-1B model.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Two tabby cats are lying on a pink couch, with remote controls nearby.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	338 tok/s	43.7 tok/s	0.84 s
Apple M4 Max (macOS)	GPU (Metal)	3878 tok/s	176.2 tok/s	0.08 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

The vision tower converts bit-faithfully to the reference — float CPU-parity end-to-end corr ≈ 1.0 (max abs diff ~1e-4), with no FLEX/CUSTOM fallback ops; int8 vision weights preserve grounding. The Qwen3-0.6B decoder uses the same blockwise-32 + OCTAV int4 recipe that scores 90.7% GSM8K on the sibling Ministral-3-3B-Reasoning build. On a reference eager run the model describes photos accurately and in detail (e.g. a black-and-white Ansel-Adams-style landscape → "dramatic mountain landscape … snow-capped peaks … a winding river through a forested valley").

On-device performance: decode/load are expected to be in line with the InternVL3-2B build on the same runtime (~20 tok/s CPU, ~45 tok/s GPU on iPhone 17 Pro for single-image VQA). Independent on-device measurement for this specific 2B/Qwe3 build is recommended before quoting exact numbers.

⚠️ Known limitation — one image per conversation on the GPU backend

Single-image VQA — the primary use case — works on GPU. But on the GPU (Metal) backend, a second image in the same conversation truncates the answer — ask about one image per chat (start a new conversation for a different image). This is GPU-delegate-specific, not a model/bundle issue: on the CPU backend, multi-image works. The same GPU truncation reproduces with Apple's litert-community/FastVLM-0.5B, so it is general to the runtime's GPU fast_vlm path, not specific to this model. For reliable multi-image, run on the CPU backend.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load InternVL3_5-1B.litertlm with the image (vision) tower enabled (modalities [.vision]), attach a photo, and ask a question.

Note for app integrators: this is a vision-only bundle (no audio tower). Bring up the engine with the vision modality only (Modality.textImage / [.vision]) — requesting the audio tower (.all) on a bundle with no audio section fails at session creation.

Run on Android — Google AI Edge Gallery

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download InternVL3_5-1B.litertlm, import it (tap +), attach an image and ask. The bundle already carries the tokenizer and prompt template.

Conversion notes

LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,448,448,3]→[1,256,4096]) + VISION_ADAPTER ([1,256,4096]→[1,256,1024], matched to the Qwen3-0.6B hidden size) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
The vision encoder bakes InternVL's ImageNet normalization and the NCHW transpose into the graph (the runtime feeds a [0,1] NHWC image).
The InternViT attention is rewritten 4D-clean (qkv split before the head reshape, avoiding a 5D intermediate) for the GPU delegate.
Decoder extracted from the InternVLChat wrapper as a standalone Qwen3ForCausalLM (dynamic rope_scaling stripped; exported with cache ≤ base max so base RoPE is exact).