Multimodal AI on Your Phone: How Offline Vision Works

A multimodal model on your phone turns a photo into a few hundred or more “image tokens” using a small vision encoder, then the language model reads those tokens next to your question and writes an answer. Both steps run on the phone’s own chip, so the photo never leaves it. Expect a few extra seconds per image, some extra memory, and answers that are good at reading text and describing scenes but shaky on fine detail and counting.

Here’s what happens under the hood, what it costs your phone, and where it goes wrong.

How does a phone AI “see” a photo? #

Language models only understand tokens, the chunks of text they read and write. To handle an image, a multimodal model adds a translation step:

1. The image is resized to the resolution the model expects.
2. A vision encoder cuts it into small patches and turns each patch into a list of numbers describing what’s there.
3. A projector maps those numbers into the same space the language model uses for words. Now the image is a sequence of tokens the model can read.
4. The language model reads the image tokens and your question together and generates the reply one token at a time.

Models differ in how tightly the vision part is built in. Many earlier models bolted a separate vision encoder onto a finished text model. Alibaba’s Qwen 3.5 family was trained on text and image tokens together from early on, which helps its small 4B and 9B versions read screenshots and documents better than their size suggests.

In phone apps built on llama.cpp, the vision part usually ships as its own file, often named mmproj, next to the main model file. That’s why some apps list “image support” as a separate download.

What does vision cost in RAM, storage and time? #

• Storage. The vision file adds roughly 195 MB to 1 GB, depending on the model.
• Memory. The encoder has to be loaded alongside the language model, and each image fills part of the model’s context window, the working memory for the conversation. Several large images in one chat use it up faster than text.
• Time. Before the first word appears, the phone has to process every image token. On a recent phone that’s a few seconds per image; on older phones it’s longer.

Here’s how the vision-capable models in one phone catalog compare. File sizes and RAM floors are from Personal LLM’s model list:

Add the image-support file on top of each download. If you’re unsure which size your phone can handle, read which AI models can run on a phone.

What can on-device vision do well? #

Small multimodal models are strongest when the answer is visible in the picture:

• Reading text: menus, signs, receipts, forms, whiteboards and screenshots.
• Translating what it reads, if the model knows both languages.
• Describing a scene or writing a caption or alt text.
• Explaining a chart or a settings screen you took a screenshot of.
• Comparing two things in the same photo, like two product labels.

Google says Gemma 4’s image understanding covers document parsing, screen and UI understanding, chart comprehension, multilingual OCR and handwriting, according to its model card.

Where small vision models fall short #

• Tiny or blurry text. Resizing loses detail. Crop the photo to the part you care about before asking.
• Counting and exact positions. “How many cars are in this lot?” often gets a confident wrong number.
• Identifying species. A small model can guess a plant or mushroom, but never trust it for anything you’d eat or touch.
• Facts it has to recall rather than see. Small models make things up often. Artificial Analysis measured the Qwen 3.5 4B and 9B giving wrong answers on most of its hard knowledge questions. Ask about what’s in the image, not trivia about it.

What about voice and audio? #

“Multimodal” can also mean speech. Google’s small Gemma 4 E2B and E4B models accept audio input on their own. Most phone chat apps, though, stick to text and images for now. Personal LLM handles typed questions, photos and attached documents, and it can read any answer aloud with on-device text-to-speech.

For voice input, your keyboard’s dictation is the practical route. On many iPhones and in many languages it runs on-device, which we cover in does iPhone speech-to-text work offline.

Cloud multimodal vs on-device multimodal #

The cloud wins on quality. On-device wins when the photo is private (an ID, a medical letter, a screenshot of messages) or when you have no signal.

How to ask your phone about a photo offline #

In Personal LLM, every catalog model supports vision:

1. On Wi-Fi, download a model that fits your phone. The app shows a “Fits your device” badge based on your actual RAM.
2. Tap the option to install image support for that model.
3. In a chat, attach a photo from your gallery or take a new one.
4. Ask a specific question: “What’s the total on this receipt?” works better than “What’s this?”

It works in airplane mode. For a side-by-side with Google Lens and Translate’s camera, see offline alternatives to Google Lens. If you want to edit the photo rather than ask about it, which AI photo editing tools work offline covers that.

Frequently asked questions #

Can AI read photos without internet? #

Yes. A vision-capable model downloaded to your phone can read text, describe scenes and answer questions about a photo in airplane mode. You need a connection once to download the model and its image-support file.

Which phone AI model is best for images? #

For most phones, Qwen 3.5 4B is the best balance of quality and speed. On phones with 8 GB of RAM or more, GLM 4.6V Flash is stronger on documents and UI screenshots. Ministral 3 3B is a quick option for documents on 3 GB phones.

Does analyzing photos use more battery? #

A little more than text, because the phone has to process hundreds of image tokens before it starts answering. A few photos in a session won’t make much difference. Long chats full of large images drain faster and fill the context window.

Is multimodal AI the same as vision AI? #

Vision is one kind of multimodal AI. “Multimodal” means the model handles more than one type of input, such as text plus images, audio or video. Most phone chat apps today are multimodal in the text-plus-images sense.