Ever been on a long-haul flight with no internet and wished you had your AI with you? Or looked at your ChatGPT invoice and wondered if you’re really getting your money’s worth? Also, where do you think all those conversations actually live?

Last week, Google released its latest “open” LLM — Gemma 4. I’ve already switched to Gemma 4, running entirely offline on Mimir, my personal mini operating system (more on that OS build in a future post).

Gemma 4 is free. It’s good enough for most everyday tasks. It works completely offline, it’s small enough to run on your phone, and it’s multimodal, meaning it understands not just text but also images and audio. And your chats with it don’t get sent to the cloud!

I’ll give you a second to pick your brain up off the floor.

It’s unfortunate how little airtime these models get compared to ChatGPT or Claude. So if you had no idea about Gemma 4, for instance, don’t beat yourself up. You’re not alone.

Most folks using AI today are still using the basic chat version of ChatGPT. They use it to draft emails, social media posts, homework assignments (which I do not condone, kids), and to generate summaries of technical documents and reports. For all of those tasks, Gemma 4 is now more than capable. Which means for many people, that $20/month suddenly becomes optional.

None of Your Beeswax

Then there’s the privacy question.

Most people treat AI like their BFF, and the chat like a private notebook. It isn’t. Your conversations live on someone else’s servers, governed by policies you probably haven’t read.

And we’ve already seen cases where user data and chat histories were exposed, like in the 2023 OpenAI breach, where 1.2% of ChatGPT Plus users’ data was exposed, including chat histories and payment details.

That’s before you consider how casually people paste in sensitive work information — internal docs, strategy notes, client data — assuming it behaves like email. It doesn’t.

And while your deepest, darkest, most personal thoughts living on someone else’s servers is probably bad enough, wait until you think about all your proprietary corporate work data you’ve so freely fed to ChatGPT. “Here’s all the confidential project info Rajesh asked for. Now write me a safe and measured corporate email response to him, telling him to go f*** himself”.

An offline model doesn’t send your data anywhere.

Then there are folks like me who still often have to fall back on the more powerful frontier models (though that too could change as these open models get smarter). But for a lot of the simple “chat” tasks on my local Mac, Gemma 4 is now my default.

Setting up

Which brings me to what you’re probably waiting for — how do you install and use Gemma 4 on your phone/laptop?

If you want to try this yourself, here’s the easiest way to start:

iOS/Android:

This is super easy — install the ‘Google AI Edge Gallery’ app from the App Store, then click the “AI Chat” button, choose and download the best Gemma 4 model the app recommends for your phone, and start chatting completely untethered and free!

macOS:

For the Mac, your best bet is Ollama, which is a 5-minute setup. Once in the app, select the Gemma 4 model that will run comfortably on your Mac, and you’ll be chatting locally with your own private LLM in no time!

Here’s a simple use case for an on-device LLM. Say you’re stuck in the mountains with no phone network and sub-zero temperatures. You need instructions on how to build a fire without freezing to death. Enter Gemma 4:

My first thought using Gemma 4 on my iPhone was: “Why the hell isn’t this the default AI model powering Siri?”

Given the recent Apple + Google deal to have Gemini power its AI, it’s not hard to imagine a future version of Siri working like this — on-device, private, and actually useful. Whether Apple uses something like Gemma or builds its own, this direction feels inevitable. We’ll find out on June 8.

We’ve had Siri on our phones for years, and we’ve all but given up on it. It is so completely useless that even the ChatGPT integration they launched two years ago couldn’t get anyone to bother with it.

Should You Switch?

Today, pretty much everyone’s using AI for a lot of basic (no judgment!) tasks that they could actually use a local, private, free model for, but they don’t know that option exists. The barrier to adoption is just awareness, at this point. So this is my attempt at introducing you, dear reader, to the wonderful world of free, private, powerful, on-device artificial intelligence.

Let me know if you’re interested in a more detailed walkthrough of my offline setup, and I’ll include that in next week’s post.

Because the best AI for you is the one that lives on your device.

Read on Substack ↗