Run Gemma 3 Locally with Ollama: Complete Setup Guide (2026)
Google's Gemma 3 is an open-weight model family optimized for efficiency and quality. Run it locally with Ollama in minutes — no API keys, no cloud dependency. This guide covers installation, model sizes (2B, 7B, 27B), hardware requirements, and how Gemma 3 compares to LLaMA 3.2 and Phi-3. (Issue #396)
Deploy Gemma 3 on a Liquid Web GPU server →
What Is Gemma 3?
Gemma 3 is Google's open-weight language model family, released in 2025–2026. Unlike closed APIs, Gemma 3 weights are publicly available — you can run them locally, fine-tune them, or deploy them on your own infrastructure. Gemma 3 models are instruction-tuned for chat and coding, with strong performance at small parameter counts.
Key characteristics:
- Open weights — Apache 2.0 / Gemma license, run anywhere
- Efficient — 2B and 7B variants fit on consumer GPUs
- Multimodal-ready — Some variants support images; text-only base is most common in Ollama
- Instruction-tuned — Optimized for chat, Q&A, and coding out of the box
Model Sizes: 2B, 7B, 27B
| Model | Parameters | VRAM (FP16) | Best For |
|---|---|---|---|
| gemma3:2b | 2B | ~4–6 GB | Low-resource machines, quick iteration |
| gemma3:7b | 7B | ~8–12 GB | General chat, coding, balanced quality/speed |
| gemma3:27b | 27B | ~24–32 GB | Best quality, complex reasoning, production |
Quantized variants (e.g. gemma3:7b-q4_0) reduce VRAM further if needed.
Hardware Requirements
| Setup | Min VRAM/RAM | Models |
|---|---|---|
| CPU only | 8 GB RAM | gemma3:2b (slow: 1–2 tok/s) |
| GTX 1660 / RTX 3050 | 6 GB | gemma3:2b |
| RTX 3060 / 4060 | 8–12 GB | gemma3:2b, gemma3:7b |
| RTX 3090 / 4090 | 24 GB | gemma3:2b, 7b, 27b |
| Cloud GPU (A10, A100) | 40–80 GB | All sizes, high throughput |
For always-on or multi-user setups, a Liquid Web GPU VPS provides dedicated VRAM and unmetered bandwidth.
Installing Ollama and Pulling Gemma 3
Step 1: Install Ollama
Linux:
curl -fsSL https://ollama.com/install.sh | sh
ollama serve # or: systemctl enable ollama && systemctl start ollama
macOS / Windows: Download the installer from ollama.com.
Step 2: Pull Gemma 3
# Smallest — best for low VRAM
ollama pull gemma3:2b
# Balanced — good quality/speed
ollama pull gemma3:7b
# Best quality — needs 24+ GB VRAM
ollama pull gemma3:27b
Download sizes: 2B ~1.5 GB, 7B ~4.5 GB, 27B ~16 GB.
Step 3: Run via CLI
ollama run gemma3:2b
Type your prompt and get responses directly in the terminal. Use /bye to exit.
Step 4: Run via Open WebUI (ChatGPT-Style Interface)
For a web-based chat UI, run Open WebUI and point it at Ollama:
docker run -d -p 3000:8080 \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
Open http://localhost:3000, select gemma3:2b (or 7b/27b), and chat.
Gemma 3 vs LLaMA 3.2 vs Phi-3: Comparison Table
| Model | VRAM | Speed (tokens/s) | Quality | Best For |
|---|---|---|---|---|
| Gemma 3 2B | ~5 GB | 40–60 | Good for size | Edge, low-end GPUs |
| Gemma 3 7B | ~10 GB | 25–40 | Strong | General chat, coding |
| Gemma 3 27B | ~28 GB | 8–15 | Excellent | Complex reasoning |
| LLaMA 3.2 8B | ~10 GB | 25–40 | Strong | General use, broad support |
| Phi-3 mini | ~4 GB | 50–70 | Good | Very low VRAM |
Verdict: Gemma 3 2B is one of the best tiny models; Gemma 3 7B competes well with LLaMA 3.2 8B. Choose Gemma 3 if you prefer Google's training and tooling; LLaMA 3.2 if you want the largest ecosystem.
Performance Benchmarks
Representative throughput on an RTX 4090 (24 GB):
| Model | TTFT (ms) | Tokens/s | Concurrent (1 user) |
|---|---|---|---|
| gemma3:2b | ~80 | ~55 | — |
| gemma3:7b | ~120 | ~35 | — |
| gemma3:27b | ~350 | ~12 | — |
On CPU (16 GB RAM): expect 1–3 tok/s for gemma3:2b. For production, use a GPU server.
Use Cases
- Coding assistant — Use gemma3:7b or 27b for code completion, debugging, and refactoring. Pair with local RAG for codebase context.
- Local chatbot — Private chat with no API costs. Use Ollama + Open WebUI for a ChatGPT-like interface.
- RAG pipelines — Embed documents with Ollama embeddings, retrieve with ChromaDB, generate with Gemma 3. See local RAG chatbot tutorial.
For high-throughput APIs with multiple concurrent users, consider vLLM instead of Ollama — it scales better under load.
Production Deployment
For 24/7 access or multi-user setups, deploy Ollama and Open WebUI on a Liquid Web GPU VPS. Use Docker Compose with persistent volumes for both Ollama model storage and Open WebUI data. Put the stack behind Nginx or Traefik for HTTPS, and restrict admin access to trusted IPs. Gemma 3 7B on a single A10 or L4 GPU typically delivers 20–40 tokens/second — sufficient for small teams. For higher concurrency, migrate to vLLM, which supports continuous batching and higher throughput.
Use ollama pull gemma3:2b first. Verify the stack works on your hardware, then upgrade to 7B or 27B if you need better quality. For production APIs, see Serve LLaMA 3 with vLLM.
Gemma 3 is Google's open-weight language model family. It comes in 2B, 7B, and 27B parameter sizes. You can run it locally with Ollama, no API keys or cloud required.
Gemma 3 2B needs ~4–6 GB VRAM; 7B needs ~8–12 GB; 27B needs ~24–32 GB. Quantized variants (e.g. q4_0) use less. On CPU, 2B runs with ~8 GB RAM but slowly.
Install Ollama, run ollama pull gemma3:2b (or 7b/27b). Then run Open WebUI with OLLAMA_BASE_URL=http://host.docker.internal:11434. Select Gemma 3 from the model dropdown. See our Ollama + Open WebUI guide.
Gemma 3 2B is among the best tiny models. Gemma 3 7B is competitive with LLaMA 3.2 8B — quality is similar; choose based on licensing and ecosystem preference. LLaMA has more fine-tunes and tooling; Gemma offers strong out-of-box performance.
Yes. Use Ollama embeddings (e.g. nomic-embed-text) to index documents in ChromaDB, then use Gemma 3 as the generation model. See our local RAG chatbot guide for the full pipeline.