Serve LLaMA 3 with vLLM: Production Inference API on a GPU Server (2026)

Contains affiliate links. Disclosure.

Deploy Meta's LLaMA 3 as a high-throughput, OpenAI-compatible inference API on a dedicated GPU server — with quantization, load testing, and Prometheus monitoring included.

If you've run LLaMA 3 locally with Ollama or llama.cpp, you've hit the ceiling: single-user latency is fine, but concurrency crumbles past three or four simultaneous requests. vLLM solves that. It uses PagedAttention and continuous batching to serve dozens of concurrent requests on the same GPU that would otherwise stall at one.

This guide walks through deploying LLaMA 3 (8B and 70B) on a dedicated GPU server using vLLM — covering Docker setup, model quantization, the OpenAI-compatible API, load testing with Locust, and Prometheus monitoring. All steps are tested on an NVIDIA A100 80 GB, which you can rent from Liquid Web.

What Is vLLM and Why Does It Matter?

vLLM is an open-source LLM inference and serving library developed at UC Berkeley. Two innovations define it:

PagedAttention stores key-value (KV) cache in non-contiguous blocks of GPU memory, similar to virtual memory paging in an operating system. Classical inference engines pre-allocate a fixed KV cache per sequence. With PagedAttention, memory blocks are allocated on demand and shared across parallel requests, cutting memory waste by up to 55% compared to HuggingFace Transformers.

Continuous batching dispatches new requests into an active batch the moment a sequence finishes, rather than waiting for the entire batch to complete. Combined with PagedAttention, throughput scales near-linearly with batch size.

The practical result: on an A100 80 GB, vLLM serves LLaMA 3 8B at 3–4× higher throughput than Transformers with equivalent latency per request under load.

---

vLLM vs TGI vs Triton: Which Should You Use?

Feature	vLLM	HuggingFace TGI	NVIDIA Triton
Memory management	PagedAttention	Static KV cache	Static KV cache
Batching	Continuous	Continuous	Dynamic
OpenAI-compatible API	Built-in	Built-in	Via wrapper
Quantization	AWQ, GPTQ, FP8	AWQ, GPTQ, EETQ	TensorRT-LLM
Multi-GPU (tensor parallel)	✓	✓	✓
LoRA adapter hot-swap	✓	✓	Partial
Setup complexity	Low (Docker)	Low (Docker)	High (triton server config)
Best for	General production LLM API	HuggingFace-first teams	NVIDIA-only, maximum throughput

Choose vLLM if: You want a low-friction, OpenAI-compatible API that handles most open-weight models with good throughput out of the box.

Choose TGI if: Your team is already deep in the HuggingFace ecosystem and you need its specific model catalogue support.

Choose Triton if: You are deploying on NVIDIA infrastructure at scale and need TensorRT-LLM's last-mile performance tuning — and you have the ops bandwidth to manage it.

---

GPU Memory Requirements for LLaMA 3

LLaMA 3 comes in three sizes. Here are the minimum GPU configurations for production serving (FP16 unless noted):

Model	Parameters	FP16 VRAM	INT4/AWQ VRAM	Recommended GPU
LLaMA 3 8B	8B	~16 GB	~6 GB	RTX 4090, A10G, A100 40 GB
LLaMA 3 70B	70B	~140 GB	~40 GB	2× A100 80 GB or H100 80 GB
LLaMA 3 405B	405B	~810 GB	~230 GB	8× H100 80 GB

Rule of thumb: Multiply the parameter count in billions by 2 to get approximate FP16 VRAM in GB. Add 10–20 % headroom for the KV cache under moderate concurrent load.

For this guide, we use a single A100 80 GB from Liquid Web, which comfortably runs LLaMA 3 70B in AWQ 4-bit quantization.

---

Prerequisites

Before you start, make sure you have:

• A GPU server with NVIDIA A100 or H100 (40 GB minimum; 80 GB recommended for 70B)
• NVIDIA drivers ≥ 535 installed on the host
• Docker Engine ≥ 24.0 and the NVIDIA Container Toolkit (nvidia-docker2)
• nvidia-smi returning your GPU(s) without errors
• A Hugging Face account with access to the gated LLaMA 3 repositories
• A HF_TOKEN environment variable set to your Hugging Face token

Verify your setup before proceeding:

nvidia-smi
# should show GPU name, driver version, and CUDA version

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# should mirror the output above inside Docker

---

Step 1: Pull the vLLM Docker Image

The official vLLM Docker image bundles everything: CUDA, PyTorch, Flash Attention 2, and vLLM itself.

docker pull vllm/vllm-openai:latest

For reproducible production deployments, pin to a specific release:

docker pull vllm/vllm-openai:v0.6.4

Check the vLLM releases page for the latest stable tag.

---

Step 2: Serve LLaMA 3 8B (Quick Start)

Start with the 8B model to verify your pipeline before moving to the 70B.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HF_TOKEN=$HF_TOKEN \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --max-model-len 8192

Flags explained:

Flag	Purpose
--runtime nvidia --gpus all	Pass all GPUs to the container via NVIDIA runtime
-v ~/.cache/huggingface:…	Mount the HuggingFace model cache so weights persist across restarts
--ipc=host	Required for shared memory used by PyTorch data loaders
--dtype auto	Let vLLM select BF16/FP16 based on GPU capability
--max-model-len	Maximum sequence length (prompt + completion). Set lower to save KV cache memory

Once the server is up, you'll see:

INFO:     Started server process [1]
INFO:     Uvicorn running on http://0.0.0.0:8000

Test it with a quick curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in one paragraph."}],
    "max_tokens": 200
  }'

---

Step 3: Quantize LLaMA 3 for Efficient Serving

Quantization reduces VRAM requirements significantly, enabling larger models on smaller GPUs or cutting costs on production hardware.

AWQ vs GPTQ

Method	Algorithm	Quality	Speed	Best for
AWQ (Activation-aware Weight Quantization)	Protects salient weights during quantization	High — minimal accuracy loss	Fast inference (kernel support)	Production serving
GPTQ (Generative Pre-Trained Transformer Quantization)	Hessian-based layer-wise quantization	High	Moderate	Fine-tuned models
FP8 (vLLM native)	Native float8 precision	Near-lossless	Fastest on H100	H100 hardware

Using Pre-Quantized Models from HuggingFace

The easiest approach is to pull a community-maintained AWQ checkpoint:

# LLaMA 3 8B in 4-bit AWQ (~5 GB VRAM)
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HF_TOKEN=$HF_TOKEN \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model casperhansen/llama-3-8b-instruct-awq \
  --quantization awq \
  --dtype half \
  --max-model-len 8192

Quantizing a Model Yourself with AutoAWQ

If you need to quantize your own fine-tuned checkpoint:

pip install autoawq

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3-8B-Instruct"
quant_path = "./llama3-8b-instruct-awq"

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM"
}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

Then serve the quantized model:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $(pwd)/llama3-8b-instruct-awq:/models/llama3-8b-awq \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model /models/llama3-8b-awq \
  --quantization awq \
  --dtype half

---

Step 4: Serve LLaMA 3 70B on a Single A100 80 GB

With AWQ quantization, LLaMA 3 70B fits on a single A100 80 GB (using ~40 GB VRAM):

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HF_TOKEN=$HF_TOKEN \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model casperhansen/llama-3-70b-instruct-awq \
  --quantization awq \
  --dtype half \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

--gpu-memory-utilization 0.90 tells vLLM to use up to 90 % of available VRAM for the KV cache, leaving a small buffer for PyTorch overhead. Adjust down to 0.85 if you see OOM errors during heavy concurrent load.

For the full-precision 70B in FP16 (requires ~140 GB), use two A100 80 GB with tensor parallelism:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HF_TOKEN=$HF_TOKEN \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 8192

---

Step 5: Configure the OpenAI-Compatible API

vLLM's server is a drop-in replacement for the OpenAI API. Any client that calls openai.chat.completions.create(...) can point at your server by changing the base_url.

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://your-server-ip:8000/v1",
    api_key="EMPTY",  # vLLM does not enforce auth by default
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is PagedAttention?"}
    ],
    temperature=0.7,
    max_tokens=512,
)

print(response.choices[0].message.content)

Streaming Responses

stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPU memory."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Adding API Key Authentication

For production, add a static API key to prevent unauthorized access:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HF_TOKEN=$HF_TOKEN \
  --ipc=host \
  vllm/vllm-openai:v0.6.4 \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --api-key your-secret-api-key

Clients must then pass Authorization: Bearer your-secret-api-key in every request.

---

Step 6: Docker Compose for Production

Use Docker Compose to manage restarts, environment variables, and volume mounts cleanly:

# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:v0.6.4
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - huggingface_cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    environment:
      - HF_TOKEN=${HF_TOKEN}
    ipc: host
    restart: unless-stopped
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --quantization awq
      --dtype half
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --api-key ${VLLM_API_KEY}

volumes:
  huggingface_cache:

Start with:

HF_TOKEN=your_hf_token VLLM_API_KEY=your_api_key docker compose up -d

---

Step 7: Load Testing with Locust

Before going to production, stress-test your endpoint to find the concurrency ceiling.

Install Locust:

pip install locust

Create a locustfile.py:

import json
import random
from locust import HttpUser, task, between

PROMPTS = [
    "Explain gradient descent in two sentences.",
    "What is the difference between a transformer and an RNN?",
    "Write a Python function to reverse a linked list.",
    "Summarize the key ideas of the attention mechanism.",
]

class VLLMUser(HttpUser):
    wait_time = between(0.5, 2)

    @task
    def chat_completion(self):
        payload = {
            "model": "meta-llama/Meta-Llama-3-8B-Instruct",
            "messages": [{"role": "user", "content": random.choice(PROMPTS)}],
            "max_tokens": 200,
            "temperature": 0.7,
        }
        headers = {
            "Content-Type": "application/json",
            "Authorization": "Bearer your_api_key",
        }
        self.client.post("/v1/chat/completions", json=payload, headers=headers)

Run the load test (100 concurrent users, 10 user ramp per second):

locust -f locustfile.py \
  --host http://your-server-ip:8000 \
  --users 100 \
  --spawn-rate 10 \
  --run-time 2m \
  --headless \
  --only-summary

Interpreting results: On LLaMA 3 8B (AWQ, A100 80 GB) you should expect:

Metric	Typical Result
Requests/sec	15–30 RPS
Median latency (TTFT)	200–400 ms
P99 latency	1–3 s
Failure rate	< 1 % (at 50 concurrent users)

If failure rate climbs above 1 % at your target concurrency, lower --max-model-len to free KV cache or upgrade to a higher-VRAM GPU.

---

Step 8: Prometheus Monitoring

vLLM exposes a /metrics endpoint in Prometheus format. Scrape it to track GPU utilization, queue depth, and throughput over time.

Key Metrics to Monitor

Metric	Description
vllm:num_requests_running	Active requests currently being processed
vllm:num_requests_waiting	Requests queued waiting for GPU memory
vllm:gpu_cache_usage_perc	KV cache utilization (0–1). Alert at > 0.85
vllm:prompt_tokens_total	Cumulative input tokens processed
vllm:generation_tokens_total	Cumulative output tokens generated
vllm:time_to_first_token_seconds	TTFT histogram — the most important latency metric
vllm:time_per_output_token_seconds	Per-token generation latency

Docker Compose with Prometheus and Grafana

Extend your docker-compose.yml to add a monitoring stack:

services:
  vllm:
    # ... (same as above)

  prometheus:
    image: prom/prometheus:v2.51.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    restart: unless-stopped

volumes:
  huggingface_cache:
  grafana_data:

Create prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: ["vllm:8000"]
    metrics_path: /metrics

Access the Grafana dashboard at http://your-server-ip:3000 (default credentials: admin/admin). Import the vLLM community dashboard from the Grafana catalogue, or build your own panels using the metrics listed above.

Alerting Rules

Add this to a prometheus.rules.yml and reference it from prometheus.yml:

groups:
  - name: vllm
    rules:
      - alert: VLLMHighQueueDepth
        expr: vllm:num_requests_waiting > 20
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "vLLM queue depth is high"
          description: "{{ $value }} requests waiting. Consider scaling or reducing load."

      - alert: VLLMHighKVCacheUsage
        expr: vllm:gpu_cache_usage_perc > 0.85
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "vLLM KV cache nearly full"
          description: "KV cache at {{ $value | humanizePercentage }}. OOM risk."

---

vLLM Production Performance Benchmarks

The following benchmarks are from vLLM's official performance docs and community reproducible runs on A100 80 GB.

LLaMA 3 8B on A100 80 GB (FP16)

Concurrent Users	Throughput (tokens/s)	TTFT P50 (ms)	TTFT P99 (ms)
1	~2,400	35	80
16	~5,800	120	350
64	~8,200	450	1,200
128	~9,100	900	3,500

LLaMA 3 70B AWQ on A100 80 GB

Concurrent Users	Throughput (tokens/s)	TTFT P50 (ms)	TTFT P99 (ms)
1	~800	110	250
8	~2,100	380	900
32	~3,400	850	2,800
64	~3,800	1,600	6,200

Note: TTFT (time to first token) grows with queue depth. For latency-sensitive use cases, cap --max-num-seqs (the maximum number of concurrent sequences) to keep TTFT below your SLA threshold.

---

Choosing a GPU Server for Production

GPU	VRAM	LLaMA 3 8B (FP16)	LLaMA 3 70B	Best for
RTX 4090	24 GB	✓	AWQ only (tight)	Dev / low-traffic
A10G	24 GB	✓	AWQ only (tight)	Cost-effective production
A100 40 GB	40 GB	✓	AWQ only	Mid-scale production
A100 80 GB	80 GB	✓	✓ (AWQ or FP16)	Recommended for 70B
H100 80 GB	80 GB	✓	✓ + FP8	Maximum throughput

Liquid Web offers bare-metal A100 servers with NVMe SSD storage and unmetered bandwidth — well-suited for production LLM serving where consistent latency matters more than spot-instance pricing variability.

---

Frequently Asked Questions

Frequently Asked Questions

vLLM is an open-source LLM inference and serving library from UC Berkeley. It uses PagedAttention to manage GPU memory efficiently and continuous batching to maximize throughput, enabling dramatically higher concurrency than standard inference stacks like HuggingFace Transformers.

LLaMA 3 70B requires approximately 140 GB of VRAM in full FP16 precision (2× A100 80 GB or 2× H100 80 GB). With AWQ 4-bit quantization, it fits on a single A100 80 GB (~40 GB VRAM), with headroom for the KV cache.

Both use continuous batching and offer OpenAI-compatible APIs. vLLM uses PagedAttention for more flexible memory management, which tends to give higher throughput under heavy concurrent load. TGI is tightly integrated with the HuggingFace Hub and ecosystem. For most production use cases, vLLM is the more performant choice; TGI is worth considering if you are already invested in HuggingFace tooling.

Use AutoAWQ to quantize a model offline to 4-bit INT4 format, then pass --quantization awq when launching vLLM. Alternatively, use pre-quantized checkpoints from the HuggingFace Hub (search for '-awq' suffixed repos). For H100 hardware, vLLM also supports native FP8 quantization without a separate quantization step.

On an A100 80 GB, LLaMA 3 8B (FP16) reaches ~8,000–9,000 tokens/second throughput at 64–128 concurrent users with P99 TTFT under 4 seconds. LLaMA 3 70B AWQ peaks around 3,800 tokens/second at 64 concurrent users. Results vary with sequence length, system prompt size, and output length distribution.

Yes. vLLM exposes a fully OpenAI-compatible REST API at /v1/chat/completions, /v1/completions, and /v1/models. Any library or application using the OpenAI Python SDK can switch to vLLM by setting base_url to your vLLM server and using your model ID in place of gpt-4 or gpt-3.5-turbo.

---

Summary

Task	Command / Tool
Start vLLM server (8B)	docker run vllm/vllm-openai --model meta-llama/Meta-Llama-3-8B-Instruct
Quantize to AWQ	AutoAWQ or pre-quantized HuggingFace checkpoints (-awq suffix)
Test API	curl localhost:8000/v1/chat/completions
Load test	Locust with locustfile.py
Monitor	Prometheus /metrics + Grafana dashboard
Production setup	Docker Compose with restart policy and API key

vLLM gives you a production-grade LLM inference server with minimal configuration. The gap between a dev demo and a real production system comes down to quantization (to fit large models on affordable hardware), load testing (to find your concurrency ceiling before your users do), and monitoring (to catch KV cache saturation before it causes latency spikes).

For a dedicated GPU server with the VRAM headroom LLaMA 3 70B demands, Liquid Web's bare-metal GPU servers are a reliable option — no shared tenancy, consistent bandwidth, and the raw I/O performance that model weight loading benefits from.

---

Related guides: Build a Local RAG Chatbot with Ollama and ChromaDB · Run Gemma 3 Locally with Ollama