Fine-Tune LLaMA 3 with QLoRA on a Rented GPU: Complete Guide (2026)

Contains affiliate links. Disclosure.

Fine-tuning adapts a pre-trained model to your domain — support tickets, legal docs, code style, or brand voice. QLoRA makes it practical on a single GPU: quantize the base model to 4-bit, train small LoRA adapters, then merge and deploy. This guide covers the full pipeline: dataset format, libraries, training script, saving and loading, evaluation, and cost estimates. For hardware, Liquid Web offers A100 and H100 GPU servers by the hour.

Why fine-tune? Task-specific improvement, shorter prompts (knowledge in weights), and cost efficiency at scale. QLoRA cuts VRAM by ~4× versus full fine-tuning, so LLaMA 3 8B fits on a single A100 40GB. See Serving with vLLM after training.

Rent a GPU from Liquid Web → | Apify for building datasets →

Why Fine-Tune and What QLoRA Does

Fine-tuning updates a small subset of weights so the model behaves differently on your task. Compared to prompting alone, you get better accuracy, fewer tokens per request, and control over style and knowledge.

QLoRA (Quantized Low-Rank Adaptation):

1. Quantize the base model to 4-bit (NF4). Cuts VRAM ~4×.
2. Train LoRA adapters — low-rank matrices that sit alongside frozen layers.
3. Merge adapters into the base model for inference, or keep them separate for multi-task setups.

Result: LLaMA 3 8B fine-tunes on an A100 40GB. LLaMA 3 70B needs an H100 80GB with QLoRA.

---

Hardware Requirements

Model	Parameters	QLoRA VRAM	Recommended GPU
LLaMA 3 8B	8B	~16–20 GB	A100 40GB, RTX 4090 24GB
LLaMA 3 70B	70B	~48–60 GB	A100 80GB, H100 80GB
Mistral 7B	7B	~14–18 GB	A100 40GB, RTX 3090 24GB

Minimum: 24 GB VRAM for 8B QLoRA. Liquid Web GPU provides A100 and H100 instances.

---

Dataset Format

Use instruction-tuning format (Alpaca style):

[
  {
    "instruction": "Extract the main claim from the following text.",
    "input": "The company announced a 15% revenue increase in Q3.",
    "output": "The company reported a 15% revenue increase in Q3."
  },
  {
    "instruction": "Classify the sentiment.",
    "input": "This product is terrible and broke after one day.",
    "output": "Negative"
  }
]

• instruction: The task description.
• input: Optional. Context or question.
• output: Expected response.

You can build datasets from scraped content with Apify — e.g. FAQ pages, support tickets, product descriptions — then format into this structure. RAG (retrieval-augmented generation) is an alternative when you have many documents but limited labeled examples: see local RAG with Ollama and ChromaDB for that approach. Fine-tuning excels when you need consistent output format, domain-specific terminology, or reduced prompt length.

---

Install Libraries

pip install transformers peft trl bitsandbytes datasets accelerate

For LLaMA 3, you need a HuggingFace token with access to the gated model:

export HF_TOKEN=hf_your_token

---

Full Training Script

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
dataset_path = "./data/train.json"  # Alpaca format
output_dir = "./llama3-qlora-out"

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    use_auth_token=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

# Load dataset
dataset = load_dataset("json", data_files=dataset_path, split="train")

def format_instruction(example):
    if example.get("input"):
        text = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
    else:
        text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return {"text": text}

dataset = dataset.map(format_instruction, remove_columns=dataset.column_names)

# Training
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        learning_rate=2e-5,
        weight_decay=0.01,
        fp16=True,
        logging_steps=10,
        save_strategy="epoch",
        report_to="none",
    ),
)

trainer.train()
trainer.save_model(output_dir)
tokenizer.save_pretrained(output_dir)

Run with HF_TOKEN set. Training time depends on dataset size: ~1–2 hours for 1k examples on A100 40GB.

---

Saving and Loading

Save LoRA only:

model.save_pretrained("./my_lora")
tokenizer.save_pretrained("./my_lora")

Merge LoRA into base for full model:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
merged = PeftModel.from_pretrained(base, "./my_lora")
merged = merged.merge_and_unload()
merged.save_pretrained("./llama3-merged")

Push to HuggingFace Hub:

merged.push_to_hub("your-username/llama3-custom")
tokenizer.push_to_hub("your-username/llama3-custom")

vLLM can load LoRA adapters without merging — useful for A/B testing multiple fine-tuned variants on the same base model. See the vLLM LoRA documentation for adapter loading. For most users, merging and serving the full model is simpler.

---

Evaluation

• Generation tasks: ROUGE, BLEU for summarization/translation.
• Classification: Accuracy, F1 on a held-out test set.
• Human eval: Sample outputs and rate quality.

Use trainer.evaluate() if you pass an eval dataset. For custom metrics, run inference on a test split and compute scores offline. If you lack a labeled test set, sample 20–50 examples and score manually (relevance, fluency, correctness). Track before/after metrics to justify the fine-tuning investment. Once satisfied, merge and serve with vLLM or load into Ollama.

---

Cost Estimate

GPU	Hourly	1k examples (~1–2 hr)	10k examples (~8–12 hr)
A100 40GB	~$1.50–2.50	~$2–5	~$12–30
H100 80GB	~$3–4	~$4–8	~$24–48
RTX 4090 (cloud)	~$0.50–1	~$1–2	~$5–12

Liquid Web and similar providers offer hourly billing. Spin up, train, save artifacts, shut down. After training, serve your model with vLLM for production APIs, or load it into Ollama for a private chat interface. The same GPU server can run ComfyUI for image generation if you need both text and image models.

Start small

Fine-tune on 100–500 examples first. If quality improves, scale to 1k–10k. Use Apify to build datasets from web data.

Frequently Asked Questions

100–500 examples can show improvement for narrow tasks. 1k–10k is typical for production. More data generally helps, but quality matters more than quantity.

Yes, with QLoRA on an 80GB GPU (A100/H100). You need ~48–60 GB VRAM. Reduce batch size to 1 and use gradient checkpointing if needed.

QLoRA: ~4× less VRAM, slightly lower ceiling, faster. Full fine-tuning: best quality, needs more VRAM. QLoRA is the practical choice for most teams.

Merge LoRA into base, then serve with vLLM or Ollama. See Serve LLaMA 3 with vLLM. For LoRA-only, vLLM supports adapter hot-swap.

Lower learning rate (1e-5 to 2e-5). Check data format. Reduce LoRA rank (try 8). Ensure tokenizer and model match (e.g. LLaMA 3 tokenizer for LLaMA 3).