vLLM serves open LLMs on GPU with PagedAttention for throughput—batch-score Apify JSON for categories, entities, or compliance flags at scale.
vLLM serves open LLMs on GPU with PagedAttention for high throughput, ideal for batch-scoring large scrape outputs. These guides cover using vLLM to classify, extract, or flag Apify JSON at scale.
When you have many records to process, vLLM's throughput beats one-off API calls on cost and speed. Apify datasets feed the batch. Below you will find tutorials for vLLM-based data processing.
Ollama and vLLM both let you run open-weight LLMs on your own hardware without sending prompts to OpenAI or Anthropic. But they are built for different points on the scale curve — Ollama for developers and small teams, vLLM for production multi-user inference workloads where throughput matters.
Self-hosting OpenClaw with a cloud API backend is easy. But cloud APIs have costs that scale with usage, and they receive every message you send. If your team uses OpenClaw heavily, or if data privacy is a concern, local GPU inference solves both problems: your data stays on your hardware, and you pay a flat server rate instead of per-token fees.
Deploy Meta's LLaMA 3 as a high-throughput, OpenAI-compatible inference API on a dedicated GPU server — with quantization, load testing, and Prometheus monitoring included.
If you've run LLaMA 3 locally with Ollama or llama.cpp, you've hit the ceiling: single-user latency is fine, but concurrency crumbles past three or four simultaneous requests. vLLM solves that. It uses PagedAttention and continuous batching to serve dozens of concurrent requests on the same GPU that would otherwise stall at one.
This guide walks through deploying LLaMA 3 (8B and 70B) on a dedicated GPU server using vLLM — covering Docker setup, model quantization, the OpenAI-compatible API, load testing with Locust, and Prometheus monitoring. All steps are tested on an NVIDIA A100 80 GB, which you can rent from Liquid Web.
vLLM is a high-throughput LLM inference server optimized for production workloads, featuring PagedAttention for efficient KV cache management, continuous batching, and an OpenAI-compatible API. It runs Llama, Mistral, and other models 10-100x faster than naive inference, making it the standard for self-hosted LLM APIs used to process scraped data.
Deploy vLLM on a GPU server, expose the OpenAI-compatible endpoint, and batch-send scraped text chunks for inference. Use async clients to parallelize requests. vLLM's continuous batching automatically groups concurrent requests for maximum throughput. For offline processing of Apify dataset exports, use vLLM's offline inference API for maximum efficiency.
NVIDIA GPU with CUDA support is required for production use (AMD ROCm support exists but is less mature). 24GB VRAM handles 7B models in FP16. 80GB for 70B models in FP16. Multi-GPU tensor parallelism scales to larger models. CPU inference is not vLLM's focus—use llama.cpp for CPU-only inference on scraped data.
vLLM is faster and more efficient for high-throughput batch processing but requires more setup. Ollama is simpler for development and low-volume use. For processing millions of scraped records overnight, vLLM's batching efficiency makes it the production choice. For a developer processing a few thousand records daily, Ollama is sufficient.
