Web Scraping Legal Guide: Public Data, GDPR, robots.txt & ToS (2026)
This guide is a practical compliance primer for engineers and operators: what courts and regulators usually care about, how to reduce risk, and where tools like Apify fit. It is not legal advice. Laws differ by country and industry, and facts control outcomes—always consult a qualified lawyer for your specific use case, especially where personal data, logins, or regulated sectors are involved.
For US/EU case context (hiQ, CFAA framing, robots/ToS) and a structured 2026 compliance framework, see Web scraping legal compliance framework (2026). For AI-era policy and traceability, see Legal architecture for AI training data (2026). This page stays focused on rules of the road and checklists you can use with your legal team.
Nothing here creates an attorney–client relationship. Use this as background for informed conversations with counsel—not as a substitute for professional advice.
Web scraping publicly available data is generally legal. Key rules: respect robots.txt, don't bypass authentication, comply with GDPR, and check ToS. Always consult a lawyer for your specific use case.
1. What “legal scraping” usually hinges on
Legality is rarely answered by “scraping yes/no.” Teams succeed when they document:
- Access — Did you use only public, unauthenticated pages and APIs intended for browsers, or did you circumvent logins, paywalls, or technical barriers?
- Volume & behavior — Could your traffic resemble abuse, scraping behind a block, or a denial-of-service pattern?
- What you copy — Facts (prices, addresses) vs. creative content (articles, photos) vs. personal data (names, emails, profiles).
- What you do next — Analytics, internal pricing, republication, model training, resale—all change risk.
Apify provides infrastructure (runs, proxies, storage). You choose targets, credentials, fields, retention, and downstream use.
2. Public pages vs. authentication
Lower-risk pattern (subject to other rules below): data visible without logging in, the same way a normal visitor would see it.
Higher-risk pattern: scraping after login, using shared or fake accounts, API tokens you were not authorized to use, or bypassing CAPTCHAs or paywalls to reach non-public content. That can implicate computer misuse laws (e.g., the U.S. CFAA), contract claims, and platform enforcement—even before privacy law enters the picture.
Engineering habit: default to no credentials; if a stakeholder demands authenticated data, stop and get legal sign-off on the exact flow and fields.
3. robots.txt and crawling etiquette
robots.txt is a voluntary protocol that expresses crawl preferences. It is not a statute, but ignoring it can increase civil, commercial, and reputational risk—and many enterprises require compliance in vendor reviews.
Practices:
- Fetch and honor
robots.txtwhere your program allows. - Snapshot the file with a timestamp when you run at scale.
- Use reasonable concurrency, backoff on errors, and identifiable user agents where appropriate.
4. Terms of Service (ToS) and contracts
Sites may restrict automated access in their Terms of Use even when pages look public. Breach of contract theories can exist alongside (not instead of) technical access rules. Account-based flows usually strengthen ToS enforcement because the user agreed to terms.
Practices:
- Archive the ToS version date you relied on.
- Separate “read for analysis” from republication or competing products built on scraped content—your counsel can map that to your jurisdiction.
5. GDPR, UK GDPR, and CCPA/CPRA (personal data)
Personal data does not lose protection because it appeared online.
If you process identifiers (names, handles, emails), profiles, or other data about identifiable individuals—especially in the EU/UK or California—you may need:
- A lawful basis (e.g., legitimate interest assessment under GDPR) where applicable
- Data minimization and retention limits
- A path for access/erasure requests and records of processing
- Clear notices where you collect directly from individuals (CCPA-style notice themes)
Scraping only non-personal business facts (e.g., public SKUs and list prices) is often a different risk profile than building a people database from social networks.
6. Copyright and reuse
Even when access is permitted, copying expressive works (long articles, images, video) for republication or model training can raise copyright questions separate from “scraping legality.” Many teams treat RAG over licensed or first-party content differently than large-scale copying of third-party news or photography.
Practice: separate facts for analytics from full-text or media reuse; involve counsel before commercializing datasets built from third-party creative content.
7. Compliance checklist (bring this to legal)
| Step | Action |
|---|---|
| 1 | Confirm pages are public and unauthenticated for the intended flow |
| 2 | Record robots.txt and ToS snapshots with dates |
| 3 | List exact fields; strip unnecessary PII |
| 4 | Define retention and deletion |
| 5 | Document purpose (analytics, ML, resale, etc.) |
| 6 | Add rate limits and monitoring for HTTP errors |
| 7 | Escalate regulated domains (health, finance, children, employment) early |
8. Where Apify fits
Apify helps you run repeatable, scheduled, observable extractions with datasets and APIs—but it does not decide whether a target or use case is lawful for your organization. Pair technical controls (schedules, concurrency, logging) with legal review on targets that include PII, logins, or restricted sectors.
There is no universal ban. Risk depends on how you access data (public vs authenticated), site rules, copyright, and privacy law. This guide is educational only—not legal advice. Always consult a lawyer for your situation.
robots.txt is not a law by itself, but ignoring it can increase legal and business risk and violates many corporate policies. Best practice is to honor it and keep a dated snapshot for audits.
Login flows usually trigger stronger terms and different computer-access analysis. Treat authenticated scraping as high risk unless counsel approves the specific method, scope, and fields.
Often yes, if you process personal data of individuals covered by GDPR. Public visibility online does not automatically mean unlimited processing—you still need appropriate legal bases, minimization, and retention controls where the regulation applies.
Apify is legitimate automation infrastructure. Whether your project is lawful depends on targets, credentials, data categories, and use—not the tool alone. See Is web scraping legal? and speak with counsel for high-risk programs.
No. This FAQ is for general education. Laws change, jurisdictions differ, and outcomes depend on facts. For disputes, contracts, or regulatory inquiries, you need advice from a qualified attorney in your jurisdiction.