Using Claude to Generate Python Scrapers for Apify (2026)

Contains affiliate links. Disclosure.

Describe your scraping target to Claude and get Python Apify Actor code. Deploy with apify push. Claude does not run the scraper for you: it generates code you review, test, and deploy. New to Claude? You can try Claude free for a week to draft and iterate on Actor code before you commit. For pre-built targets, check the Apify Store first. Claude shines when you need a custom target: an internal portal, a niche marketplace, or a site with a unique layout. The Store covers popular sites (Indeed, Amazon, LinkedIn); Claude fills the gaps. Expect 15–30 minutes from prompt to first successful run for a simple target.

The Workflow

1. Describe target URL, desired fields, auth, and pagination.
2. Claude generates Python code (Crawlee, Apify SDK).
3. You review selectors, error handling, and storage.
4. Run apify run locally, then apify push to deploy.
5. Iterate: fix broken selectors, add proxy support, tune limits.

What Claude Needs in the Prompt

Input	Why it matters
Target URL	Claude infers site structure and typical selectors
Desired fields	e.g. job title, company, salary, location. Drives output schema
Auth	Login, cookies, API key. Affects whether to use Playwright for sessions
Pagination	Page numbers, "Load more", infinite scroll. Determines crawl logic
Rate limits	Helps Claude add delays and maxConcurrency

Example prompt:

Build a Python Apify Actor that scrapes Indeed job listings from a search URL. Fields: job title, company name, location, salary (if shown), job URL. Handle pagination (next button). Use CheerioCrawler for static HTML or Playwright if the page is JS-rendered. Input: startUrls (list of Indeed search URLs), maxItems (default 50). Output to default dataset.

Example: Claude's Output (Scaffold)

Claude typically produces a structure like this:

import asyncio

from apify import Actor
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    async with Actor:
        actor_input = await Actor.get_input() or {}
        start_urls = actor_input.get("startUrls", [{"url": "https://example.com"}])
        max_items = actor_input.get("maxItems", 50)

        crawler = PlaywrightCrawler(max_requests_per_crawl=max_items)

        @crawler.router.default_handler
        async def handler(context: PlaywrightCrawlingContext) -> None:
            context.log.info(f"Scraping {context.request.url}")
            for item in await context.page.locator(".job-card").all():
                await context.push_data({
                    "title": await item.locator("h2").text_content(),
                    "company": await item.locator(".company").text_content(),
                    "url": context.request.url,
                })

        await crawler.run([u["url"] for u in start_urls])


if __name__ == "__main__":
    asyncio.run(main())

This is illustrative. Real selectors depend on the site. See the building Apify Actor guide for structure (TypeScript, but concepts map to Python). Python Actors use Crawlee for Python or the apify-client for simple runs. The apify create -t python-crawlee-playwright template wires up the Actor context, the Crawlee router, and push_data out of the box (use python-beautifulsoup for static HTML). Claude's code should follow that pattern: register a handler on crawler.router, then call crawler.run([...]) with your start URLs. Note that Crawlee's context.push_data() writes to the Actor's default dataset when running on Apify. If Claude produces BeautifulSoup-based code, ensure it integrates with Apify's storage: push items to the default dataset so they appear in the Console.

Reviewing Claude's Code

Check	Action
Selectors	Inspect live page. Use DevTools. Selectors change often.
Error handling	Add try/except around extraction. Use maxRequestRetries and failedRequestHandler.
Storage	Ensure Actor.push_data() with consistent schema. Avoid pushing partial or malformed records.
Input schema	Define .actor/input_schema.json for startUrls, maxItems, proxy.
Proxy	Sites like Amazon need residential proxies. Add proxyConfiguration if blocked.

Claude does not test against the live site. You must validate. Run apify run with a single test URL first. Inspect the dataset in the Apify Console or via apify storage dataset list. If fields are empty, the selectors are wrong. Share the HTML snippet and selector with Claude for a fix. For pagination, test that the "next" logic actually advances. Some sites use JavaScript navigation that Cheerio cannot handle. Switch to Playwright if needed.

Pushing to Apify via CLI

apify login
apify create my-job-scraper -t python-crawlee-playwright   # or use Claude's files
# Edit src/main.py with Claude's code
apify run
apify push

If you already have a local Actor from Claude, copy the files into an apify create scaffold or use a minimal structure:

• src/main.py
• .actor/actor.json
• .actor/input_schema.json
• requirements.txt

Deploy to Apify after apify push.

Iterating with Claude

When selectors break or pages change, re-prompt:

The selector .job-card h2 returns empty. Here's the relevant HTML: [snippet]. Suggest a more robust selector.

Claude can suggest alternatives. For proxy support:

Add Apify proxy configuration to this Actor. Use the default proxy group from input.

Refer to the proxy configuration guide for schema and usage.

Limitations

• Claude does not run the scraper. It generates code. You deploy and run.
• Selectors may be wrong. Sites change. Always verify.
• Claude may use outdated Crawlee or Apify APIs. Check docs.
• Complex sites (heavy JS, captchas, login) often need manual tuning. For login-required pages, include session handling in the prompt. Claude can generate Playwright login flows, but you must test against the live site. Captchas usually require a human or a dedicated solving service; Claude cannot bypass them.

Claude-Generated vs Apify Store Pre-Built

	Claude-generated	Apify Store Actor
Time to first run	15–30 min (prompt, review, push)	1–2 min (run existing Actor)
Customization	Full. You own the code.	Limited to input params
Maintenance	You fix selectors when sites change	Actor author may update
Edge cases	You add error handling, proxies	May already handle
Winner / Best for	Best for: custom targets, learning	Winner: popular sites (Indeed, Amazon, Google)

For Indeed, Amazon, LinkedIn, Google, use the Apify Store. For niche sites or internal tools, Claude-generated scrapers are viable. See error handling before production. Once your Actor is deployed, you can trigger it from Claude via MCP ("Run my job scraper for Austin") without leaving the chat. Claude-generated plus MCP is a strong combo for custom targets with an interactive interface.

Next step

Run Claude-generated Actors from Claude Desktop via MCP. No need to leave the chat to trigger scrapes.

Frequently Asked Questions

No. Claude generates code. You run apify run locally or apify push and run in the Apify Console. Claude MCP can trigger pre-deployed Actors from chat.

Crawlee for Python: PlaywrightCrawler for JS sites, BeautifulSoupCrawler or ParselCrawler for static HTML. All integrate with Apify.

Inspect the page in DevTools. Test selectors in the console. Re-prompt Claude with the HTML snippet and error. Add fallback selectors.

Use the Store for Indeed, LinkedIn, etc. They're maintained. Use Claude for custom job boards or internal sites.

Yes. apify create, apify run, and apify push work for Python. Install with pip or npm install -g apify-cli.