Can Claude Code Scrape Websites? Native Limits and MCP Extensions

Claude Code can fetch a web page and read its HTML. That’s a start, but it’s not scraping. Real scraping involves finding the right pages, handling pagination, extracting structured data, and dealing with sites that render content with JavaScript. Claude Code needs help for most of that.

What Claude Code Can Do Out of the Box

Claude Code has a built-in WebFetch tool. Hand it a URL, and it pulls the page content. It can then parse that HTML, extract the data you care about, and even write a script to automate the process. For a single, static page, this works fine.

> Fetch https://example.com/pricing and extract the plan names and prices into a table.

Claude Code fetches the page, reads the HTML, finds the pricing elements, and formats them. For this kind of one-off extraction from a known URL, you don’t need anything extra.

It can also write scraping scripts. Ask Claude Code to build a Python script using BeautifulSoup or a Node script using Cheerio, and it will generate working code. If you give it a sample of the HTML, it can figure out the right selectors and write the extraction logic.

Where It Falls Short

Discovery. Scraping usually starts with finding the right pages. Claude Code can’t search the web natively, so you’d need to provide every URL yourself. Real scraping workflows start with a search query or a sitemap, not a hand-curated list of links.

JavaScript-rendered content. Many sites load their content via JavaScript after the initial HTML loads. WebFetch pulls raw HTML, so React apps, SPAs, and dynamically loaded product listings come back as empty shells. You’d need a headless browser like Playwright or Puppeteer for those, and while Claude Code can write scripts that use these tools, it can’t run a headless browser through WebFetch.

Pagination and crawling. Multi-page scraping (following “next page” links, iterating through search results) requires orchestration that goes beyond fetching a single URL. Claude Code can write the code to do this, but it can’t execute a multi-step crawl natively within a single tool call.

Rate limiting and politeness. Production scraping requires delays between requests, retry logic, and respect for robots.txt. Claude Code doesn’t manage any of that on its own.

The Practical Scraping Pipeline

Here’s where things get interesting. Claude Code’s strength isn’t in being a scraper itself. It’s in building and orchestrating scraping workflows. The most effective pattern combines MCP tools with Claude Code’s code-writing ability:

  1. Search to find target pages (via an MCP web search tool)
  2. Fetch the pages to inspect their structure (via WebFetch or an MCP fetch tool)
  3. Write a scraping script tailored to that structure (Claude Code generates the code)
  4. Run the script in your terminal (Claude Code executes it)
  5. Process the results (Claude Code analyzes, filters, and formats the data)

This pipeline turns Claude Code from a simple page fetcher into a scraping assistant that can handle real workflows.

Adding Search and Fetch with AgentPatch

The missing piece in that pipeline is step one: discovery. AgentPatch provides Google Search and other tools through MCP, so Claude Code can find pages before it scrapes them.

The AgentPatch CLI is designed for AI agents to use via shell access. Install it, and your agent can discover and invoke any tool on the marketplace.

Install (zero dependencies, Python 3.10+):

pip install agentpatch

Set your API key:

export AGENTPATCH_API_KEY=your_api_key

Example commands your agent will use:

ap search "web search"
ap run google-search --input '{"query": "test"}'

Get your API key from the AgentPatch dashboard.

Install the AgentPatch skill — it teaches Claude Code when to use AgentPatch and how to use the CLI:

/plugin marketplace add fullthom/agentpatch-claude-skill
/plugin install agentpatch@agentpatch

MCP Server (Alternative)

If you prefer raw MCP tool access instead of the skill:

claude mcp add -s user --transport http agentpatch https://agentpatch.ai/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

Replace YOUR_API_KEY with your actual key from the AgentPatch dashboard.

Example Workflow

Say you want to collect pricing data from several competitors:

> Search for the pricing pages of Vercel, Netlify, and Railway,
  then extract their plan names and prices into a CSV.

Searching Google for "Vercel pricing page 2026"...
Searching Google for "Netlify pricing page 2026"...
Searching Google for "Railway pricing page 2026"...

Found pricing pages. Fetching each one...

Here's the extracted data:

| Provider | Plan     | Price      |
|----------|----------|------------|
| Vercel   | Hobby    | $0/mo      |
| Vercel   | Pro      | $20/mo     |
| Netlify  | Starter  | $0/mo      |
| Netlify  | Pro      | $19/mo     |
| Railway  | Trial    | $0 (500h)  |
| Railway  | Pro      | $5/mo base |

I've also saved this as competitors-pricing.csv.

Claude Code searched, fetched, extracted, and formatted. You didn’t provide any URLs or write any code. For more complex scraping (dynamic sites, pagination, hundreds of pages), Claude Code can write a dedicated script using Playwright or Scrapy and run it for you.

Wrapping Up

Claude Code can fetch and parse individual web pages, and it’s great at writing scraping scripts. What it lacks out of the box is web search for page discovery. MCP tools fill that gap, turning Claude Code into a capable scraping assistant. AgentPatch adds search and other tools through a single config block. Check it out at agentpatch.ai.

← Back to blog