---
name: scrapingdog
description: |
  Scrapingdog gives AI agents and apps live web data: Google search and
  its verticals (Maps, News, Shopping, Images, Jobs, Scholar, Trends,
  Patents, AI Overview, AI Mode), Bing, Baidu, Amazon, Walmart, eBay,
  X, YouTube, ChatGPT answers, screenshots, and a general web
  scraper with proxies and JavaScript rendering. Use this skill to
  connect the hosted Scrapingdog MCP server, get an API key, or call the
  REST API from code.
---

# Scrapingdog

Scrapingdog returns live web data as structured JSON. It handles proxies,
CAPTCHAs, headless browsers, and retries. Agents use it in two ways:

- **During a session**, through the hosted MCP server at
  `https://mcp.scrapingdog.com/mcp`, with 35 tools. Nothing to install.
- **From code**, through the REST API at `https://api.scrapingdog.com`,
  one HTTP GET per request.

Both are billed to the same API key at the same credit cost. A failed
request is not charged.

## Setup Order

1. **Get an API key.** Reuse `SCRAPINGDOG_API_KEY` if it is already in the
   environment or the project's `.env`, and check that it works (see
   [Check a key](#check-a-key)). Otherwise follow
   [Get an API Key](#get-an-api-key).
2. **Connect the MCP server** in the harness you are running in
   ([Path A](#path-a-live-web-data-in-this-session)). Configure other
   harnesses only when the user asks.
3. **Verify** with one real tool call that returns results before you
   call the setup done.

Do each check yourself when you have the tools to run it. Involve the
human only for the steps an agent cannot do: creating the account and
copying the key out of the dashboard.

## Choose Your Path

- **Need web data while you work** -> Path A (MCP tools)
- **Building a product that calls Scrapingdog** -> Path B (REST API from code)
- **Cannot or should not install anything** -> Path C (REST API from the shell)
- **No API key yet** -> [Get an API Key](#get-an-api-key) first

---

## Get an API Key

Scrapingdog has no device-code or browser-callback flow for agents. The
human creates the account and gives you the key:

1. Ask the human to sign up, or sign in, at
   **https://api.scrapingdog.com/register**. A new account starts on a
   free trial with no card required.
2. The API key is shown on the dashboard at
   **https://api.scrapingdog.com/**. Ask the human to paste it to you.
3. Save it where the project keeps secrets, never in source control:

```dotenv
SCRAPINGDOG_API_KEY=your_api_key
```

Team members have their own keys. Usage is billed to the team owner's
plan.

### Check a key

`/account` returns the plan, the credits left, and the live concurrency.
It does not spend credits, so call it as often as you need:

```bash
curl -s "https://api.scrapingdog.com/account?api_key=$SCRAPINGDOG_API_KEY"
```

`requestLimit - requestUsed` is the credit balance, and `threadCount`
against `concurrency_limit` is how many requests are in flight. A `4xx`
with `"success": false` means the key is wrong; ask the human to copy it
again.

The response body also contains the API key itself, so read the fields
you need instead of printing the whole response into chat, a log, or a
commit.

---

## Path A: Live Web Data in This Session

The MCP server is hosted, so there is nothing to install or run. It
speaks Streamable HTTP and takes the key as a bearer token on every
request.

- **URL:** `https://mcp.scrapingdog.com/mcp`
- **Auth header:** `Authorization: Bearer <SCRAPINGDOG_API_KEY>`

### Claude Code

```bash
claude mcp add --transport http scrapingdog https://mcp.scrapingdog.com/mcp \
  --header "Authorization: Bearer $SCRAPINGDOG_API_KEY"
```

Then run `/mcp` and check that `scrapingdog` shows as connected. Add
`--scope user` to make it available in every project.

### Cursor, Windsurf, and other clients with an `mcp.json`

```json
{
  "mcpServers": {
    "scrapingdog": {
      "url": "https://mcp.scrapingdog.com/mcp",
      "headers": { "Authorization": "Bearer YOUR_API_KEY" }
    }
  }
}
```

Cursor reads `~/.cursor/mcp.json` (all projects) or `.cursor/mcp.json`
(one project). Do not commit a file that holds a real key.

### VS Code (Copilot agent mode)

`.vscode/mcp.json`, which prompts for the key so it never lands in the
file:

```json
{
  "inputs": [
    { "type": "promptString", "id": "scrapingdog-key", "description": "Scrapingdog API key", "password": true }
  ],
  "servers": {
    "scrapingdog": {
      "type": "http",
      "url": "https://mcp.scrapingdog.com/mcp",
      "headers": { "Authorization": "Bearer ${input:scrapingdog-key}" }
    }
  }
}
```

### Codex

`~/.codex/config.toml`, with the key read from the environment:

```toml
[mcp_servers.scrapingdog]
url = "https://mcp.scrapingdog.com/mcp"
bearer_token_env_var = "SCRAPINGDOG_API_KEY"
```

### ChatGPT and claude.ai

Their connector directories need OAuth, and the Scrapingdog MCP server
takes an API key, so these chat apps cannot connect to it yet. Use a
coding harness above, or Path C.

### Verify the connection

1. The tool list should contain `google_search`, `web_scrape`, and the
   others in the [tool guide](#which-tool-to-use). If your harness
   cannot list tools, this works from any shell:

```bash
curl -s https://mcp.scrapingdog.com/mcp \
  -H "Authorization: Bearer $SCRAPINGDOG_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

2. Call `google_search` with `query: "scrapingdog"` and check that it
   returns organic results with links. Only then mark Path A ready.

A `401` with `missing_api_key` means the header did not reach the
server, so check how your client sends headers. A `401` with
`invalid_api_key` means the key itself is wrong.

---

## Which Tool to Use

Pick the most specific tool. It returns parsed fields, where
`web_scrape` returns a raw page.

| Need | Tool |
| --- | --- |
| Any URL, no dedicated tool | `web_scrape` (set `markdown: true` for clean text, `dynamic: true` for JavaScript pages) |
| Web search | `google_search`, `bing_search`, `baidu_search`, `universal_search` |
| Google's AI answers | `google_ai_overview`, `google_ai_mode` |
| An answer from ChatGPT | `chatgpt` |
| Places and businesses | `google_maps` |
| News, jobs, papers, patents | `google_news`, `google_jobs`, `google_scholar`, `google_patents` |
| Search interest over time | `google_trends` |
| Products and prices | `google_shopping`, `google_immersive_product`, `amazon_search`, `amazon_product`, `amazon_offers`, `amazon_reviews`, `walmart_search`, `ebay_search` |
| Images and visual search | `google_images`, `google_lens` |
| Stocks and markets | `google_finance` |
| Hotels | `google_hotels` |
| Video | `youtube_search`, `youtube_video`, `youtube_channel`, `youtube_comments`, `youtube_transcripts`, `google_shorts` |
| People and profiles | `x_profile` |
| What a page looks like | `screenshot` |

`amazon_reviews` is temporarily unavailable: Amazon moved reviews behind
a login wall, so the API cannot return review data at the moment. Do not
build a flow that depends on it, and tell the human rather than retrying.

Default flow:

1. **Search** when you need to discover pages. **Scrape** when you
   already have the URL.
2. Leave the cost-increasing options off unless the plain request fails
   or lacks what you need: `dynamic` and `premium` on `web_scrape`,
   `advance_search` on `google_search`. Each one costs more credits, and
   `advance_search` makes responses much larger.
3. Use `country` (and `domain` on Google tools) when results should
   come from one market.
4. Cite the source URLs that the results return.

---

## Path B: Call Scrapingdog From App Code

Use this path when the integration will keep running inside the user's
product (a backend, a script, an agent loop) after your session ends.
There is no SDK to install. Every API is one GET request with the key in
the query string.

- **Base URL:** `https://api.scrapingdog.com`
- **Auth:** the `api_key` query parameter, read from
  `SCRAPINGDOG_API_KEY`

```bash
curl -s "https://api.scrapingdog.com/google?api_key=$SCRAPINGDOG_API_KEY&query=web+scraping&country=us"
```

```python
import os, requests

r = requests.get("https://api.scrapingdog.com/google", params={
    "api_key": os.environ["SCRAPINGDOG_API_KEY"],
    "query": "web scraping",
    "country": "us",
}, timeout=120)
r.raise_for_status()
results = r.json()
```

```javascript
const params = new URLSearchParams({
  api_key: process.env.SCRAPINGDOG_API_KEY,
  query: 'web scraping',
  country: 'us',
});
const res = await fetch(`https://api.scrapingdog.com/google?${params}`);
if (!res.ok) throw new Error(`Scrapingdog ${res.status}: ${await res.text()}`);
const results = await res.json();
```

For an existing project, read the repo first and add Scrapingdog where
it already calls external APIs and loads secrets. Then make one real
request as a smoke test.

Every MCP tool is one REST endpoint, but the path does not always follow
from the tool name. Most are what you would guess (`google_search` is
`/google`, `web_scrape` is `/scrape`, `amazon_product` is
`/amazon/product`), and these are not:

| Tool | Endpoint |
| --- | --- |
| `google_news` | `/google_news/v2` |
| `universal_search` | `/search` |
| `youtube_search` | `/youtube` |
| `x_profile` | `/x/profile` |
| `google_ai_mode` | `/google/ai_mode` |
| `google_ai_overview` | `/google/ai_overview` |

Look the endpoint up rather than guessing it. The parameters, the credit
cost per call, and the response schemas for every API are in the
documentation: **https://www.scrapingdog.com/documentation/**

### Status codes

| Status | Meaning | What to do |
| --- | --- | --- |
| `200` | Success — data returned | Charged |
| `202` | Accepted and queued for processing | Not a failure. Poll or retry rather than erroring out |
| `400` | Bad parameters, or the target could not be scraped | Not charged. Fix the parameters, or retry |
| `401` | Invalid or missing API key | Not charged. Check the key with `/account` |
| `403` | Your plan does not include this API | Not charged. A human has to upgrade |
| `404` | The requested item does not exist | Not charged. A real answer, not a failure |
| `410` | Gone — the target URL returned no content | Not charged. Retry once, then treat the page as empty |
| `429` | Rate limit exceeded for the plan's concurrency | Not charged. Lower the concurrency and back off |

Requests time out after **60 seconds**. Set the client timeout at least
that high, and retry rather than failing the task on one timeout.

Retry `410`, `429`, and occasional `400`s with backoff. Do not retry a
`401` or `403` in a loop; they need a human.

---

## Path C: No Install, From the Shell

The same REST API works from `curl` with nothing installed. It suits a
one-off lookup when MCP is not connected:

```bash
curl -s "https://api.scrapingdog.com/scrape?api_key=$SCRAPINGDOG_API_KEY&url=https://example.com&markdown=true"
```

You still need a key. See [Get an API Key](#get-an-api-key).

---

## Limits and Etiquette

- **Credits:** every successful call spends credits from the key's plan.
  Check the balance with `/account` before a large batch, and tell the
  human before a run that would use up most of it.
- **Concurrency:** each plan allows a fixed number of requests in
  flight, and going over returns `429`. Batch work within that limit
  rather than firing everything at once.
- **Running out:** if credits run out or a `403` names the plan, report
  it to the human with the upgrade link in the error. Do not work around
  it by retrying or switching keys.
- **Keys are secrets:** do not print a key back into chat, commit it, or
  put it in a URL that will be shared or logged.

## Links

- Sign up: https://api.scrapingdog.com/register
- Dashboard and API key: https://api.scrapingdog.com/
- Documentation: https://www.scrapingdog.com/documentation/
- Pricing: https://www.scrapingdog.com/pricing/
- MCP server source: https://github.com/Darshan972/Scrapingdog-mcp
- Support: info@scrapingdog.com