Google Search is one of the most valuable sources of publicly available data, from SERP rankings and competitor analysis to keyword research and lead generation. But scraping it is not straightforward. Google actively blocks bots, changes its HTML structure, and heavily relies on JavaScript rendering.
In this guide, you’ll learn how to scrape Google Search results using Node.js and later with the Google SERP API. If you prefer Python, check out our guide on scraping Google Search with Python.
TL;DR
If you’re short on time or already understand the scraping logic, you can simply copy this code and use it in your environment.
const axios = require('axios');
const params = {
api_key: 'your-api-key',
query: 'what is scraping?',
country: 'us',
advance_search: 'true',
domain: 'google.com'
};
axios.get('https://api.scrapingdog.com/google', { params })
.then(response => {
console.log(response.data);
})
.catch(error => {
console.error('Error:', error.message);
});
Don’t forget to use your own API key.
Challenges of Scraping Google Search
- Bot Detection — Google fingerprints requests using IP reputation, request headers, and behavioral patterns.
- JavaScript Rendering — Results are JS-rendered, so raw HTTP libraries like Axios return incomplete HTML. A full browser environment is required.
- CAPTCHA — Suspicious traffic triggers CAPTCHA, which becomes a major blocker at scale.
- Frequent DOM Changes — Google updates its HTML structure without notice, breaking selectors regularly.
- IP Blocking — High request volumes from a single IP get it banned fast. Proxy rotation is almost mandatory.
- Geo-Restricted Results — SERPs vary by location, so region-specific scraping requires geo-targeted proxies.
Prerequisites
Before getting started, make sure you have the following:
- Node.js (v18 or above) and npm installed on your machine. If you are new to JavaScript scraping, you can first read our guide on web scraping with Node.js.
- Plawright for rendering the Google search page. You can install it with the command
npm install playwright. - Axios for making an HTTP connection with the host website. You can install it with the command
npm install axios. - Cheerio for parsing the raw HTML. Cheerio can be installed via
npm install cheerio. - A Scrapingdog API key. You can get this by signing up for the trial pack.
Method 1: Scraping Google Search with Playwright and Cheerio
First, we will fetch the raw HTML using Playwright. Then, we will parse that HTML using Cheerio. If you want to learn the basics of Playwright first, check out our guide on web scraping with Playwright.
For this tutorial, we will scrape this page from Google.
const { chromium } = require('playwright');
const cheerio = require('cheerio');
async function scrapeGoogleSearch(query) {
const browser = await chromium.launch({
headless: false,
channel: 'chrome'
});
const page = await browser.newPage();
await page.setViewportSize({ width: 1920, height: 1080 });
// Navigate to search results
const searchUrl = `https://www.google.com/search?q=${query}&gl=us&hl=en`;
await page.goto(searchUrl, { waitUntil: 'domcontentloaded' });
const html = await page.content();
await browser.close();
return html;
}
// Run
scrapeGoogleSearch('spaceX').then(results => {
console.log(JSON.stringify(results, null, 2));
}).catch(console.error);
Here is what the code is doing:
1. Launch a real Chrome browser- Playwright launches an actual Chrome instance (channel: 'chrome') with headless: false, meaning the browser runs visibly. This avoids the bot detection that headless mode often triggers.
2. Open a new page and set viewport- A new browser tab is created and the viewport is set to 1920×1080, a standard desktop resolution that mimics a real user.
3. Build the search URL and navigate- The query is embedded into a standard Google Search URL with gl=us&hl=en to lock results to English (US). Playwright navigates to it and waits until the DOM is loaded.
4. Extract the full page HTML- Once the page is rendered, page.content() grabs the complete HTML including all JS-rendered content — something raw HTTP libraries cannot do.
5. Close the browser and return HTML- The browser is closed cleanly and the raw HTML is returned.
If you see this after running the code, then it means your Playwright setup is working.
Parsing with Cheerio
We will parse three things:
- Title
- Description
- Link
The title is wrapped inside the h3 element with the class LC20lb.
The description is wrapped inside the div tag with the class VwiC3b.
The link is wrapped inside a tag with the class zReHs.
The entire page is wrapped inside a div container with id search.
According to the information gathered above, the Cheerio logic will be:
const $ = cheerio.load(html);
const results = [];
$('#search .tF2Cxc').each((i, el) => {
const title = $(el).find('h3.LC20lb').first().text();
const link = $(el).find('a.zReHs').first().attr('href');
const description = $(el).find('div.VwiC3b').first().text();
if (title && link) {
results.push({ title, link, description });
}
});
return results;
After running the complete code, you get this output:
[
{
"title": "SpaceX",
"link": "https://spacex.com/",
"description": "SpaceX's Starship spacecraft and Super Heavy rocket is a fully reusable transportation system designed to carry both crew and cargo to Earth orbit, the Moon, ..."
},
{
"title": "SpaceX",
"link": "https://en.wikipedia.org/wiki/SpaceX",
"description": "SpaceX was founded by Elon Musk in 2002 with a vision of decreasing the costs of space launches, paving the way to a self-sustaining colony on Mars. In 2008, ...Read more"
},
{
"title": "SpaceX (@spacex) • Instagram photos and videos",
"link": "https://www.instagram.com/spacex/",
"description": "17M followers · 3 following · 996 posts · @spacex: “SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft”"
}
]
Complete Code
const { chromium } = require('playwright');
const cheerio = require('cheerio');
async function scrapeGoogleSearch(query) {
const browser = await chromium.launch({
headless: false,
channel: 'chrome'
});
const page = await browser.newPage();
await page.setViewportSize({ width: 1920, height: 1080 });
// Navigate to search results
const searchUrl = `https://www.google.com/search?q=spaceX&gl=us&hl=en`;
await page.goto(searchUrl, { waitUntil: 'domcontentloaded' });
const html = await page.content();
await browser.close();
// Parse results with Cheerio
const $ = cheerio.load(html);
const results = [];
$('div#search .tF2Cxc').each((i, el) => {
const title = $(el).find('h3.LC20lb').first().text();
const link = $(el).find('a.zReHs').first().attr('href');
const description = $(el).find('div.VwiC3b').first().text();
if (title && link) {
results.push({ title, link, description });
}
});
return results;
}
// Run
scrapeGoogleSearch('spaceX').then(results => {
console.log(JSON.stringify(results, null, 2));
}).catch(console.error);
Method 2: Scraping Google Search with Scrapingdog
The approach above works well if you’re scraping a few hundred pages. But once you scale to thousands or millions of pages, a Playwright + Cheerio setup starts to break down. Here, we’ll use a Google SERP API that handles the hard parts for us, including proxies, headers, and browser management.
Here, we will use Axios to make the GET request to the Scrapingdog’s Google SERP API.
Want to see the full workflow in action? Watch this video tutorial to learn how to Scrape Google Search Results with Node.js & Scrapingdog’s Google SERP API
const axios = require('axios');
async function scrapeGoogleSearch(query) {
const response = await axios.get('https://api.scrapingdog.com/google', {
params: {
api_key: 'YOUR_API_KEY',
query: query,
results: 10,
country: 'us',
}
});
const results = response.data.organic_results.map(result => ({
title: result.title,
link: result.link,
description: result.snippet
}));
return results;
}
scrapeGoogleSearch('web scraping tools').then(results => {
console.log(JSON.stringify(results, null, 2));
}).catch(console.error);
As you can see, with Scrapingdog, we only had to make a simple GET request. There was no need to manage a browser or write custom parsing logic. Running this code will produce a beautiful JSON response that contains every little detail, like Google Ads, AI overview, Video links, people also ask, etc.
{
"search_information": {
"time_taken": 0.3,
"total_results": 2110,
"query_displayed": "spaceX",
"organic_results_state": "Results for exact spelling",
"original_query_yields_zero_results": false,
"url": "https://www.google.com/search?q=spaceX&oq=spaceX&gl=us&hl=en&sourceid=chrome&ie=UTF-8"
},
"peopleAlsoAskedFor": [
{
"question": "Is SpaceX fully owned by Elon Musk?",
"rank": 1
},
{
"question": "Can I buy stock in SpaceX?",
"rank": 2
},
{
"question": "What is the real purpose of SpaceX?",
"rank": 3
},
{
"question": "What's the average salary of a SpaceX employee?",
"rank": 4
}
],
"organic_results": [
{
"title": "SpaceX",
"link": "https://spacex.com/",
"displayed_link": "https://spacex.com",
"favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA10lEQVR4Ad2QIQiEQBBFfweDxXrNds1qsJoslu1mk71tEOxBMG41W8yb7WzcaBeEvRkXRLy7zt1LH97O/mHwO6TK4AvVsjkHhPXEdNEpgrgjs+uUc7c6QhdeRUVPatUlPJVma2VMWSh6us01ThLFdq2QK0thkSGuSOuMTHrDb4YMN0SbimXjbhHgnYHV3uMDpXGEzVKZ40Yodx5b0Fhnm9up/LYK2fH1EF+X0YdrAzx6y3EWp6t8XRP4ozrCSHjq9ZgT50G4fhufR984E9Nlx3qaiS7GH/ECeAedlKpi2RoAAAAASUVORK5CYII=",
"source": "SpaceX",
"snippet": "SpaceX's Starship spacecraft and Super Heavy rocket is a fully reusable transportation system designed to carry both crew and cargo to Earth orbit, the Moon, ...",
"highlighted_keywords": [
"SpaceX's"
],
"extended_sitelinks": [
{
"title": "Launches",
"link": "https://www.spacex.com/launches",
"snippet": "Home to SpaceX headquarters and one of the world's first ..."
},
{
"title": "Careers",
"link": "https://www.spacex.com/careers",
"snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
},
{
"title": "Mission",
"link": "https://www.spacex.com/mission",
"snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
},
{
"title": "Starship",
"link": "https://www.spacex.com/vehicles/starship",
"snippet": "Starship is the world's most powerful launch vehicle ever ..."
},
{
"title": "Updates",
"link": "https://www.spacex.com/updates",
"snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
}
],
"rank": 1,
"page_rank": 1
},
{
"title": "SpaceX",
"link": "https://en.wikipedia.org/wiki/SpaceX",
"displayed_link": "https://en.wikipedia.org › wiki › SpaceX",
"favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b+u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz/6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1+flswMu1EQZ3xlE7lK8eKDkjtwE+crBMV+wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk/9ryhFMAAAAASUVORK5CYII=",
"source": "Wikipedia",
"snippet": "SpaceX was founded by Elon Musk in 2002 with a vision of decreasing the costs of space launches, paving the way to a self-sustaining colony on Mars. In 2008, ...Read more",
"highlighted_keywords": [
"SpaceX"
],
"rank": 2,
"page_rank": 2
},
{
"title": "SpaceX (@spacex) • Instagram photos and videos",
"link": "https://www.instagram.com/spacex/",
"displayed_link": "17.1M+ followers",
"favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAMAAABF0y+mAAABcVBMVEVHcEx3GPlwGPx1FvyDC/OWBumqA+K+AeDQANvVAs7rDa7jAtfoAMflAriKGP2ZCvunAvLEG/TgV9/oZt72G8nxB8nxALCKHPymGf7Pg/H86vf////zds79AcD9ALT3DZ30t+j63fP1st31quT8AKf8AJn98/jaO973Lr38AIvAAOz8Drn4V7370PHiGe34E6n7HZv9AHv5wOP1Wpr8D5H9AGz9GYX6q8r2YKL5dZP3tMT8D2/8KW31Y5f9AV/9Inn+G17+KFf3v879J0f+MUf5g2v3wbr9Nyz6RUP9Ml/9Qxr+SjX7zcr6Gkb8AEL+UQ/9Wiv98uT70rD8XQT8ZxD9ZCr40sP4Tn/9cAz+egj9ciX1km/1X1L9O0j4nD7+hAT9hRb8fh34unb9jQT3jZX73a78dwX+mwL6ymPzoZD8YSf9qAD+sQD7wBL8zEn5tV79jwb+uAD+vgH+wwD+LHP8owX+ygD6Olj7rQz+twf8UjMSMf5jAAAAe3RSTlMAW9L////////FW///2P///////////8b/////////xf///////////////////////////v/////////////////////////////////////////////////////////////////F////W////f//2f///9fY/9hbxcXA5ZvtAAACBUlEQVR4AUTPRaLkNhSF4f9IV3Kp/JjfCzMzM8xCG8lWMs9SsoIkTdNmZubiKtvqduMnZl0BkgCBEI+zhiBwD5f0ZA3wQj1Eu1YiMZQADyAA9SRXgpCEaGXcODFKoq/NEXRHXYmnshMak0xdGC/PiO1Mi1rTbA0LyM/BOOWyuISrHyLPTRoKZDaBndvoQrezQA/IWlBe6DXCsOdgong+bdCLyzW5F3VzpY7TckLH4ESRlvsbNlj2V4DNu3klk/Ed8HUvYlVKwa4NSmedWzNX1dbUcxlzsJMgxNppsywKv6l5M9YJZtizxOfObknxVOkVaAblle1BB0JNssXaJhA05tlKMOrcIXcMSOBc8IBvqzbQZC8SjIfMwxxz0iLnn0vQS+cx7ye0nPcPT7qPBmk2my7b4A0XlmiNzEMDQfD53vLF6sigLHLGMyzBCMzaawXf/HcAPqpmvqbDq5dGlQVjlexMYvqD0C3P+iVOccGBM0L1+d6uYozzRc6TbvA3/cI0r7g4dgoKy/SCee+akH32ltf38muG0kzR+PUfPl+DHq1in3jnSp8ZtjAxgz+u7OUJfb51cmk0Tqf158QwgcQUmM5zy9+tqnr9+nCf+3seU4hFCKHsptStxoEQg4fPhp7dP1dB8u6+LFzjqOq6afof/IWH/38O40irXXZ1g2+q99K9TWBgAACS27uoe/IOhgAAAABJRU5ErkJggg==",
"source": "Instagram · spacex",
"snippet": "17M followers · 3 following · 996 posts · @spacex: “SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft”",
"highlighted_keywords": [
"spacex",
"SpaceX"
],
"rank": 3,
"page_rank": 3
},
{
"title": "SpaceX",
"link": "https://www.youtube.com/spacex",
"displayed_link": "6.8M+ followers",
"favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAcElEQVR4AWP4//8/RZh6BgCZAkDsAMUNWDFCXgDFACCV8J/B+D8pGKwHRAKRAUyQDEMMQAYEUGBAAsiABpwKHjz4/9/BAZ8BDXgNgIMNGyg04MABkg1AeCEgAK8XKA5EiqORooSELykXEJuUBz43AgAIA1ZhBoG9vwAAAABJRU5ErkJggg==",
"source": "YouTube · SpaceX",
"snippet": "SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft. ...more SpaceX designs, manufactures and launches the world's ...Read more",
"highlighted_keywords": [
"SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft"
],
"rank": 4,
"page_rank": 4
},
{
"title": "SpaceX (@SpaceX) / Posts / X",
"link": "https://x.com/SpaceX",
"displayed_link": "41.6M+ followers",
"favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA/0lEQVR4AbXPIazCMACE4d+L2qoZFEGSIGcRc/gJJB5XMzGJmK9EN0HMi+qaibkKVF1txdQe4g0YzPK5yyWXHL9TaPNQ89LojH87N1rbJcXkMF4Fk31UMrf34hm14KUeoQxGArALHTMuQD2cAWQfJXOpgTbksGr9ng8qluShJTPhyCdx63POg7rEim95ZyR68I1ggQpnCEGwyPicw6hZtPEGmnhkycqOio1zm6XuFtyw5XDXfGvuau0dXHzJp8pfBPuhIXO9ZK5ILUCdSvLYMpc6ASBtl3EaC97I4KaFaOCaBE9Zn5jUsVqR2vcTJZO1DdbGoZryVp94Ka/mQfE7f2T3df0WBhLDAAAAAElFTkSuQmCC",
"source": "X · SpaceX",
"snippet": "SpaceX (@SpaceX) - Posts - SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft | X (formerly Twitter)",
"highlighted_keywords": [
"SpaceX",
"SpaceX",
"SpaceX"
],
"rank": 5,
"page_rank": 5
}
],
"relatedSearches": [
{
"query": "Spacex news",
"link": "https://www.google.com/search?gl=us&q=Spacex+news"
},
{
"query": "Spacex falcon heavy launch",
"link": "https://www.google.com/search?gl=us&q=Spacex+falcon+heavy+launch"
},
{
"query": "SpaceX owner",
"link": "https://www.google.com/search?gl=us&q=SpaceX+owner"
},
{
"query": "SpaceX launch",
"link": "https://www.google.com/search?gl=us&q=SpaceX+launch"
},
{
"query": "SpaceX Starship",
"link": "https://www.google.com/search?gl=us&q=SpaceX+Starship"
},
{
"query": "SpaceX stock",
"link": "https://www.google.com/search?gl=us&q=SpaceX+stock"
},
{
"query": "SpaceX careers",
"link": "https://www.google.com/search?gl=us&q=SpaceX+careers"
},
{
"query": "SpaceX IPO",
"link": "https://www.google.com/search?gl=us&q=SpaceX+IPO"
}
]
}
How to Handle Pagination?
To scrape all the pages from Google, you can use the page parameter from Scrapingdog. You can read more about it in the documentation.
async function scrapeMultiplePages(query, pages = 3) {
const allResults = [];
for (let i = 0; i < pages; i++) {
const response = await axios.get('https://api.scrapingdog.com/google', {
params: {
api_key: 'YOUR_API_KEY',
query: query,
results: 10,
country: 'us',
page: i
}
});
const results = response.data.organic_results.map(result => ({
title: result.title,
link: result.link,
description: result.snippet
}));
allResults.push(...results);
}
return allResults;
}
scrapeMultiplePages('web scraping tools', 3).then(results => {
console.log(`Total results: ${results.length}`);
console.log(JSON.stringify(results, null, 2));
}).catch(console.error);
With this approach, pagination is just incrementing the page parameter; no browser delays, no block risk, no extra complexity.
Parsing Different Result Types
The Scrapingdog API response contains much more than just organic results. Here’s how to parse each result type from the JSON:
1. Organic Result
const organic = response.data.organic_results.map(result => ({
rank: result.rank,
title: result.title,
link: result.link,
snippet: result.snippet
}));
console.log('Organic Results:', organic);
2. People Also Ask
const paa = response.data.peopleAlsoAskedFor.map(item => ({
rank: item.rank,
question: item.question
}));
console.log('People Also Ask:', paa);
3. Related Searches
const related = response.data.relatedSearches.map(item => ({
query: item.query,
link: item.link
}));
console.log('Related Searches:', related);
4. Sitelinks (only on top branded results)
const topResult = response.data.organic_results[0];
if (topResult.extended_sitelinks) {
const sitelinks = topResult.extended_sitelinks.map(s => ({
title: s.title,
link: s.link,
snippet: s.snippet
}));
console.log('Sitelinks:', sitelinks);
}
5. Search Metadata
const meta = response.data.search_information;
console.log(`Query: ${meta.query_displayed}`);
console.log(`Total Results: ${meta.total_results}`);
console.log(`Time Taken: ${meta.time_taken}s`);
The advantage of the Scrapingdog API is that all these result types come back in a single structured response, no need to write separate CSS selectors for each one or worry about Google changing its DOM structure.
Export the scraped data to CSV
Node.js has a built-in fs module so no extra dependencies are needed for this step.
const fs = require('fs');
const axios = require('axios');
function convertToCSV(results) {
const headers = ['Rank', 'Title', 'Link', 'Snippet'];
const rows = results.map(r => [
r.rank,
`"${r.title.replace(/"/g, '""')}"`,
r.link,
`"${r.snippet.replace(/"/g, '""')}"`
]);
return [headers.join(','), ...rows.map(row => row.join(','))].join('\n');
}
async function scrapeAndExport(query) {
const response = await axios.get('https://api.scrapingdog.com/google', {
params: {
api_key: 'YOUR_API_KEY',
query: query,
results: 10,
country: 'us'
}
});
const results = response.data.organic_results.map(r => ({
rank: r.rank,
title: r.title,
link: r.link,
snippet: r.snippet
}));
const csv = convertToCSV(results);
const filename = `${query.replace(/\s+/g, '_')}_results.csv`;
fs.writeFileSync(filename, csv, 'utf8');
console.log(`Saved ${results.length} results to ${filename}`);
}
scrapeAndExport('spaceX').catch(console.error);
Once you run this code, you will find a CSV file inside the folder.
Conclusion
In this guide, we covered two ways to scrape Google Search results with Node.js: Playwright and the Scrapingdog API. Playwright is good for low-volume testing and learning, but breaks easily at scale due to Google’s aggressive bot detection and frequent DOM changes.
For production use, the Scrapingdog’s Google SERP API is the clear winner. It handles proxies, CAPTCHA, and browser fingerprinting for you, returning clean, structured JSON with organic results, People Also Ask, sitelinks, and pagination in a single call, so you can focus on using the data rather than fighting to get it.
Frequently Asked Questions
Is scraping Google Search results legal?
Scraping public Google data is legal. For commercial use at scale, using an API like Scrapingdog is the safer route.
Why does my scraper get blocked so quickly?
Google detects bots through IP reputation, request frequency, and browser fingerprinting. Without proxy rotation and human-like behavior, even a few requests will trigger a block.
Can I scrape Google Search results for free?
Yes, using Playwright or Puppeteer, but you’ll hit blocks fast without proxies. Scrapingdog offers a free tier to get started without any infrastructure setup.
How many results can I scrape per page?
Google returns 10 results per page by default. You can paginate using the page parameter page=0 for page 1, page=1 for page 2, and so on.
Do I need to parse HTML when using the Scrapingdog API?
No. The API returns structured JSON directly, so there’s no need to write CSS selectors or worry about Google changing its DOM structure.
