How to Scrape Google News Using Python

When people talk about scraping Google News, they’re usually referring to one of two things: the news tab in regular Google Search (google.com/search?tbm=nws) or the dedicated news aggregator at news.google.com. These are two completely different pages with different structures, different data, and different scraping challenges and most tutorials only cover one of them.

In this guide, we’ll show you how to scrape both using Python and Scrapingdog’s Google News API, so you get structured data like headlines, sources, publication times, and article links, regardless of which source you need.

Google News results are news articles that Google surfaces from multiple publishers across the web based on a user’s query, interests, and freshness of content. Unlike regular Google Search results, which mix blogs, product pages, and websites, Google News focuses specifically on journalistic and news-style content.

When you search a keyword in Google News, you typically see:

• Headlines from multiple publishers covering the same story

• Source names (BBC, Reuters, niche blogs, local media, etc.)

• Publication time (minutes ago, hours ago, days ago)

• Article snippets summarizing the story

• Images or thumbnails tied to the article

• Topic clusters, where related coverage is grouped together

Media Monitoring — Track brand or competitor mentions across thousands of news sources in real time.

Sentiment Analysis — Feed headlines into NLP models to measure how media coverage shifts over time.

Competitive Intelligence — See which stories about your competitors are gaining traction in the press.

Market Research — Correlate breaking news with stock movements, industry trends, or consumer behavior.

AI Training Data — Google News surfaces authoritative, publisher-verified content ideal for LLM training datasets.

Academic Research — Analyze how stories evolve across outlets, detect misinformation, or study media bias at scale.

We will be discussing two ways of extracting news from Google:

1. Using Python and BeautifulSoup

2. Using Scrapingdog’s Google News API

Let’s start by creating a basic scraper using Python and BS4.

In this section, we’ll build a Python script that extracts up to 10 Google News results, including the title, snippet, link, source, and publication date, and finally saves them to a CSV file.

Make sure you have Python 3.7+ installed along with the following libraries:

Once installed, create a new Python file and import them at the top:

Next, define the core scraping function. This function sends a request to Google News with a spoofed User-Agent header to avoid being immediately blocked, then parses the returned HTML with BeautifulSoup.

⚠️ A static User-Agent works for testing, but Google will block repeated requests at scale. For production use, rotate User-Agents or route requests through a proxy.

Now we need to identify the right HTML elements to target. If you open Chrome DevTools (right-click → Inspect) on a Google News results page, you’ll find that each individual news card is wrapped in a div with the class SoaBEf. We loop over all of them to extract data from each result:

This loop iterates over every news card on the page and appends an empty dictionary for each one. We’ll fill in the actual fields title, source, link, snippet, and date in the next steps.

Let’s find the headline of the news article by inspecting it.

As you can see in the above image, the title is under the div container with the class MBeuO.

Add the following code in the append block to get the news title.

Similarly, we can extract the News source and link from the HTML.

Each news card contains an anchor tag whose href attribute holds the article URL, and the publisher name sits inside a span within .NUnG9d:

The article snippet is inside .GI74Re and the publication date is inside .LfVVr:

Here’s the complete scraper putting it all together:

Run this in your terminal and you should see a JSON array of news results. Here’s what the output looks like:

Instead of printing to the terminal, let’s save the results directly to a CSV file. Add pandas to your imports at the top:

Then, you can replace the print line with the following code.

Running the script now will create a news_data.csv file in your project directory with one row per news result.

Hurray!!!

The bare requests + beautifulSoup approach works well for testing and small runs, but it has a hard ceiling; Google will block repeated requests, class names change without warning, and maintaining the scraper becomes a job in itself.

Scrapingdog’s Google News API handles all of that behind the scenes: proxy rotation, CAPTCHA bypassing, and header management are taken care of automatically. Instead of raw HTML, you get back clean, structured JSON that’s ready to use without any parsing.

This approach makes sense when you need:

• More than a few hundred requests per day

• Reliable, uninterrupted data collection

• Consistent JSON output without worrying about Google’s layout changes

To get started, you’ll need a Scrapingdog API key.

Head to Scrapingdog’s registration page and create a free account. You’ll get 1,000 credits to test with, no credit card required.

Once registered, your API key will be waiting on the dashboard. Copy it somewhere handy as you’ll need it in the next step.

Since the API returns structured JSON, you only need the requests library , no BeautifulSoup required:

Here’s what a single result looks like:

Each result contains five fields — title, snippet, source, lastUpdated, and urlwhich map directly to what you see on the Google News page. The full list of supported parameters (language, country, date range, topic filtering) is covered in the API documentation.

By default the API returns 10 results per request. To collect more, use the page parameter to paginate through results:

This loops through 20 pages for the query “Elon Musk”, appending each page’s results to all_results. Adjust range(0, 20) to control how many pages you collect.

If you want to scrape data from news.google.com using Scrapingdog, then you have to use Google news Scraper API.

Once you run this code, you will get this data:

• Google News exposes headlines, snippets, sources, publication dates, and URLs. All is scrapable for brand monitoring, market research, and competitive analysis.

• A requests + BeautifulSoup setup is sufficient for learning and small-scale runs.

• DIY scrapers are fragile; Google’s class names change, CAPTCHAs trigger quickly, and IP blocks follow at scale.

• Scaling the DIY approach means managing proxies, header rotation, retries, and pagination yourself, which adds significant engineering overhead.

• Scrapingdog’s Google News API handles all of that automatically, returning clean JSON with no HTML parsing required.

This article discussed two ways of scraping news from Google using Python. Data collectors looking to have an independent scraper and want to maintain a certain amount of flexibility while scraping data can use Python as an alternative to interacting with the web page.

Otherwise, Google News API is a simple solution that can quickly extract and clean the raw data obtained from the web page and present it in structured JSON format.

We also learned how this extracted data can be used for various purposes, including brand monitoring and competitor analysis.

If you like this article please do share it on your social media accounts. If you have any questions, please contact me at [email protected].

• Scrape Bing Search Results using Python (A Detailed Tutorial)

• Scrape Google Short Videos using Python

• 10 Best Google Search Scraper APIs

• Web Scraping Google Maps using Python

• Web Scraping Google Search Results using Python

• Web Scraping Google Images using Python

• Scrape Google Finance using Python

• Web Scraping Google Scholar using Python

• How To Scrape Google Shopping using Python

• How To Scrape Google Patents

• Web Scraping Google Jobs using Nodejs

• Building A Stock Automation To Predict The Daily Stock Analysis Using News API

• How to scrape Google AI Overviews using Python

• How to Scrape Google AI Mode Using Python