Author: Manthan Koolwal

Airbnb is one of the major websites that travelers go to. Scraping Airbnb data with Python can give you a lot of insights into how the travel market is working currently. Further, trends can be analyzed in how the pricing deviates over the period.

Alrighty, now before diving into the main context of this blog, If you are new to web scraping, I would recommend you to go through the basics of web scraping with Python.

Let’s get started!!

Setting up the prerequisites for scraping Airbnb

For this tutorial, you will need Python 3.x on your machine. If it is not installed then you can download it from here. We will start by creating a folder where we will keep our Python script.

mkdir airbnb

Inside this folder, we will install two libraries that will be used in the course of this tutorial.

• Selenium– It is used to automate web browser interaction from Python.
• Beautiful Soup is a Python library for parsing data from the raw HTML downloaded using Selenium.
• Chromium– It is a web browser that will be controlled through selenium. You can download it from here.

pip install bs4
pip install selenium

Remember that you will need the exact version of the Chromium driver as your Chrome browser. Otherwise, it will not run and throw an error like below.

Once all the libraries are installed we can now create a Python file where we will write the code for the scraper. I am naming this file as scraper.py.

What are we going to scrape?

For this tutorial, we are going to scrape this page.

We are going to scrape mainly four data points from this page.

• Name of the property
• Rating
• Price per night
• Price per night with tax

The procedure to execute this task is very simple.

• First, we are going to download the raw HTML using Selenium.
• Then using BS4 we are going to parse the required data.

Downloading the raw data using Selenium

Before writing the script let’s import all the libraries that will be used in this tutorial.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time

1. from bs4 import BeautifulSoup Imports the BeautifulSoup class from the bs4 module. BeautifulSoup is a popular Python library for parsing HTML and XML documents.
2. from selenium import webdriver Imports the webdriver module from the selenium package. Selenium is a web automation tool that allows you to control web browsers programmatically.
3. from selenium.webdriver.chrome.service import Service Imports the Service class from the selenium.webdriver.chrome.service module. This is used to configure the ChromeDriver service.
4. import time Imports the time module, which provides various time-related functions. We will use it for the timeout.

PATH = 'C:\Program Files (x86)\chromedriver.exe'

Then we have declared a PATH variable that shows the location of the chrome driver. In your case, the location string can be different.

service = Service(executable_path=PATH)
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)

The executable_path parameter specifies the path to the chromedriver executable file. This is necessary for Selenium to know where to find the chromedriver executable.

The second line creates a new instance of ChromeOptions, which allows you to configure various options for the Chrome browser.

The third line creates a new instance of the Chrome WebDriver. It takes two parameters: service, which specifies the ChromeDriver service that was set up earlier, and options, which specifies the ChromeOptions object that was created. This line effectively initializes the WebDriver with the specified service and options.

# Navigate to the target page
driver.get("https://www.airbnb.co.in/s/Sydney--Australia/homes?adults=1&checkin=2024-05-17&checkout=2024-05-18")
time.sleep(5)

html_content = driver.page_source
print(html_content)

driver.quit()

The first line tells the WebDriver to navigate to the specified URL, which is an Airbnb search page for homes in Sydney, Australia. It includes query parameters for the number of adults and the check-in and check-out dates.

This second line pauses the execution of the script for 5 seconds, allowing time for the webpage to load completely. This is a simple way to wait for dynamic content to be rendered on the page.

The third line retrieves the HTML content of the target webpage using the page_source attribute of the WebDriver.

The last line closes the browser and terminates the WebDriver session. This line is necessary because it can save your memory too while doing mass scraping.

Once you run this code you will see this on your console.

We have successfully managed to download the raw HTML from our target page. Now, we can proceed with the next step of parsing.

Parsing the data with BS4

Before we write the code for parsing, let’s find out the DOM location of each target data point.

Finding the Property Title

As you can see in the above image, the title of each property is located inside the div tag with attribute data-testid and value listing-card-title.

Finding the Rating

The rating can be found inside a div tag with class t1a9j9y7.

Finding the Price Without Tax

Price without tax can be found inside a span tag with class _14y1gc.

Finding the Price with Tax

Price with tax can be found inside a div tag with class _10d7v0r.

Finding all the property Boxes

Now on every page, there are a total of 24 properties and to get data from each of these properties we have to run a for loop. But before that, we have to find the location where all these property boxes are located.

All these properties are located inside a div tag with an attribute itemprop and value itemListElement.

soup=BeautifulSoup(html_content,'html.parser')

Here we have created an instance of BS4 to parse the data from the raw HTML.

allData = soup.find_all("div",{"itemprop":"itemListElement"})

Then we search for all the properties present on that particular target page.

for i in range(0,len(allData)):
    try:
        o["property-title"]=allData[i].find('div',{'data-testid':'listing-card-title'}).text.lstrip().rstrip()
    except:
        o["property-title"]=None

    try:
        o["rating"]=allData[i].find('div',{'class':'t1a9j9y7'}).text.split()[0]
    except:
        o["rating"]=None

    try:
        o["price"]=allData[i].find('span',{"class":"_14y1gc"}).text.lstrip().rstrip().split()[0]
    except:
        o["price"]=None

    try:
        o["price_with_tax"]=allData[i].find('div',{'class':'_i5duul'}).find('div',{"class":"_10d7v0r"}).text.lstrip().rstrip().split(" total")[0]
    except:
        o["price_with_tax"]=None

    l.append(o)
    o={}

print(l)

Then we are running a for loop to extract the data from their respective locations. Finally, we store all the data inside a list l.

Once you run this code you will see an array with all the data on your console.

Saving the data to a CSV file

We can save this data to a CSV file by using the pandas library. You can install this library like this.

Now, import this library at the top of your script.

import pandas as pd

First, we have to create a Pandas DataFrame named df from the list l. Each element of the list l becomes a row in the DataFrame, and the DataFrame’s columns are inferred from the data in the list.

df = pd.DataFrame(l)
df.to_csv('airbnb.csv', index=False, encoding='utf-8')

The second line exports the DataFrame df to a CSV file named “airbnb.csv”. The to_csv() method is used to save the DataFrame to a CSV file.

Again once you run the code you will find a CSV file inside your working directory.

Complete Code

You can scrape many more data points from this page but for now, the code will look like this.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time
import pandas as pd


l=[]
o={}

PATH = 'C:\Program Files (x86)\chromedriver.exe'


service = Service(executable_path=PATH)
options = webdriver.ChromeOptions()

driver = webdriver.Chrome(service=service, options=options)


driver.get("https://www.airbnb.co.in/s/Sydney--Australia/homes?adults=1&checkin=2024-05-17&checkout=2024-05-18")
time.sleep(2)
html_content = driver.page_source

driver.quit()


soup=BeautifulSoup(html_content,'html.parser')

allData = soup.find_all("div",{"itemprop":"itemListElement"})



for i in range(0,len(allData)):
    try:
        o["property-title"]=allData[i].find('div',{'data-testid':'listing-card-title'}).text.lstrip().rstrip()
    except:
        o["property-title"]=None

    try:
        o["rating"]=allData[i].find('div',{'class':'t1a9j9y7'}).text.split()[0]
    except:
        o["rating"]=None

    try:
        o["price"]=allData[i].find('span',{"class":"_14y1gc"}).text.lstrip().rstrip().split()[0]
    except:
        o["price"]=None

    try:
        o["price_with_tax"]=allData[i].find('div',{'class':'_i5duul'}).find('div',{"class":"_10d7v0r"}).text.lstrip().rstrip().split(" total")[0]
    except:
        o["price_with_tax"]=None

    l.append(o)
    o={}







df = pd.DataFrame(l)
df.to_csv('airbnb.csv', index=False, encoding='utf-8')
print(l)

Limitations

The above approach for scraping Airbnb is fine but this approach will not work if you want to scrape millions of pages. Airbnb will either block your IP or your resources will not be enough for you to get the data at high speed.

To overcome this problem you can use web scraping APIs like Scrapingdog. Scrapingdog will handle all the hassle of JS rendering with headless chrome and rotation of IPs. Let’s see how you can use Scrapingdog to scrape Airbnb at scale.

Scraping Airbnb with Scrapingdog

Using Scrapingdog is super simple. To get started with Scrapingdog you have to sign up for an account. Once you sign up you will get free 1000 API credits which is enough for initial testing.

On successful account creation, you will be redirected to your dashboard where you will find your API key.

Using this API key you can easily integrate Scrapingdog within your coding environment. For now, we will scrape Airbnb using Scrapingdog in our Python environment. You can even refer to this Web Scraping API Docs before proceeding with the coding.

import requests
from bs4 import BeautifulSoup
import pandas as pd

l=[]
o={}




target_url="https://api.scrapingdog.com/scrape?api_key=YOUR-API-KEY&url=https://www.airbnb.co.in/s/Sydney--Australia/homes?adults=1&checkin=2024-05-17&checkout=2024-05-18&wait=3000"



resp = requests.get(target_url)

soup=BeautifulSoup(resp.text,'html.parser')

allData = soup.find_all("div",{"itemprop":"itemListElement"})



for i in range(0,len(allData)):
    try:
        o["property-title"]=allData[i].find('div',{'data-testid':'listing-card-title'}).text.lstrip().rstrip()
    except:
        o["property-title"]=None

    try:
        o["rating"]=allData[i].find('div',{'class':'t1a9j9y7'}).text.split()[0]
    except:
        o["rating"]=None

    try:
        o["price"]=allData[i].find('span',{"class":"_14y1gc"}).text.lstrip().rstrip().split()[0]
    except:
        o["price"]=None

    try:
        o["price_with_tax"]=allData[i].find('div',{'class':'_i5duul'}).find('div',{"class":"_10d7v0r"}).text.lstrip().rstrip().split(" total")[0]
    except:
        o["price_with_tax"]=None

    l.append(o)
    o={}







df = pd.DataFrame(l)
df.to_csv('airbnb.csv', index=False, encoding='utf-8')
print(l)

As you can see we have removed Selenium and in place of that we have imported requests library that will be used for requesting the Scrapingdog Scraping API. Scrapingdog will now handle all the JS rendering and IP rotation.

Conclusion

In this tutorial, we learned how we can create a scraper for Airbnb using Selenium and BS4. We also covered how using Scrapingdog, we can bypass the limitations of a Python Scraper.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on social media.

Frequently Asked Questions

Additional Resources

• Web Scraping with Selenium & Python
• Web Scraping booking.com using Python
• Web Scraping Expedia using Python

LinkedIn is the best platform when it comes to finding new clients, new employees, etc. It has around 1 Billion users from more than 200 countries. Data from LinkedIn can enrich any data pipeline because it provides fresh information on any individual or company. You can find out who works where, what skills they have, where were they working a year back, where they graduated, etc.

This information helps managers or marketing teams find the best prospects for their product, and hiring or consultancy firms can find the best employee for any job vacancy in their company. There are many use cases for scraping LinkedIn. Let’s discuss that in detail.

Use cases of Scraping LinkedIn

• You can build products that can help marketing teams find leads by scraping LinkedIn at scale. Once you scrape any profile you will get information like where he/she works, what experience they hold, how many followers/connections they have, etc.
• You can scrape LinkedIn Jobs to find the desired sector you would want to go in.
• You can identify people with just their emails by scraping LinkedIn profiles. This is called enrichment. You can identify who the person is behind a certain email.
• You can find out sector-wise companies. Then you can target them through cold emails to sell your B2B products.
• You can find contact details like emails and phone numbers by scraping LinkedIn.
• Investors can find a suitable company for their next investments. They just have to filter out data and they select companies from the list. Like if they want to find companies in Mumbai, India then they can filter it out.
• You can analyze the sentiments of any individual through his/her activity on LinkedIn. What kinds of comments they make, and what kinds of articles they like can help you identify the person better.
• Other data like volunteer work can help you identify which prospect has done social work in the past.
• Certificates can help one identify which person has what kind of specialization.

Is It Legal to Scrape LinkedIn?

Yes, It is legal to scrape LinkedIn if you are doing it ethically. I would help you with this by giving a few examples that have happened in the past that would help you to clear your doubts.

Some companies like Mantheos a Singapore-based company faced the heat from Linkedin for Scraping its data. You can read more about the case over here. Mainly they were using fake cards to bypass their payment system and they were then using the premium plans of Linkedin to scrape prospect details from it. This type of method is of course illegal and the person behind this idea should be punished.

The fact is Google also works by crawling over all the websites/domains. It has large crawlers that keep scraping websites to make its search result algorithm smarter. But they are quite large and they can deal with legal matters very easily as compared to small companies or any individual.

There is another ongoing matter between hiQ and LinkedIn. hiQ was scraping LinkedIn by creating fake accounts. In November 2022 the court said that “hiQ already knew that they were illegally acquiring Linkedin’s data through web scraping”. But but but… in 2022 California District Court found some evidence of the exchange of emails among LinkedIn employees where it was very clear that LinkedIn was aware of this scraping back in 2014 and let it happen until 2017. In the end, both parties agreed to a private settlement.

This whole mess concludes that you should not do the following:

1. Do not create fake accounts on Linkedin.
2. Do not bypass their payment wall with fake credit cards.
3. Respect server bandwidth.

This is what you can do:

You can scrape everything available to the public according to the Computer Fraud and Abuse Act(CFAA). This can be concluded by hiQ vs LinkedIn case as both the parties had a very weak allegation against each other.

Limitations of Scraping LinkedIn

LinkedIn has placed a limit of 50 profiles per user in recent years. This means that you can only scrape 50 profiles in a day. But LinkedIn is too smart and it will block your account because of data extraction activity.

Other than this if you are willing to scrape LinkedIn without creating a user then Linkedin will redirect you to either a Login screen or it will through a captcha like this.

So, it is next to impossible to scrape LinkedIn at scale with a normal infrastructure on your own.

Quick Solution To Scraping LinkedIn at Scale

Scrapingdog can help you scrape LinkedIn at scale without you getting blocked. By using Scrapingdog you will never face any kind of legal issues as your IP will never be used. You can scrape up to 1 Million profiles per day and our scraper is capable of bypassing any kind of captcha.

Other than Scrapingdog there is Brightdata which offers Linkedin Scraper but it is too expensive. Our per profile cost is $0.009 and goes below that if you want to scrape more than 100k profiles per month. On the other hand, Bightdata will charge you $0.01 per profile which makes it extremely expensive.

There are other solutions also in the market but they provide data from 2016 and this makes it very old.

Scrapingdog also offers a generous free pack through which you can test the LinkedIn Profile Scraper API. Other than this we also offer a LinkedIn Jobs Scraping API which can be used for scraping LinkedIn Jobs at scale with a simple GET request.

Also, do check out our compiled list of best LinkedIn Scrapers.

Happy Scraping!!

Additional Resources

• How To Bypass 999 responses when scraping LinkedIn profiles?
• How To Scrape Email Addresses from Any Website using Python

Bright Data provides a large proxy pool and various tools for web scraping with flexible plans. This is great, but their high prices and complicated tool management can be frustrating at times.

So, why not look at the alternatives to Bright Data? There may be other options that offer similar capabilities at a lower cost and with easier-to-use tools. 

By looking at these alternatives, you might find a better fit for your specific needs and budget.

Let’s jump in.

Scrapingdog

Scrapingdog is a web scraping API and can be used to scrape almost any website. If you are currently on Bright data plan and looking for a better alternative for web scraping you should give it a try. 

The API takes care of rotating proxies, headless browsers, & CAPTCHAs thus making it easy for you to do blockless data extraction. 

With that being said, we provide dedicated APIs for different sources to get the output in JSON format. 

Here are some of the dedicated APIs

• LinkedIn Scraper API
• LinkedIn Jobs API
• Zillow Scraper API
• Google Search Scraper API
• Twitter Scraper API
• Amazon Scraper API

Other than that, we also provide unmetered datacenter proxies, that can be used in large-scale data extraction. The proxies are set to have 99.99% uptime. We have around 15 Million sets of these proxies so that if some of them get detected, your scraping process doesn’t get disturbed. 

Scrapingdog has an average rating of 4.8 stars on Trustpilot. You can check the reviews to see how satisfied the customer is with the service. 

Oxylabs

The second on this list is Oxylabs. The product has pricing in the premium segment offering almost all types of proxies & has multiple APIs for web scraping. 

They offer proxies from over 192+ countries making them one of the largest proxy pools.  

They do provide dedicated scraper APIs, but proxies being their main strength, the APIs aren’t the best when compared to top players. 

Oxylabs takes a different approach to pricing compared to Bright Data. It offers a subscription model for all its services, while residential and mobile proxies are available on a pay-as-you-go basis. 

Some proxies are priced per GB of data, while others are charged per IP address. They are upfront about their pricing, with no hidden costs or upsells. Additionally, you can save 10% by opting for an annual subscription. 

Smartproxy

It can be one of the best options to save money if you are looking for a pool of rotating proxies.

Smartproxy offers around 40 Million residential proxies. They offer cheaper smaller plans as compared to other proxy providers. 

After successfully driving customers to their proxy business, they are now also providing APIs for scaping. Some of them are, social media scraping APIs, SERP APIs, and no code scraper. They offer best-in-class proxies, but the dedicated APIs are mediocre.

Read More: Best SERP APIs in 2024

NetNut

A great proxy provider and they are very old in the business. NetNut can be your best choice if you are looking for a provider with a high volume of proxies. They offer guaranteed uptime and request-based plans. 

With a proxy pool of 85M+ residential proxies, they claim to be the fastest proxy provider around the globe. 

Their interactive dashboard offers real-time proxy management, an analytics tool and you can easily customize your proxies via the setting option. Other than the proxies they offer, they do offer a dedicated API for search engine scraping. 

SOAX Proxy

Compared to other proxy providers SOAX proxies is very new. They provide residential proxies, ISP proxies, and Mobile proxies. However, the proxy provider doesn’t provide datacenter and dedicated proxies. 

Offering 24/7 support depending upon the type of plan you choose. Having residential proxies with over 155M +, they are a good choice for startups and medium-sized companies. 

On their website, you can find quick start guides, an FAQ section, and setup instructions so it is easy for users with minimal to no experience with proxies. Besides having a blog, they now run a podcast that talks about the proxies and related stuff.

Wrapping Up: Brighdata Alternatives

Bright Data is one of the well-known proxy provider in the domain. It has a large proxy pool and some of the great features. However, as said before, the cost is one of the factors that users would switch to an alternative. 

In this article, we have provided some of the best brightdata alternatives that are known well and are economical as compared to Brightdata. To choose the best one, give a test trial to each of them & see what goals you have and which of them solves all of these. 

Having this clarity will let you select the perfect proxy solution for you. 

Additional Resources

• Best Web Scraping Tools
• Best Datacenter Proxies in 2024
• Best LinkedIn Scraper in 2024

In today’s online world, there’s a lot of information out there, but finding and organizing it can be tough. That’s where web crawlers come in. These are tools that go through the internet to find and collect data for us like the ones search engines use to know what’s on the web. JavaScript, a prevalent programming language, especially with Node.js, makes building these web crawlers easier and more effective.

In this blog, we will be using javascript to create a web crawler and further we will understand how to store this data.

Setting Up the Environment

Before you start to build a JavaScript web crawler, you need to set up a few things on your system. For starters, you will need to have Node.js and npm set up locally to create and develop the project.

Once you have these in place, you can start by creating a new project. Run the following commands to create a new directory for your project and initialize a new Node.js project in it:

mkdir js-crawler
cd js-crawler
npm init -y

Once the NPM project is initialized, you can now start by creating a new file named app.js in the project directory. This is where you will write the code for the JavaScript web crawler.

Before you start to write the code, you should install two key dependencies you will use for this tutorial: Axios for HTTP requests and Cheerio for HTML parsing. You can do that by running the following command:

npm install axios cheerio

Now, you are ready to start building your JavaScript web crawler.

Crawling Basics

In this section, you will learn how to set up a Node.js script that crawls through all the pages under the scraping sandbox https://quotes.toscrape.com/. This page lists quotes from famous people and links to their about pages. There are multiple quotes from the same people, so you will learn how to handle duplicate links and there is a link to the source of the quotes (GoodReads) as well to help you understand how to handle external links.

Here’s what the homepage looks like:

Fetching a Web Page

To fetch the homepage, you will need to paste the following code snippet in your app.js file:

const axios = require("axios")

const baseURL = "https://quotes.toscrape.com"

function main() { 
    const pageHTML = await axios.get(baseURL + "/")
} 
 
main()

This will retrieve the HTML of the page at https://quotes.toscrape.com/ and store it in pageHTML. You can now use Cheerio to traverse through the DOM and extract the data that you need. To initialize Cheerio with the HTML of the quotes page, update the code in app.js to look like the following:

const axios = require("axios")
// Add the import for Cheerio
const cheerio = require("cheerio"); 

const baseURL = "https://quotes.toscrape.com"

function main() { 
    const pageHTML = await axios.get(baseURL + "/")
    
    // Initialize Cheerio with the page's HTML
    const $ = cheerio.load(pageHTML.data)
} 
 
main()

You can now use the $ variable to access and extract HTML elements similar to how you would work with jQuery. You will learn more about how to extract links and text in the following sections.

Extracting Links

The next step is to start extracting links from the page. There are three types of links that you will extract from this website:

• /author: These are URLs for author profile pages. You will collect these from quotes and use these to extract author information.
• /tag: These are URLs for tag pages. You will find quotes that are related to a particular tag on these pages. You can use these to extract lists of related quotes.
• /page: These are URLs to help you navigate through the pagination of the website. You will need to collect and use these to navigate through the website and collect all quotes.

First of all, create two arrays to store pagination URLs and visited URLs and two Sets to store author page URLs and tag URLs in the main() function:

const paginationURLs = ["/"]
    const visitedURLs = []
    
    const authorURLs = new Set()
    const tagURLs = new Set()

The visitedURLs array will help avoid crawling the same URL twice. The reason for using Sets instead of arrays for author and tag URLs is to avoid storing the same URL twice in the list. Set automatically removes duplicate elements from the list.

The next step is to identify the locators for the links from the page. You will need to inspect the HTML of the webpage in a web browser and find the appropriate class names, IDs, and elements to locate the elements.

As mentioned before, you will be extracting author profile links and tags from quotes and tag links from the “Top Ten tags” list on each page:

To write the selector for extracting the author profile link from a quote, right anywhere on the webpage and click Inspect. The developer tools window will open. On this window, click on the Sources tab and click on the Inspect button at the top:

Now, click on the author link to view its source in the developer tools window:

You can now use the DOM structure to write a selector query for this element. For instance, a selector query that works for this element would be .quote a. However, this will also extract links from the tags as well. It is not possible to make the query more specific since the target anchor element does not have any ID or class added to it. Therefore, you will need to filter out the tags’ URLs while processing the results of the extraction from Cheerio.

Following a similar process, you will need to write the selector queries for the top ten tags and pagination as well. A selector query that works for Top Ten tags is .tag-item a and one that works for pagination is .pager > .next a. You can use these to write the logic for extracting and filtering the links.

Next, create a new function named crawlQuotes and define it as the following:

const crawlQuotes = async (paginationURLs, visitedURLs, authorURLs, tagURLs) => {

    let currentURLIndex = 0
    
    while (visitedURLs.length !== paginationURLs.length) {

        const pageHTML = await axios.get(baseURL + paginationURLs[currentURLIndex])

        const $ = cheerio.load(pageHTML.data)

        // Whenever a URL is visited, add it to the visited URLs list
        visitedURLs.push(paginationURLs[currentURLIndex])

        // Extracting all author links and tag links from each quote
        $(".quote a").each((index, element) => {
            const URL = $(element).attr("href")
    
            if (URL.startsWith("/tag")){
                if (URL.endsWith("/page/1/"))
                    tagURLs.add(URL.split("/page/1/")[0])
                else
                    tagURLs.add(URL)
            } else if (URL.startsWith("/author"))
                authorURLs.add(URL)
    
        })
    
        // Extracting all tag links from the top ten tags section on the website
        $(".tag-item a").each((index, element) => {
            const URL = $(element).attr("href")
    
            if (URL.startsWith("/tag")) {
                if (URL.endsWith("/page/1/"))
                    tagURLs.add(URL.split("/page/1/")[0])
                else
                    tagURLs.add(URL)
            } else if (URL.startsWith("/author"))
                authorURLs.add(URL)
        })
    
        // Extracting the links from the "next" button at the bottom of the page
        $("li.next a").each((index, element) => {
            const URL = $(element).attr("href")
    
            if (URL.startsWith("/page"))
                paginationURLs.push(URL)
        })

        // Once the processing is complete, move to the next index
        currentURLIndex += 1
    }
}

This function takes care of extracting all tags, authors, and page links from the homepage. It also takes care of iterating over all page links to extract all data from the website, and the use of Sets in the URL queues for author and tag pages ensures that any link encountered twice does not end up creating duplicate entries in the queue.

You can run this function by updating your main() function to look like this:

async function main() { 
    const paginationURLs = ["/"]
    const visitedURLs = []
    
    const authorURLs = new Set()
    const tagURLs = new Set()

    await crawlQuotes(paginationURLs, visitedURLs, authorURLs, tagURLs)

    console.log(authorURLs)
    console.log(tagURLs)
    console.log(paginationURLs)
}

You can run the script by running the command node app.js in a terminal window. You should see the extracted author, tag, and pagination URLs in the output:

Set(50) {
  '/author/Albert-Einstein',
  '/author/J-K-Rowling',
  '/author/Jane-Austen',
  '/author/Marilyn-Monroe',
  '/author/Andre-Gide',
  ... 45 more items
}
[
  '/',        '/page/2/',
  '/page/3/', '/page/4/',
  '/page/5/', '/page/6/',
  '/page/7/', '/page/8/',
  '/page/9/', '/page/10/'
]
Set(147) {
  '/tag/change',
  '/tag/deep-thoughts',
  '/tag/thinking',
  '/tag/world',
  '/tag/abilities',
  ... 142 more items
}

This confirms that your web crawler is working correctly. As you saw, designing a web crawler greatly depends on the structure of the target website. Studying the DOM structure and writing the right selector queries is an integral part of the process.

Scheduling and Processing

While you have a working web crawler already, there are good chances it will run into errors and rate limits in real-world use cases. You need to prepare for such cases as well.

It is considered a good practice to avoid overloading target servers by sending too many requests simultaneously. To do that, consider introducing a delay in your pagination loop. You can use a “waiting” function like the following:

function halt(duration) {
  return new Promise(r => setTimeout(r, duration))
}

Calling the halt function with a duration like 300 or 400 will help you slow down the rate at which your code crawls through the entire target website. The duration can be adjusted based on the estimated number of URLs to be crawled and any known rate limits of the target website.

Error handling is important to ensure the smooth functioning of your web crawlers. Make sure to always wrap network requests in try-catch blocks and provide appropriate error handling logic for known issues, like HTTP 403 (forbidden) or HTTP 429 (rate limit exceeded).

Data Extraction and Storage

Now that you have a set of URLs extracted from the target website, it becomes very easy to extract information from those pages using Cheerio. For instance, if you want to extract the details of each author and store it in a JSON file locally, you can do that with a few lines of code with Cheerio.

First of all, open an author page to understand its structure and figure out the right Cheerio selectors.

You will notice that the name, date of birth, location of birth, and a description of the author can be found under an H3 with the class “author-title”, a span with the class “author-born-date”, a span with the class “author-born-location”, and a div with the class “author-description” respectively. You can use these selectors to write a new function in the same index.js file that iterates over all author URLs and extracts this information:

const scrapeAuthors = async (authorURLs) => {
    const authors = []
    for (let url of authorURLs) {
        const pageHTML = await axios.get(baseURL + url)

        const $ = cheerio.load(pageHTML.data)

        const author = {
            name: $("h3.author-title").text(),
            dateOfBirth: $("span.author-born-date").text(),
            locationofBirth: $("span.author-born-location").text(),
            description: $("div.author-description").text()
        }

        authors.push(author)

    } 

    return authors
}

You can then update your main() function to call this function right after the crawlQuotes function:

Now when you run the app (using node index.js), you will be able to view the information for all authors printed as an array of author objects in the terminal. You can now choose to store this in a database or a JSON file as you require.

You can find the complete code developed in this tutorial in this GitHub repo.

Conclusion

This article has shown you the basics of building a web crawler using JavaScript and Node.js. Web crawlers are essential for sorting through massive amounts of data online, and with JavaScript and Node.js, you have a powerful set of tools to do just that.

Keep practicing and experimenting with what you’ve learned. The more you work with these tools, the better you’ll get at managing and making sense of the web’s enormous resources.

Additional Resources

• Web Crawling with Python
• Web Scraping with Nodejs
• Web Scraping Python vs Nodejs
• Web Scraping with Puppeteer & Nodejs

Web scraping is a process of collecting data from the Internet for price aggregation, market research, lead generation, etc. But web scraping is mainly done by major programming languages like Python, Nodejs, or PHP due to this many non-coders found it very difficult to collect data from the internet. They have to hire a developer to complete small data extraction tasks.

In this article, we will learn how we can scrape a website using Google Sheets without using a single line of code. Google Sheets provides built-in functions like IMPORTHTML, IMPORTXML, and IMPORTDATA that allows you to import data from external sources directly into your spreadsheet. It is a great tool for web scraping. Let’s first understand these built-in functions one by one.

Google Sheets Functions

It is better to discuss the capabilities of Google Sheets before scraping a live website. It offers three functions as explained above. Let’s discuss those functions in a little detail.

IMPORTHTML– This function provides you with the capability to import a structured list or a table from a website directly into the sheet. Isn’t that great?

=IMPORTHTML("url", "query", index)

• “url” is the URL of the webpage containing the table or list you want to import data from.
• “query” specifies whether to import a table (“table”) or a list (“list”).
• index the index of the table or list on the webpage. For example, if there are multiple tables on the page, you can specify which one to import by providing its index (e.g., 1 for the first table).

IMPORTXML– This function can help you extract text/values or specific data elements from structured HTML or XML.

=IMPORTXML(url, xpath_query)

• url is the URL of the webpage or XML file containing the data you want to import.
• xpath_query is the query used to specify the data element or value you want to extract from the XML or HTML source.

IMPORTDATA– This function can help you import data from any external CSV or a TSV file directly into your Google sheet. It will not be discussed in this article later because the application of this function in web scraping is too small.

Scraping with Google Sheets

This section will be divided into two parts. In the first part, we will use IMPORTXML for scraping, and in the next section, we will use IMPORTHTML for the same.

Scraping Using IMPORTXML

The first step would be to set up an empty or blank Google Sheet. You can do it by visiting https://sheets.google.com/.

You can click on Blank Spreadsheet to create a blank sheet. Once this is done we have to analyze the structure of the target website. For this tutorial, we are going to scrape this website https://scrapeme.live/shop/.

We are going to scrape the name of the Pokemon and its listed price. First, we will learn how we can scrape data for a single Pokemon and then later we will learn how it can be done for all the Pokemons on the page.

Scraping Data for a Single Pokemon

First, we will create three columns Name, Currency, and Price in our Google Sheet.

As you know IMPORTXML function takes two inputs as arguments.

• One is the target URL and in our case the target URL is https://scrapeme.live/shop/
• Second is the xpath_query which specifies the XPath expression used to extract specific data from the XML or HTML source.

I know you must be wondering how you will get this xpath_query, well that is super simple. We will take advantage of Chrome developer tools in this case. Right-click on the name of the first Pokemon and then click on Inspect to open Chrome Dev Tools.

Now, we need an XPath query for this element. Well this can be done by a right click on that h2 tag and then click on the Copy button and finally click on the Copy XPath button.

This is what you will get once you copy the XPath.

//*[@id="main"]/ul/li[1]/a[1]/h2

We can use this XPath query to get the name of the first Pokemon.

Remember to replace any double quotes in the xpath_query with single quotes otherwise, you will get this error in Google Sheets like the one in the image below.

Formula parse error can be resolved by passing single quotes in xpath_query. So, once you type the right function, Google Sheets will pull the name of the first Pokemon.

=IMPORTXML("https://scrapeme.live/shop/", "//*[@id='main']/ul/li[1]/a[1]/h2")

We can see Bulbasaur being pulled from the target web page in the A2 cell of the sheet. Well, this was fast and efficient too!

Now, the question is how to pull all the names. Do we have to apply a different xpath_query for each Pokemon present on the target page?

Well, the answer is NO. We just have to figure out an XPath query that selects all the names of the Pokemon at once.

If you notice our current xpath_query you will notice that it is pulling data from the li element with an index 1. If you remove that index you will notice that it selects all the name tags.

Great! Now, our new xpath_query will look like this.

//*[@id='main']/ul/li/a[1]/h2

Let’s change our xpath_query in the IMPORTXML function.

=IMPORTXML("https://scrapeme.live/shop/", "//*[@id='main']/ul/li/a[1]/h2")

Let’s use this in the Google Sheet now.

In just a few seconds Google Sheet was able to pull all the data from the target page and populate it in the sheet itself. This was super COOL! Similarly, you can pull the currency and price.

xpath_query for all the price tags will be //*[@id=’main’]/ul/li/a[1]/span/span.

=IMPORTXML("https://scrapeme.live/shop/", "//*[@id='main']/ul/li/a[1]/span/span")

Let’s apply this to our currency column.

Let’s see whether we can scale this process by scraping more than one page. When you click on the II page by scrolling down you will notice that the website URL changes to https://scrapeme.live/shop/page/2/ and when you click on the III page the URL changes to https://scrapeme.live/shop/page/3/. We can see the pattern that the number after page/ increases by 1 on every click. This much information is enough for us to scale our current scraping process.

Create another column Page in your spreadsheet.

We have to make our target URL dynamic so that it can pick the page value from the E2 cell. This can be done by changing our target URL to this.

"https://scrapeme.live/shop/page/"&E2

Remember you have to change the target URL to the above URL for both the Name and Price columns. Now, the target URL changes based on the value you provide to the E2 cell.

This is how you can scale the web scraping process by concatenating the static part of the URL with the cell reference containing the dynamic part.

Scraping with IMPORTHTML

Create another sheet within your current spreadsheet by clicking the plus button at the bottom.

For this section, we are going to use https://en.wikipedia.org/wiki/World_War_II_casualties as our target URL. We are going to pull country-wise data from this table.s

=IMPORTHTML("https://en.wikipedia.org/wiki/World_War_II_casualties", "table", 1)

The above function will pull this data.

This function helps you quickly import the data from a table.

Overall, IMPORTHTML is a versatile function that can save you time and effort by automating the process of importing data from HTML tables or lists on web pages directly into your Google Sheets. It’s especially useful for tasks that involve data scraping, reporting, analysis, and monitoring of external data sources.

However, IMPORTHTML may not always format imported data as expected. This can result in inconsistent formatting or unexpected changes to the data once it’s imported into Google Sheets. Users may need to manually adjust formatting or use additional formulas to clean up the imported data.

Limitations of using IMPORTXML and IMPORTHTML

• IMPORTXML and IMPORTHTML are designed for simple data extraction tasks and may not support advanced scraping requirements such as interacting with JavaScript-generated content, handling dynamic web pages, or navigating complex website structures.
• Google Sheets imposes rate limits on the frequency and volume of requests made by IMPORTXML and IMPORTHTML functions. Exceeding these limits can result in errors, delays, or temporary suspensions of the functions. This makes it challenging to scrape large volumes of data or scrape data from multiple websites rapidly.
• Imported data may require additional formatting, cleaning, or transformation to make it usable for analysis or integration with other systems. This can introduce complexity and overhead, particularly when dealing with inconsistent data formats or messy HTML markup.

An alternative to scraping with Google Sheets – Scrapingdog

As discussed above scraping with Google Sheets at scale has many limitations and Scrapingdog can help you bypass all of those limitations. With a pool of more than 13 million proxies and headless Chrome, you can scrape any website at scale without getting blocked.

Scrapingdog provides a web scraping API that can help you bypass CAPTCHAs and IP limits very easily. Its API can handle JS rendering and proxies for you so that you can focus on data collection rather than on these inefficient processes.

You can try the free pack by signing up here. You will get 1000 FREE credits which are enough for testing the API on any website.

Conclusion

We’ve explored the capabilities of IMPORTXML and IMPORTHTML functions in Google Sheets for web scraping. These powerful tools provide a convenient and accessible way to extract data from websites directly into your spreadsheets, eliminating the need for complex coding or specialized software.

However, it’s important to be mindful of the limitations of IMPORTXML and IMPORTHTML, such as rate limits, HTML structure dependencies, and data formatting challenges.

To eliminate such challenges you are advised to use a web scraping API which can take the web scraping process to autopilot mode for you.

Additional Resources

• Scrape LinkedIn Jobs using Scrapingdog’s API without Coding
• What is Search Engine Scraping
• Web Scraping vs Data Mining: What’s the Difference?
• Best Use Cases & Applications of Web Scraping

Having data by your side is the biggest asset one can have.

Every decision today is backed up by data, & therefore the value of data cannot be understated. Unless you are informed in advance, you can’t make a decision. 

Search engines index a lot of data, and gaining access to that data can be your upper hand in competing against others in your industry. And this is where the power data from search engine scraping can become a game-changer.

A recent research study revealed that the search engine giant Google contains over 100,000,000 GB worth of data. 

That’s an enormous amount of data! Let’s jump in and understand what search engine scraping is and how it can help you.

What is Search Engine Scraping?

Web Scraping as a whole is the process of extracting data from a particular source, however when we scrape or extract data from search engines (i.e. Google, Yahoo, Yandex, etc.) then the process is referred to as search engine scraping.

This data extracted can be analyzed and used for various purposes. Search engine scrapers are typically the tools that are designed to extract data from them. 

By now, you might be questioning whether scraping should be an option or whether you can do it the old-fashioned manual way. 

Well, you can do it manually and there are other ways to do it. I have discussed them in the later section of this blog.

What Type of Data Can You Scrape From Search Engines?

Search engines, each with their unique algorithms and features, offer a wealth of information in various formats. Generally, these platforms provide access to a diverse array of data types, including web pages, news articles, images, videos, and more. Essentially, anything that appears on a search engine result page (SERP) is potentially scrapable.

By analyzing the data from SERPs, one can understand how different websites rank for specific keywords, track changes in search engine algorithms, and gather data on consumer engagement with various types of content.

Furthermore, scraping news sections can provide up-to-date information on current events, industry developments, and market shifts. This can be valuable for businesses looking to stay ahead in a rapidly changing environment.

Images and video content scraped from search engines can also be used for various purposes, from digital marketing to machine learning applications. By analyzing visual content, companies can gain insights into consumer preferences, and emerging trends, and even perform competitive analysis.

In addition to these, search engines also index forums, academic papers, patents, and other specialized databases, offering a wide knowledge and information that can be extracted and utilized for research, development, and strategic planning.

Use cases of Search Engine Scraping

SEO and Digital Marketing

SEO is one of the mainstream channels for most of the businesses. According to a study conducted, it generates 34% of the qualified leads for B2B businesses.

By extracting data from SERPs (search engine result pages), professionals can analyze which competitor websites rank higher for keywords and understand the factors contributing to their success. This information is crucial for developing effective SEO strategies, including keyword optimization, content creation/optimization, and link building. Engaging with a white-label SEO company can provide businesses with the expertise and services needed to implement these strategies effectively, under their brand name.

Additionally, digital marketers can use this data to craft more targeted and effective advertising campaigns, understanding what content resonates with audiences and how to position their brand effectively in the domain.

Lead Generation and Sales Intelligence

Search engines can play a significant role in generating leads. Scraping Google Maps of your target potential customers can give you the phone numbers. Similarly, there are other Google products you can web scrape to generate leads.

Learn More: Web Scraping for Lead Generation

Brand Protection

Building a brand from the ground up is a considerable achievement, and naturally, protecting its reputation is of utter importance. Today threats to your brand’s image require serious attention and proactive measures. 

Many companies utilize search engine scraping to detect instances of brand misuse or imitation. This technique is particularly effective in identifying unauthorized use of proprietary business elements, such as images or videos, by competitors or other entities.

Challenges of Search Engine Scraping

Scraping data from Search Engine Results Pages (SERPs) offers significant value to businesses across various industries. However, this data extraction process has challenges, often complicating the scraping process. 

A key issue lies in search engines’ difficulty differentiating between beneficial and harmful bots. As a result, legitimate web scraping activities are frequently misidentified as malicious, leading to unavoidable obstructions. 

IP Blocks: A Common Hurdle

One major obstacle is the risk of IP blocking. Search engines can easily detect a user’s IP address. During web scraping, a large number of requests are sent to servers to retrieve needed information. 

If these requests consistently originate from the same IP address, search engines may block it, perceiving it as non-human traffic. This necessitates careful planning to avoid IP-related issues.

CAPTCHAs

CAPTCHAs represent another prevalent security measure. Search engines throw CAPTCHAs when their system detects unusual or bot activity. Standard tools struggle to bypass CAPTCHAs, often leading to IP blocks. Only the most sophisticated scraping technologies can effectively bypass CAPTCHA challenges.

Dealing with Unstructured Data

Successfully extracting data from search engines is just the right start. However, the real challenge lies in handling the fetched data, especially if it is unstructured and difficult to interpret. Therefore, it’s crucial to consider the desired data format before choosing the right web scraping tool. The utility of the scraped data hinges on its readability and structure, making this an important factor in your scraping strategy.

Frequent Changes in SERP Layouts and Algorithms

Search engines frequently update their algorithms and change the layout of their result pages. These updates can significantly impact scraping efforts, as existing scripts or tools become unusable overnight. 

Keeping up with these changes requires constant monitoring and quick adaptation of scraping tools and techniques. Businesses must invest in agile and adaptable scraping solutions capable of quickly responding to these changes to maintain uninterrupted data collection.

Rate Limiting and Throttling

Another challenge in scraping is rate limiting and throttling implemented by search engines. These mechanisms limit the number of requests an IP address can make within a certain timeframe. Exceeding these limits can result in temporary blocks or slowed responses from the server. 

Effective scraping requires a strategy that either rotates IP addresses or schedules requests in a manner that respects these rate limits, thereby avoiding throttling and ensuring continuous data access.

Tools to Scrape Search Engines 

There are a couple of ways to extract search results. The very basic way would be to do it manually, however, this method is time-consuming, is prone to make mistakes, and is not scalable.

Further, there are no-code readily available tools, these tools can be used by someone who has zero experience in scraping. These tools have some limitations, that can be overcome by using a Web scraping API.

Although some programming background needs to be there to run these APIs, they are a great way to scale the process of scraping search results. Recently, I have made a dedicated Google Scraping API, the output it gives is in JSON format. 

Conclusion

Search engines are indeed a great source of information. The value they can provide is immense. Built-in tools can help you to help you in this process. I at Scrapindog have an experience of over 8 years of scraping & have been constantly evolving in the web scraping space.

Over time we have built more stable APIs for different sources. Also, you can check out my article published on the best Google SERP APIs to see which API would suit you. I have compared different aspects and listed them in a table.

Happy Scraping!!

Additional Resources

• Scrape Google Search Results using Python
• Scraping Yahoo Search Engine Results using R
• Scrape Bing Search Results using Python
• How To Scrape Website using Google Sheets without Coding

Wanna Buy some items on Amazon? But afraid that the current pricing of the product doesn’t match your budget?

Well, you can wait for the price to drop. But when will it drop?

In this tutorial, we are going to build an Amazon Price Tracker using our old favorite programming language ‘Python’. We will first scrape prices from any Amazon product and then will set an alert when the price drops a certain number.

Requirements

I hope you have already installed Python 3.x on your machine. If not then you can download it from here. After this, we will create a folder in which we will keep this project. I am naming the folder as scraping.

mkdir scraping
cd scraping

Now, we will install all the libraries that we need in this tutorial.

• requests– This will be used to make a GET request to the API which provides Amazon pricing.
• smtplib– This will be used to email when the price drops.
• time– To create a delay between two consecutive function calls. We will talk about this later.

We just need to install requests as the other two libraries are pre-installed.

pip install requests

We also have to sign up for a free Amazon Scraping API which will help us pull pricing of any product in real time. The trial package comes with a generous 1000 free credits. So, it is enough for testing.

Scraping Amazon Product Pricing

We will be scraping the price of this product page. In this section, we are going to scrape Amazon prices using the API.

import requests
import time
import smtplib
from email.mime.text import MIMEText

def check_price():
    api_url = 'http://api.scrapingdog.com/amazon/product?api_key=YOUR-API-KEY&domain=com&asin=B0BSHF7WHW'

    try:
        response = requests.get(api_url)

        price = int(float(response.json()['price'].replace("$","").replace(',', '')))

        print(f"Current price: {price}")

        if int(price) < 2147:
            print(price)
    except requests.RequestException as e:
        print(f"Error fetching price: {e}")

check_price()

1. Imports:

• requests– A Python library used for making HTTP requests.
• time– A Python library providing various time-related functions (though it’s not used in this function).
• smtplib and email.mime.text.MIMEText-Libraries for sending emails, but these are not used in the provided function.

2. The check_price Function

• api_url– This is a URL string that points to an API endpoint. It appears to be for an Amazon product’s price, fetched using the Scrapingdog API service.
• The function makes an HTTP GET request to the API using requests.get(api_url).
• It then parses the JSON response to extract the product’s price.
• response.json()[‘price’] gets the price from the JSON response.
• .replace(“$”,””).replace(‘,’, ”) removes the dollar sign and commas from the price string. This is necessary to convert the price from a formatted string (like $2,147.42) to a number.
• int(float(…)) first converts the cleaned string to a float (to handle decimal points) and then to an integer. This effectively truncates any decimal part.
• Our budget is anything below $2147. So, we have set a condition that if the price is less than 2147 we will send ourselves an email. We will write the email function in the next section.
• The try-except block catches exceptions related to the network request (like connectivity issues, or errors returned by the API).

3. Calling check_price

• Finally, the function check_price() is called to execute the code.

Remember you have to use your API key in the above code.

Once you run the code you will get this.

Read More: A Comprehensive Guide on Amazon Price Scraping

Setting Price Alert Email

In this section, we will write an email function that will help us send an email if the price of the product is under our budget.

def send_email(price):
    sender_email = "[email protected]"
    receiver_email = "[email protected]"
    password = "your-password"

    message = MIMEText(f"The price has dropped below $2147! Current price is: ${price}")
    message['Subject'] = "Price Alert!"
    message['From'] = sender_email
    message['To'] = receiver_email

    try:
        with smtplib.SMTP('smtp.gmail.com', 587) as server:
            server.starttls()
            server.login(sender_email, password)
            server.sendmail(sender_email, receiver_email, message.as_string())
            print("Email sent successfully!")
    except Exception as e:
        print(f"Error sending email: {e}")

Let me explain the above code step by step.

• def send_email(price)– This line defines a function named send_email that takes one argument, price.
• sender_email– The email address that will send the notification.
• receiver_email– The email address that will receive the notification.
• password– The password for the sender’s email account. For security reasons, hardcoding passwords like this is not recommended in a production environment.
• MIMEText– This is a class from the email.mime.text module used to create an email body. The email body in this case includes a message about the price drop and the current price.
• The Subject, From, and To headers of the email are set to appropriate values.
• The function uses smtplib.SMTP to connect to Gmail’s SMTP server at smtp.gmail.com on port 587 (the standard port for SMTP with TLS).
• server.starttls()– This command starts TLS encryption for the connection, ensuring that the email contents and login credentials are securely transmitted.
• server.login(sender_email, password)– Logs into the SMTP server using the provided credentials.
• server.sendmail(sender_email, receiver_email, message.as_string())– Sends the email.
• If the email is sent successfully, “Email sent successfully!” is printed. If there’s an error during this process, it’s caught in the except block and printed.

Setting a Time Interval

Only one thing is left which is to set a time interval between two consecutive calls. For this, we will use the time library.

while True:
    check_price()
    time.sleep(3600)

This code will run the check_price() function every hour. Our motive is to constantly monitor the pricing of the product and as soon as the price comes under our budget we will send an email to ourselves.

Complete Code

import requests
import time
import smtplib
from email.mime.text import MIMEText

def send_email(price):
    sender_email = "[email protected]"
    receiver_email = "[email protected]"
    password = "your-password"

    message = MIMEText(f"The price has dropped below $2147! Current price is: ${price}")
    message['Subject'] = "Price Alert!"
    message['From'] = sender_email
    message['To'] = receiver_email

    try:
        with smtplib.SMTP('smtp.gmail.com', 587) as server:
            server.starttls()
            server.login(sender_email, password)
            server.sendmail(sender_email, receiver_email, message.as_string())
            print("Email sent successfully!")
    except Exception as e:
        print(f"Error sending email: {e}")

def check_price():
    api_url = 'http://api.scrapingdog.com/amazon/product?api_key=YOUR-API-KEY&domain=com&asin=B0BSHF7WHW'

    try:
        response = requests.get(api_url)

        price = int(float(response.json()['price'].replace("$","").replace(',', '')))

        print(f"Current price: {price}")

        if int(price) < 2147:
            send_email(price)
    except requests.RequestException as e:
        print(f"Error fetching price: {e}")

while True:
    check_price()
    time.sleep(3600)

Test Email

For testing purposes, we can set the price budget above or equal to $2147 and see if the code works or not.

So, once you run the code you should see this on your console.

And you should receive an email.

With this our Amazon Price Tracker is ready. Now, you can track the pricing of any product and send yourself an email.

You can even create an app using this Amazon Scraping API which helps other people track the pricing of their favourite products.

Conclusion

In this blog, we understood how you can keep track of products on Amazon when they drop a certain level. Just like Amazon, you can use Python to track prices from other marketplaces as well, such as walmart & eBay.

If you are a beginner in Python, I would recommend you go through this comprehensive guide on web scraping with Python.

If you liked the article do share it with your social network.

Happy Tracking!!

Additional Resources

• How To Bypass Captcha When Scraping Amazon
• Web Scraping Walmart using Python
• Web Scraping Yellow Pages using Python
• Web Scraping Yelp using Python
• Web Scraping eBay using Python

Yelp is a platform that gathers customer opinions about businesses. It started in 2004 and helps people see what real customers think about local businesses. It ranks as the 44th most visited website & has over 184 million reviews. (source)

As of now, there are more than 5 million businesses listed on Yelp, making it a valuable resource for finding information about these businesses. To extract details from any listing from Yelp there are different ways.

For the sake of this tutorial, we’ll use Python to extract business listing details from Yelp.

Requirements For Scraping Yelp Data

Generally, web scraping is divided into two parts:

1. Fetching data by making an HTTP request
2. Extracting important data by parsing the HTML DOM

Libraries & Tools

1. Beautiful Soup is a Python library for pulling data out of HTML and XML files.
2. Requests allow you to send HTTP requests very easily.
3. Web scraping API extracts the HTML code of the target URL.

Know more: Learn Web Scraping 101 with Python!!

Setup

Our setup is pretty simple. Just create a folder and install BeautifulSoup & requests. For creating a folder and installing libraries, type the below-given commands. I assume that you have already installed Python 3. x (The latest version is 3.9 as of April 2022).

mkdir scraper
pip install beautifulsoup4
pip install requests

Now, create a file inside that folder by any name you like. I am using scraping.py.

from bs4 import BeautifulSoup
import requests

Data Points We Are Going To Scrape From Yelp

We are going to scrape data from this restaurant.

We will extract the following information from our target page.

1. Name of the Restaurant
2. Address of the Restaurant
3. Rating
4. Phone number

Let’s Start Scraping

Now, since we have all the ingredients to prepare the scraper, we should make a GET request to the target URL to get the raw HTML data.

We will scrape Yelp data using the requests library below.

from bs4 import BeautifulSoup
import requests

l={}
u=[]

r = requests.get('https://www.yelp.com/biz/sushi-yasaka-new-york').text

This will provide you with an HTML code of that target URL.

Parsing the raw HTML

Now we will use BS4 to extract the information we need. But before this, we have to find the DOM location of each data element. We will take advantage of Chrome developer tools to find the location.

Let’s start with the name first.

So, the name is located inside the h1 tag with the class css-1se8maq.

Similarly, the address can be found inside the p tag with the class css-qyp8bo from the image above.

The star rating can be found in the div tag with the class css-1v6kfrx. Inside this class, there is an attribute aria-label inside which this star rating is hidden.

The phone number is located inside the second div tag with the class css-djo2w.

Now, we have the location of each data point we want to extract from the target page. Let’s now use BS4 to parse this information.

soup = BeautifulSoup(r,'html.parser')

Here we have created a beautifulSoup object.

try:
    l["name"]=soup.find("h1",{"class":"css-1se8maq"}).text
except:
    l["name"]=None
try:
    l["address"]=soup.find("p",{"class":"css-qyp8bo"}).text
except:
    l["address"]=None
try:
    l["stars"]=soup.find("div",{"class":"css-1v6kfrx"}).get('aria-label')
except:
    l["stars"]=None
try:
    l["phone"]=soup.find_all("div",{"class":"css-djo2w"})[1].text.replace("Phone number","")
except:
    l["phone"]=None


u.append(l)
l={}
print({"data":u})

Once you run the above code you will get this output on your console.

{'data': [{'name': 'Sushi Yasaka', 'address': '251 W 72nd St New York, NY 10023', 'stars': '4.2 star rating', 'phone': '(212) 496-8460'}]}

There you go!

We have the Yelp data ready to manipulate and maybe store somewhere like in MongoDB. But that is out of the scope of this tutorial.

Complete Code

You can scrape other information like reviews, website addresses, etc from the raw HTML we downloaded in the first step. But for now, the code will look like this.

from bs4 import BeautifulSoup
import requests
l={}
u=[]
r = requests.get('https://www.yelp.com/biz/sushi-yasaka-new-york').text

soup = BeautifulSoup(r,'html.parser')



try:
    l["name"]=soup.find("h1",{"class":"css-1se8maq"}).text
except:
    l["name"]=None
try:
    l["address"]=soup.find("p",{"class":"css-qyp8bo"}).text
except:
    l["address"]=None
try:
    l["stars"]=soup.find("div",{"class":"css-1v6kfrx"}).get('aria-label')
except:
    l["stars"]=None
try:
    l["phone"]=soup.find_all("div",{"class":"css-djo2w"})[1].text.replace("Phone number","")
except:
    l["phone"]=None

u.append(l)
l={}
print({"data":u})

How to scrape Yelp without getting blocked?

Scrapingdog’s API for web scraping can help you extract data from Yelp at scale without getting blocked. You just have to pass the target url and Scrapingdog will create an unbroken data pipeline for you, that too without any blockage.

Once you sign up you will get an API key on your dashboard.

You have to use this API key in the below provided code.

from bs4 import BeautifulSoup
import requests
l={}
u=[]
r = requests.get('https://api.scrapingdog.com/scrape?dynamic=false&api_key=Your-API-key&url=https://www.yelp.com/biz/sushi-yasaka-new-york').text

soup = BeautifulSoup(r,'html.parser')



try:
    l["name"]=soup.find("h1",{"class":"css-1se8maq"}).text
except:
    l["name"]=None
try:
    l["address"]=soup.find("p",{"class":"css-qyp8bo"}).text
except:
    l["address"]=None
try:
    l["stars"]=soup.find("div",{"class":"css-1v6kfrx"}).get('aria-label')
except:
    l["stars"]=None
try:
    l["phone"]=soup.find_all("div",{"class":"css-djo2w"})[1].text.replace("Phone number","")
except:
    l["phone"]=None

u.append(l)
l={}
print({"data":u})

As you can see the code is the same except the target url. With the help of Scrapingdog, you can scrape endless data from Yelp.

Conclusion

In this step-by-step guide, you’ve gained a comprehensive understanding of how to create a Python scraper capable of retrieving Yelp data efficiently. As demonstrated throughout this tutorial, the process is surprisingly straightforward.

Additionally, you’ve explored an alternative approach using the Web Scraper API, which can help bypass anti-bot protection mechanisms and extract Yelp data with ease. The techniques outlined in this article not only apply to Yelp but can also be employed to scrape data from similarly complex websites without the risk of being blocked.

Happy Scraping!!

Additional Resources

• Web Scraping Yellow Pages using Python
• Web Scraping Walmart using Python

Today many websites cannot be scraped with a normal XHR request because they render data through Javascript execution. Traditional web scraping libraries like requests and Scrapy are excellent for extracting data from websites where the content is directly embedded in the HTML of the page. However, a growing number of modern websites use JavaScript to dynamically load and render content. When you make a simple HTTP request (like an XHR request) using requests or similar libraries, you get the initial HTML, but it might lack the dynamically loaded content.

Headless browsers or browser automation tools can help. These tools simulate a browser environment and can execute JavaScript on a page.

In this tutorial, we will learn how to scrape dynamic websites using Selenium and Python. For this tutorial, we are going to scrape myntra.com.

Requirements

I assume you already have Python 3.x installed on your machine. If not then you can install it from here. Let’s start with creating a folder in which we are going to keep our Python script.

mkdir seleniumtut

Then after creating the folder, we have to install two libraries that will be needed in the course of this tutorial.

• Selenium– It is used to automate web browser interaction from Python.
• Beautiful Soup is a Python library for parsing data from the raw HTML downloaded using Selenium.
• Chromium– It is a web browser that will be controlled through selenium. You can download it from here.

pip install bs4
pip install selenium

Remember that you will need the exact version of the Chromium driver as your Chrome browser. Otherwise, it will not run and throw an error.

Now, create a Python file where we can write our code. I am naming the file as tutorial.py.

What are we going to scrape?

For this tutorial, we are going to scrape this page.

We are going to scrape three data points from this page.

• Name of the product
• Price of the product
• Rating of the product

First, we will download the raw HTML from this page using selenium, and then using BS4 we are going to parse the required data.

Downloading the data with Selenium

First step would be import all the required libries inside our file tutorial.py.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

Here we first imported the BeautifuSoup, webddriver module from selenium and the Service class from selenium.

PATH = 'C:\Program Files (x86)\chromedriver.exe'

Here I have declared the path where I have installed my chromium driver. Your location might be different.

service = Service(executable_path=PATH)
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)

The executable_path parameter is used to provide the path to the ChromeDriver executable on your system. This is necessary for Selenium to locate and use the ChromeDriver executable.

webdriver.ChromeOptions is a class in Selenium that provides a way to customize and configure the behavior of the Chrome browser.

webdriver.Chrome is a class in Selenium that represents the Chrome browser. The service parameter is used to specify the ChromeDriver service, which includes the path to the ChromeDriver executable. The options parameter allows you to provide the ChromeOptions instance created earlier, customizing the behavior of the Chrome browser according to your needs.

driver.get("https://www.myntra.com/trousers/the+indian+garage+co/the-indian-garage-co-men-black-slim-fit-solid-joggers/9922235/buy")

html_content = driver.page_source
print(html_content)


# Close the browser
driver.quit()

The get method is used to navigate the browser to a specific URL, in this case, the URL of a Myntra product page for trousers.

driver.page_source is a property of the WebDriver that returns the HTML source code of the current page. After this step we are printing the result on the console.

driver.quit() closes the browser, ending the WebDriver session.

Once you run this code you will get this on your console.

This means that we have successfully managed to download the raw HTML from the target website. Now, we can use BS4 for parsing the data.

To understand more you should refer to this guide on scraping with Python & Selenium.

Parsing the data with BS4

Since we have already decided what we are going to extract from this raw data let’s find out the DOM location of each of these data points.

The title of the product is located inside the h1 tag with class pdp-title.

The rating is located inside the first div tag of the parent div with the class index-overallRating.

Similarly, you can see that the pricing text is located inside the span tag with class pdp-price.

We can now implement this into our code using BS4.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time

l=[]
o={}

PATH = 'C:\Program Files (x86)\chromedriver.exe'

# Set up the Selenium WebDriver
service = Service(executable_path=PATH)
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
driver = webdriver.Chrome(service=service, options=options)

# Navigate to the home page
driver.get("https://www.myntra.com/trousers/the+indian+garage+co/the-indian-garage-co-men-black-slim-fit-solid-joggers/9922235/buy")

html_content = driver.page_source

soup=BeautifulSoup(html_content,'html.parser')


try:
    o["title"]=soup.find('h1',{'class':'pdp-title'}).text.lstrip().rstrip()
except:
    o["title"]=None

try:
    o["rating"]=soup.find('div',{'class':'index-overallRating'}).find('div').text.lstrip().rstrip()
except:
    o["rating"]=None

try:
    o["price"]=soup.find('span',{'class':'pdp-price'}).text.lstrip().rstrip()
except:
    o["price"]=None


l.append(o)


print(l)


# Close the browser
driver.quit()

soup = BeautifulSoup(html_content, ‘html.parser’) initializes a BeautifulSoup object (soup) by parsing the HTML content (html_content) using the HTML parser.

• o[“title”] = soup.find(‘h1’, {‘class’: ‘pdp-title’}).text.lstrip().rstrip() finds an <h1> element with the class ‘pdp-title’ and extracts its text content after stripping leading and trailing whitespace. If not found, sets ‘title’ to None.
• o[“rating”] = soup.find(‘div’, {‘class’: ‘index-overallRating’}).find(‘div’).text.lstrip().rstrip() finds a <div> element with the class ‘index-overallRating’, then finds another <div> inside it and extracts its text content. If not found, sets ‘rating’ to None.
• o[“price”] = soup.find(‘span’, {‘class’: ‘pdp-price’}).text.lstrip().rstrip() finds a <span> element with the class ‘pdp-price’ and extracts its text content after stripping leading and trailing whitespace. If not found, sets ‘price’ to None.

At the end, we pushed the complete object o to the list l using .append() method.

Once you run the code you will get this output.

At the end, we pushed the complete object o to the list l using .append() method.

Once you run the code you will get this output.

[{'title': 'The Indian Garage Co', 'rating': '4', 'price': '₹545'}]

Complete Code

You can of course extract many other data points from this page by finding the location of those elements. But for now, the code will look like this.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time

l=[]
o={}

PATH = 'C:\Program Files (x86)\chromedriver.exe'

# Set up the Selenium WebDriver
service = Service(executable_path=PATH)
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
driver = webdriver.Chrome(service=service, options=options)

# Navigate to the home page
driver.get("https://www.myntra.com/trousers/the+indian+garage+co/the-indian-garage-co-men-black-slim-fit-solid-joggers/9922235/buy")

html_content = driver.page_source

soup=BeautifulSoup(html_content,'html.parser')


try:
    o["title"]=soup.find('h1',{'class':'pdp-title'}).text.lstrip().rstrip()
except:
    o["title"]=None

try:
    o["rating"]=soup.find('div',{'class':'index-overallRating'}).find('div').text.lstrip().rstrip()
except:
    o["rating"]=None

try:
    o["price"]=soup.find('span',{'class':'pdp-price'}).text.lstrip().rstrip()
except:
    o["price"]=None


l.append(o)


print(l)


# Close the browser
driver.quit()

Conclusion

In this article, we learn how we can easily scrape JS-enabled websites using Selenium.

• Selenium is highlighted for its capability to automate browser interactions seamlessly.
• The focus was to parse HTML with elegance using Selenium to extract valuable insights.
• The scope of insights includes a wide range, from product details to user ratings.
• Overall, the article demonstrates how Selenium can be a powerful tool for web scraping tasks involving dynamic content and JavaScript execution.

Some websites may force you to use a web scraping API for heavy scraping. Web Scraping APIs will take your headless browser and proxy management part on an autopilot mode and you will just have to focus on data collection.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on social media. 

Additional Resources

• Scraping Amazon using Python
• Web Scraping eBay using Python
• Web Scraping Walmart using Python
• What is Price Scraping & How It Can Help Your Business
• A Complete Guide on Web Scraping with Python

In the vast ecosystem of the World Wide Web, no single entity dominates the search engine space quite like Google. Serving as the internet’s principal gatekeeper, Google processes an astounding 8.5 billion searches per day, firmly maintaining over 90% of the global search engine market share. Its groundbreaking algorithms and unprecedented ability to process and present information make Google not just a search engine, but a fundamental pillar of the digital age.

However, as comprehensive as the Google search engine may be, extracting specific, manageable data from this colossal repository poses a significant challenge.

In this article, we will explore the 10 best SERP APIs. These APIs provide the ability to provide enhanced efficiency, flexibility, and scalability in a wide array of applications and services.

Know More: What is Search Engine Scraping

Why do you need SERP scraping API?

Indeed, crafting your own Google web scraper using your preferred programming language is an intriguing possibility for extracting search results. This bespoke approach can yield significant results, especially for smaller-scale operations or niche applications.

That being said, to scale Google scraping you would need a search engine API. Google scraping can have many use cases, one of them we use commonly is keyword position tracking in SEO strategy.

Using your Google web scraper can be a risky proposition, especially considering Google’s protective measures. Rapid, large-scale data extraction can lead to an immediate IP ban, effectively halting your data pipeline. Moreover, ensuring your scraper’s longevity requires altering headers with each request – a labor-intensive process.

Seeking a Google scraping service may be driven by the difficulty of scraping Google and the need for quick access to data. Unless you have an abundance of time, it is generally not advisable to develop a Google scraper.

Additionally, data parsing becomes your responsibility. It is no easy task. Furthermore, Google is continuously evolving, rendering scrapers ineffective over time. Maintaining a Google scraper indefinitely is an unappealing prospect.

You can use the official API from Google but it is way too expensive. Then the alternative is to use the 3rd party scraping APIs. With these, you can get Google search results anonymously. They provide great flexibility and are faster in response.

Advantages of using SERP API for Google

There are various advantages of using these APIs to scrape Google search results:

1. You will always stay anonymous. On each request, a new IP will be used to scrape a page. Your IP will always be hidden.
2. The price is too low compared to the official API.
3. You can get parsed JSON as well as raw HTML data from these APIs.
4. You can even ask the vendors for API customization.
5. 24X7 support.

10 Best APIs for Scraping Google in 2024

We will be judging these APIs based on 5 attributes.

• Scalability means how many pages you can scrape in a day.
• Pricing of the API. What is the cost of one API call?
• Speed means how fast an API can respond with results.
• Developer-friendly refers to the ease with which a software engineer can use the service.
• Stability refers to how much load a service can handle or for how long the service is in the market.

Scrapingdog’s Google SERP API

Scrapingdog’s Google SERP API provides raw and parsed data from Google search results with a breeze. Along with a general web scraper API, you get dedicated scrapers for websites like Google, LinkedIn, Zillow, etc.

Details

• With this API you get more than a billion API requests every month which makes this API a healthy choice.
• Per API call cost for scraping Google starts from $0.003 and goes below $0.00125 for higher volumes.
• For testing the speed of the API we are going to test the API on POSTMAN.

It took around 2.98 seconds to complete the request.

• Scrapingdog has very clean documentation in multiple languages. From curl to Java, you will find a code snippet in almost every language.
• Scrapingdog has been in the market for more than 3 years and rated 4.8 on Trustpilot. This Google SERP API is stable.
• You can even test the API for free.

Data For SEO

Data for SEO provides the data required for creating any SEO tool. They have APIs for backlinks, keywords, search results, etc.

Details

• Documentation is too noisy which makes integration of the API too time-consuming.
• The pricing is not clear. Their pricing changes based on the speed you want. But the high-speed pack will cost $0.002 per search. The minimum investment is $2k per month.
• Since they have been into scraping for a very long time then I can say that this API is both scalable and stable.
• Cannot comment on the speed as we were unable to test the API because of very confusing documentation.

Bright Data

Bright Data as we all know is a huge company focused on data collection. They provide proxies, data scrapers, etc.

Details

• Their documentation is quite clear and testing is super simple.
• We tested their API and the average response time was close to 1.9 seconds.
• Per API call cost starts from $0.005.
• The success rate is pretty great which makes this API scalable and stable. They have been in the market for a very long time. You can consider them experts in data extraction.

Scrape-It Cloud

Scrape-it Cloud is another great option if you are looking for search engine API. Their dashboard makes your onboarding pretty simple.

Details

• Documentation is pretty simple and easy to understand.
• Per API call response time is around 4.7 seconds.
• If you hit the same API multiple times they show a lag. So, I think the API is not scalable at the moment.
• Per API call price starts from $0.003 and goes around $0.0004 with higher volumes.
• They are pretty new to the market and lack regular updates to the API.

Scrapingbee’s Google Search API

Scrapingbee is also a web scraping API that offers a dedicated scraper for Google.

Details

• The documentation is pretty clear and contains code snippets in almost every language.
• The average time taken by the API to respond was close to 5 seconds. API delays response if multiple requests are made at the same time.

• They have been in the market for more than 4 years now and this makes them reliable and stable.
• The average time might go up if you need high concurrency.
• Pricing starts from $0.0081 per search and goes down to $0.001875 per search for the Business pack.

Know Why: Scrapingdog is a Better Alternative to Scrapingbee.

Serpdog

Serpdog was initially launched as a free search engine API but now it is a fully-fledged freemium API. They offer multiple dedicated APIs around Google and its other services like Maps, Images, News, etc.

Details

• Documentation is very clear and has code snippets for all major languages like Java, NodeJS, etc. This makes testing this API super easy.
• The average response time was around 3.2 seconds. And it continues to be the same even for high concurrency.

• Pricing starts from $0.00375 per search and goes below $0.00166 for bigger packs.
• API is reliable and can be used for bigger projects.

SerpAPI

SerpAPI is the fastest Google search scraper API with the highest variety of Google-related APIs.

Details

• The documentation is very clear and concise. You can quickly start scraping and Google service within minutes.
• The average response time was around 3.8 seconds. API is fast and reliable. This API can be used for any commercial purpose which requires a high volume of scraping.

• Pricing starts at $0.01 per request and it goes down to $0.0083! Now, you might be thinking, ‘Wow, that’s expensive!’ But fear not, because here’s the SerpAPI alternative that’s not only fast but also easy on your wallet.
• SerpAPI has been in this industry since 2016 and they have immense experience in this market. If you have a high-volume project then you can consider them.

Zenserp

Zenserp is a search engine API that offers multiple dedicated scrapers for Google, Bing, etc.

• The documentation is not that great clear. You have to spend a little time to get an idea. Only curl code snippets are available.
• The average response time was around 5.2 seconds. Time might go up with high concurrency.

• Pricing starts from $0.058 and goes below $0.03325. Seems very expensive.
• No support and zero updates to the API make it non-reliable.

Apify

Apify is a web scraping and automation platform that provides tools and infrastructure to simplify data extraction, web automation, and data processing tasks. It allows developers to easily build and run web scrapers, crawlers, and other automation workflows without having to worry about infrastructure management.

Details

• The documentation is pretty clear and makes integration simple.
• The average response time was around 8.2 seconds.

• Pricing starts from $0.003 per search and goes below $0.0019 per search in their Business packs.
• They have been in this industry for a very very long time which indicates they are reliable and scalable.

Serpwow

Serpwow is an API service that provides access to search engine result page (SERP) data from various search engines. It allows developers to retrieve organic search results, paid search results, and other relevant data related to search engine rankings.

Details

• Documentation is neat and API integration is simple.
• The average response time for this API was around 10 seconds. Which is quite high.

• Pricing starts from $0.012 per search and goes below $0.0048 per search for bigger packs.
• API seems unreliable as the time taken by the API kept increasing with the load.

Final Verdict

Based on multiple factors we have prepared a list of the best SERP APIs, many of which offer free trials to test. This is the report based on my thorough analysis of each API. They all will look a little similar on the surface but once you dig deeper while testing you will find very few APIs(two or three) are stable and can be used for production purposes.

At this point, it is crucial to evaluate the options based on your specific requirements and select the most suitable one from the provided list.

Additional Resources

• Best Twitter Scraper in 2024
• Best LinkedIn Scraper in 2024
• The Best Web Scraping Tools
• Scrape Bing Search Results using Python
• Scrape Website using Google Sheets without Coding

Scraping social media websites has become quite imperative for multiple companies. You can track people’s sentiments through their comments, posts, likes, etc. Companies can track feedback from their customers or they can track activities of their competitors.

Today influencer marketing is at its peak and by scraping the number of followers and their content reach one can decide to promote their products with them.

PR agencies can scrape social media to analyze hot topics and then they can ask their clients to create content on the same. On the other hand, political parties can identify which topic should be on the agenda for the election campaign.

Among all social media websites, Twitter is the most dynamic one. It has around 450M active users which makes it a great data dump. From movie celebrities to top political figures all are pretty active over here and hence there will be a lot of user-generated data over here due to reactions from common people.

But scraping Twitter at scale is a challenge in itself because your IP will be blocked in no time and your data pipeline will be blocked. So, it is always advisable to use a Twitter Scraping API.

In this blog, we will discuss the top 4 Twitter Scrapers which can help you to get the data you need.

Scrapingdog

Scrapingdog is known for its exceptional APIs for data-driven apps. We provide an API for web scraping that is capable of scraping public data from almost any source on the internet. Apart from general web scraping API, we have a dedicated Twitter scraper API, LinkedIn Profile Scraper API, LinkedIn Jobs API & Amazon Scraper API.

Features

• Scrape any tweet with its tweet ID or scrape any public page to extract details like followers count, the following count, website link, etc.
• Pricing: The per-page cost of scraping Twitter is $0.00075 per page which is the best available considering the top APIs.
• It can return results in both HTML and parsed JSON.
• A free trial is available with 1000 free credits.

Brightdata

Brightdata is another API in the list with great scraping infrastructure. It specializes in providing proxies but also provides ready-made scrapers to provide data from selected websites.

Features

• They also provide particular tweet scraper, Twitter page scraper, and scraping tweets for any keyword.
• Results will appear in JSON format only.
• Pricing: per page, the cost is around $0.01. This makes it pretty expensive.
• A free trial is not available.

Read More: 5 Best Bright Data Alternatives for Web Scraping

Scrapingbee

Features

• Does not provide any dedicated Twitter Scraper but a general scraper can be used.
• The data you receive will be in HTML format.
• Pricing: Per page cost for scraping Twitter is $0.0016.
• A free trial is available.

Know How: Scrapingdog is the Best Alternative to Scrapingbee.

ScraperAPI

ScraperAPI also provides a dedicated Twitter Scraper.

Features

• They have dedicated APIs for scraping trends and Twitter profile data.
• You will get JSON data with all the necessary data properties.
• Pricing: Not clear on the website but it should be around $ 0.0049.
• A free trial is available.

Know Why: Scrapingdog is the best Alternative to Scraper API!!

Conclusion

We have curated a list of all the Twitter scrapers available in the market. Of course, there are a few more but they were not as good as these four. We will keep updating this list as we find better Twitter scraping solutions.

Additional Resources

• Best Google SERP APIs
• Best LinkedIn Scrapers in 2024
• Best Web Scraping Tools

For businesses, investors, and even curious individuals, real-time insights into the housing market can be invaluable. Redfin, a prominent player in the real estate sector, offers a mine of such data, spanning across more than 100 markets in both the United States and Canada. With a commendable 0.80% market share in the U.S. (Wikipedia), as gauged by the number of units sold, and boasting a network of approximately 2,000 dedicated lead agents, Redfin stands as a significant source of real estate intelligence.

In this blog, we will see how we can scrape data from Redfin using Python, further, I will show you how you can scale this process.

Let’s start!!

Collecting all the Ingredients for Scraping Redfin

Assuming that you have already installed Python 3.x on your machine and if not then please install it from here. Once this is done create a folder in which we will keep our Python scripts.

mkdir redfin
cd redfin

Once you are inside your folder install these public Python libraries.

• Requests– This will be used for making the HTTP connection with redfin.com. Using this library we will install the raw HTML of the target page.
• BeautifulSoup– Using this we will parse the important data from the raw HTML downloaded using the requests library.

pip install requests
pip install beautifulsoup4

Now create a python file inside this folder where you can write the script. I am naming the file as redfin.py.

With this our project setup is complete and now we can proceed with the scraping.

What are we going to scrape?

In this tutorial, we are going to scrape two types of pages from redfin.com.

1. Redfin Search Page
2. Redfin Property Page

Scraping Redfin Search Page

It is always a great practice to decide in advance what data you want from the page. For this tutorial, we are going to scrape this page.

Download Raw HTML from the Page

Our first task would be to download the raw HTML from the target web page. For this, we are going to use the requests library.

import requests
from bs4 import BeautifulSoup


l=[]
o={}



head={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
target_url="https://www.redfin.com/city/30749/NY/New-York/filter/status=active"



resp = requests.get(target_url,headers=head,verify=False)
print(resp.status_code)

First, we imported all the libraries that we installed earlier. Then I declared one empty list and one empty object.

The head variable is a dictionary containing the User-Agent header. The target_url variable contains the URL of the webpage to be scraped.

The requests.get function is used to send an HTTP GET request to the specified URL (target_url). The headers parameter is set to include the User-Agent header from the head dictionary. The verify=False parameter disables SSL certificate verification. The response object (resp) contains the server’s response to the request.

Once you run this code and see a 200 on the logs then that means you have successfully scraped the target web page.

Now, we can parse the data using BS4.

Parsing the Raw HTML

BeautifulSoup will now help us extract all the data points from the raw HTML downloaded in the previous section. But before we start coding we have to identify the DOM location of each element.

We will use the Chrome developer tool to find the DOM location. If you inspect and analyze the design of the page then you will find that all the property box is inside the div tag with the class HomeCardContainer. So, first, we should find all these elements using find_all() method of BS4.

soup=BeautifulSoup(resp.text,'html.parser')

allBoxes = soup.find_all("div",{"class":"HomeCardContainer"})

The BeautifulSoup constructor is used to create a BeautifulSoup object (soup). The find_all method of the BeautifulSoup object is used to find all HTML elements that match the class HomeCardContainer.

allBoxes is a list that contains all the property data elements. Using for loop we are going to reach every property container and extract the details. But before we write our for loop let’s find the DOM location of each data point.

Let’s start with the property price.

Once you right-click on the price you will see that the price is stored inside the span tag with the class homecardV2Price.

Similarly, the configuration of the property can be found inside the div tag with class HomeStatsV2.

Agent/Broker name can be found inside the div tag with the class brokerageDisclaimerV2.

Individual property links can be found inside the a tag. This a tag is the only a tag inside each property container.

for box in allBoxes:
    try:
        o["property-price"]=box.find("span",{"class":"homecardV2Price"}).text.strip()
    except:
        o["property-price"]=None

    try:
        o["property-config"]=box.find("div",{"class":"HomeStatsV2"}).text.strip()
    except:
        o["property-config"]=None

    try:
        o["property-address"]=box.find("div",{"class":"homeAddressV2"}).text.strip()
    except:
        o["property-address"]=None

    try:
        o["property-broker"]=box.find("div",{"class":"brokerageDisclaimerV2"}).text.strip()
    except:
        o["property-broker"]=None

    try:
        o["property-link"]="https://www.redfin.com"+box.find("a").get('href')
    except:
        o["property-link"]=None

    l.append(o)
    o={}


print(l)

For each home card container, it extracts specific pieces of information, such as property price, configuration, address, broker details, and a link to the property.

for loop iterates through each element (box) in the list of home card containers. For each piece of information (property price, configuration, address, broker, link), a try block attempts to find the corresponding HTML element within the current home card container (box). If successful, it extracts the text content, strips leading and trailing whitespaces, and assigns it to the corresponding key in the dictionary (o). If the extraction fails (due to an attribute not being present or other issues), the except block sets the value to None.

After extracting information from the current home card container, the dictionary o is appended to the list l. Then, the dictionary o is reset to an empty dictionary for the next iteration.

Once you run this code you will get this response.

Saving the data to a CSV file

For better visibility of this data, we are going to save this data to a CSV file. For this task, we are going to use the pandas library.

df = pd.DataFrame(l)
df.to_csv('properties.csv', index=False, encoding='utf-8')

The code uses the pandas library to create a DataFrame (df) from the list of dictionaries (l) that contains the scraped data. After creating the DataFrame, it is then exporting the DataFrame to a CSV file named 'properties.csv'.

After running the code you will find a CSV file inside your working folder by the name properties.csv.

Saving the data from a list to a CSV file was super simple with Pandas.

Complete Code

You can scrape many more details from the page but for now, the code will look like this.

import requests
from bs4 import BeautifulSoup
import pandas as pd

l=[]
o={}



head={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
target_url="https://www.redfin.com/city/30749/NY/New-York/filter/status=active"



resp = requests.get(target_url,headers=head,verify=False)
print(resp.status_code)
soup=BeautifulSoup(resp.text,'html.parser')

allBoxes = soup.find_all("div",{"class":"HomeCardContainer"})

for box in allBoxes:
    try:
        o["property-price"]=box.find("span",{"class":"homecardV2Price"}).text.strip()
    except:
        o["property-price"]=None

    try:
        o["property-config"]=box.find("div",{"class":"HomeStatsV2"}).text.strip()
    except:
        o["property-config"]=None

    try:
        o["property-address"]=box.find("div",{"class":"homeAddressV2"}).text.strip()
    except:
        o["property-address"]=None

    try:
        o["property-broker"]=box.find("div",{"class":"brokerageDisclaimerV2"}).text.strip()
    except:
        o["property-broker"]=None

    try:
        o["property-link"]="https://www.redfin.com"+box.find("a").get('href')
    except:
        o["property-link"]=None

    l.append(o)
    o={}


print(l)
df = pd.DataFrame(l)
df.to_csv('properties.csv', index=False, encoding='utf-8')

Scraping Redfin Property Page

From the property page, we are going to gather this information.

• Property Price
• Property Address
• Is it still available(True/False)
• About section of the property

Download Raw HTML from the Page

import requests
from bs4 import BeautifulSoup



l=[]
o={}
available=False


head={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
target_url="https://www.redfin.com/NY/New-York/112-E-35th-St-10016/home/45333496"



resp = requests.get(target_url,headers=head,verify=False)
print(resp.status_code)

This Python code performs web scraping on a Redfin property page using the requests library to make an HTTP GET request and the BeautifulSoup library to parse the HTML content. The script initializes empty data structures (l and o) to store scraped information and sets a User-Agent header to simulate a Chrome browser request. The target URL is specified, and an HTTP GET request is sent with SSL certificate verification disabled.

After running the code if you get 200 on your console then that means your code was able to scrape the raw HTML from the target web page.

Let’s use BS4 to parse the data.

Parsing the Raw HTML

As usual, we have to first find the location of each element inside the DOM.

Price is stored inside the div tag with class statsValue.

The address is stored inside the h1 tag with the class full-address.

Property sale status is located inside the div tag with the class ListingStatusBannerSection.

About section of the property can be found inside the div tag with id marketing-remarks-scroll.

soup=BeautifulSoup(resp.text,'html.parser')

try:
    o["property-price"]=soup.find("div",{"class":"statsValue"}).text.strip()
except:
    o["property-price"]=None

try:
    o["property-address"]=soup.find("h1",{"class":"full-address"}).text.strip()
except:
    o["property-address"]=None

check = soup.find("div",{"class":"ListingStatusBannerSection"}).text.strip()
if "ACTIVE" in check:
    available=True
else:
    available=False

try:
    o["property-available"]=available
except:
    o["property-available"]=False

try:
    o["about-property"]=soup.find("div",{"id":"marketing-remarks-scroll"}).text.strip()
except:
    o["about-property"]=None

print(l)

By default available is set to False and it is set to True if the string ACTIVE is present inside the check string. We have used strip() function to remove the unwanted spaces from the text value.

Once you run the code you should get this.

Finally, we were able to extract all the desired information from the target page.

Complete Code

The complete code for this property page will look like this.

import requests
from bs4 import BeautifulSoup


l=[]
o={}
available=False


head={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
target_url="https://www.redfin.com/NY/New-York/112-E-35th-St-10016/home/45333496"



resp = requests.get(target_url,headers=head,verify=False)
print(resp.status_code)
soup=BeautifulSoup(resp.text,'html.parser')

try:
    o["property-price"]=soup.find("div",{"class":"statsValue"}).text.strip()
except:
    o["property-price"]=None

try:
    o["property-address"]=soup.find("h1",{"class":"full-address"}).text.strip()
except:
    o["property-address"]=None

check = soup.find("div",{"class":"ListingStatusBannerSection"}).text.strip()
if "ACTIVE" in check:
    available=True
else:
    available=False

try:
    o["property-available"]=available
except:
    o["property-available"]=False

try:
    o["about-property"]=soup.find("div",{"id":"marketing-remarks-scroll"}).text.strip()
except:
    o["about-property"]=None
l.append(o)

print(l)

Bonus Section

While scrolling down on the product page you will find information regarding agents, down payment, calculator, etc. This information loads through an AJAX injection.

This cannot be scraped through a normal XHR request. At this point, many of you will think that this information can be scraped easily through a headless browser but the problem is that these headless browsers consume too much CPU resources. Well, let me share the alternate for this.

Redfin renders this data from the API calls it makes from the second last script tag of any property page. Let me explain to you what I mean over here.

The raw HTML you get after making the GET request will have a script tag in which all this data will be stored.

The script tag you see above is the second last script tag of the raw HTML downloaded from the target property page. Here is how using regular expression you can access the data from this tag.

try:
    o["other-details"]=soup.find_all('script')[-2]
except:
    o["other-details"]=None

config_match = re.search(r'reactServerState\.InitialContext\s*=\s*({.*?});', str(o["other-details"]))

if config_match:
    config_data = config_match.group(1)
    print(config_data)

Using regular expression we are finding a string that matches the pattern reactServerState\.InitialContext\s*=\s*({.*?});

Once you run this code you will find all the information inside this string.

How to scrape Redfin at scale?

The above approach is fine until you are scraping a few hundred pages but this approach will fall flat when your scraping demands reach millions. Redfin will start throwing captcha pages like this.

To avoid this situation you have to use a web scraping API like Scrapingdog. This API will handle proxy rotations for you. Proxy rotation will help you maintain the data pipeline.

You can sign up for the free account from here. The free account comes with a generous 1000 credits which is enough for testing the API.

Once you are on the dashboard you will find an API key that will be used in the below code.

For this example, I am again using the Redfin search page.

import requests
from bs4 import BeautifulSoup
import pandas as pd

l=[]
o={}


target_url="https://api.scrapingdog.com/scrape?dynamic=false&api_key=YOUR-API-KEY&url=https://www.redfin.com/city/30749/NY/New-York/filter/status=active"

resp = requests.get(target_url)
print(resp.status_code)
soup=BeautifulSoup(resp.text,'html.parser')

allBoxes = soup.find_all("div",{"class":"HomeCardContainer"})

for box in allBoxes:
    try:
        o["property-price"]=box.find("span",{"class":"homecardV2Price"}).text.strip()
    except:
        o["property-price"]=None

    try:
        o["property-config"]=box.find("div",{"class":"HomeStatsV2"}).text.strip()
    except:
        o["property-config"]=None

    try:
        o["property-address"]=box.find("div",{"class":"homeAddressV2"}).text.strip()
    except:
        o["property-address"]=None

    try:
        o["property-broker"]=box.find("div",{"class":"brokerageDisclaimerV2"}).text.strip()
    except:
        o["property-broker"]=None

    try:
        o["property-link"]="https://www.redfin.com"+box.find("a").get('href')
    except:
        o["property-link"]=None

    l.append(o)
    o={}

print(l)
df = pd.DataFrame(l)
df.to_csv('properties.csv', index=False, encoding='utf-8')

Did you notice something? The code is almost the same as above we just replaced the target URL with the Scrapingdog API URL. Of course, you have to use your personal API key above to run this program successfully.

It is a very economical solution for large-scale scraping. You just have to focus on data collection and the rest will be managed by Scrapingdog.

Conclusion

In this blog, I have scraped two distinct types of pages on Redfin: the search page and the property page. Moreover, I have included a bonus section that sheds light on extracting information that’s dynamically loaded through AJAX injections.

Just like Redfin, I have extracted data from other real estate giants. (find their links below)

1. Scraping Zillow Real Estate Property Data using Python
2. Scraping Idealista.com using Python
3. Web Scraping Realtor Property Data using Python
4. Web Scraping Airbnb Data using Python

If this article resonates with you and you appreciate the effort put into this research, please share it with someone who might be on the lookout for scalable real estate data extraction solutions from property sites.

In the future, I will be making more such articles. If you found this article helpful, please share it. Thanks for reading!

Nodejs offers multiple HTTP client options and Axios is one of them. Axios has by far the best community support compared to other libraries like Fetch, Unirest, etc. It provides a stable API through which you can make XMLHttpRequests. Using a middleware-like function you can customize the pattern of request. It supports GET, POST, DELETE, PUT, etc methods. Axios is built on top of Promises, making it easy to work with asynchronous code and response handling.

Axios is a very powerful tool when it comes to web scraping. In this article, we will understand how we can scrape a website with the combination of proxy and Axios.

Requirements

For this tutorial, I hope you have already installed the latest version of Nodejs on your machine. If not then you can download it from here.

Now, let’s set up our coding playground. First, we will create a folder to keep our Nodejs script.

mkdir axiostut
cd axiostut
npm init

Then we have to install Axios inside this folder.

npm i axios

Once this is done we can create a JS file in which we will learn how Axios works with proxy. I am naming the file as axiostut.js.

How Axios works?

In this section, I will explain step by step how Axios works.

How to make a Basic GET Request

Let’s first start with the basic HTTP GET request. For this entire tutorial, we are going to use this website for scraping.

const axios = require('axios');

async function scraping() {
    try {
        // Make a GET request to https://httpbin.org/ip
        const response = await axios.get('https://httpbin.org/ip');

        // Check if the request was successful (status code 200)
        if (response.status === 200) {
            // Parse JSON response
            const ipAddress = response.data.origin;

            // Log or use the IP address
            console.log('Your IP address is:', ipAddress);
        } else {
            console.error(`Error: ${response.status} - ${response.statusText}`);
        }
    } catch (error) {
        // Handle errors
        console.error('Error:', error.message);
    }
}

// Call the function to get the IP address
scraping();

Let me explain the code in brief.

1. We use the axios.get method to make a GET request to https://httpbin.org/ip.
2. The response is checked for a successful status code 200.
3. If the request is successful, the IP address is extracted from the JSON response and logged.
4. If there’s an error during the request, it is caught and logged.

How to use a proxy with Axios?

For this example, we can use any public proxy. Take any free proxy, from this list.

const axios = require('axios');

let config= {
    method: 'get',
    url: 'https://httpbin.org/ip',
    proxy: {
      host: '20.157.194.61',
      port: 80
    }
  };

async function getIpAddress() {
    try {
        // Make a GET request to https://httpbin.org/ip
        const response = await axios(config);

        // Check if the request was successful (status code 200)
        if (response.status === 200) {
            // Parse JSON response
            const ipAddress = response.data.origin;

            // Log or use the IP address
            console.log('Your IP address is:', ipAddress);
        } else {
            console.error(`Error: ${response.status} - ${response.statusText}`);
        }
    } catch (error) {
        // Handle errors
        console.error('Error:', error.message);
    }
}

// Call the function to get the IP address
getIpAddress();

In the above code, config is an object that contains configuration options for making an HTTP request using the Axios library. Here’s a breakdown of the properties in the config object.

1. method: 'get' -Specifies the HTTP method for the request. In this case, it is set to ‘get,’ indicating an HTTP GET request.
2. url: 'https://httpbin.org/ip' -Specifies the target URL for the HTTP request. The request will be made to the ‘https://httpbin.org/ip‘ endpoint, which returns information about the requester’s IP address.
3. proxy: { host: '20.157.194.61', port: 80 } -Specifies a proxy configuration for the request. The proxy property is an object that includes the host (IP address) and port of the proxy server. This configuration is optional and is used here to demonstrate how to request a proxy.

How to use a password-protected proxy with Axios?

To use the proxy which is protected by a password you can simply pass the username and the password to the config object.

const axios = require('axios');

let config= {
    method: 'get',
    url: 'https://httpbin.org/ip',
    proxy: {
      host: '94.103.159.29',
      port: 8080,
      auth: {
        username: 'Your-Username',
        password: 'Your-Password',
    }
    }
  };

async function getIpAddress() {
    try {
        // Make a GET request to https://httpbin.org/ip
        const response = await axios(config);

        // Check if the request was successful (status code 200)
        if (response.status === 200) {
            // Parse JSON response
            const ipAddress = response.data.origin;

            // Log or use the IP address
            console.log('Your IP address is:', ipAddress);
        } else {
            console.error(`Error: ${response.status} - ${response.statusText}`);
        }
    } catch (error) {
        // Handle errors
        console.error('Error:', error.message);
    }
}

// Call the function to get the IP address
getIpAddress();

We have just passed an auth object with the properties username and password. Once you run this code you will get this output.

Your IP address is: 94.103.159.29

How to rotate proxies with Axios?

Many crawler-sensitive websites like Amazon, Walmart, LinkedIn, etc will block you if you keep scraping these websites with just a single IP. Headers are important too but changing the IPs on every request is as critically important.

const axios = require('axios');

let proxy_arr=[
{host: '69.51.19.191',
port: 8080,
auth: {
  username: 'Your-Username',
  password: 'Your-Password',
}},
{host: '69.51.19.193',
port: 8080,
auth: {
  username: 'Your-Username',
  password: 'Your-Password',
}},
{host: '69.51.19.195',
port: 8080,
auth: {
  username: 'Your-Username',
  password: 'Your-Password',
}},
{host: '69.51.19.207',
port: 8080,
auth: {
  username: 'Your-Username',
  password: 'Your-Password',
}},
{host: '69.51.19.220',
port: 8080,
auth: {
  username: 'Your-Username',
  password: 'Your-Password',
}}]

let config= {
    method: 'get',
    url: 'https://httpbin.org/ip',
    proxy:proxy_arr[Math.floor(Math.random() * 5)]
  };


async function getIpAddress() {
    try {
        // Make a GET request to https://httpbin.org/ip
        const response = await axios(config);

        // Check if the request was successful (status code 200)
        if (response.status === 200) {
            // Parse JSON response
            const ipAddress = response.data.origin;

            // Log or use the IP address
            console.log('Your IP address is:', ipAddress);
        } else {
            console.error(`Error: ${response.status} - ${response.statusText}`);
        }
    } catch (error) {
        // Handle errors
        console.error('Error:', error.message);
    }
}

// Call the function to get the IP address
getIpAddress();

In the above code, I have created a proxy array that contains five proxy objects. Using Math.random function we are passing these proxies on a random basis to the config object.

Now, every request will go through a different proxy and the chances of getting your scraper getting blocked will be very low.

How to use Scrapingdog Proxy with Axios?

For small-scale scraping above methods are fine and will do the job. But if you want to scrape millions of pages then you have to go with some premium web scraping API that can take this proxy management on an autopilot mode. You simply have to send a GET request and the API will handle all these headaches for you.

In this section, I will show you how Scrapingdog proxy can be used for scraping purposes. First of all, you have to sign up for the free pack from here.

The free pack will provide you with a generous 1000 credits and that is enough for you to test the service before proceeding with the paid plan. Once you sign up you will get an API key on your dashboard.

You have to pass this API key in the below code as your proxy password. You can read more about the proxies from the documentation.

const axios = require('axios');



let config= {
    method: 'get',
    url: 'http://httpbin.org/ip',
    proxy:{
    host: 'proxy.scrapingdog.com',
    port: 8081,
    auth: {
      username: 'scrapingdog',
      password: 'Your-API-key'
    }
  }
}


async function getIpAddress() {
    try {
        // Make a GET request to https://httpbin.org/ip
        const response = await axios(config);

        // Check if the request was successful (status code 200)
        if (response.status === 200) {
            // Parse JSON response
            const ipAddress = response.data.origin;

            // Log or use the IP address
            console.log('Your IP address is:', ipAddress);
        } else {
            console.error(`Error: ${response.status} - ${response.statusText}`);
        }
    } catch (error) {
        // Handle errors
        console.error('Error:', error.message);
    }
}

// Call the function to get the IP address
getIpAddress();

Scrapingdog has a proxy pool of more than 10M proxies which makes large-scale scraping seamless. Once you run this code after placing your API key, every run will print a new IP on the console.

It is a very economical solution for large-scale scraping. You just have to focus on data collection and the rest will be managed by Scrapingdog.

Conclusion

Axios is a very popular choice when it comes to web scraping and in this article, we saw how Axios can be used with proxies in different scenarios. We also understood the importance of proxy rotation and how APIs like Scrapingdog can take all the hassle of proxy rotation on autopilot mode.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on social media.

Additional Resources

And there’s the list! At this point, you should feel comfortable writing your first web scraper to gather data from any website. Here are a few additional resources that you may find helpful during your web scraping journey:

• Web Scraping with Nodejs
• Best Javascript web scraping libraries
• Best HTML Javascript parsing libraries
• Node-Unblocker for web scraping

We often discuss proxies in the context of web scraping. We understand the significance of proxy rotation when scraping millions of pages. However, in addition to proxies, headers also play an equally important role in web scraping.

With the help of other headers, User-Agents can help you scrape a tremendous amount of data from the internet. In this article, we will discuss what a user agent is, how it is used for normal/small web scraping projects, and how it can help you with advanced scraping.

What is a User Agent?

If I talk in the context of web scraping then User-Agent is a header that mimics a real browser. This makes a request look more legitimate and influences how the host server responds to the request. It provides information about the client making the request, such as the browser type, version, and sometimes the operating system.

But why are User Agents important?

Well, the User Agent in most cases is the deciding factor for the host server to respond with status code 200(OK) and allow access to the requested resource. A server can send a 4xx error if it identifies the User Agent as suspicious.

What does User Agent look like?

A User Agent looks like this- Mozilla/5.0 (X11; U; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5399.183 Safari/537.36.

Let me break the above string and explain to you what every part means in detail.

1. Mozilla/5.0 – This is a legacy token that most web browsers include in their User-Agent strings for historical reasons. It’s a reference to the original Mosaic web browser and is used to ensure compatibility with websites.
2. (X11; U; Linux x86_64)-This part typically represents the operating system and platform information. In this case, it indicates that the browser is running on a Linux (X11) system using a 64-bit x86 architecture.
3. AppleWebKit/537.36-The AppleWebKit part denotes the layout engine used by the browser. This engine is used to render web pages. Apple’s Safari browser also uses the WebKit engine. The “537.36” number is the version of the WebKit engine.
4. (KHTML, like Gecko)– This is an additional detail to ensure compatibility with some websites. “KHTML” refers to the open-source layout engine used by the Konqueror web browser. “Gecko” is the layout engine used by Mozilla Firefox. This part helps the browser appear compatible with a wider range of web content.
5. Chrome/108.0.5399.183– This part indicates that the browser is Chrome, and “108.0.5399.183” is the version of Google Chrome. This detail allows websites to detect the browser and version, which may be used to optimize content or detect compatibility issues.
6. Safari/537.36– The final part specifies that the browser is compatible with Safari. The “537.36” version number is a reference to WebKit, indicating the version of the engine. Including “Safari” in the User-Agent helps with rendering content designed for Safari browsers.

If you want to break down and test more user agent strings then use this website.

How to use User Agents with Python

Web scraping with Python is the most common way for many new coders to learn web scraping. During this journey, you will come across certain websites that are quite sensitive to scrapers and you might have to pass headers like User Agents. Let’s understand how you can pass a User-Agent with a simple example.

import requests

target_url='https://httpbin.org/headers'
headers={'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361675787112'}
resp = requests.get(target_url,headers=headers)

print(resp.text)

Here I am passing a custom User-Agent to the target URL https://httpbin.org/headers. Once you run this code you will get this as output.

{
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Host": "httpbin.org",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361675787112",
    "X-Amzn-Trace-Id": "Root=1-6548b6c9-4381f1cb1cb6dc915aa1268f"
  }
}

So, this way you can pass a user agent to websites that are sensitive to scraping with Python.

How to avoid getting your scraper banned?

You might be thinking that you can simply avoid this situation by using a rotating proxy and that will certainly solve the problem. But this is not the case with many websites like Google, amazon, etc.

Along with proxy rotation, you have to also focus on header rotation (especially User-Agent). In some cases, you might have to use the latest User-Agents to spoof the request. Let’s see how we can rotate user agents in Python.

User Agent rotation with Python

For this example, I am going to consider these five User Agents.

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361675787110',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5412.99 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5361.172 Safari/537.36',
'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5388.177 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5397.215 Safari/537.36',

We are going to use the random library of Python. This is a legacy library and you don’t have to install it separately. Also, if you need more latest User Agents then visit this link.

import requests
import random


userAgents=['Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.361675787110',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5412.99 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5361.172 Safari/537.36',
'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5388.177 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5397.215 Safari/537.36']


target_url='https://httpbin.org/headers'
headers={'User-Agent':random.choice(userAgents)}

resp = requests.get(target_url,headers=headers)

print(resp.text)

In this code, every request will go through a separate User Agent. Now, you can use this code with a rotating proxy to give more strength to the scraper. Techniques like this will help you scrape Amazon, scrape Google, etc. effectively.

List of Best User-Agents for Web Scraping

Conclusion

Many websites have started using a protective layer that prevents scraping. Therefore passing proper headers has become necessary and in this tutorial, I showed you how with the help of User Agents you can bypass that layer and extract the data.

But of course, for mass scraping this will not be enough and you have to consider using a Web Scraping API. This API will handle all the headers, proxy rotation, and headless chrome for you.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on social media. You can also follow us on Twitter.

Additional Resources

• Tips to Avoid Getting Blocked when Web Scraping
• Challenges in Web Scraping & How To Avoid Them

Every business be it a small organization, a medium-sized team, or an enterprise. Revenue is generated when businesses have customers paying them. 

To reach those customers businesses reach out to potential people who can be interested in the problem you are solving. 

In this blog, we will know how web scraping can be one of the ways to generate leads, what are its advantages, and how different industries can use it. 

What is Lead Scraping?

Lead Scraping is the process of collecting information such as emails, names, contact numbers, etc. using web scraping. The process involves using third-party software to do this very part of web scraping. 

This way businesses can enjoy the freedom to focus on other tasks, while with scraping at the backend they can generate leads at a generous amount each day. The whole process has the advantage that it can be done at scale and the output can be sent to the CRM a company is using. 

Challenges in Lead Scraping

Scraping leads can be a valuable strategy for businesses looking to expand their client base. However, it comes with its own set of challenges. Here are some common challenges faced in lead scraping and how to overcome them.

IP blocking

Lead Scraping is not a simple and straightforward task. You have to crawl websites that are quite sensitive to bots. These websites can easily block your scraper by blocking your IP which can stop your data pipeline.

Complex Website Structure

Websites like LinkedIn are not easy to scrape. They require a completely different infrastructure in order to crawl public profiles. LinkedIn is a lead warehouse and you can find tons of leads for your product over here. A dedicated LinkedIn Scraper can help you bypass the hurdle of maintaining a scraper.

Website Design

Sometimes websites change their design due to which HTML tags also change. Due to this, your scraper can break because the tags you are picking and parsing are now gone.

Methods to scrape Leads

Scraping leads, or extracting contact information from various sources, is a common practice for sales and marketing teams looking to generate leads and grow their customer base. Here are some methods to scrape leads:

No-code Web Scraping

Let’s say I want to scrape emails from yellowpages.com then I can simply use the combination of Airtable, Web Scraping API, and data fetcher to pull fresh data and save it in a CSV file. Here Data Fetcher is an extension that can be downloaded when you sign up for an account on Airtable. Using this extension you can pull data through Web Scraping APIs and then save it to a sheet inside Airtable.

You can even schedule your tasks in Airtable. So, even when you are sleeping your scraper will keep running and it will keep scraping the leads. Isn’t that amazing?

Creating your own Scrapers

If you are a developer then you can build a scraper using Python or Nodejs. Websites like Yelp and Yellowpages can be scraped using these languages. You can crawl and parse the data to extract details like phone number, email, etc.

However, there is a limitation to this method because this method is not scalable. Normal scrapers get blocked easily due to IP blocking. We have discussed this above too in the Challenges section.

We can overcome this problem by using Web Scraping APIs.

Web Scraping API

Web Scraping API will handle all the hassles like the rotation of proxies, passing custom headers, or even headless browsers. This will help you create a constant flow of data through the data pipeline. You just have to send a GET request and you will get the data without worrying about getting blocked. Even websites like LinkedIn can be scraped at a speed of 1 million profiles per day using these services.

Websites where you can find your target prospects

Yelp

Scraping leads from Yelp involves the extraction of business names, addresses, phone numbers, and websites, from Yelp listings. These leads can be potential clients or prospects. You can kickstart your journey by reading web scraping with Yelp.

Google

When you make a search on Google it shows the Title, link, and description. Inside this description, you will sometimes find emails or phone numbers. This data can help you generate leads. You can extract the data from the box of Google Business. You can learn how to scrape Google search results with Python to start scraping leads.

Google Maps

Google Maps can also provide you with email, phone numbers, addresses, etc. Although scraping Google Maps is not an easy task, you should always use scraping APIs to scrape it. You can read this guide on scraping Google Maps with Python.

LinkedIn

Personally, I prefer LinkedIn for generating leads. LinkedIn is like an ocean of leads and you can find your target customers over here. It is used by more than 900M users which means you can extract thousands of leads every day from Linkedin. If you want to scrape LinkedIn person and company profiles then read Scraping LinkedIn Profiles with Python.

Advantages of collecting leads via web scraping

• When you collect leads by scraping websites, the process becomes relatively faster than collecting leads manually.
• You reach out fast and the lead pipeline never goes empty.
• Ultimately customers and revenue go up.

How companies can take advantage of lead generation via web scraping?

Large enterprise companies have a set goal for customer acquisition. Here marketing and sales teams can lower their burden by web scraping websites from where they think their market-fit audience can be found.

Otherwise, they have to manually visit forums, social media, and other websites to collect leads. This becomes a tedious and manually dependent task and can leave the lead pipeline dry for a period of time.

Companies can even save bucks by not using a paid tool like LinkedIn Sales Navigator, Snov, etc. IT team can create a dedicated web scraper or can use a third-party tool like Scrapingdog that can be used by the sales team.

After this sales team can refine leads by cold calling or by either emailing them. If somebody is interested then it can be passed on to the upper management for closing the prospect.

Conclusion

Web scraping can sometimes be a little rough and therefore you should avoid scraping those particular websites. For example, LinkedIn does not allow the crawling of public profiles. Do remember that the quality of the audience is far better than the quantity.

Check Out: These 10 tips to avoid getting blocked while web scraping!!

Our main focus should be on finding leads that can become our customers in the future otherwise there is no point in collecting emails like a robot just for the sake of keeping the lead pipeline open.

Additional Resources

• Best Web Scraping Tools for Generating Leads
• Know How You Can Use Web Scraping for Market Research
• Why You Should Enrich Your Data

You don’t need to know coding to scrape LinkedIn jobs at scale!!

If you’ve found your way to this article, chances are you’re not a developer but eager to learn how to scrape a website and seamlessly store its JSON data into Airtable.

What is Airtable – It’s a digital platform that acts like a supercharged spreadsheet, making it easy to organize and store data.

Quickly now, I will run through the steps you need to follow along. We will be using Scrapingdog’s LinkedIn Jobs API & will save the data in Airtable without coding anywhere in between!!

Requirements

First, you have to sign up on Scrapingdog, you can do that from here. Then you have to sign up on Airtable.

Process of saving the data

Once you sign up on Airtable you will see this on your screen.

You have to click on the “Start from scratch” button. Then a screen will appear where Airtable will ask you to name the project. You can use any name here. I am using Demo-Project.

After that, you can click “Skip setup” and then on Confirm.

This will directly take you to your project page.

Quickly Install this extension to your Chrome browser. This extension is Data Fetcher and it can help you import data from Scrapingdog’s API directly to Airtable. Isn’t that fantastic?

Just click on Add extension and let it launch.

After clicking on Add extension you will be redirected to your project page on Airtable. Now, click on Add extension again.

Now, you will be asked to create an account on Data Fetcher.

Once you sign up this box will open on your screen

Click on “Create your first request” and proceed ahead.

Then click on Custom under the Application tab. It will open a box that might appear similar to POSTMAN or Insomnia.

Now, we can start scraping. I would request you go through the documentation of LinkedIn Jobs API. This will give you an idea about the API. How it works and what data it needs to return the response. Below I am attaching the image from the documentation, wherein you can see what inputs you need to give and what they are.

For this tutorial, we are going to focus on this API.

This API will return a list of jobs based on the parameters you pass. Now, let’s say I want to find jobs in Python for geoid 101473624(you can find it on LinkedIn URL) and I need all the data on the first page. Then in this case the API URL will look like this:

http://api.scrapingdog.com/linkedinjobs?page=1&geoid=101473624&field=python&api_key=Your-API-Key

Once you have this link, you need to place it in the Data Fetcher Box. (please refer image below)

But before that, in “http://api.scrapingdog.com/linkedinjobs?page=1&geoid=101473624&field=python&api_key=Your-API-Key”, see the last part “Your-API-Key”, here you have to paste your own API key.

You can find your API key on the dashboard of Scrapingdog. (please refer image)

Now, paste the API link inside the Data Fetcher box.

Once done click on Save and Run from the bottom right. After that a tab will appear with the name “Response field Mapping”, you need to click Save & Run again here. (please refer screenshot below)

Data Fetcher will request data from scrapingdog’s API

The JSON result from Scrapingdog API has been saved inside the table. You can find this data inside your Airtable project too.

If you want to download this data in a CSV file then click on “Grid view” on the top left and then click “download CSV”.

This will download all of your results in CSV format.

Conclusion

In this journey, we’ve ventured into the world of web scraping and data automation without the need for coding skills. By harnessing the power of Scrapingdog’s LinkedIn Jobs API, we’ve unlocked a treasure trove of JSON data from LinkedIn, and then seamlessly funneled it into Airtable for organized storage and analysis. What seemed like a complex task has now become accessible to non-coders, empowering them to gather valuable insights, monitor job trends, or curate data for their projects.

Of course, you can use any API to store data inside Airtable using the Data Fetcher extension. Data Fetcher works great by creating a bridge between the data collection process and non-coders.

If you want to learn more about this extension then you should refer to their documentation. For more such tutorials keep visiting our Blog section. We will be releasing more such content so keep your heads up.

Additional Resources

• How To Scrape Website with Google Sheets Without Coding

In today’s digital age, online reviews have become an integral part of our decision-making process. Whether we’re searching for a cozy restaurant, a reputable doctor, or a five-star hotel, we often turn to platforms like Google Maps to read user reviews and gauge the quality of services.

For businesses, these reviews are not just feedback but a vital aspect of their online presence. So, what if you could harness the power of Python to extract and analyze these valuable insights from Google Maps? In this article, we’ll explore how to scrape Google Maps reviews using Python, opening up a world of possibilities for businesses, researchers, and data enthusiasts alike.

Scraping Google Maps reviews can offer a wealth of information. You can uncover trends, sentiments, and preferences of customers, providing businesses with actionable insights to enhance their services.

Whether you’re looking to gather competitive intelligence, track your own business’s performance, or conduct market research, Python offers a versatile toolkit to automate the extraction of Google Maps reviews efficiently. Join us on this journey as we delve into the fascinating world of web scraping, data extraction, and analysis to unlock the hidden treasures of Google Maps reviews.

Web scraping Google Maps reviews can be achieved by using Playwright and Beautiful Soup Python libraries. The first is an emerging headless browser and the second is a widely recognized web scraping library that offers extensive documentation.

Playwright and Beautiful Soup: Why choose this team?

Playwright is a library developed by Microsoft, initially intended for JavaScript applications. but it has since been extended to support Python, serving as a good alternative to Selenium when it comes to headless browser automation.

Playwright allows you to control browser behavior testing, web scraping, and other automation tasks. To install Playwright in your virtual environment, you’ll need to run the following commands.

pip install pytest-playwright
playwright install

It can easily be paired with web scraping libraries, such as Beautiful Soup. Which is a well-known library that parses data from HTML and XML files. To install it, you can run the following pip command.

pip install beautifulsoup4

How to automate Google?

To scrape reviews from Google Maps, a set of automation tasks need to be taken beforehand, such as clicks, scrolls, and changing pages. Take a look at the required imports.

import time
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

Now let’s specify three variables, one for the category for which we want reviews, another for the location, and finally the Google main URL.

# the category for which we seek reviews
CATEGORY = "vegan restaurants"


# the location
LOCATION = "Lisbon, Portugal"


# google's main URL
URL = "https://www.google.com/"

All set to start our Playwright instance.

with sync_playwright() as pw:
# creates an instance of the Chromium browser and launches it
      browser = pw.chromium.launch(headless=False)


      # creates a new browser page (tab) within the browser instance
      page = browser.new_page()

Playwright supports both synchronous and asynchronous variations, in this case, we are using synchronous for the sake of better understanding each step, since in this mode, each command is executed one after the other.

In addition, Playwright is compatible with all modern rendering engines: Chromium, Webkit and Firefox. In this case, we’ll be using Chromium which is the most used.

A new instance of the latter browser is created with the headless mode set to False, allowing the user to see the automation live on a GUI (Graphic User Interface). Finally, the new browser page is created, this instance will be responsible for most of the actions.

# go to url with Playwright page element
page.goto(URL)


# deal with cookies
page.click('.QS5gu.sy4vM')


# write what you're looking for
page.fill("textarea", f"{CATEGORY} near {LOCATION}")


# press enter
page.keyboard.press('Enter')


# change to english
page.locator("text='Change to English'").click()
time.sleep(4)


# click in the "Maps" HTML element
page.click('.GKS7s')
time.sleep(4)

Above, we can see several automation actions applied with the page instance. The first task (page.goto(URL)), moves the browser’s tab to the Google main URL. Then, in some cases, Google might display a cookies window, depending on your location or proxy.

In that case, you can use the function .click() on the HTML class (‘.QS5gu.sy4vM’) which owns the button to continue.

At this point, we have reached Google’s main page, and we can write what are we looking for. The variables CATEGORY and LOCATION were introduced before, and they can be used in the .fill() function. Writing is not enough, and that’s why just below we see the .keyboard.press() function to press Enter.

If you’re running the script from a non-English country without a proxy, and you want the reviews in English, you might need to click on some HTML element that changes the language. In this case, this was achieved by using the .locator() function to track the text Change to English and click on it.

The .sleep() functions are important to add loading time just after the actions. Sometimes they take more time than expected and the following steps do not occur, resulting in error.

Finally, we can head to the Google Maps page, by clicking on the respective HTML class (‘.GKS7s’).

The Google Maps page shows the different vegan restaurants in Lisbon. But only a few are presented. To see more we need to start scrolling, and it is an infinite scroll situation, meaning that not all restaurants are loaded at the same time.

# scrolling
for i in range(4):
      # tackle the body element
      html = page.inner_html('body')


      # create beautiful soup element
      soup = BeautifulSoup(html, 'html.parser')


      # select items
      categories = soup.select('.hfpxzc')
      last_category_in_page = categories[-1].get('aria-label')


      # scroll to the last item
      last_category_location = page.locator(
          f"text={last_category_in_page}")
      last_category_location.scroll_into_view_if_needed()


# get links of all categories after scroll
links = [item.get('href') for item in soup.select('.hfpxzc')]

The code snippet shows a loop to scroll the page 4 times. The higher the number, the more restaurants we have.

This is where we start using Beautiful Soup, not to scrape reviews just yet, but to grab a string that is needed to apply scrolling. The html instance contains the HTML information, and the soup element is created to be able to parse it.

Playwright owns other functions to do scrolling such as .mouse.wheel(), but in this case, we have it on the left and another strategy had to be applied by using the function .scroll_into_view_if_needed(). This takes a locator element and scrolls to it. In this case, the element is the last restaurant title available on the page. This triggers the loading of more restaurants. The step is repeated until the desired number of restaurants is reached.

At the very end of the loop, we can obtain all the restaurant URLs (links), by selecting the same HTML element as before (‘.hfpxzc’) and getting the href of each.

See the code below.

for link in links:
# go to subject link
      page.goto(link)
      time.sleep(4)


      # load all reviews
      page.locator("text='Reviews'").first.click()
      time.sleep(4)


      # create new soup
      html = page.inner_html('body')


      # create beautiful soup element
      soup = BeautifulSoup(html, 'html.parser')


      # scrape reviews
      reviews = soup.select('.MyEned')
      reviews = [review.find('span').text for review in reviews]


      # print reviews
      for review in reviews:
      print(review)
            print('\n')

Another loop is needed to extract the reviews from each restaurant. This time we navigate to each link. Then we locate the ‘Reviews’ tab and click on it. We need to make another soup instance, otherwise, we would be reading the HTML information from the previous page.

The first reviews of each restaurant are presented in the ‘.MyEned ‘ class. From here we take the text of all span elements (reviews).

See the output below:

As a tourist I really recommend this place a super nice family business with delicious vegan food and a mix of different 
cultures as well. We had the Brazilian dish (Feijoada) and the mushrooms 🍄 calzone with salad as well and the apple 🍏 ...


A perk in Lisboa, where is a bit hard to find vegan food. This restaurant is managed by very lovely people, the owner is so 
kind and her wife too. Quality is top of the edge, they do not use much spices neither much salt or sugar, but yet ...

Good solid vegan food. Not inventive just very good.  Very nice out of the way location.

Complete Code

Of course, you can scrape more valuable data from the page but for the current scenario, the code will look like this.

import time
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
from rich import print


# the category for which we seek reviews
CATEGORY = "vegan restaurants"
# the location
LOCATION = "Lisbon, Portugal"
# google's main URL
URL = "https://www.google.com/"


if __name__ == '__main__':
    with sync_playwright() as pw:
        # creates an instance of the Chromium browser and launches it
        browser = pw.chromium.launch(headless=False)
        # creates a new browser page (tab) within the browser instance
        page = browser.new_page()
        # go to url with Playwright page element
        page.goto(URL)
        # deal with cookies page
        page.click('.QS5gu.sy4vM')
        # write what you're looking for
        page.fill("textarea", f"{CATEGORY} near {LOCATION}")
        # press enter
        page.keyboard.press('Enter')
        # change to english
        page.locator("text='Change to English'").click()
        time.sleep(4)
        # click in the "Maps" HTML element
        page.click('.GKS7s')
        time.sleep(4)
        # scrolling
        for i in range(2):
            # tackle the body element
            html = page.inner_html('body')
            # create beautiful soup element
            soup = BeautifulSoup(html, 'html.parser')


            # select items
            categories = soup.select('.hfpxzc')
            last_category_in_page = categories[-1].get('aria-label')
            # scroll to the last item
            last_category_location = page.locator(
                f"text={last_category_in_page}")
            last_category_location.scroll_into_view_if_needed()
            # wait to load contents
            time.sleep(4)


        # get links of all categories after scroll
        links = [item.get('href') for item in soup.select('.hfpxzc')]


        for link in links:
            # go to subject link
            page.goto(link)
            time.sleep(4)
            # load all reviews
            page.locator("text='Reviews'").first.click()
            time.sleep(4)
            # create new soup
            html = page.inner_html('body')
            # create beautiful soup element
            soup = BeautifulSoup(html, 'html.parser')
            # scrape reviews
            reviews = soup.select('.MyEned')
            reviews = [review.find('span').text for review in reviews]
            # print reviews
            for review in reviews:
                print(review)
                print('\n')

Once you run the code it will look like this on your screen.

Conclusion

In the age of information, data is power, and Python equips us with the tools to access that power. With the knowledge you’ve gained from this article, you’re now equipped to scrape Google Maps reviews with ease, transforming raw data into actionable insights. Whether you’re a business owner aiming to monitor your online reputation, a researcher seeking to analyze customer sentiments, or simply a Python enthusiast looking for a practical project, the ability to extract and analyze Google Maps reviews is a valuable skill.

You can also use Selenium but to be honest, I was getting bored with Selenium (of course it’s a great library). Playwright brings flexibility and consumes much fewer resources than selenium does.

I hope you like this tutorial and if you do then please do not forget to share it with your friends and on your social media.

Additional Resources

• How To Scrape Google Search Results using Python
• How To Scrape Google Finance using Python
• How To Scrape Website using Google Sheets without Coding

There is a lot of data available on the internet and almost all of that is pretty useful. You can make an analysis based on that data, make better decisions, and even predict changes in the stock market. But there is a gap between this data and your decision-making graphs and that gap can be filled with HTML parsing.

If you want to use this data for your personal or business needs then you have to scrape it and clean it. All this data is not human readable therefore you need a mechanism to clean that raw data and make it human readable. This technique is called HTML parsing. Optimal performance, reliability, scalability, security, and ease of management for Python web apps using HTML parsers like Beautiful Soup, lxml, html5lib, and PyQuery hinge on selecting the best Python hosting provider. A reliable host is crucial for the success of your projects, providing the necessary infrastructure and support.

In this blog, we will talk about the best python html parsing libraries available. Many new coders get confused while choosing a suitable parsing library. Python is supported by a very large community and therefore it comes with multiple options for parsing html.

Here are some common criteria and reasons for selecting specific HTML parsing libraries for this blog.

• Ease of Use and Readability
• Performance and Efficiency
• Error Handling and Robustness
• Community and Support
• Documentation and Learning Resources

Top 4 Python HTML Parsing Libraries

BeautifulSoup

It is the most popular one among all the html parsing libraries. It can help you parse HTML and XML documents with ease. Once you read the documentation you will find it very easy to create parsing trees and extract useful data out of them.

Since it is a third-party package you have to install it using pip in your project environment. You can do it using pip install beautifulsoup4. Let’s understand how we can use it in Python with a small example.

The first step would be to import it into your Python script. Of course, you have to first scrape the data from the target website but for this blog, we are just going to focus on the parsing section. You can refer to web scraping with Python in order to learn more about the web scraping part using the best Python web scraping libraries.

Example

Let’s say we have the following simple HTML document as a string.

<!DOCTYPE html>
<html>
<head>
    <title>Sample HTML Page</title>
</head>
<body>
    <h1>Welcome to BeautifulSoup Example</h1>
    <p>This is a paragraph of text.</p>
    <ul>
        <li>Item 1</li>
        <li>Item 2</li>
        <li>Item 3</li>
    </ul>
</body>
</html>

Here’s a Python code example using BeautifulSoup.

from bs4 import BeautifulSoup

# Sample HTML content
html = """
<!DOCTYPE html>
<html>
<head>
    <title>Sample HTML Page</title>
</head>
<body>
    <h1>Welcome to BeautifulSoup Example</h1>
    <p>This is a paragraph of text.</p>
    <ul>
        <li>Item 1</li>
        <li>Item 2</li>
        <li>Item 3</li>
    </ul>
</body>
</html>
"""

# Create a BeautifulSoup object
soup = BeautifulSoup(html, 'html.parser')

# Accessing Elements
print("Title of the Page:", soup.title.text)  # Access the title element
print("Heading:", soup.h1.text)  # Access the heading element
print("Paragraph Text:", soup.p.text)  # Access the paragraph element's text

# Accessing List Items
ul = soup.ul  # Access the unordered list element
items = ul.find_all('li')  # Find all list items within the ul
print("List Items:")
for item in items:
    print("- " + item.text)

Let me explain the code step by step:

1. We import the BeautifulSoup class from the bs4 library and create an instance of it by passing our HTML content and the parser to use (in this case, 'html.parser').
2. We access specific elements in the HTML using the BeautifulSoup object. For example, we access the title, heading (h1), and paragraph (p) elements using the .text attribute to extract their text content.
3. We access the unordered list (ul) element and then use .find_all('li') to find all list items (li) within it. We iterate through these list items and print their text.

Once you run this code you will get the following output.

Title of the Page: Sample HTML Page
Heading: Welcome to BeautifulSoup Example
Paragraph Text: This is a paragraph of text.
List Items:
- Item 1
- Item 2
- Item 3

You can adapt similar techniques for more complex web scraping and data extraction tasks. If you want to learn more about BeautifulSoup, you should read web scraping with BeautifulSoup.

LXML

LXML is considered to be one of the fastest parsing libraries available. It gets regular updates with the last update released in July of 2023. Using its ElementTree API you can access libxml2 and libxslt toolkits(for parsing HTML & XML) of C language. It has great documentation and community support.

BeautifulSoup also provides support for lxml. You can use it by just mentioning the lxml as your second argument inside your BeautifulSoup constructor.

lxml can parse both HTML and XML documents with high speed and efficiency. It follows standards closely and provides excellent support for XML namespaces, XPath, and CSS selectors.

In my experience, you should always prefer BS4 when dealing with messy HTML and use lxml when you are dealing with XML documents.

Like BeautifulSoup this is a third-party package that needs to be installed before you start using it in your script. You can simply do that by pip install lxml.

Let me explain to you how it can used with a small example.

Example

<bookstore>
  <book>
    <title>Python Programming</title>
    <author>Manthan Koolwal</author>
    <price>36</price>
  </book>
  <book>
    <title>Web Development with Python</title>
    <author>John Smith</author>
    <price>34</price>
  </book>
</bookstore>

Our objective is to extract this text using lxml.

from lxml import etree

# Sample XML content
xml = """
<bookstore>
  <book>
    <title>Python Programming</title>
    <author>Manthan Koolwal</author>
    <price>36</price>
  </book>
  <book>
    <title>Web Development with Python</title>
    <author>John Smith</author>
    <price>34</price>
  </book>
</bookstore>
"""

# Create an ElementTree from the XML
tree = etree.XML(xml)

# Accessing Elements
for book in tree.findall("book"):
    title = book.find("title").text
    author = book.find("author").text
    price = book.find("price").text
    print("Title:", title)
    print("Author:", author)
    print("Price:", price)
    print("---")

Let me explain you above code step by step.

1. We import the etree module from the lxml library and create an instance of it by passing our XML content.
2. We access specific elements in the XML using the find() and findall() methods. For example, we find all <book> elements within the <bookstore> using tree.findall("book").
3. Inside the loop, we access the <title>, <author>, and <price> elements within each <book> element using book.find("element_name").text.

The output will look like this.

Title: Python Programming
Author: Manthan Koolwal
Price: 36
---
Title: Web Development with Python
Author: John Smith
Price: 34
---

If you want to learn more about this library then you should definitely check out our guide Web Scraping with Xpath and Python.

html5lib

HTML5lib is another great contender on this list which works great while parsing the latest HTML5. Of course, you can parse XML as well but mainly it is used for parsing html5.

It can parse documents even when they contain missing or improperly closed tags, making it valuable for web scraping tasks where the quality of HTML varies. html5lib produces a DOM-like tree structure, allowing you to navigate and manipulate the parsed document easily, similar to how you would interact with the Document Object Model (DOM) in a web browser.

Whether you’re working with modern web pages, and HTML5 documents, or need a parsing library capable of handling the latest web standards, html5lib is a reliable choice to consider.

Again this needs to be installed before you start using it. You can simply do it by pip install html5lib. After this step, you can directly import this library inside your Python script.

Example

import html5lib

# Sample HTML5 content
html5 = """
<!DOCTYPE html>
<html>
<head>
    <title>HTML5lib Example</title>
</head>
<body>
    <h1>Welcome to HTML5lib</h1>
    <p>This is a paragraph of text.</p>
    <ul>
        <li>Item 1</li>
        <li>Item 2</li>
        <li>Item 3</li>
    </ul>
</body>
</html>
"""

# Parse the HTML5 document
parser = html5lib.HTMLParser(tree=html5lib.treebuilders.getTreeBuilder("dom"))
tree = parser.parse(html5)

# Accessing Elements
title = tree.find("title").text
heading = tree.find("h1").text
paragraph = tree.find("p").text
list_items = tree.findall("ul/li")

print("Title:", title)
print("Heading:", heading)
print("Paragraph Text:", paragraph)
print("List Items:")
for item in list_items:
    print("- " + item.text)

Explanation of the code:

1. We import the html5lib library, which provides the HTML5 parsing capabilities we need.
2. We define the HTML5 content as a string in the html5 variable.
3. We create an HTML5 parser using html5lib.HTMLParser and specify the tree builder as "dom" to create a Document Object Model (DOM)-like tree structure.
4. We parse the HTML5 document using the created parser, resulting in a parse tree.
5. We access specific elements in the parse tree using the find() and findall() methods. For example, we find the <title>, <h1>, <p>, and <ul> elements and their text content.

Once you run this code you will get this.

Title: HTML5lib Example
Heading: Welcome to HTML5lib
Paragraph Text: This is a paragraph of text.
List Items:
- Item 1
- Item 2
- Item 3

You can refer to its documentation if you want to learn more about this library.

Pyquery

With PyQuery you can use jQuery syntax to parse XML documents. So, if you are already familiar with jQuery then pyquery will be a piece of cake for you. Behind the scenes, it is actually using lxml for parsing and manipulation.

Its application is similar to BeautifulSoup and lxml. With PyQuery, you can easily navigate and manipulate documents, select specific elements, extract text or attribute values, and perform various operations on the parsed content.

This library receives regular updates and has growing community support. PyQuery supports CSS selectors, allowing you to select and manipulate elements in a document using familiar CSS selector expressions.

Example

from pyquery import PyQuery as pq

# Sample HTML content
html = """
<html>
  <head>
    <title>PyQuery Example</title>
  </head>
  <body>
    <h1>Welcome to PyQuery</h1>
    <ul>
      <li>Item 1</li>
      <li>Item 2</li>
      <li>Item 3</li>
    </ul>
  </body>
</html>
"""

# Create a PyQuery object
doc = pq(html)

# Accessing Elements
title = doc("title").text()
heading = doc("h1").text()
list_items = doc("ul li")

print("Title:", title)
print("Heading:", heading)
print("List Items:")
for item in list_items:
    print("- " + pq(item).text())

Understand the above code:

1. We import the PyQuery class from the pyquery library.
2. We define the HTML content as a string in the html variable.
3. We create a PyQuery object doc by passing the HTML content.
4. We use PyQuery’s CSS selector syntax to select specific elements in the document. For example, doc("title") selects the <title> element.
5. We extract text content from selected elements using the text() method.

Once you run this code you will get this.

Title: PyQuery Example
Heading: Welcome to PyQuery
List Items:
- Item 1
- Item 2
- Item 3

Conclusion

I hope things are pretty clear now. You have multiple options for parsing but if you dig deeper you will realize very few options can be used in production. If you want to mass-scrape some websites then Beautifulsoup should be your go-to choice and if you want to parse XML then lxml should your choice.

Of course, the list does not end here there are other options like requests-html, Scrapy, etc. but the community support received by BeautifulSoup and lxml is next level.

You should also try these libraries on a live website. Scrape some websites and use one of these libraries to parse out the data to make your own conclusion. If you want to crawl a complete website then Scrapy is a great choice. We have also explained web crawling in Python, it’s a great tutorial you should read it.

I hope you like this tutorial and if you do then please do not forget to share it with your friends and on your social media.