List of Web Scraping Libraries we will go through

1. request-promise-native
2. Unirest
3. Cheerio
4. Puppeteer
5. Osmosis

Request-Promise-Native

const request = require(‘request-promise-native’);let scrape = async() => {
 var respo = await request(‘http://books.toscrape.com/')
 return respo;
}scrape().then((value) => {
 console.log(value); // HTML code of the website
});

1. It provides proxy support
2. Custom headers
3. HTTP Authentication
4. Support TLS/SSL Protocol

Unirest

1. GET request

var unirest = require('unirest');let scrape = async() => {
 var respo = await unirest.get(‘http://books.toscrape.com/')
 return respo.body;
}scrape().then((value) => {
 console.log(value); // Success!
});

var unirest = require(‘unirest’);let scrape = async() => {
 var respo = await unirest.post(‘http://httpbin.org/anything').headers({'X-header': ‘123’})
 return respo.body;
}scrape().then((value) => {
 console.log(value); // Success!
});

{
 args: {},
 data: ‘’,
 files: {},
 form: {},
 headers: {
 ‘Content-Length’: ‘0’,
 Host: ‘httpbin.org’,
 ‘X-Amzn-Trace-Id’: ‘Root=1–5ed62f2e-554cdc40bbc0b226c749b072’,
 ‘X-Header’: ‘123’
 },
 json: null,
 method: ‘POST’,
 origin: ‘23.238.134.113’,
 url: ‘http://httpbin.org/anything'
}

var unirest = require(‘unirest’);let scrape = async() => {
 var respo = await unirest.put(‘http://httpbin.org/anything').headers({'X-header': ‘123’})
 return respo.body;
}scrape().then((value) => {
 console.log(value); // Success!
});

{
 args: {},
 data: ‘’,
 files: {},
 form: {},
 headers: {
 ‘Content-Length’: ‘0’,
 Host: ‘httpbin.org’,
 ‘X-Amzn-Trace-Id’: ‘Root=1–5ed62f91-bb2b684e39bbfbb3f36d4b6e’,
 ‘X-Header’: ‘123’
 },
 json: null,
 method: ‘PUT’,
 origin: ‘23.63.69.65’,
 url: ‘http://httpbin.org/anything'
}

1. support all HTTP Methods (GET,POST,DELETE,etc.)
2. support forms uploads
3. supports both streaming and callback interfaces
4. HTTP Authentication
5. Proxy Support
6. Support TLS/SSL Protocol

Cheerio

const cheerio = require(‘cheerio’)let scrape = async() => {
 var respo = await request(‘http://books.toscrape.com/')
 return respo;
}scrape().then((value) => {const $ = cheerio.load(value)
 var numberofbooks = $(‘ol[class=”row”]’).find(‘li’).length
 console.log(numberofbooks); // 20!
});

• Familiar syntax: Cheerio implements a subset of core jQuery. It removes all the DOM inconsistencies and browser cruft from the jQuery library, revealing its truly gorgeous API.
• Lightening Quick: Cheerio works with a very simple, consistent DOM model. As a result, parsing, manipulating, and rendering are incredibly efficient. Preliminary end-to-end benchmarks suggest that cheerio is about 8x faster than JSDOM.
• Stunningly flexible: Cheerio can parse nearly any HTML or XML document.

Puppeteer

• Puppeteer is a Node.js library that offers a simple but efficient API that enables you to control Google’s Chrome or Chromium browser.
• It also enables you to run Chromium in headless mode (useful for running browsers in servers) and can send and receive requests without the need for a user interface.
• It has better control over the Chrome browser as it does not use any external adaptor to control Chrome plus it has Google support too.
• The great thing is that it works in the background, performing actions as instructed by the API.

let scrape = async () => {
 const browser = await puppeteer.launch({headless: true});
 const page = await browser.newPage(); await page.goto(‘http://books.toscrape.com/'); await page.waitFor(1000); var result = await page.content(); browser.close();
 return result;
};scrape().then((value) => {
 console.log(value); // complete HTML code of the target url!
});

1. This will launch a chrome browser.
2. Second-line will open a new tab.
3. The third line will open that target URL.
4. We are waiting for 1 second to let the page load completely.
5. We are extracting all the HTML content of that website.
6. We are closing the Chrome browser.
7. returning the results.

• Click elements such as buttons, links, and images
• Automate form submissions
• Navigate pages
• Take a timeline trace to find out where the issues are in a website
• Carry out automated testing for user interfaces and various front-end apps, directly in a browser
• Take screenshots
• Convert web pages to pdf files

Osmosis

• Osmosis is HTML/XML parser and web scraper.
• It is written in node.js which packed with css3/xpath selector and lightweight HTTP wrapper
• No large dependencies like Cheerio

osmosis(‘https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population').set({ heading: ‘h1’, title: ‘title’}).data(item => console.log(item));

{ heading: ‘List of U.S. states and territories by population’, title: ‘List of U.S. states and territories by population — Wikipedia’ }

• Supports CSS 3.0 and XPath 1.0 selector hybrids
• Load and search AJAX content
• Logs URLs, redirects, and errors
• Cookie jar and custom cookies/headers/user agent
• Login/form submission, session cookies, and basic auth
• Single proxy or multiple proxies and handles proxy failure
• Retries and redirect limits

Conclusion

Additional Resources

• List of free web scraping tools
• The 10 Best web scraping proxy services
• Web Scraping with Python
• Guide to web scraping