< Back to Blog Overview

Scraping and Crawling Yelp reviews

2020-08-27
Scraping and Crawling Yelp reviews

In this post, we are going to learn web scraping with python. Using python we are going to scrape Yelp reviews. This is a great source for public reviews. We will code a scraper for that. Using that scraper you would be able to scrape reviews of any company from yelp. To make things simple we will use web scraping API.

Why this API? This tool will help us to scrape dynamic websites using millions of rotating proxies so that we don’t get blocked. It also provides a captcha clearing facility. It uses headerless chrome to scrape dynamic websites.

Requirements

Generally, web scraping is divided into two parts:

  1. Fetching data by making an HTTP request
  2. Extracting important data by parsing the HTML DOM

Libraries & Tools

  1. Beautiful Soup is a Python library for pulling data out of HTML and XML files.
  2. Requests allow you to send HTTP requests very easily.
  3. web scraping API extract the HTML code of the target URL.

Setup

Our setup is pretty simple. Just create a folder and install Beautiful Soup & requests. For creating a folder and installing libraries type below given commands. I am assuming that you have already installed Python 3.x.

mkdir scraper
pip install beautifulsoup4
pip install requests

Now, create a file inside that folder by any name you like. I am using scraping.py.

Firstly, you have to sign up for the scrapingdog API. It will provide you with 1000 FREE credits. Then just import Beautiful Soup & requests in your file. like this.

from bs4 import BeautifulSoup
import requests

What we are going to scrape

We are going to scrape public reviews for this restaurant.

  1. Name of the person
  2. Location of the person
  3. stars
  4. review

Preparing the Food

Now, since we have all the ingredients to prepare the scraper, we should make a GET request to the target URL to get the raw HTML data. If you are not familiar with the scraping tool, I would urge you to go through its documentation. Now we will scrape Yelp for review data using requests library as shown below.

r = requests.get('https://api.scrapingdog.com/scrape?api_key=5ea541dcacf1b0b4b4042&url=https://www.yelp.com/biz/sushi-yasaka-new-york').text

this will provide you with an HTML code of that target URL.

Now, you have to use BeautifulSoup to parse HTML.

soup = BeautifulSoup(r,’html.parser’)

Now, all the reviews are in the form of a list. We have to find all those lists.

Image for post
allrev = soup.find_all(“li”,{“class”:”lemon — li__373c0__1r9wz margin-b3__373c0__q1DuY padding-b3__373c0__342DA border — bottom__373c0__3qNtD border-color — default__373c0__3-ifU”})

We will run for a loop to reach each and every reviewer. To extract name, place, stars, and reviews we have to first find the tags where this data is stored. For example “Name” is stored in “lemon — a__373c0__IEZFH link__373c0__1G70M link-color — inherit__373c0__3dzpk link-size — inherit__373c0__1VFlE”. Just like this using chrome developer tools you can find the rest of the tags.

Image for post
for i in range(0,len(allrev)): try:
l["name"]=allrev[i].find("a",{"class":"lemon--a__373c0__IEZFH link__373c0__1G70M link-color--inherit__373c0__3dzpk link-size--inherit__373c0__1VFlE"}).text
except:
l["name"]=None
try:
l["place"]=allrev[i].find("span",{"class":"lemon--span__373c0__3997G text__373c0__2Kxyz text-color--normal__373c0__3xep9 text-align--left__373c0__2XGa- text-weight--bold__373c0__1elNz text-size--small__373c0__3NVWO"}).text
except:
l["place"]=None
try:
l["stars"]=allrev[i].find("div",{"class":"lemon--div__373c0__1mboc i-stars__373c0__1T6rz i-stars--regular-5__373c0__N5JxY border-color--default__373c0__3-ifU overflow--hidden__373c0__2y4YK"}).get('aria-label')
except:
l["stars"]=None
try:
l["review"]=allrev[i].find("span",{"class":"lemon--span__373c0__3997G raw__373c0__3rKqk"}).text
except:
l["review"]=None
u.append(l)
l={}
print({"data":u})

The output of the above code will be

{
“data”: [
{
“review”: “If you’re looking for great sushi on Manhattan’s upper west side, head over to Sushi Yakasa ! Best sushi lunch specials, especially for sashimi. I ordered the Miyabi — it included a fresh oyster ! The oyster was delicious, served raw on the half shell. The sashimi was delicious too. The portion size was very good for the area, which tends to be a pricey neighborhood. The restaurant is located on a busy street (west 72nd) & it was packed when I dropped by around lunchtimeStill, they handled my order with ease & had it ready quickly. Streamlined service & highly professional. It’s a popular sushi place for a reason. Every piece of sashimi was perfect. The salmon avocado roll was delicious too. Very high quality for the price. Highly recommend! Update — I’ve ordered from Sushi Yasaka a few times since the pandemic & it’s just as good as it was before. Fresh, and they always get my order correct. I like their takeout system — you can order over the phone (no app required) & they text you when it’s ready. Home delivery is also available & very reliable. One of my favorite restaurants- I’m so glad they’re still in business !”,
“name”: “Marie S.”,
“stars”: “5 star rating”,
“place”: “New York, NY”
},
{
“review”: “My friends recommended for me to try this place for take out as I was around the area. I ordered the Miyabi, all the sushi and sashimi was very fresh and tasty. They also gave an oyster which was a bonus! The price is great for the quality and amount of fish. I was happily full.”,
“name”: “Lydia C.”,
“stars”: “5 star rating”,
“place”: “Brooklyn, Brooklyn, NY”
},
{
“review”: “Best sushi on UWS and their delivery is quicker than any I’ve seen! I ordered their 3 roll lunch special around 1:40pm and by 2, I was thoroughly enjoying my sushi! Granted, I live only a few blocks away but I was BLOWN away by the quick services. I had, spicy yellowtail, jalapeño yellowtail and tuna avocado roll. Great quality of fish for such a reasonable price. $16 for 3 rolls. This has certainly come by go-to place for amazing, fresh sushi on UWS.”,
“name”: “Ella D.”,
“stars”: “5 star rating”,
“place”: “Manhattan, New York, NY”
},
{
“review”: “One of my favorite sushi places in the city. They have high-quality fish for affordable prices, something you don’t often see in the big apple. I’ve ordered in from here a handful of times and dined in only once. Both modes are equally as delicious. I actually dined in here alone for lunch on my day off and had a great experience. The waiter was friendly and the sushi lunch special (sushi + one roll + oyster) was fabulous. Fresh, affordable, and delicious. Can’t wait to return!”,
“name”: “William S.”,
“stars”: “5 star rating”,
“place”: “New Orleans, LA”
},
{
“review”: “I’d bookmarked this place for omakase a while back. I haven’t had it there yet, but I’d dined in and gotten takeout a couple times. Really fresh sushi. Everything is on point. I would absolutely do omakase here with the quality of fish and freshness. Service was pretty good to. I wish they’d deliver to my apt lol. But I’ll have to wait to be in the area to get my sushi fix there.”,
“name”: “Esther T.”,
“stars”: null,
“place”: “New York, NY”
},
{
“review”: “Sushi Yasaka is the only sushi in town (upper west side) in my mind!!! Very reasonable prices and great quality!! We order from them once a week for the past 5 years…. never disappoint!”,
“name”: “Kaori K.”,
“stars”: “5 star rating”,
“place”: “New York, NY”
},
{
“review”: “This is our go-to sushi delivery place. Very good fish and prompt service. I’ve been ordering from Yasaka for several years and it’s been consistently good. The one time I ate lunch in the restaurant it was hectic and noisy, but it’s great for delivery.”,
“name”: “Sam H.”,
“stars”: “5 star rating”,
“place”: “New York, NY”
},
{
“review”: “BEWARE!!!!! This restaurant gave out my personal information including my phone number to a third party that called the restaurant inquiring about the order that I placed. Total violation of privacy to disclose a customer’s personal information!”,
“name”: “Gina G.”,
“stars”: null,
“place”: “Kahului, HI”
},
{
“review”: “This place was on a top New York City Japanese list and also had strong Yelp review average. The place is underground — has that minimal Japanese feeling and is also cozy and dark. Bento boxes for $16 at lunch is a good deal in Manhattan. Especially for a clean and cozy spot, making this a score in value — given its atmosphere and food quality. The sushi chefs yelled at me when I walked in. I felt kinda shogun warrior. The place was empty at 12 o’clock which I thought was a weird time to open, but the place filled up fast. On the bento I chose mixed tempura for my main dish with brown rice and a spicy tuna roll. It came with a shrimp shumai dumpling and I chose a salad instead of miso soup. Everything was very good. The shrimp shumai were large with a tasty sauce. The spring roll was tasty and not greasy. The tempura also had a light batter, crunchy yet somehow melt in your mouth, with 2 shrimps, an onion, an acorn squash as well as a broccoli floret. The food was so good and reasonably priced. Lots of young bankers and real estate agents surrounded me. Service was swift and kind. This place is better than average in a cozy atmosphere with great prices.”,
“name”: “Scott L.”,
“stars”: null,
“place”: “Manhattan, NY”
},
{
“review”: “My boyfriend and I came for Valentine’s Day and each had the regular omakase, which comes with 12 pieces of nigiri and one roll, at $55 each. While the fish was fresh and satisfactorily tasty, we probably could have gotten something of better value for our experience. Because the bar was fully occupied, we didn’t have the full omakase experience. They put the pieces all on one plate — we probably have been better off ordering a set instead. I really wish that the server had recommended something else for us. Food was a 4, but experience a 3.”,
“name”: “Regina L.”,
“stars”: null,
“place”: “New York, NY”
},
{
“review”: “Its been a while since I had some good sushi so we decided to have a to-go order on a very hot summer day. Sushi delux and a spicy maki combo were packed nicely with a small ice pack. Everything wes fresh and delicious!! Love this place so much for being consistent with their quality and freshness every time we go. Highly recommended”,
“name”: “Sarah C.”,
“stars”: “5 star rating”,
“place”: “New York, NY”
},
{
“review”: “If you want high quality sushi and don’t want to break bank, go to Sushi Yasaka — small, cozy place located at Upper West Side. I came around 1:30 pm and it was packed! I’d recommend making reservations online. Here’s what we ordered: -Sushi Lunch ($16) which comes with 6 pc of sushi and one roll choice, choice of soup or salad -Chirashi Lunch ($20) which comes with 14 pc of sashimi over rice, choice of soup or salad -Two chu-toro sushi ($7/piece) Their lunch special deals are fantastic and affordable. We spent around $55 for all this! I also loved the food presentation as it was brought out on a wooden board. Service was amazing too; our host was very professional and friendly. Sushi was very fresh and for the price and quality, Sushi Yasaka is incredible! We’re definitely going to be regulars here.”,
“name”: “Judy L.”,
“stars”: “5 star rating”,
“place”: “Manhattan, New York, NY”
},
{
“review”: “I can’t eat sushi anymore without comparing it to Sushi Yasaka. The best sushi I have yet to try. Do yourself a favor and hop on over.What makes it great?No attitude or pretense — service is kind and swift. Though many of the guests are Upper West side resident and bankers, it doesn’t feel high brow. You can dress casual. Preparation — the soy sauce and/or wasabi is preapplied to the fish (when it ought). The temperature, acidity, texture of the rice are on point. The rice is evenly molded and consistent. Not dense. Fish is properly sized and proportionate to the rice. Fish is the right texture and brightness — no fishy smell. Variety — Lots of different options to choose from. There are various cuts of the same fish. Lean, medium, and fatty cuts of tuna are available, offering different flavors and textures. There are dessert options for an \”Americanized palate\” such as red bean and green tea fried ice cream. Great selection of wine and Japanese beer to complement your meal. High quality appetizers — Though the sushi is the star of the meal, the appetizers are a close second. Highly recommend the agedashi tofu.”,
“name”: “Penelope H.”,
“stars”: “5 star rating”,
“place”: “Hoboken, NJ”
},
{
“review”: “Love coming back here. It’s always packed so it’s best to come with a reservation.Instead of the omakase, we got the sushi deluxe (9 pc with roll of choice), sushi lunch, and miyabi lunch. The fish is always fresh. The eel avocado roll is some of the best I’ve had. The appetizers also come out hot and fresh (we got the age dashi tofu and the mixed tempura).This is one of the few places I’ve come back to multiple times in the city.”,
“name”: “Rebecca C.”,
“stars”: “5 star rating”,
“place”: “New York, NY”
},
{
“review”: “We came here for an early dinner with a reservation. It got pretty packed quickly so it’d be safe to make a reservation before you go. It’s a pretty small place so probably not great for large groups.For the appetizer, we got the squid. It was really good — tasted fresh and yummy sauce. I’d highly recommend you try it if you like squid.We got a mix of maki rolls (salmon, shrimp tempura, and eel) and sashimi. Everything tasted really fresh and everything is what’d you expect from a classic sushi place. Everything was decently priced too which was a bonus.I’d definitely go again if I’m craving sushi and in the area!”,
“name”: “Chengru L.”,
“stars”: null,
“place”: “Boston, MA”
},
{
“review”: “BEST. OMAKASE. SUSHI. IN. NYC. I live deep in Brooklyn and aside from Levain and the women’s march, there’s pretty limited reason for me to go to the Upper West Side. Sushi Yasaka is that reason. I’ve made the hour long commute twice and both have been exceptionally worth it. Every piece here is delectable, fresh, largely portioned, and deliciously made. The pieces range from toro, salmon, tuna, yellowtail, uni, scallop, snapper, eel, and more, and all of the highest grade and quality. The other thing that makes this place special is the price. It’s only $55 for the 12 piece omakase, but tastes like the sushi you’d get for $200. This is 100% my place of choice to wow out of towners when it comes to sushi .”,
“name”: “Lindsey O.”,
“stars”: “5 star rating”,
“place”: “Brooklyn, NY”
},
{
“review”: “Sushi Yasaka is a fun spot for decently priced sushi!I dined here on a Saturday afternoon w/ a party of 2 and no reservation. The restaurant was full, but we were able to sit at the bar. We ordered our lunch specials (a good deal for the amount of food) and had soup and salad immediately.Our meals took a little longer (maybe 20–30 min) but were worth the wait. We had a tempura bento box and a sushi roll platter. All were delicious! Servers were friendly and very aware of timing. Try this spot!”,
“name”: “Caitlin O.”,
“stars”: null,
“place”: “Coopersburg, PA”
},
{
“review”: “One of the best omakase deals in the city. We were able to snag bar seats at 5:30 PM on a Saturday by calling ahead the day of (although I recommend making a reservation online ahead of time). Each piece of sushi was delectable, and the sushi chefs weren’t overbearing. I especially enjoyed the toro sushi! We also ordered a cold bottle of sake to complement our sushi, and it was relatively affordable but delicious. Only docking down a star because I’m not DREAMING about the sushi or dying to go back as soon as possible. The non-omakase menu did look delicious and beautiful though, so perhaps I’ll try it out again at some point!”,
“name”: “Grace L.”,
“stars”: null,
“place”: “Boston, MA”
},
{
“review”: “My favorite sushi in NYC so far! We fell in love with their lunch section and have already been there twice in the same week! Their Miyabi lunch has an impressive selection of items and includes salad or soup as appetizer, rolls, and sashimi including tuna, salmon and yellowtail. A random musing but their sushi rice is my personal favorite out of all the sushi places we have tried around the country Everything was very fresh and For the price and quality this spot was incredibly tasty and affordable. Aside from the above They also have other lunch combinations with sashimi or sushi rolls only. We have brought our friends here and recommend it highly!”,
“name”: “Linda Y.”,
“stars”: “5 star rating”,
“place”: “Brooklyn, NY”
},
{
“review”: “Solid. Like your mom. Food: 4$ to portion: 3Quality: 4 Service: 3 Ambiance: busy & loud Wait time: insane BB level: 3 Date spot: 3 Han Solo: 3 This just might be the best bargain in the city for the quality of fish vs what they charge. Never had a bad meal, yet. Maybe that’s why they’re always packed! I wouldn’t say it’s a destination, but if you’re in the area, it won’t disappoint for that random Wednesday night post work pedestrian sushi night. Get reservations. You’re welcome.”,
“name”: “Your Mom’s Y.”,
“stars”: null,
“place”: “Manhattan, NY”
}
]
}

There you go! We have the yelp reviews ready to manipulate and maybe store somewhere like in MongoDB. But that is out of the scope of this tutorial.

Remember that if you aren’t using Python but other programming languages like Ruby, Nodejs, or PHP. You can easily find HTML parsing libraries to parse the results from Scrapingdog API.

We hope you enjoyed this tutorial and we hope to see you soon in Scrapingdog. Happy Scraping!

Conclusion

In this article, we understood how we can scrape data using the data scraping tool & BeautifulSoup regardless of the type of website.

Feel free to comment and ask me anything. You can follow me on Twitter and Medium. Thanks for reading and please hit the like button! 👍

Additional Resources

And there’s the list! At this point, you should feel comfortable writing your first web scraper to gather data from any website. Here are a few additional resources that you may find helpful during your web scraping journey:

Scrapingdog Logo

Try Scrapingdog for Free!

Free 1000 API calls of testing.

No credit card required!