Async Crawler Documentation

Asynchronous Scraping using Webhooks.

Points to consider while using Asynchronous Scraping

  • You have to save the webhook URL on your dashbaord
  • Webhook URL should support POST request and should respond within 5 seconds with NO CONTENT and status as 200, 204 or 201.
  • You just have two use an additional param other than the Scraping API in Async API that is scraper
  • Using scraper we will identify the Webhook POST URL.
  • You can create 7 webhooks.
  • You can identify the success of the request by sd-status. It will be 200 for every successful request.
  • You can make 5 requests each second.

On Demand Data Delivery.

You provide a webhook endpoint and the scraping data will be delivered there with your total control.


curl "https://api.scrapingdog.com/webhook?api_key=5ea317fee83fsdsd2231583332581a67&url=url&scraper=scraperName"

This will generate a Scraping Id as response as shown in Results.


Results


      {"sid":"8566d6fe-92dc-45a3-b5c2-8910c71bb766"}