alihan/stack-on-a-budget
1
0
Fork 0
mirror of https://github.com/255kb/stack-on-a-budget.git synced 2021-05-13 03:06:20 +03:00
Code Issues Projects Releases Wiki Activity
Files
master
stack-on-a-budget/pages/web-scraping.md
Guillaume 6aff704132 Update web-scraping.md
2021-05-04 08:31:09 +02:00

1.4 KiB
Raw Permalink Blame History

Web Scraping

Apify

Apify Home Page

  • Free tier: 5,000 Free Page Crawls Per Month
  • Pros: Handles proxies, provides simple interface and url classification to make scraping easier
  • Limitations: Rate limited to 1 crawler at a time, data retention of 7 days.

ProxyCrawl

ProxyCrawl Home Page

  • Free tier: 1,000 Free API Calls
  • Pros: Bypass blocks, captchas and allows to scrape javascript pages with a simple API without managing proxies. Supports all websites.
  • Limitations: Rate limited to 20 requests per second, may be increased upon request.

Scraper API

Scraper API Home Page

  • Free tier: 1,000 Free API Calls Per Month
  • Pros: Handles proxies, browsers, and CAPTCHAs for you, so you can scrape any web page with a simple API call.
  • Limitations: Rate limited to 5 requests per second, may be increased upon request.

PhantomJsCloud

PhantomJsCloud Home Page

  • Free tier: About 500 pages/day ($0.05/day in credit)
  • Pros: Browser Automation, PDF Generation, Screenshots, Full request/response details in JSON. Scales to hundreds of parallel requests.
  • Limitations: Free tier blocked from using the Builtin Anonymous Proxy system.