Skip to content
#

web-scraping

Star

Here are 5,786 public repositories matching this topic...

Language: All
Filter by language
All 5,786 Python 2,811 Jupyter Notebook 1,244 JavaScript 461 HTML 256 TypeScript 170 R 119 Java 95 Go 88 PHP 60 Ruby 40
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated
changedetection.io

dgtlmoon / changedetection.io

Star 19.1k

The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

notifications monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection changedetection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
  • Updated Nov 10, 2024
  • Python
crawlee

apify / crawlee

Star 15.6k

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
  • Updated Nov 13, 2024
  • TypeScript
Douyin_TikTok_Download_API

Evil0ctal / Douyin_TikTok_Download_API

Sponsor
Star 9.3k

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
  • Updated Sep 26, 2024
  • Python
SeleniumBase

seleniumbase / SeleniumBase

Star 5.4k

📊 Blazing fast Python framework for web crawling, scraping, testing, and reporting. Supports pytest. Stealth options: UC Mode and CDP Mode. Multiple tools and integrations.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin behave bot-detection unittests web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
  • Updated Nov 14, 2024
  • Python

apify / crawlee-python

Star 4.6k

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
  • Updated Nov 14, 2024
  • Python

Improve this page

Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."

Learn more