238 Open Source WebscrAPIng Software Projects

Create agents that monitor and act on your behalf. Your agents are standing by!

Web Scraper in Go, similar to BeautifulSoup

Creating Scrapy scrapers via the Django admin interface

Take the hassle out of web scraping

Suck the InTernet

a class that uses scraped proxies to make http GET/POST requests (Python requests)

🥫 The simple, fast, and modern web scraping library

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

An R web crawler and scraper

Be nice on the web

Open Source web scraping API. Falkor turns web pages into queryable JSON

LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

📲 Bot to help solve HQ trivia

An extensible API for breaking captchas

Extract price and indicator data from TradingView charts to create ML datasets

An exploration of New York Times crossword answers from 1994-2017, i.e. the Will Shortz era.

Scrapes g4g and creates PDF

Github stargazers information gathering tool

A php crawler that finds emails on the internets

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Code for the second edition Web Scraping with Python book by Packt Publications

:clapper: A Crunchyroll show/season ripper

🗽 A Simple Demonstration of the New York Times App 📱 using Jsoup web crawler with MVVM Architecture 🔥

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Chemical Information from the Web

operating systems three easy pieces by Rezmi

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

A tkinter GUI collating various data

Perceptual image hashing for Node.js

extract videos from youtube in audio format using webscraping techniques :notes:

可视化任务调度系统，精简到一个二进制文件 (Web visual task scheduler system , yes ! just one binary solve all the problems !)

This repository contains data and code used to get and clean data from https://github.com/CSSEGISandData/COVID-19 and https://www.worldometers.info/coronavirus/

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

A TikTokBot that downloads trending tiktok videos and compiles them using FFmpeg

In this tutorial we will build a web scraping program that will scrape a Github user profile and get the Repository names and the Languages for the pinned repositories.

robots.txt file parsing and checking for R

A CSS Selector library in Julia

Web scraper project to collect the useful Hong Kong weather data from HKO website

Plugin based RSS feed generator for sites that don't offer any. Serves RSS, Atom and JSON Feeds.

Nunux Keeper core API

:earth_americas: :hotel: Scrape Booking.com :hotel: :earth_americas:

Web-scraping script that writes the data of all players from FutHead and FutBin to a CSV file or a DB

Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram

Syllabus and example code for 7-week class at NYU/ITP

News extraction and scraping. Article Parsing

App that scrapes the Footlocker website to construct URLs for upcoming sneaker releases and adds the shoe to your cart if it is available. Uses Python and Selenium Webdriver. *Chrome and Chromedriver must be installed and Chromedriver must be on main path

Predict stock price based on financial news feeds

[NOT WORKING] 🤖 CLI Twitter Bot. It's made to reach more engagement based on your interests. See https://github.com/edoardottt/twitterbot2

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

One repo to rule them all !!?!?!! 🤓 😎

State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.

Scrape public FaceBook posts from any group or user into a .csv file without needing to register for any API access

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

Let's you to access your FB account from the command line and returns various things number of unread notifications, messages or friend requests you have.

Documentation and example scripts for SeeShell Automation

The goal of this project/app is to let the user watch anime without ads. It uses Jsoup to extract data from the website and Exoplayer to show videos.

a work-in-progress guide to web scraping as an artistic and critical practice

Perform Google Dork search with Dorkify

data and code for scrapping and cleaning data on covid-19 in India from https://www.mohfw.gov.in/ and https://www.covid19india.org/

Reworked NBA Predictions (in Python)

Change cover photo of all your songs automatically using python

Scrap Medium Articles using tags.

:books: A GoodReads.com Scraper script to get books reviews including text and rating.

This repo contains regression and classification projects. Examples: development of predictive models for comments on social media websites; building classifiers to predict outcomes in sports competitions; churn analysis; prediction of clicks on online ads; analysis of the opioids crisis and an analysis of retail store expansion strategies using Lasso and Ridge regressions.

THE TELEGRAM BOT FOR COMPETITIVE PROGRAMMERS

Scrape a Tumblr blog for personal posts

[really old and probably doesn't work] Slack bot to post relevant Toronto apartment listings from Kijiji & Craigslist

Using Selenium, Neha scraped data about 35 top selling sneakers of Nike and Adidas from stockx.com. She used this data to draw insights about sneaker resales.

Blog do Brasil.IO

Simple datasets and notebooks for data visualization, statistical analysis and modelling - with write-ups here: http://projectosyo.wix.com/datadoubleconfirm.

Animeflv is a custom API that has the entire catalog of the animeflv.net website. You can enjoy all the content with subtitles in Spanish and the latest in the world of anime for free.

Web Scraping Framework

Covid19 Data API (JSON) - LIVE

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

Sometimes we need to scrap web data from our Android App. To achieve this goal jsoup library is a good option. I wrote a blog post on this topic in my personal blog. If you know Bengali language then you can visit this link.

Get those fastest finger deals with python [Price tracker with email & sms alerts]

🗽 NY Times is an Minimal News 🗞 iOS app 📱 built to describe the use of SwiftSoup and CoreData with SwiftUI🔥

An Artificial Intelligence Chat Bot and Service Provider written in Python and AIML.

Python implementation of web scraping of TripAdvisor with Selenium in a new 2019 website

[partially working] Scrape and add anime episode stream URLs to uGet (Linux) or IDM (Windows) ~ Python3

A web app that helps Kickstarter creators increase their chances of being funded

Android Web Scraper is a simple library for android web automation. You can perform web task in background to fetch website data programmatically.

web scraping in parallel with Selenium Grid and Docker

A python web scraper for public email lists.

Browser automation API for repetitive web-based tasks, with a friendly user interface. You can use it to scrape content or do many other things like capture a screenshot, generate pdf, extract content or execute custom Puppeteer, Playwright functions.

Download all the ebooks with indexed csv of "allitebooks.com"

Fast and robust date extraction from web pages, with Python or on the command-line

Introduction to web scraping

Burger King - Free Burger Code Generator

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

:package: R package to easily web scrape Glassdoor company reviews. Write up of demo:

A tutorial to setup and deploy a simple Serverless Python workflow with REST API endpoints in AWS Lambda.

A library to get images from social media

A lightweight Python library (and command-line utility) for downloading anime.

GUI app for downloading whole online courses with folder structure from one url

This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.

A python script to get class schedules on UFLA's SIG and convert to a .CSV file to use in Google Calendar

Pawn package manager for SA-MP