A collection of awesome web crawler,spider in different languages
-
Updated
Nov 3, 2021
#
web-scraper
Here are 548 public repositories matching this topic...
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api
php
http
client
json
framework
curl
xml
proxy
restful
class
http-client
http-proxy
api-client
web-scraper
requests
web-scraping
php-curl
web-service
php-curl-library
-
Updated
Dec 29, 2021 - PHP
Web Scraper in Go, similar to BeautifulSoup
-
Updated
Jan 18, 2022 - Go
A list of practical knowledge-building projects.
javascript
python
processing
c
java
search-engine
music-player
programming
csharp
projects
web-scraper
cpp11
-
Updated
Jun 4, 2021
-
Updated
Jan 19, 2022 - JavaScript
Faster requests on Python 3
python
curl
high-performance
cython
python-library
web-scraper
python3
speed
open-data
http-requests
web-scraping
scrapy
ndjson
python-requests
urllib
download-file
urllib3
faster-than-requests
requests3
requests-toolbelt
-
Updated
Jan 19, 2022 - Nim
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
-
Updated
Jun 23, 2021 - Ruby
Bulk download your favourite anime episodes from your favourite anime websites
ffmpeg
anime
web-scraper
anime-search
anime-fans
anime-downloader
anime-scraper
9anime
hls-downloader
animeultima
animepahe-downloader
animepahe
4anime
monkey-dl
-
Updated
May 30, 2021 - Python
Generate and download e-books from online sources.
-
Updated
Dec 23, 2021 - Python
A framework for creating semi-automatic web content extractors
python
crawler
tutorial
extractor
scraping
web-scraper
selector
css-selector
web-scraping
scrapy
scrapers
beautifulsoup
xpath-expression
lxml
selector-expression
-
Updated
Oct 24, 2020 - Python
A list of scrapers from around the web.
-
Updated
Oct 22, 2021
NBA Stats API via Basketball Reference
-
Updated
Dec 27, 2021 - HTML
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
web-scraper
web-scraping
newsletter
reuters
bloomberg
futures
web-scrapers
scrapper
financial-data
news-websites
data-scraping
news-scraper
futures-historical-data
data-scraper
sraping
python-web-scraper
financial-times
options-data
wall-street-journal
wallstreetbets
-
Updated
Jun 28, 2021 - Python
OnlyFans content downloader
-
Updated
Jan 19, 2022 - Python
A simple browser/client-side web scraper.
-
Updated
Apr 24, 2017 - TypeScript
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
Updated
Oct 21, 2021 - Python
spekulatius
commented
Aug 28, 2020
It would be beneficial to return the URL of the sitemap.xml file directly.
A collection of awesome web scaper, crawler.
-
Updated
Mar 7, 2021
MetaData html scraper and parser for Node.js (supports Promises and callback style)
-
Updated
Aug 15, 2020 - JavaScript
Fetch user's data across social media
python
social-media
pinterest
web-scraper
web-scraping
request
instagram-scraper
twitter-scraper
facebook-scraper
scrapping-python
selenium-python
reddit-scraper
quora-scraper
tiktok-scraper
medium-scraper
pinterest-scrapper
-
Updated
Dec 28, 2021 - Python
Go cascadia package command line CSS selector
tsv
command-line
curl
extract
web-scraper
css-selector
web-scraping
html-source
command-line-tool
csv-table
cascadia
html-text
-
Updated
Jan 2, 2022 - Go
Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!
python
instagram
spam
winning
selenium
macros
web-scraper
python3
cheating
posts
instagram-scraper
comments
mentions
selenium-webdriver
hacktoberfest
giveaways
instagram-bot
selenium-python
instagram-script
hacktoberfest2020
-
Updated
Nov 29, 2021 - Python
PHP Library for detecting CMS
-
Updated
Feb 24, 2020 - PHP
Powerful web scraping framework for Crystal
-
Updated
Jun 21, 2020 - Crystal
Adult XXX Addons (18+) for the Kodi Media Center - Kodi is a registered trademark of the XBMC Foundation. We are not connected to or in any other way affiliated with Kodi - DMCA: [email protected]
-
Updated
Nov 10, 2021 - Python
A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.
-
Updated
Nov 2, 2021 - Python
A command line interface for downloading Bollywood and punjabi songs
python
music
youtube
mp3
python-script
songs
web-scraper
download-songs
music-download
singer
bollywood
tqdm
hollywood
song-downloader
song-download-script
music-download-script
song-pypi
mr-jatt
top-songs
song-download
-
Updated
Jan 9, 2020 - Python
sudiptosarkar
commented
Oct 24, 2017
Yes, I know the class name needs refactor. We need a test class made for this class.
Tests should not make real calls. Instead, calls must be mocked with Mockito and/or PowerMockito.
Examples of the testing pattern can be found in the other three dependency repos mentioned in the README.md.
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
python
scrapy-spider
web-scraper
craigslist
web-scraping
scrapy
web-crawling
scrapy-crawler
scrapy-tutorial
-
Updated
Aug 5, 2017 - Python
Improve this page
Add a description, image, and links to the web-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraper topic, visit your repo's landing page and select "manage topics."
Hello,
Thanks for new update in personal_info section,
I found out that the attribute 'certifications' return empty list []
Test url:
https://www.linkedin.com/in/an-nguyen-9b3248122/Results:
`{'personal_info': {'name': 'An Nguyen',
'headline': 'Data Scientist/Machine Learning Engineer',
'company': 'PERSOL PROCESS & TECHNOLOGY CO., LTD.',
'school': 'National Chiao Tung University',