AlphaUncommon
brendonboshell/supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
- Owner
- brendonboshell (personal account)
- Stars
- Stars: 382
- Forks
- Forks: 62
- Language
- JavaScript
- License
- Apache-2.0
- Created
- 2016
- Topics
- web-crawlerrobotcrawlersitemapdistributed-crawler
- Tier
- Alpha (250 to 699 ★) What this tier means
Nearby in JavaScript
All JavaScript repos- burakorkmez/chatifyStars: 382JavaScriptAlpha
- devswha/patinaStars: 382JavaScriptAlpha
- WenXiaoWendy/timeline-for-agentStars: 382JavaScriptAlpha
- auxesis/visageStars: 382JavaScriptAlpha
- endor/kettuStars: 382JavaScriptAlpha
- aichaos/rivescript-jsStars: 382JavaScriptAlpha