# brendonboshell/supercrawler

<https://repo.masters-masters.com/en/repos/brendonboshell/supercrawler>

[Home](https://repo.masters-masters.com/en) › [brendonboshell](https://repo.masters-masters.com/en/owners/brendonboshell) › [supercrawler](https://repo.masters-masters.com/en/repos/brendonboshell/supercrawler)

brendonboshell/supercrawler has 382 stars on GitHub. Its RepoMasters card: Alpha tier. A web crawler. Supercrawler automatically crawls websites. Define…

| Field | Value |
| --- | --- |
| Owner | [brendonboshell](https://repo.masters-masters.com/en/owners/brendonboshell) |
| Stars | 382 |
| Forks | 62 |
| Language | [JavaScript](https://repo.masters-masters.com/en/languages/javascript) |
| License | Apache-2.0 |
| Created | 2016 |
| Topics | web-crawler, robot, crawler, sitemap, distributed-crawler |
| Tier | [Alpha (250 to 699 ★)](https://repo.masters-masters.com/en/ranks/b_movie) |
| View on GitHub | [brendonboshell/supercrawler](https://github.com/brendonboshell/supercrawler) |

## Repository

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

The tier for repos with 250 to 699 ★. Published odds per card pulled: 20%. Rarity: Uncommon.

## Nearby in JavaScript

- [devswha/patina · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/devswha/patina)
- [lwjjike/xbsReverseSkill · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/lwjjike/xbsReverseSkill)
- [WenXiaoWendy/timeline-for-agent · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/WenXiaoWendy/timeline-for-agent)
- [auxesis/visage · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/auxesis/visage)
- [endor/kettu · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/endor/kettu)
- [TooTallNate/node-time · ★ 382 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/TooTallNate/node-time)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.