# PuerkitoBio/fetchbot

<https://repo.masters-masters.com/en/repos/PuerkitoBio/fetchbot>

[Home](https://repo.masters-masters.com/en) › [PuerkitoBio](https://repo.masters-masters.com/en/owners/PuerkitoBio) › [fetchbot](https://repo.masters-masters.com/en/repos/PuerkitoBio/fetchbot)

PuerkitoBio/fetchbot has 791 stars on GitHub. Its RepoMasters card: Beta tier. A simple and flexible web crawler that follows the robots.txt policies and…

| Field | Value |
| --- | --- |
| Owner | [PuerkitoBio](https://repo.masters-masters.com/en/owners/PuerkitoBio) |
| Stars | 791 |
| Forks | 92 |
| Language | [Go](https://repo.masters-masters.com/en/languages/go) |
| License | BSD-3-Clause |
| Created | 2014 |
| Topics | crawler, robots-txt |
| Tier | [Beta (700 to 1,999 ★)](https://repo.masters-masters.com/en/ranks/box_office) |
| View on GitHub | [PuerkitoBio/fetchbot](https://github.com/PuerkitoBio/fetchbot) |

## Repository

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

The tier for repos with 700 to 1,999 ★. Published odds per card pulled: 6%. Rarity: Rare.

## More repos by PuerkitoBio

- [PuerkitoBio/goquery · ★ 14,990 · Go · Stable](https://repo.masters-masters.com/en/repos/PuerkitoBio/goquery)
- [PuerkitoBio/gocrawl · ★ 2,052 · Go · RC](https://repo.masters-masters.com/en/repos/PuerkitoBio/gocrawl)
- [PuerkitoBio/purell · ★ 485 · Go · Alpha](https://repo.masters-masters.com/en/repos/PuerkitoBio/purell)
- [PuerkitoBio/rehttp · ★ 223 · Go · Prototype](https://repo.masters-masters.com/en/repos/PuerkitoBio/rehttp)

## Nearby in Go

- [insaaniManav/prompt-forge · ★ 791 · Go · Beta](https://repo.masters-masters.com/en/repos/insaaniManav/prompt-forge)
- [ShahabSL/Skirk · ★ 791 · Go · Beta](https://repo.masters-masters.com/en/repos/ShahabSL/Skirk)
- [MichaelKinsy/PiG · ★ 791 · Go · Beta](https://repo.masters-masters.com/en/repos/MichaelKinsy/PiG)

---
Data: GitHub · Illustration: the project’s public image, pixelated by RepoMasters.