# gnemoug/distribute\_crawler

<https://repo.masters-masters.com/en/repos/gnemoug/distribute_crawler>

[Home](https://repo.masters-masters.com/en) › [gnemoug](https://repo.masters-masters.com/en/owners/gnemoug) › [distribute\_crawler](https://repo.masters-masters.com/en/repos/gnemoug/distribute_crawler)

gnemoug/distribute\_crawler has 3,235 stars on GitHub. Its RepoMasters card: RC tier. 使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫…

| Field | Value |
| --- | --- |
| Owner | [gnemoug](https://repo.masters-masters.com/en/owners/gnemoug) |
| Stars | 3,235 |
| Forks | 1,552 |
| Language | [Python](https://repo.masters-masters.com/en/languages/python) |
| License | None |
| Created | 2013 |
| Tier | [RC (2,000 to 4,999 ★)](https://repo.masters-masters.com/en/ranks/blockbuster) |
| View on GitHub | [gnemoug/distribute\_crawler](https://github.com/gnemoug/distribute_crawler) |

## Repository

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

The tier for repos with 2,000 to 4,999 ★. Published odds per card pulled: 3%. Rarity: Epic.

## More repos by gnemoug

- [gnemoug/sina\_reptile · ★ 474 · Python · Alpha](https://repo.masters-masters.com/en/repos/gnemoug/sina_reptile)
- [gnemoug/ComPerformance · ★ 387 · JavaScript · Alpha](https://repo.masters-masters.com/en/repos/gnemoug/ComPerformance)

## Nearby in Python

- [thearn/webcam-pulse-detector · ★ 3,235 · Python · RC](https://repo.masters-masters.com/en/repos/thearn/webcam-pulse-detector)
- [google/model\_search · ★ 3,235 · Python · RC](https://repo.masters-masters.com/en/repos/google/model_search)
- [rmurai0610/MASt3R-SLAM · ★ 3,235 · Python · RC](https://repo.masters-masters.com/en/repos/rmurai0610/MASt3R-SLAM)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.