# JFanZhao/spider

<https://repo.masters-masters.com/en/repos/JFanZhao/spider>

[Home](https://repo.masters-masters.com/en) › [JFanZhao](https://repo.masters-masters.com/en/owners/JFanZhao) › [spider](https://repo.masters-masters.com/en/repos/JFanZhao/spider)

JFanZhao/spider has 233 stars on GitHub. Its RepoMasters card: Prototype tier. 使用java+httpclient+httpcleaner，多线程、分布式爬去电商网站商品信息，数据存储在hbase上，并使用solr对商品建立索引，使用red…

| Field | Value |
| --- | --- |
| Owner | [JFanZhao](https://repo.masters-masters.com/en/owners/JFanZhao) |
| Stars | 233 |
| Forks | 149 |
| Language | [Java](https://repo.masters-masters.com/en/languages/java) |
| License | None |
| Created | 2016 |
| Tier | [Prototype (100 to 249 ★)](https://repo.masters-masters.com/en/ranks/rush) |
| View on GitHub | [JFanZhao/spider](https://github.com/JFanZhao/spider) |

## Repository

使用java+httpclient+httpcleaner，多线程、分布式爬去电商网站商品信息，数据存储在hbase上，并使用solr对商品建立索引，使用redis队列存储一个共享的url仓库；使用zookeeper对爬虫节点生命周期进行监视等。

The tier for repos with 100 to 249 ★. Published odds per card pulled: 70%. Rarity: Common.

## Nearby in Java

- [FPSMasterTeam/FPSMaster-Edge · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/FPSMasterTeam/FPSMaster-Edge)
- [SongJian-99/dj-uav · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/SongJian-99/dj-uav)
- [bkhtmltopdf/bkhtmltopdf · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/bkhtmltopdf/bkhtmltopdf)
- [paul-hammant/paranamer · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/paul-hammant/paranamer)
- [OpenGamma/OG-Platform · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/OpenGamma/OG-Platform)
- [apache/commons-validator · ★ 233 · Java · Prototype](https://repo.masters-masters.com/en/repos/apache/commons-validator)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.