We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.
UbiCrawler : a scalable fully distributed web crawler / M. Santini, S. Vigna, P. Boldi, B. Codenotti. - In: SOFTWARE-PRACTICE & EXPERIENCE. - ISSN 0038-0644. - 34:8(2004), pp. 711-726. [10.1002/spe.587]
UbiCrawler : a scalable fully distributed web crawler
M. SantiniPrimo
;S. VignaSecondo
;P. BoldiPenultimo
;
2004
Abstract
We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.
| File | Dimensione | Formato | |
|---|---|---|---|
|
587_ftp.pdf
accesso solo dalla rete interna
Tipologia:
Publisher's version/PDF
Dimensione
184.24 kB
Formato
Adobe PDF
|
184.24 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
