TwitterFacebookPinterestGoogle+

Tag Archives: Scrapy Cluster

Introduction to Scrapy Cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster. The goal is to distribute seed URLs among many waiting spider instances, whose requests are coordinated via Redis. Any other crawls those trigger, as a result of frontier expansion or depth traversal, will also be distributed among all workers in…

Read more

Sections

Shows

Local News

Tools

About Us

Follow Us

Skip to toolbar