RW Spider: Ruby Web Spider

What is it?

RW Spider is an multithreading spider client written in Ruby. The library was designed to make easy the development of programs that spider the web.
RW Spider design comes from the direct experiences of the development of another PHP library that is currently used as the engine for an freeware SEO tool.

Features

Multithreading spider
Customizable options for the spider job
Robots.txt support
Indexing of web pages and others files (images, CSS, JavaScript, PDF and more)
Following redirects

How to install RW Spider?

This library is intended to be installed via the Gemcutter system.

$ gem install rwspider

You might need administrator privileges on your system to install it.

Code example

require 'rwspider'

# set the option values
opts = {
   :useragent => 'My user agent',
   :robot_name => 'my_spider_name',
   :scan_documents_limit => 100,
   :scan_domain_limit => nil,
   :scan_images => true,
   :scan_other_files => false,
   :follow_robotstxt_directive => true,
   :follow_HTTP_redirection => true,
   :timeout => 5
 }
# Run the Rwspider::start method to start the spider
 Rwspider.start('http://www.rwspider.com', opts) do |d|
    puts 'Current URL ' + d.url.normalize.to_s
 end

Requirements

Ruby >= 1.8.7
Hpricot >= 0.8.2
Robotstxt >= 0.5.2 (Robotstxt on Gemcutter)

Who wrote it?

RW Spider is written by Simone Rinzivillo.
RW Spider is free to use under the terms of the MIT License.

RW Spider