nz.ac.waikato.mcennis.rat.crawler
Class WebCrawler.Spider

java.lang.Object
  nz.ac.waikato.mcennis.rat.crawler.CrawlerBase
      nz.ac.waikato.mcennis.rat.crawler.WebCrawler.Spider

All Implemented Interfaces:

java.lang.Runnable, Crawler

Enclosing class:

WebCrawler

public class WebCrawler.Spider
extends CrawlerBase
implements java.lang.Runnable

Helper class that is used for threads. Crawls sites in order. Each crawler recieves an equal number f sites to crawl, not based on curent load.

Field Summary

Fields inherited from class nz.ac.waikato.mcennis.rat.crawler.CrawlerBase

parser

Constructor Summary

WebCrawler.Spider(WebCrawler p)
Base constructor that stores a reference to the parent in each thread

Method Summary

protected void	add(SiteReference site) add a new site to be crawled by this thread
protected void	doParse(java.io.InputStream raw_data, Properties parsers) Helper function separated from public parse to allow easy overloading.
protected boolean	isEmpty() is this thread idle and waiting for more sites to crawl
boolean	isRunning() return whether or not threads are actively crawling the Internet
void	run() starts the thread executing, parsing web sites in its queue until it recieves a stop request.

Methods inherited from class nz.ac.waikato.mcennis.rat.crawler.CrawlerBase

block, block, crawl, crawl, getCrawler, getFilter, getParsers, getProperties, getProxyHost, getProxyPort, getProxyType, isCaching, isSpidering, set, setCaching, setCrawler, setFilter, setParsers, setProxy, setProxyHost, setProxyPort, setProxyType, setSpidering

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebCrawler.Spider

public WebCrawler.Spider(WebCrawler p)

Base constructor that stores a reference to the parent in each thread

Parameters:

p - parent which allows communication between thread and the parent object.

Method Detail

add

protected void add(SiteReference site)

add a new site to be crawled by this thread

Parameters:

entry - Site to be crawled

isEmpty

protected boolean isEmpty()

is this thread idle and waiting for more sites to crawl

run

public void run()

starts the thread executing, parsing web sites in its queue until it recieves a stop request.

Specified by:

run in interface java.lang.Runnable

isRunning

public boolean isRunning()

return whether or not threads are actively crawling the Internet

Returns:

Is this crawler currently crawling?

doParse

protected void doParse(java.io.InputStream raw_data,
                       Properties parsers)
                throws java.io.IOException,
                       java.lang.Exception

Description copied from class: CrawlerBase

Helper function separated from public parse to allow easy overloading. Parses the given data into a byte array and passes a copy to every parser.

Overrides:

doParse in class CrawlerBase

Throws:

java.io.IOException

java.lang.Exception