Spider plugin.. Spider websites and makes an index into them.
taken from http://code.activestate.com/recipes/576551-simple-web-crawler/
—BHJTW 15-11-2011 Adapted for JSONBOT
commands in this plugin: spider-stop .. spider
spider .. [handle_spider]
description: [] permissions: OPER code: jsb.plugs.common.spider_spider_command. examples: <user> ;spider http://jsonbot.org/handbook <output> calling fetcher on Sat Mar 24 19:28:18 2012
spider-stop .. [handle_spiderstop]
description: [] permissions: OPER code: jsb.plugs.common.spider_spider-stop_command. examples: <user> ;spider-stop <output> stopped 1 spiders