Class WebRobots::RobotsTxt
In: lib/webrobots/robotstxt.rb
Parent: Object

Methods

allow?   error!   new   options   unfetchable  

Classes and Modules

Class WebRobots::RobotsTxt::AccessControlLine
Class WebRobots::RobotsTxt::AgentLine
Class WebRobots::RobotsTxt::AllowLine
Class WebRobots::RobotsTxt::CrawlDelayLine
Class WebRobots::RobotsTxt::DisallowLine
Class WebRobots::RobotsTxt::ExtentionLine
Class WebRobots::RobotsTxt::Line
Class WebRobots::RobotsTxt::Parser
Class WebRobots::RobotsTxt::Record

Constants

DISALLOW_ALL = <<-TXT User-Agent: * Disallow: / TXT

Attributes

error  [RW] 
site  [R] 
sitemaps  [R] 
timestamp  [R] 

Public Class methods

Public Instance methods

[Validate]