PHP 5: Robots_txt
Andy Pieters (contact me)
This class can be used to check whether a page may be crawled by looking at the robots.txt file of its site.
It takes the URL of a page and retrieves the robots.txt file of the same site.
The class parses the robots.txt file and looks up for the rules defined in that file to see if the site allows crawling the intended page.
The class also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.
| Utility
| Consistency
| Docs
| Examples
| Tests
| Videos
| Overall
| Rank
|
| Sufficient (66.7%)
| Sufficient (66.7%)
| Not sure (50.0%)
| -
| -
| -
| Not sure (45.0%)
| 1484 |
Click here for detailed information about this class on phpclasses.org
Php Handleiding