This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: robotparser.py fails on some URLs
Type: Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: loewis Nosy List: calvin, gvanrossum, loewis
Priority: normal Keywords: patch

Created on 2002-01-04 18:21 by calvin, last changed 2022-04-10 16:04 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
robotparser.patch calvin, 2002-01-04 18:22 robotparser_readlines_patch
robotparser_with_copyright.patch calvin, 2002-01-04 19:02 robotparser_readlines_patch_and_copyright
Messages (6)
msg38635 - (view) Author: Bastian Kleineidam (calvin) Date: 2002-01-04 18:21
I am using Python 2.1.1.

The URL http://www.chaosreigns.com/robots.txt results
in an empty RobotParser object.
Reason is that the file object returned from the
URLOpener does not have a readlines() attribute.

I patched the robotparser.py to use readline() instead
of readlines().
Furthermore I removed the unnecessary redirection limit
code which is already in FancyURLopener.


Greetings, Bastian
msg38636 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-01-04 18:49
Logged In: YES 
user_id=6380

I'll gladly apply your patch.

Would you mind to also supply a patch for the copyright
statement?
It says "Python 2.0 open source license" but that's no
longer the current license. How about the PSF license
agreement for Python 2.2?
msg38637 - (view) Author: Bastian Kleineidam (calvin) Date: 2002-01-04 19:02
Logged In: YES 
user_id=9205

Updated patch with copyright
msg38638 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-03-18 10:43
Logged In: YES 
user_id=21627

Thanks for the patch. Committed as robotparser.py 1.12.
msg38639 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-05-19 22:32
Logged In: YES 
user_id=21627

Backport to 2.2.
msg38640 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-05-20 06:19
Logged In: YES 
user_id=21627

Backport committed as 1.10.16.1.
History
Date User Action Args
2022-04-10 16:04:50adminsetgithub: 35863
2002-01-04 18:21:59calvincreate