Calling f.xreadlines() multiple times returns the same 
xreadlines object.

A file is an iterator - __iter__() returns self and next() calls 
the cached xreadlines object's next method.

Logged In: YES 
user_id=6380

I posted some comments to python-dev.

Logged In: YES 
user_id=562624

Now invalidates cache on a seek.

Logged In: YES 
user_id=6380

I'm reviewing this and will check it in, or something like
it (probably).

Logged In: YES 
user_id=6380

Alas, there's a fatal flaw. The file object and the
xreadlines object now both have pointers to each other,
creating an unbreakable cycle (since neither participates in
GC). Weak refs can't be used to resolve this dilemma. I
personally think that's enough to just stick with the status
quo (I was never more than +0 on the idea of making the file
an interator anyway). But I'll leave it to Oren to come up
with another hack (please use this same SF patch).

Oren, if you'd like to give up, please say so and I'll close
the item in a jiffy. In fact, I positively encourage you to
give up. But I don't expect you to take this offer. :-)

Logged In: YES 
user_id=562624

The version of the patch still makes a file an iterator but it no 
longer depends on xreadlines - it implements the readahead 
buffering inside the file object.

It is about 19% faster than xreadlines for normal text files and 
about 40% faster for files with 100k lines.

The methods readline and read do not use this readahead 
mechanism because it skews the current file position (just like 
xreadlines does).

Logged In: YES 
user_id=6380

This begins to look good.

What's a normal text file?  One with a million bytes? :-)

Have you made sure this works as expected in Universal
newline mode?

I'd like a patch that doesn't use #define WITH_READAHEAD_BUFFER.

You might also experiment with larger buffer sizes (I
predict that a larger buffer doesn't make much difference,
since it didn't for xreadlines, but it would be nice to
verify that and then add a comment; at least once a year
someone asks whether the buffer shouldn't be much larger).

Logged In: YES 
user_id=562624

> What's a normal text file?  One with a million bytes? :-)
I meant 100kBYTE lines... Some apps actually use such long 
lines.

Yes, it works just fine with universal newlines.

Ok, the #ifdefs will go.

Strange, a bigger buffer seems to actually slow it down... I'll 
have to investigate this further.

Logged In: YES 
user_id=6380

OK, I'll await a new patch.

Logged In: YES 
user_id=31435

Just FYI, in apps that do "read + process" in a loop, a small 
buffer is often faster because the data has a decent shot at 
staying in L1 cache.  Make the buffer very large (100s of 
Kb), and it won't even stay in L2 cache.

Logged In: YES 
user_id=562624

Updated patch.

What to do about the xreadlines method? The patch doesn't 
touch it but It could be made an alias to __iter__ and the 
dependency of file objects on the xreadlines module will be 
eliminated.

On my linux machine the highest performance is achieved for 
buffer sizes somewhere around 4096-8192.  Higher or lower 
values are significantly slower.

Logged In: YES 
user_id=6380

Thanks!

Making xreadlines an alias for __iter__ sounds about right,
for backwards compatibility.

Then we should probably deprecate xreadlines, despite the
fact that it could be useful for other file-like objects;
it's just not a pretty enough interface.

Logged In: YES 
user_id=562624

Logged In: YES 
user_id=6380

Hm, test_file fails on a technicality. I'll take it from
here. Thanks!

Logged In: YES 
user_id=6380

Thanks!  Checked in.