This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Patch to add tempfile.SpooledTemporaryFile (for #415692)
Type: Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: arigo, collinwinter, djmitche
Priority: normal Keywords: patch

Created on 2007-01-07 20:36 by djmitche, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
SpooledTemporaryFile.patch djmitche, 2007-03-18 18:02
Messages (9)
msg51691 - (view) Author: Dustin J. Mitchell (djmitche) * Date: 2007-01-07 20:36
Attached please find a patch that adds a SpooledTemporaryFile class to tempfile, along with the corresponding documentation (optimistically labeling the feature as added in Python 2.5) and some test cases.
msg51692 - (view) Author: Dustin J. Mitchell (djmitche) * Date: 2007-01-07 20:37
File Added: SpooledTemporaryFile.patch
msg51693 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2007-01-08 08:26
The __getattr__ magic makes the following kind of code fail with SpooledTemporaryFile:

  f = SpooledTemporaryFile(max_size=something)
  rd = f.read
  wr = f.write
  for x in y:
      ...use rd(size) and wr(data)...

The problem is that the captured 'f.read' method is the one from the StringIO instance, even after the write() rolled the file over to disk.  Given that capturing bound methods is a semi-official speed hack advertised in some respected places, we might have to be careful about it.  About such matters I am biased towards first getting it right and then getting it fast...

Also, Python 2.5 is already out, so this will probably be a 2.6 addition.
msg51694 - (view) Author: Dustin J. Mitchell (djmitche) * Date: 2007-01-08 15:53
I agree it would break in such a situation, but I'm not clear on which direction your bias leads you (specifically, which do we get right -- don't use bound methods, or don't use the __getattr__ magic?).

I could fix this by defining "proxy" functions (and some properties) for the whole file interface, rather than just the methods that potentially trigger rollover.  That would lose a little efficiency, but mostly only in reading (calling e.g., f.read() will always result in two function applications; in the current model, after the first call it runs at "native" speed).  It would also lose forward compatibility if the file protocol changes, although I'm not sure how likely that is.

Would you like me to do that?
msg51695 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2007-01-11 07:55
Reimplementing the whole file interface as a proxy functions might be the safest route, yes.

I realized that the __getattr__() magic is also used to serve at least one special method, namely the __iter__() of the file objects.  This only works with old-style classes.  In the long-term future, when old-style classes disappear and these classes become new-style, this is likely to introduce a subtle bug.
msg51696 - (view) Author: Dustin J. Mitchell (djmitche) * Date: 2007-02-03 17:22
Sounds good -- here's a new version.
File Added: SpooledTemporaryFile.patch
msg51697 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2007-03-09 01:55
Did you mean to leave SpooledTemporaryFile out of tempfile.py's __all__? Also, would it be possible for you to add some tests for the pathological cases you and Armin discussed related to bound .read() and .write() methods?
msg51698 - (view) Author: Dustin J. Mitchell (djmitche) * Date: 2007-03-18 18:02
Slightly updated version to add STF to __all__ and add a test that bound methods work correctly.
File Added: SpooledTemporaryFile.patch
msg51699 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2007-03-19 18:53
Applied as r54439. Thanks for your patch!
History
Date User Action Args
2022-04-11 14:56:21adminsetgithub: 44424
2007-01-07 20:36:07djmitchecreate