This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: zipfile.testzip() using progressive file reads
Type: resource usage Stage: patch review
Components: Library (Lib), Tests Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: languishing Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alanmcintyre, gradha, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2007-06-19 10:56 by gradha, last changed 2022-04-11 14:56 by admin.

Files
File name Uploaded Description Edit
testzip_in_chunks.diff gradha, 2007-06-19 10:56
zipfile_testzip_amcintyre.diff alanmcintyre, 2008-01-04 14:47 updated patch
testzip-patch2.diff alanmcintyre, 2009-09-06 18:56 Addition of test for ZipFile.testzip
testzip-patch3.diff alanmcintyre, 2010-08-14 20:41 patch updated against python3 trunk
testzip-patch4.patch serhiy.storchaka, 2012-04-29 06:36 Updated and reloaded for review review
Messages (12)
msg52784 - (view) Author: Grzegorz Adam Hankiewicz (gradha) Date: 2007-06-19 10:56
testzip() still fails for big files. This patch implements reading small buffers. Also corrected main() where the return value of testzip() is ignored, hiding errors in zipfiles.
msg55306 - (view) Author: Grzegorz Adam Hankiewicz (gradha) Date: 2007-08-26 09:30
Any progress report on this issue, please?
msg59226 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2008-01-04 14:47
This sort of change definitely needs to be made to avoid reading huge
files into memory for testzip().  I've attached a modified patch as
zipfile_testzip_amcintyre.diff; it didn't seem appropriate to have a
module-level variable for the size of the test read, and I made a few
other minor tweaks.
msg87724 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2009-05-13 23:58
It would appear that the change to testzip has already been made:

http://mail.python.org/pipermail/python-checkins/2008-August/072892.html
http://mail.python.org/pipermail/python-3000-checkins/2008-August/004310.html

I can add some tests, though.  I'll submit another patch shortly.
msg92325 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2009-09-06 18:56
Attached is a patch that makes the zipfile module check the result of
testzip when run as __main__ as well as in test_zipfile.py, and adds
some tests in test_zipfile64.py.  The changes to test_zipfile64.py
increased its runtime from ~671 sec to ~1060 sec on my machine; I'm not
sure if that's too much.
msg111534 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-25 10:48
Tried the patch against 2.7 and got "RuntimeError: Attempt to read ZIP archive that was already closed" for TestsWithSourceFile test_deflated and test_stored.  The patch needs updating for py3k.
msg111541 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2010-07-25 13:02
Ok, I'll see if I can update that in the next week or so.
msg113930 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2010-08-14 20:41
Here's an updated patch against the py3k trunk (testzip-patch3.diff) that passes all tests (including test_zipfile64) on Linux on a 64-bit machine.

I can backport this to 2.x, but I'll wait until somebody indicates a need for it before I spend any time on it.
msg143972 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2011-09-13 14:48
I re-checked testzip-patch3.diff since some time has passed since I last commented on it, and it still seems to work ok (the small test_zipfile.py block failed to apply, but that's easy enough to do manually).  Passes full test run, test_zipfile64.py, etc., on Linux x86_64.

If there is something about the patch that still needs to be addressed before it can be committed, let me know and I'll see what I can do.
msg159551 - (view) Author: Alan McIntyre (alanmcintyre) * (Python committer) Date: 2012-04-29 00:14
I'd be glad to do some code reviews or something in exchange for the time of somebody with commit rights. :-)  If anybody is interested in getting this change committed, please let me know and I'll check that the patch is still valid.  Thanks!
msg159583 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-04-29 06:36
I'm not sure about new tests, but the changes in the docstring and old tests look necessary.
msg176742 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-12-01 12:20
I added comments in Rietveld.
History
Date User Action Args
2022-04-11 14:56:25adminsetgithub: 45108
2014-02-03 19:53:56BreamoreBoysetnosy: - BreamoreBoy
2012-12-01 12:20:58serhiy.storchakasetmessages: + msg176742
components: + Tests
versions: + Python 3.3, Python 3.4, - Python 3.1
2012-04-29 06:36:58serhiy.storchakasetfiles: + testzip-patch4.patch

messages: + msg159583
2012-04-29 00:14:33alanmcintyresetstatus: open -> languishing

messages: + msg159551
2012-04-28 21:57:54serhiy.storchakasetnosy: + serhiy.storchaka
2011-09-13 14:48:00alanmcintyresetmessages: + msg143972
2010-08-14 21:52:26pitrousetnosy: + pitrou
2010-08-14 20:41:37alanmcintyresetfiles: + testzip-patch3.diff

messages: + msg113930
2010-07-25 13:02:03alanmcintyresetmessages: + msg111541
2010-07-25 10:48:15BreamoreBoysetversions: + Python 3.1
nosy: + BreamoreBoy

messages: + msg111534

stage: test needed -> patch review
2009-09-06 18:56:26alanmcintyresetfiles: + testzip-patch2.diff

messages: + msg92325
2009-05-13 23:58:02alanmcintyresetmessages: + msg87724
2009-05-13 21:10:28ajaksu2setstage: test needed
type: resource usage
versions: + Python 2.7, Python 3.2
2008-01-04 14:47:17alanmcintyresetfiles: + zipfile_testzip_amcintyre.diff
nosy: + alanmcintyre
messages: + msg59226
2007-08-26 09:30:29gradhasetmessages: + msg55306
2007-06-19 10:56:48gradhacreate