This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: zipfile -- too many files?
Type: Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ronaldoussoren, shmengie
Priority: normal Keywords:

Created on 2006-07-09 05:57 by shmengie, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (7)
msg29102 - (view) Author: Joe Brown (shmengie) Date: 2006-07-09 05:57
While creating a huge zipfile (probably too big) the
following error appeared.

Exception exceptions.OverflowError: 'long int too large
to convert to int' in <bound method ZipFile.__del__ of
<zipfile.ZipFile instance at 0x009BD198>> ignored

I assume this is during the closing of the zipfile,
further assuming too many files were added to the archive.
msg29103 - (view) Author: Joe Brown (shmengie) Date: 2006-07-09 06:01
Logged In: YES 
user_id=729883

Platform: Windows SBS 2003
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit
(Intel)] on win32
msg29104 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2006-07-11 17:47
Logged In: YES 
user_id=580910

How large is the resulting file? Zipfile in python2.4 cannot handle zipfiles that 
require the zip64 extensions and fails misserably when you do cross that limit.

This was fixed in python 2.5.
msg29105 - (view) Author: Joe Brown (shmengie) Date: 2006-07-11 20:08
Logged In: YES 
user_id=729883

The zip file was on order of 6-9 GBs, surely over 32000
files.  I will give Python2.5 a shot and see how that works.

I've stumbled on another bug with tarfile.  It's not
reporting an error, but I only find 350 files when I
unarchive.  That's entirely different bug tho :-o
msg29106 - (view) Author: Joe Brown (shmengie) Date: 2006-07-12 02:17
Logged In: YES 
user_id=729883

Windows 2.5 beta 2 responded like this:

Traceback (most recent call last):
  File "C:\python_scripts\archive.py", line 27, in <module>
    main()
  File "C:\python_scripts\archive.py", line 23, in main
    zipper(zip, folder)
  File "C:\python_scripts\archive.py", line 15, in zipper
    zip.write(os.path.join(root, item))
  File "C:\Python25\lib\zipfile.py", line 561, in write
    self._writecheck(zinfo)
  File "C:\Python25\lib\zipfile.py", line 536, in _writecheck
    raise LargeZipFile("Zipfile size would require ZIP64
extensions")
zipfile.LargeZipFile: Zipfile size would require ZIP64
extensions
C:\Python25\lib\struct.py:63: DeprecationWarning: struct
integer overflow maskin
g is deprecated
  return o.pack(*args)
msg29107 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2006-07-13 07:47
Logged In: YES 
user_id=580910

The problem with zipfile is not the number of files, but the size of the archive. 
The basic zip file format stores file sizes as 32 bit (signed) integers, there is an 
extension for using 64 bit integers, but that isn't supported in python 2.4.
msg29108 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2006-07-13 08:17
Logged In: YES 
user_id=580910

hmm, I should read all new comments before responding :-(

To use the ZIP64 extensions with python2.5 you must add 'allowZip64=True' to 
arguments of the zipfile constructor. ZIP64 is disabled by default because the 
unix zip tools don't support zip64.
History
Date User Action Args
2022-04-11 14:56:18adminsetgithub: 43637
2006-07-09 05:57:45shmengiecreate