This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: file.write(x) where len(x) > 64*1024**2 is unreliable
Type: Stage:
Components: Windows Versions:
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, gfe, tim.peters
Priority: normal Keywords:

Created on 2005-04-25 14:51 by gfe, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (5)
msg25116 - (view) Author: Martin Gfeller Martin Gfeller (gfe) Date: 2005-04-25 14:51
Large file writes (over 64MB) are unreliable under 
Windows.

On my Windows 2000 SP4, it succeeds when writing to 
a file on a local driver, or on a network drive.

Howver, if I map the share of the local drive c$ as 
another drive, the same write fails with:

IOError: [Errno 22] Invalid argument


Filemon (www.sysinternals.com) shows that this is a 
STATUS_INSUFFICIENT_RESOURCES returned by 
IRP_MJ_WRITE.

Although this is probably a Windows bug, my 
suggestion is that Python should abstract from such 
problems, and break the writes into suitable chunks.

Best regards,
Martin 
msg25117 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2005-04-25 16:48
Logged In: YES 
user_id=31435

Python's file.write() just calls the platform C's fwrite(), and 
MS's fwrite() docs don't admit to any size limitations.

If this is a documented bug on Windows, then your complaint 
is with Microsoft.

If it's not documented, how is anyone supposed to guess 
what "suitable" means in all cases?

IOW, this report is either misdirected, or impossible to fix 
sanely.  Sorry, but I'm closing as "3rd Party, WontFix" on 
these grounds.

If you care enough to research it in depth and create a patch, 
that may be a different story.
msg25118 - (view) Author: Martin Gfeller Martin Gfeller (gfe) Date: 2005-04-25 17:03
Logged In: YES 
user_id=884167

As you confirm that fwrite() has no documented size 
limitations, it must be a Windows bug. 

So I will need to find a more practical solution instead of 
assuming that Python will protect me from the rough edges of 
OS. 
msg25119 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2005-04-25 17:14
Logged In: YES 
user_id=849994

Well, when there's no bug documented on the Windows side,
Python cannot work around it.
msg25120 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2005-04-25 17:18
Logged In: YES 
user_id=31435

Python hides all the rough edges it can, where "can" means 
a combination of knowing in advance precisely where rough 
edges are, and of workarounds being practical.

In this case I'd _guess_ that it really has nothing to do with 
fwrite(), it has to do with limitations introduced by specific 
ways of mapping drives.  For all I know (and I don't know, one 
way or the other), Microsoft does document such limitations 
somewhere.  If so, they'll say it's a limitation, and not a bug.  
That's common enough.  It may even be that there's some 
non-obvious knob you could change on Windows to increase 
whatever resource it is that you're running out of when writing 
> 64MB to a map of a share of the local c$ drive.

But Python isn't the OS regardless, and there are hundreds 
of OS and platform C limitations Python can't sanely hide.  
This looks like one of 'em.
History
Date User Action Args
2022-04-11 14:56:11adminsetgithub: 41896
2005-04-25 14:51:17gfecreate