The shutils.rmtree() implementation uses an excessive
amount of memory when deleting large directory heirarchies.

Before actually deleting any files, it builds up a list
of (function, filename) tuples for all the files that
it is going to remove.  If there are a lot of files,
this results in a lot of memory for a large heirarchy
(I had a Python process using 800MB in one case).

I'm not sure why it is doing things this way.  It isn't
using the list to avoid recursion, so the depth of
directories it can remove is still limited by Python's
recursion limit.

Replacing _build_cmdtuple() with a generator might be a
good way to reduce the memory usage while leaving the
rest of the code unchanged.

I checked in CVS, and this issue is still present on HEAD.

Logged In: YES 
user_id=31435

Rewrite it using os.walk() (not os.path.walk()) with 
topdown=False.

Logged In: YES 
user_id=146903

Attached is a Python file including a fixed up
shutils.rmtree() using os.walk().  It seems to work for me,
and should have the same error behaviour.

Logged In: YES 
user_id=469548

Please attach changes as a patch next time. I've attached
shutils-rmtree.py as a patch this time.

I gave it a quick review and added a test to test_shutil.py
to ensure some not-very-obvious behavior (don't delete a
path passed to rmtree if it's a file) would be preserved.
The new version seems fine to me. Tim, could you take a look
at it as well?

Logged In: YES 
user_id=31435

I don't really have time for a thorough review.  I'll note that 
stuff like

func = something1
arg = something2
func(arg)

looks, to my eye, like a convoluted way to say

something1(something2)

I suppose that's to keep the onerror= gimmick working, 
though.

Logged In: YES 
user_id=469548

I just looked at my own patch again, added the _raise_err
function back in and checked it in as rev 1.33 of shutil.py.