Looks like an out-of-bounds array access... might be a security problem.

The attached file includes a script which, when executed, tries to pack the two given directories with a bz2 compressor. On my machine (stock 32 bit Ubuntu Edgy), the program fails with the following exception: 
--------------------
Traceback (most recent call last):
  File "test.py", line 13, in ?
    block = compressor.compress(open("compress-0067/"+file, "rb").read())
ValueError: the bz2 library has received wrong parameters
--------------------

The problem occurs under either python2.4 or 2.5 (I don't have other versions to test with).

Sorry, the file is large... I tried to reduce the example to smaller number of files etc., but no such luck. The file is too large to be submitted here as an attachment, so I have uploaded it to http://www.cs.technion.ac.il/~gsasha/testcase.tar.bz2

I don't understand why you think this is an array out of bounds.  I ran your test case under valgrind and it reported no problems.

I can reproduce the problem.  I have attached a patch that fixes the problem for me.  I am not certain it's the correct fix however.  The unit tests pass with this modification.
File Added: bz2.diff

I saw that the effect occurs only when one compressor finishes and is destroyed, and then the second one gives the "wrong parameters". Usually, for such an effect to happen, the first compressor must have some permanent effect on the system - and since it was destroyed before the second one was created, it was only natural to assume that the reason was an out of bounds access.

My approach is that such a bug should be suspected as an out-of-bounds access until shown othervise, especially in a language like Python where it is natural to assume that such problems shouldn't occur and that it is safe to use security-wise. Of course I'll be happy to learn that it's just a simple bug.

On a related note, will the fix be backported to all the relevant Python versions? If you understand why the bug happens (I don't, sorry, have no idea of internals of Python), you might be able to generate a compact test case that would capture the erroneous behavior.

Best regards,
Alex

I had the same problem with the BZ2Compressor. (reproducible under 2.5)
It was working fine with BZ2File.

Unfortunately I cannot check the proposed fix as I don't have the environment to compile python...

I think Neal's patch is correct, the equivalent needed to be applied in several other places as well.

Committed as rev. 54336, 54337 (2.5).