This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: list(xrange(sys.maxint/4)) again
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: arigo, gvanrossum, loewis, tim.peters
Priority: normal Keywords:

Created on 2002-10-04 16:02 by arigo, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Messages (11)
msg12593 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2002-10-04 16:02
SF bug #556025 triggers again in the latest 2.2 branch:

>>> list(xrange(sys.maxint/4))
Segmentation fault (core dumped)

on my Linux box.
msg12594 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-10-05 09:57
Logged In: YES 
user_id=21627

I can't reproduce this on SuSE 8.1. Can you analyse this in
more detail?
msg12595 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2002-10-05 13:41
Logged In: YES 
user_id=4771

Right, the bug doesn't show up on my SuSE 7 either, but only
on an old Mandrake box.  I found out that it is caused by a
bug in malloc().  Any malloc(n) with n greater than
(1<<31)-12 will pretend it succeeded, but actually only
allocate a few bytes, hence the immediately following
segmentation fault.  As it seems to be fixed in recent
versions of the libc I suggest to ignore this problem.

However, there are other overflow bugs in listobject.c which
are not malloc()'s fault.  Would you like me to go over the
whole code in detail and submit a patch?  Here are a couple
of examples:

>>> (sys.maxint/16+1) * range(16)
SystemError: Objects/listobject.c:63: bad argument to
internal function
>>> (sys.maxint/2+1) * range(16)
Segmentation fault (core dumped)
msg12596 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-10-05 21:07
Logged In: YES 
user_id=21627

Good. I suggest we take no action for 2.2. For the other
bugs you found, would you like to propose a patch?
msg12597 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2002-10-05 23:51
Logged In: YES 
user_id=4771

Hunting all overflow bugs seems like a long-term project
:-)  I can got along the source and submit patches as I find
them, but at some point we will need a policy, like a common
set of overflow-checking macros to compute the size of the
memory to allocate or resize.

Just for the fun, a couple more bugs:

>>> '%2147483647d' % -123
Segmentation fault
>>> 1L<<2147483647
SystemError: NULL object passed to PyObject_InitVar
msg12598 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2002-10-10 19:46
Logged In: YES 
user_id=31435

Note that the original problem here (the test_b1 
segfault) "should have been" fixed, in 2.2.2 and 2.3, by a 
change to the test made yesterday.
msg12599 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-11 00:47
Logged In: YES 
user_id=6380

Fixed '%2147483647d' % -123 in 2.2.2 and 2.3.
msg12600 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-11 20:44
Logged In: YES 
user_id=6380

Fixed 1L<<sys.maxint too.  (I had a hard time reproducing
this because at first it would actually allocate that much
space rather than fail. :-)

Armin, can you comment on your request for a policy?  So far
I haven't seen many similarities between the bugs you reported.

I'll go hunt down the first two you mentioned now:

(sys.maxint/16+1) * range(16)
(sys.maxint/2+1) * range(16)
msg12601 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-11 21:13
Logged In: YES 
user_id=6380

OK, I think I see what you mean. We could use macros or
functions that add or multiply two ints with an overflow
check (perhaps assuming the args are >= 0 and setting the
result to a negative number when there's overflow).

I've fixed the two list repeat bugs (with one test), added
an overflow test for list+list, and also one for
tuple+tuple. All backported to 2.2.2.

Closing this now (but making a mental note about those
overflow macros).
msg12602 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2002-10-12 10:12
Logged In: YES 
user_id=4771

Yes, exactly.  I can see a couple of approaches based on
values that mean "overflow":

1) there is just one special "overflow" value, e.g.
((size_t)-1), that is returned and propagated by the macros
when an overflow is detected.  This might be error-prone
because if we forget once to use the macros to add a few
bytes to the size, this special value will wrap down to a
small legal value -- and segfault.

2) same as above, but with a whole range of overflow
values.  For example, just assume (or decide) that no malloc
of more than half the maximum number that fits into a size_t
can succeed.  We don't need any macro to add a (resonable)
constant to a size;  we only need a macro for multiplication
that -- upon overflow -- returns the first number of the
"overflow" range.

3) we compute all sizes with signed integers (int or long),
as is currently (erroneously?) done at many places.  Any
negative integer is regarded as "overflow", but the
multiplication macro returns the largest negative integer in
case of overflow, so that as above no addition macro is
needed.

This will require a "multiplication hunt party" :-)

Also, approaches 2 and 3 require fixes to ensure that
mallocs of sizes in the overflow range always fail, for any
of the several implementations of malloc found in the code.
msg12603 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-12 13:24
Logged In: YES 
user_id=6380

I'd like to take this back to a mailing list.

Can you repost that to python-dev?
History
Date User Action Args
2022-04-10 16:05:43adminsetgithub: 37264
2002-10-04 16:02:49arigocreate