This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: obmalloc,structmodule: 64bit, big endian
Type: Stage:
Components: Extension Modules Versions: Python 2.2
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: tim.peters Nosy List: gvanrossum, tim.peters, tzot
Priority: normal Keywords: patch

Created on 2002-08-28 15:24 by tzot, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
64bit.diff tzot, 2002-08-28 15:24 diif -c of obmalloc.c and structmodule.c
Messages (6)
msg41047 - (view) Author: Χρήστος Γεωργίου (Christos Georgiou) (tzot) * Date: 2002-08-28 15:24
Issue 1: in Objects/obmalloc.c, SYSTEM_PAGE_SIZE 
is assumed to be 4K.  I corrected (#if _MIPS_ABI>=4) 
the page size at least for 64bit SGI Irix machines.

Issue 2: in Modules/structmodule.c, most of the 
un/packing functions assumed that the char *p 
argument (destination or source for un/packing) is 
correctly aligned for the concerned type.  It is no 
problem for x86 processors (2 cycle access instead of 1 
if unaligned), but it is a problem for 64bit MIPS 
processors (where an int is 64bit and it must be at least 
on a 4-byte boundary).
All the functions handling types larger than 1 byte use 
memcpy now, taking care of endianess with an 
intermediate variable y wherever needed.
The sparse comments stating "p may not be properly 
aligned" are removed, and a more thorough comment is 
inserted before the definitions of the un/packing 
functions.
The patched module passes test_struct fine in 64-bit 
and 32-bit builds on an SGI Octane with MIPS cc; so it 
does on a Pentium Linux with gcc.  I have no way to 
make a Windows build.  Pack formats prefixed with '!' 
or '>' produce the same sizes everywhere.
msg41048 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-03 18:43
Logged In: YES 
user_id=6380

Tim sez, the obmalloc.c patch should not go in unless
supported by measurements.

I do like the structmodule.c patch, and have check it in.
Thanks!

PS the structmodule patch might apply to Python 2.2.x too.
Somebody should backport it.
msg41049 - (view) Author: Χρήστος Γεωργίου (Christos Georgiou) (tzot) * Date: 2002-09-04 14:23
Logged In: YES 
user_id=539787

Tim is probably right.

Compiled obmalloc.c with PYMALLOC_DEBUG and running
test.testall with PYTHONMALLOCSTATS, here are some
statistics results:

4K maximum numbers:
# bytes in allocated blocks        =           33,964,192
# bytes in available blocks        =               63,056
# bytes lost to pool headers       =              405,216
# bytes lost to quantization       =              145,968
# bytes lost to arena alignment    =              548,864

16K maximum numbers:
# bytes in allocated blocks        =           33,970,384
# bytes in available blocks        =              289,824
# bytes lost to pool headers       =              100,800
# bytes lost to quantization       =               45,392
# bytes lost to arena alignment    =            2,293,760

The overall size of the python process remained in both
cases at 223MB the most.
I don't think I should research more into it.  I will try to
make a patch for the current 2.2 branch too.
msg41050 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-04 14:55
Logged In: YES 
user_id=6380

I'll leave it to Tim to comment on those numbers. A
different question is, did you measure a speedup by matching
the system page size?
msg41051 - (view) Author: Χρήστος Γεωργίου (Christos Georgiou) (tzot) * Date: 2002-09-04 16:11
Logged In: YES 
user_id=539787

3 runs of test.pystone.main(), averaged:
16K pool size: 5920
4K pool size: 6120
msg41052 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2002-09-04 19:10
Logged In: YES 
user_id=31435

AFAIK, you're the first person ever to try changing 
SYSTEM_PAGE_SIZE.  It may (or may not <wink>) work 
better to boost that while simultaneously changing 
POOL_SIZE to remain at 4K, or perhaps 8K.

In any case, there's no reason I can see to believe that the 
SYSTEM_PAGE_SIZE value should make any difference in 
general, provided only that it's a reasonably large power of 
2:  the system malloc sees Python asking for the much 
larger ARENA_SIZE bytes at a time regardless of what 
value SYSTEM_PAGE_SIZE has.

Closing this, since there's nothing clear left to be persued 
here.
History
Date User Action Args
2022-04-10 16:05:37adminsetgithub: 37099
2002-08-28 15:24:37tzotcreate