This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: PyBuffer_New() memory not aligned
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: nascheme Nosy List: arigo, giacometti, jhylton, nascheme, tim.peters
Priority: normal Keywords:

Created on 2001-10-18 21:31 by anonymous, last changed 2022-04-10 16:04 by admin. This issue is now closed.

Messages (9)
msg7033 - (view) Author: Nobody/Anonymous (nobody) Date: 2001-10-18 21:31
Memory buffer areas created by PyBuffer_New are
missaligned on 32 bit machines, for doubles (64 bit).

This typically generates a bus error crash on most RISC
machines when a library function tries to write a
double in a memory buffer allocated by PyBuffer_New().

When looking at the bufferobject.c code, this seems to
come from the fact that  the memory buffer points at
'malloc() + sizeof  struct PyBufferObject'; 	[line:
b->b_ptr = (void *)(b + 1); ]

To the extent that struct PyBufferObject is not a
multiple of sizeof (double), the largest value type;
thus the misalignment, and the crashes on most RISC
processors.

A quick and temporary solution would consist in adding
a dummy double field as the last field of the
PyBufferObject structure:

struct PyBufferObject {
...
double dummy; 
} ;

and setting
b->b_ptr = (void*)&b->dummy;  /*was (void*(b+1)*/

In doing so, b->b_ptr will always by aligned on sizeof
(double), on either 32 and 64 bit architectures.


Since I'm on the buffer type problem: It would be nice
(and probably easy to do) to augment the buffer
protocol interface with the ability to specify the
basic value type stored in the buffer.

A possible list of values (enum ...) would be:

- undefined (backward compatibility)
- char, unsigned char, short, ....int, ... long, long
long, float, double, and void* (memory address).

This would enable to check at runtime the type of
values stored in a buffer (and prevent missalignement
buserrors, as well as catching garbage situations when
improper array types are passed by means of the buffer
interface [e.g.: int/float/double/short...).

Frederic Giacometti


Frederic Giacometti




msg7034 - (view) Author: Frederic Giacometti (giacometti) Date: 2001-10-18 21:35
Logged In: YES 
user_id=93657

I wasn't looged in when I submitted the item. Don't think
I'm becoming anonymous :))
msg7035 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2001-10-22 20:19
Logged In: YES 
user_id=31392

Note to whomever take this bug: PyBuffer_New() is not called
anywhere in the Python source tree; nor are there any tests
for buffer objects that I'm aware of.  A few simple test
cases would have caught this bug already.  (And for the case
of the builtin buffer() call, it might be good if it used
PyBuffer_New().)
msg7036 - (view) Author: Frederic Giacometti (giacometti) Date: 2001-11-01 00:31
Logged In: YES 
user_id=93657

A portable solution (im[provement over what I proposed)
would consitst in declaring 'dummy' with a union type,
'unionizing' all C-ANSI value types (and including 'long
long' optionally by mean of an #ifdef).

 { .... union { int Int; long Long; double Double; void*
Pvoid ...} dummy; }

All (void*)obj->dummy can be replaced with obj->dummy.Pvoid

FG
msg7037 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2001-11-01 01:31
Logged In: YES 
user_id=31435

The only portable way to fix this (assuming it's broken -- 
I don't see any alignment promises in the docs, and since 
we never use it we can't mine the source code for clues 
either) is to have PyBuffer_New do a second separate malloc
(size) and set b_ptr to that.  The C std guarantees memory 
returned by malloc is suitably aligned for all purposes; it 
doesn't promise that any standard type captures the 
strictest alignment requirement (indeed, at KSR we returned 
128-byte aligned memory from malloc, to cater to 
our "subpage" type extension).
msg7038 - (view) Author: Frederic Giacometti (giacometti) Date: 2001-11-01 17:21
Logged In: YES 
user_id=93657

The solution Tim proposes (2d malloc) will be very fine with
what we do.

Here are some more details on what we're doing (and this is
a standard operation):

- We want to create an array of double that we pass to a C
function, and then return this array to Python as buffer
object (the buffer is passed latter on as arg to other
functions using the buffer interface); and do it so that
Python takes ownership of the buffer memory management.

- We don't want to require Numerical to operate the package;
just for memory allocation.

- We should actually be using the Python array module
interface.
Unfortunately:
  * the Python array object C definitions are not exported
in a .h file
  * the Array python interface does not provide the ability
to create a new array of an arbitrary size (and certainly
initialized to 0). One has to provide a list or a string to
create an array of a given size.  IThis is not workable if
we work we large arrays (e.g.: an array of 1.000.000 doubles
is only 8 MB RAM ...).

Another  solution, then, would consist in extending the
array Python interface, so as to enable the creation of
arrays of arbitrary sizes (prefereably initialized to 0 or
to something alse with a calloc or a memset).

The extension of the array.array() function could be the
better solution, taking into account our needs as well as
Tim's concerns.

FG
msg7039 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2002-06-17 15:30
Logged In: YES 
user_id=31435

Unassigned -- I don't expect to work on this.
msg7040 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2003-09-22 18:57
Logged In: YES 
user_id=4771

No more activity around here. I suggest to deprecate this
bug report. Here is a note for the docs saying the memory
PyBuffer_New() gets you isn't specifically aligned. (cannot
attach it, sorry)
http://arigo.tunes.org/api_concrete.diff
msg7041 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2004-06-08 03:00
Logged In: YES 
user_id=35752

Added note to documentation as suggested by Armin.
History
Date User Action Args
2022-04-10 16:04:32adminsetgithub: 35351
2001-10-18 21:31:53anonymouscreate