This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: float/atof have become locale aware
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: bernhard, georg.brandl, georg.brandl, gustavo, loewis
Priority: high Keywords:

Created on 2006-01-29 01:04 by bernhard, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python-1417699-20060610.diff bernhard, 2006-06-10 19:18
Messages (10)
msg27376 - (view) Author: Bernhard Herzog (bernhard) Date: 2006-01-29 01:04
The builtin float and the function string.atof have
become locale aware in Python 2.4:

Python 2.4.2 (#1, Nov 29 2005, 16:07:55) 
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for
more information.
>>> import locale
>>> import string
>>> locale.setlocale(locale.LC_ALL, "de_DE")
'de_DE'
>>> float("1,5")
1.5
>>> string.atof("1,5")
1.5


This doesn't match what's specified in pep 331:

        - float() and str() stay locale-unaware.

It also doesn't match the documentation for atof:

  Convert a string to a floating point number.
  The string must have the standard syntax for a
  floating point literal in Python, optionally
  preceded by a sign ("+" or "-"). Note that this
  behaves identical to the built-in function float()
  when passed a string.

The documentation for float() doesn't state which
format is accepted by float(), though.

The reason for this behavior is ultimately, that
PyOS_ascii_strtod accepts the locale specific
convention in addition to the "C"-locale/python
convention, even though the comment in the code
preceding its definition states:

  This function behaves like the standard strtod() 
  function does in the C locale.


msg27377 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-02-17 12:19
Logged In: YES 
user_id=1188172

Martin, you checked in the patch which is mentioned in PEP
331. Is this correct behavior?
msg27378 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-05-04 04:42
Logged In: YES 
user_id=21627

It's a bug. As bernhard says, it originates from
PyOS_ascii_strtod accepting both forms, which in turn
happens because it just leaves the locale-specific decimal
point in the string, only replacing the . with the
locale-specific point. Then, the platform's strtod will
happily accept the locale-specific version. 

I'd like Gustavo Carneiro to comment.
msg27379 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2006-05-04 09:47
Logged In: YES 
user_id=908

My comment is, is "PyOS_ascii_strtod accepting both forms"
something that is harmful and should be fixed?

I didn't exactly write the PyOS_ascii_strtod code; I only
integrated it with Python.  But if we really need to fix
this, I'll try to figure out how to make it not accept
floating points in locale aware format; if necessary I'll
ask help from  the original author of the code, Alexander
Larsson.
msg27380 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-05-08 18:00
Logged In: YES 
user_id=849994

I grepped around and found PyOS_ascii_strtod in the
following places:

* stropmodule::atof
* cPickle::load_float (the writing of locale dependent
floats was fixed recently)
* floatobject::PyFloat_FromString
* complexobject::complex_subtype_from_string

In my opinion, all these should not tolerate localized
floats, so Gustavo, please ask the original author how to
achieve this.
msg27381 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-05-12 14:14
Logged In: YES 
user_id=21627

Unassigning myself - I don't plan to work on this anytime soon.
msg27382 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2006-05-13 11:59
Logged In: YES 
user_id=908

I'm quite busy at this moment, but I'll send a patch soon,
promise.
msg27383 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2006-05-16 16:35
Logged In: YES 
user_id=908

It seems I can't upload files?!
I have put the patch to fix this problem, including unit
test, here: http://www.gnome.org/~gjc/python-1417699.diff
msg27384 - (view) Author: Bernhard Herzog (bernhard) Date: 2006-06-10 19:18
Logged In: YES 
user_id=2369

gustavo, your patch seems to work fine.  However, it leads
to some test failures because the testsuite already tests
for the wrong behaviour.  I've uploaded a version of your
patch that in addtion to your changes contains modifications
to the test suite so that it tests for the new behavior and
also tests some more variations of floating point literals
accepted by float().

It would be nice if this patch could go into 2.5 before beta1
msg27385 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-07-03 12:21
Logged In: YES 
user_id=21627

Thanks for the report and the patch. Committed as r47212.

I don't think it should get backported to 2.4, as it may
break existing code.
History
Date User Action Args
2022-04-11 14:56:15adminsetgithub: 42848
2006-01-29 01:04:39bernhardcreate