This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Allow use of non-latin1 chars in interactive shell
Type: Stage:
Components: IDLE Versions:
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: loewis, noamr
Priority: normal Keywords: patch

Created on 2005-10-21 00:49 by noamr, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (4)
msg48880 - (view) Author: Noam Raphael (noamr) * Date: 2005-10-21 00:49
If I type a unicode string, such as u"éÜÕÝ" in the
interactive interpreter in the terminal, it does what I
expect - return the unicode string (in this case,
u'\u05e9\u05dc\u05d5\u05dd')

However, if I type it in IDLE's interactive
interpreter, I get something funny - I get a unicode
string (u'\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d'), which is
actually u"éÜÕÝ".decode('utf8').encode('latin1').

This is caused by the PyShell.runsource method, which
checks if the source string it gets is unicode, and if
so, encodes it using the 'ascii' codec. If this check
is removed, so that the unicode object itself gets
compiled, everything works out fine.

The bottom line: remove the if block starting with "if
isinstance(source, types.UnicodeType)", at
PyShell.py:589, and everything is fine.

Have a good day,
Noam
msg48881 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2005-11-02 04:43
Logged In: YES 
user_id=21627

This isn't right, though: it will assume that the source is
UTF-8 encoded, even though the user would expect a different
encoding.

The "right" way would be if IDLE compiled the source with
IOBindings.encoding.

Rejecting this patch.
msg48882 - (view) Author: Noam Raphael (noamr) * Date: 2005-11-02 11:27
Logged In: YES 
user_id=679426

But IOBindings.encoding is a hard coded 'latin1', which is 
certainly not better than 'utf8'.

Is there a way to detect the encoding from the current locale?

Noam
msg48883 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2005-11-02 18:13
Logged In: YES 
user_id=21627

No, IOBindings.encoding is not hard coded 'latin1'. It is
locale.getdefaultlocale()[1] on Windows, and
locale.nl_langinfo(locale.CODESET) on Unix. If these fail,
it is 'ascii'.
History
Date User Action Args
2022-04-11 14:56:13adminsetgithub: 42507
2005-10-21 00:49:50noamrcreate