This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: null source chars handled oddly by tokenize
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: Alexandre.Zani, ajaksu2, gvanrossum, rcharney
Priority: normal Keywords:

Created on 2005-01-20 07:35 by rcharney, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
invalidChars.py rcharney, 2005-01-20 07:35
Messages (4)
msg24004 - (view) Author: Reginald B. Charney (rcharney) Date: 2005-01-20 07:35
When null characters appear in the source, outside
literals, tokenize seems to either: skip the null
character and the next two following characters; or
ignore the remainder of the line, including the newline
character.

(To see the invalid characters, use vim, or an editor
that displays control characters when needed.)
msg24005 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-01-10 23:07
Logged In: YES 
user_id=1188172

Confirmed with current SVN heads.
msg82112 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-02-14 19:09
Confirmed on trunk at rev69546.
msg396005 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-06-17 15:19
Closing in favor of issue20115.
History
Date User Action Args
2022-04-11 14:56:09adminsetgithub: 41466
2021-06-17 15:19:21gvanrossumsetstatus: open -> closed

nosy: + gvanrossum
messages: + msg396005

resolution: duplicate
stage: needs patch -> resolved
2020-09-19 19:04:20georg.brandlsetnosy: - georg.brandl
2012-05-21 03:25:25Alexandre.Zanisetnosy: + Alexandre.Zani
2010-08-19 18:52:07BreamoreBoysettitle: null source chars handled oddly -> null source chars handled oddly by tokenize
stage: test needed -> needs patch
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-02-14 19:09:46ajaksu2setnosy: + ajaksu2
stage: test needed
type: behavior
messages: + msg82112
versions: + Python 2.6
2005-01-20 07:35:04rcharneycreate