This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: subprocess.Popen(cmd, stdout=sys.stdout) fails
Type: Stage:
Components: Library (Lib) Versions: Python 2.5
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: niemeyer Nosy List: anthonybaxter, jfmeinel, loewis, niemeyer, nnorwitz
Priority: normal Keywords:

Created on 2006-07-31 16:53 by jfmeinel, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess.diff jfmeinel, 2006-07-31 16:53 Patch to subprocess.py to not close stdout if stdout is passed in
Messages (13)
msg29392 - (view) Author: John A Meinel (jfmeinel) Date: 2006-07-31 16:53
I'm currently using subprocess.Popen() to run a
command, and I allow the caller to specify where the
output should go.

One valid output is to send it to sys.stdout (fileno == 1)

The subprocess module seems to unconditionally close
stdout if a file handle is passed (even if it stdout).

Compare:
python -c "import subprocess,sys; \
  subprocess.Popen(['echo', 'hello'])"

versus
python -c "import subprocess,sys; \
  subprocess.Popen(['echo', 'hello'], stdout=sys.stdout)"

or even
python -c "import subprocess,sys; \
  subprocess.Popen(['echo', 'hello'], stdout=1)"

The first one prints 'hello' as expected.

The latter two give an error:
echo: write error: Bad file descriptor

Attached is a possible patch to subprocess.py
msg29393 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-08-16 04:11
Logged In: YES 
user_id=33168

If stderr == stdout, this patch won't fix that, will it? 
Shouldn't you add 1, 2 to the blacklist for stderr?  (The
patch adds 2, I think 1 may also be required.)
msg29394 - (view) Author: Anthony Baxter (anthonybaxter) (Python triager) Date: 2006-08-16 04:16
Logged In: YES 
user_id=29957

Making it check for particular FD numbers is a bad idea.
Instead, it should check that any FD that's being closed
isn't in the set
(sys.stdin.fileno(), sys.stdout.fileno(), sys.stderr.fileno()) 


msg29395 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-06 01:44
Logged In: YES 
user_id=7887

Neal, I'm preparing a patch which fixes this problem as well.

Anthony, we should really be checking fd numbers, since
they're the ones dup'ed in the child. If sys.stdout has a
fileno() not in (0, 1, 2), that's not an issue.
msg29396 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-06 02:06
Logged In: YES 
user_id=7887

Fixed in 51758, backported to 2.5 in 51759.
msg29397 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-09-06 02:34
Logged In: YES 
user_id=33168

This change has broken many (all?) of the buildbots.
http://www.python.org/dev/buildbot/all/
msg29398 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-09-06 03:59
Logged In: YES 
user_id=33168

I have reverted both of these changes since all the
buildbots were broken.
msg29399 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-06 12:48
Logged In: YES 
user_id=7887

Wow.. this is strange. I don't see any reason why the build
bots would break with this change, except for the reason
that the test needs to output data to stdout/stderr to check
if it's working or not.  Is the buildbot checking these
somehow?  Is there any additional information about these
failures?
msg29400 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-06 12:52
Logged In: YES 
user_id=7887

Notice that in all buildbots that reported the failure,
subprocess tests still pass correctly.
msg29401 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-07 00:50
Logged In: YES 
user_id=7887

Problem fixed again in 51797 (trunk) and 51794 (2.5), after
removing tests which were offending buildbot due to
sys.stdout being set to a StringIO.
msg29402 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-09-10 09:53
Logged In: YES 
user_id=21627

How did you run the test cases yourself? They should have
failed for you as well; buildbot has nothing to do with that.
msg29403 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2006-09-10 22:01
Logged In: YES 
user_id=7887

Interesting. I was running tests directly most of the times
and also in verbose mode, which indeed was hiding the
problem that would happen when stdout is replaced.

Now, one thing I wonder is about this:

http://www.python.org/dev/buildbot/all/g4%20osx.4%20trunk/builds/1424/step-test/0

Why is it failing?  In the first run, when all buildbots
were broken, this is what was happening.  It does look like
tests are succeeding, and the implementation is the same as
the currently committed one, but buildbot still says it's
broken for some reason, and no error messages. Strange.
msg29404 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-09-11 05:16
Logged In: YES 
user_id=21627

It often happens that the tests fail first and then pass
when rerun; it is not that clear why that happens. In many
cases, this has turned out to be a consequence of the
sequence in which the tests are run, where earlier tests
"break" some state information to let later tests fail.

It should be possible to enhance regrtest/unittest to keep
more information about a failure, e.g. which specific test
method failed. This might give a clue in cases like this.
History
Date User Action Args
2022-04-11 14:56:19adminsetgithub: 43754
2006-07-31 16:53:53jfmeinelcreate