Right now you can use subprocess.Popen.communicate() to
make communication with subprocesses much easier and
less likely to block than communicating directly with
.stdin, .stdout, etc.  However, that requires
completely buffering the input and output.

The functionality of communicate() (which is somewhat
complex because of platform issues) could be made more
general fairly easily.  The current functionality of
communicate could then be implemented in terms of that
new method.

I attached a function I'm using which does that for the
posix systems (basically turning Popen's posix
communicate into a function with some modifications). 
Replace "proc" with "self" (and give the function a
better name) and you'd have a method.

If patch 1175984 was accepted, then this wouldn't be
that much of an issue:
http://sourceforge.net/tracker/index.php?func=detail&aid=1175984&group_id=5470&atid=305470

Logged In: YES 
user_id=341410

Would an asynchronous subprocess (which you would poll
manually) be better/sufficient?

http://python.org/sf/1191964

Logged In: YES 
user_id=341410

Also, what you post is not a 'bug', it is a 'feature request'.

Logged In: YES 
user_id=1188172

Reclassifying as RFE. Assigning to Peter.

Closed issue14872 as a duplicate of this.

Please note the work being performed on #1191964 which was first referenced in msg54591.

The non-blocking read and write features proposed in Issue 1191964 wouldn’t be sufficient. Perhaps they might be useful for implementing this feature on Windows though; I’m not sure. And a non-blocking write on Posix may be slightly more efficient as well, removing the limitation of copying in chunks of PIPE_BUF bytes.

Ian’s original patch posted here proposes a version of communicate() which accepts a file reader for the subprocess’s input, and file writers for the subprocess’s outputs. Another option could be something like my SubprocessWriter class <https://github.com/vadmium/pacman-tools/blob/9ffdd88/makeaur#L179>. It provides a file writer interface for the caller to write to the subprocess’s input, while copying data from the subprocess’s output behind the scenes. I used it to stream a tar file, as it is written by the “tarfile” module, into a GPG subprocess to create a digital signature, while writing the output of GPG into a BytesIO buffer. Using Unix command pipeline pseudocode, it is used a bit like this:

tarfile.open(mode="w|") | Popen("gpg") | BytesIO()

since the time this was filed, subprocess has evolved a lot and third party options for child process have appeared as well as modern things like:

https://docs.python.org/3/library/asyncio-subprocess.html (stdlib)

https://trio.readthedocs.io/en/stable/reference-io.html?highlight=subprocess#options-for-starting-subprocesses (trio, easier async framework to use than asyncio)

plumbing stdin/stdout/stderr data to python file-like objects from the stdlib subprocess APIs themselves is better left to third party solutions building on top of the subprocess module today.