[GRASS-dev] Continuos output from grass python script of subprocess

[Glynn Clements]

Michel Wortmann wrote:

> could you tell me why this simple example script doesn�t show me the
> output of my program that I�m launching through subprocess? It�s
> only giving me the output once the program has finished. As the
> program runs for quite some time, I would like to see the output
> while the script is running through the grass module interface. (In
> the actual script, I�m preparing some input and postprocess output
> before/after the program has run.)

The most likely reason is that the child program's stdout is
fully-buffered, meaning that the stdio functions will buffer the data
and only pass it to the OS when the buffer is full (for GNU libc, the
default buffer size is 8192 bytes).

Typically, stdout is line-buffered if it is associated with a
terminal, and fully buffered otherwise (e.g. if it is associated with
a file or pipe).

The exact wording of the C standard is (§7.19.3p7)

	As initially opened, the standard error stream is not fully
	buffered; the standard input and standard output streams are
	fully buffered if and only if the stream can be determined not
	to refer to an interactive device.

If the program is one you can modify, you can force line buffering
using the setvbuf() function, or you can explicitly flush the buffer
after each line with the fflush() function. Or you could write
progress output to stderr, which is typically unbuffered by default.

If the script doesn't need to process the program's output you could
just remove the stdout=subprocess.PIPE, allowing the child process to
inherit the script's stdout.

If the script needs to process the program's output, you can't modify
the program, and you're using Linux, you could use Python's "pty"
module (which is only available on Unix and only supported on Linux)
to create a pseudo-tty (pty), and set the child process' stdout to the
pty slave. This will cause the stdio library to make stdout
line-buffered.

-- 
Glynn Clements <glynn at gclements.plus.com>