[GRASS-dev] Re: [GRASS-user] r.in.xyz can now read from stdin

[Glynn Clements]

Hamish wrote:

> I just added redirection from stdin support to r.in.xyz. Everything seems to
> work, but of course testing is appreciated.
> 
> This bypasses the bulk of the code which may trigger LFS issues for
> very large input files (many GB). ie it skips scanning the filesize --
> which was only needed for G_percent() anyway.

Reading from stdin and the progress indication should be orthogonal. I
would have thought that the easiest solution would be to attempt to
determine the file size with fseek/ftell, and disable progress if that
fails, e.g. because the input is a pipe or is too large for a "long"
(fseek/ftell use long, not off_t).

> You can't rewind a piped stream so the percent= multi-pass option won't
> work,

If you redirect stdin from a file, stdin will be a file, not a pipe. 
There is no inherent reason why stdin cannot be rewound. Conversely,
someone could use input=/dev/tape, which cannot be rewound (in the
sense of rewind() or fseek()).

If you want to determine whether it's possible to seek on a stream,
either try seeking on it, or use fstat(), e.g.:

	struct stat st;
	if (fstat(filename, &st) != 0)
		/* error */
	if (S_ISREG(st.st_mode))
		/* it's a regular file */

> and it must keep 100% of the raster map in memory. (limits region
> size) Hopefully if you are working with massive datasets you already
> have lots of RAM installed.
> 
> I used G_clicker()- I'm pretty sure it does not have GUI hooks like
> G_percent() does for the progress bar, but then you can't feed data from
> stdin using the GUI so you can't trigger it anyway. (this is the
> \b\b\b\b\b... GUI window output problem)
> 
> 
> Q: is realloc() needed here? it works for me but I'm not sure if it'll
> segfault for someone someday.
> 
> char   *infile;
> ...
> infile = input_opt->answer;
> ...
> if (strcmp ("-", infile) == 0) {
>    from_stdin = TRUE;
>    ...
>    strcpy(infile, "stdin"); /* filename for history metadata */

Yes. There's no guarantee that infile will be large enough. I suggest:

	infile = G_store("stdin");

ISTR that the malloc() in GNU libc always effectively rounds up
allocations to multiples of 8 bytes (blocks are always aligned to
8-byte multiples, so nothing else will be stored in the 8 bytes
following the start of the block), so you'll get away with it on
Linux, but it's conceivable that other platforms might use 4 bytes
(the x86 architecture itself doesn't have *any* alignment
constraints).

-- 
Glynn Clements <glynn at gclements.plus.com>