[GRASS-dev] r.in.xyz can now read from stdin

[Hamish]

> Hamish:
> >I just added redirection from stdin support to r.in.xyz. Everything
> >seems to work, but of course testing is appreciated.

Eric wrote: 
> Thanks very much for this. I've tested r.in.xyz on a massive Simrad
> EM710 swath bathymetry dataset. Results follow:
> 
> $ g.region -p
..
> nsres:      10
> ewres:      10
> rows:       14885
> cols:       23544
> cells:      350452440
> 
> $ r.out.xyz input=Matthew_EM710_2007_July4_10m.grd fs=, output=- |
> r.in.xyz input=- output=TESTING_R_IN_XYZ fs=, Scanning data ...
>  100%
> Writing to map ...
>  100%
> r.in.xyz complete. 35315212 points found in region.
>
> $ r.info TESTING_R_IN_XYZ
..
> The newly-gridded bathy looks fine; identical to the original.

r.univar results the same? (n and sum)
 or
r.mapcalc diff_map=old-new
r.univar diff_map

maybe we need a r.md5sum option added to r.info?

 
> Before I leave work today I'm going to halve the resolution (quadruple
> the number of cells), and start another r.out.xyz --> r.in.xyz run,
> and let it run overnight. I'll let you know how it goes tomorrow
> morning (wee hours Saturday morning your time). Just for fun, I'll
> 'time' it as well.
..
> Well, never mind. We won't have to wait that long. I ran out of memory
> trying to pipe the text file in question to r.in.xyz:
..
> nsres:      5
..
> cells:      1401809760
..
> I realize I'm using a ridiculously large region. Should r.in.xyz have
> been able to accept a file this large?

It's not anything to do with the input file size, it's the region size
that is the problem. r.in.xyz from stdin has to keep the entire region
in memory at once. See the (updated) help page section about memory
issues.

If the input comes from a file and not a stream, you can use the input=
filename and percent=25 to run it using 4 passes using a quarter of the
memory. You can get the same effect by importing a number of subregions
and r.patch-ing them back together.


thanks for the feedback,
Hamish