[GRASS-user] Large File Support (LFS)

[John Tate]

Hamish wrote:
>
> how did it fail? what was the exact error message and command
> line used? what does "g.region -p" say?

The successful r.in.xyz results below vary slightly due to merging different 
swaths to get a progressively larger file:

g.region -p

projection: 0 (x,y)
zone:       0
north:      406000
south:      392000
west:       401000
east:       412000
nsres:      1
ewres:      1
rows:       14000
cols:       11000
cells:      154000000

1.3Gb file
r.in.xyz -s -g input=/home/bob2/rawdata/clean/test.asc output=test method=min 
type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=100
n=405857.470000 s=392290.270000 e=410776.540000 w=401171.890000 b=174.140000 
t=631.580000

1.5Gb file
r.in.xyz -s -g input=/home/bob2/rawdata/clean/test2.asc output=test2 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=100
n=405857.470000 s=392290.270000 e=410776.540000 w=401171.890000 b=174.010000 
t=726.180000

1.9Gb file
r.in.xyz -s -g input=/home/bob2/rawdata/clean/test3.asc output=test3 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=100
n=405857.470000 s=392290.270000 e=411861.910000 w=401171.890000 b=174.010000 
t=726.180000

2.1Gb file
r.in.xyz -s -g input=/home/bob2/rawdata/clean/test4.asc output=test4 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=100
Unable to open input file </home/bob2/rawdata/clean/test4.asc>

r.in.xyz -s -g input=/home/bob2/rawdata/clean/test4.asc output=test4 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=25
Unable to open input file </home/bob2/rawdata/clean/test4.asc>

3.4Gb file
r.in.xyz -s -g input=/home/bob2/rawdata/clean/allmerged.asc output=intest 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=100
Unable to open input file </home/bob2/rawdata/clean/allmerged.asc>

r.in.xyz -s -g input=/home/bob2/rawdata/clean/allmerged.asc output=intest 
method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0 percent=25
Unable to open input file </home/bob2/rawdata/clean/allmerged.asc>

> 
> I suspect you are running out of memory, for very large regions
> with extended stats (median,percentile,skewness,trimmean) you
> should make use of percent=25 or so as needed.

So anything above 2Gb fails, even using percent=25, which for the 2.1Gb file 
should have had an effect if memory was an issue.

It does not really use the memory (or swap file) when using '-s' flag.
Memory hangs around 45%, swap file=0% but the CPU hits 100% varying between 
processors (watched in system monitor).
If '-s' flag is removed to create a raster then it uses 100% of memory and <2Gb 
files start to process fine (didn't continue processing, to save time in 
testing), but >2Gb give the same message: 'Unable to open input file...'

This 32bit machine only has 1Gb RAM (the 64bit machine has 2Gb RAM and never 
had a problem), but (as observed in system monitor) I don't think memory is an 
issue for scanning the data with the '-s' flag.

> Or if there's a bug, I'd like to know about it.

What do you reckon?
I compiled and installed yesterday (20/03/10) following the specific Ubuntu 
page, but added yes to largefile support (and ignored the slight typo over 
folder locations, 'grass_current / grass_trunk'.)

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/grass-user/attachments/20100321/1138c697/attachment-0001.html