[GRASS-user] Large File Support (LFS)
hamish_b at yahoo.com
Tue Mar 23 06:42:03 EDT 2010
> g.region -p
> rows: 14000
> cols: 11000
> cells: 154000000
for type = float this will use 154000000 * 4 / 1024000 ~= 600mb
RAM for method = min. (floats are 4 bytes each) It will be
twice that for mean. so memory and the percent= option are not
an issue here. (memory use is dependent on the region rows x
columns, not the number of input points)
> 2.1Gb file
> r.in.xyz -s -g input=/home/bob2/rawdata/clean/test4.asc
> output=test4 method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0
> Unable to open input file </home/bob2/rawdata/clean/test4.asc>
> So anything above 2Gb fails, even using percent=25, which for > the 2.1Gb file should have had an effect if memory was an
as Glynn explained, I was wrong- glibc's fopen() does care if it
is not used on a fully 64bit software stack.
if you have a 32bit machine, the solution/workaround for this is
to pipe from stdin instead. try this:
cat /home/bob2/rawdata/clean/test4.asc | r.in.xyz input=- \
output=test4 method=min fs=, x=6 y=7 z=8
r.in.xyz input=- output=test4 method=min fs=, x=6 y=7 z=8 < \
> It does not really use the memory (or swap file) when using
> '-s' flag.
right, the scan flag only has to remember a single set of
min/max values for x,y,z. that's just a few bytes. if it sees
a smaller/larger value it can forget the old one and just
remember the current leader.
> Memory hangs around 45%, swap file=0% but the CPU hits 100%
> varying between processors (watched in system monitor).
> If '-s' flag is removed to create a raster then it uses 100%
> of memory
of, then you should employ the percent= option until it does
not. (start with percent=50 to make it run in 2 passes)
> and <2Gb files start to process fine (didn't continue
> processing, to save time in testing), but >2Gb give the same
> message: 'Unable to open input file...'
try sending the data via stdin rather than reading the file
directly, as in the examples above.
> This 32bit machine only has 1Gb RAM (the 64bit machine has
> 2Gb RAM and never had a problem), but (as observed in system
> monitor) I don't think memory is an issue for scanning the
> data with the '-s' flag.
right, as scanning doesn't have to remember much at all.
> I compiled and installed yesterday (20/03/10) following the
> specific Ubuntu page, but added yes to largefile support (and
> ignored the slight typo over folder locations, 'grass_current
> / grass_trunk'.)
you mean the wiki page?
... so changing to "input=-" should make it work on the 32bit
More information about the grass-user